Image processing system and method

Abstract

The embodiments of the present specification provide an image processing system and method. The method is implemented on at least one machine, each machine having at least one processor and at least one storage device, and includes: generating an initial image by filtering image data generated by an image acquisition device; performing a first iterative operation on the initial image to generate an intermediate image, the first iterative operation including sparse deconvolution of the intermediate image; and performing a second iterative operation on the intermediate image to generate a target image based on a second objective function, the second objective function being associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image. The system includes an initial image generation module, a sparse reconstruction module, and a deconvolution module.

Description

[0001] Description of the case

[0002] This case is a divisional application of the Chinese patent application entitled “An Image Processing System and Method” and application number 202080012667.7 filed on June 8, 2020. Technical Field

[0003] The present invention relates to the field of image processing, and more particularly to a system and method for reconstructing a super-resolution image to extract a spatial resolution exceeding that of image data generated by an image acquisition device. Background Art

[0004] The raw images generated or collected by image acquisition devices (e.g., microscopes, telescopes, cameras, web cameras) usually have relatively poor resolution and / or relatively low contrast due to problems such as photobleaching, phototoxicity, etc. Image processing techniques are widely used in image reconstruction to produce images with higher quality. However, conventional image processing techniques are limited by factors such as signal-to-noise ratio (SNR), physical optical limitations, etc., so that the reconstructed images usually have artifacts or the reconstruction speed may be relatively low. Therefore, it is desirable to provide a system and method for effectively processing images to effectively improve image resolution and reconstruction speed. Summary of the invention

[0005] In one aspect of the present application, an image processing method is provided. The method is implemented on at least one machine, each machine having at least one processor and at least one storage device, and includes: generating an initial image by filtering image data generated by an image acquisition device; performing a first iterative operation on the initial image to generate an intermediate image based on a first objective function, the first objective function being associated with the image fidelity of the intermediate image, the continuity of the intermediate image, and the sparsity of the intermediate image; and performing a second iterative operation on the intermediate image to generate a target image based on a second objective function, the second objective function being associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image.

[0006] In some embodiments, filtering the image data generated by the image acquisition device includes filtering the image data by performing Wiener inverse filtering on the image data.

[0007] In some embodiments, the first iterative operation comprises a sparse deconvolution of the initial image.

[0008] In some embodiments, the first objective function comprises a first weighting factor associated with image fidelity of the intermediate image or an L-2 norm of a first difference between the intermediate image and the initial image.

[0009] In some embodiments, the method further includes: generating an estimated background image by performing an iterative wavelet transform operation on the initial image, and the first objective function is also associated with the estimated background image.

[0010] In some embodiments, the first objective function further comprises an L-2 norm of a first difference between the intermediate image and the initial image and a fourth difference between the estimated background image.

[0011] In some embodiments, the iterative wavelet transform operation includes one or more iterations, and each current iteration includes: determining an input image based on the initial image or an estimated image generated in a previous iteration; generating a decomposed image by performing a multi-scale wavelet decomposition operation on the input image; generating a transformed image by performing an inverse wavelet transform operation on the decomposed image; generating an updated transformed image based on the transformed image and the input image; generating an estimated image of the current iteration by performing a cutoff operation on the updated transformed image; and in response to determining that a termination condition is met, designating the estimated image as a background image.

[0012] In some embodiments, the first iterative operation includes: determining an initial estimated intermediate image based on the initial image; and performing at least two iterations of the first objective function based on the initial estimated intermediate image to update the estimated intermediate image, wherein, in each of the at least two iterations, in response to determining that a termination condition is satisfied, the intermediate image is determined.

[0013] In some embodiments, the second iterative operation comprises an iterative deconvolution and the second objective function is associated with a third difference between the intermediate image and the target image multiplied by the system matrix.

[0014] In another aspect of the present application, an image processing system is also provided. The system includes: an initial image generation module, which is configured to generate an initial image by filtering image data generated by an image acquisition device; a sparse reconstruction module, which is configured to perform a first iterative operation on the initial image to generate an intermediate image based on a first objective function, wherein the first objective function is associated with the image fidelity of the intermediate image, the continuity of the intermediate image, and the sparsity of the intermediate image; and a deconvolution module, which is configured to perform a second iterative operation on the intermediate image to generate a target image based on a second objective function, wherein the second objective function is associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The exemplary embodiments of the present application are further described in detail. These exemplary embodiments will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0016] Figure 1 is a schematic diagram of an exemplary application scenario of an image processing system according to some embodiments of the present application;

[0017] Figure 2 is a schematic diagram of an exemplary computing device according to some embodiments of the present application;

[0018] Figure 3 is a block diagram of an exemplary mobile device in which a terminal is located according to some embodiments of the present application;

[0019] Figure 4 is a schematic diagram of an exemplary processing device according to some embodiments of the present application;

[0020] Figure 5 is an exemplary flowchart of image processing according to some embodiments of the present application;

[0021] Figure 6 is an exemplary flow chart of background estimation according to some embodiments of the present application;

[0022] Figure 7 is an exemplary result of a sparse deconvolution algorithm flow chart according to some embodiments of the present application;

[0023] Figures 8A-8Q It is an exemplary result shown according to some embodiments of the present application that the spatiotemporal resolution of sparse-SIM in living cells can substantially reach 60 nm and milliseconds;

[0024] Figures 9A-9G It is an exemplary result showing that the complex dynamics within mitochondria and between endoplasmic reticulum mitochondria are visualized by two-color sparse-SIM according to some embodiments of the present application;

[0025] Figures 10A-10L The exemplary results shown in some embodiments of the present application show that sparse SD-SIM can achieve three-dimensional, multi-color and less than 90nm resolution SR imaging of living cells;

[0026] Figures 11A-11K It is an exemplary result according to some embodiments of the present application showing that upsampling can enable sparse SD-SIM to overcome the Nyquist sampling limitation to achieve multi-color three-dimensional SR imaging of living cells;

[0027] Fig.12is an exemplary result showing a detailed workflow of sparse deconvolution according to some embodiments of the present application;

[0028] Figures 13A-13D The exemplary results of the spatial resolution benchmark according to the synthetic image at different steps of sparse deconvolution are shown according to some embodiments of the present application;

[0029] Figures 14A-14D It is an exemplary result shown in some embodiments of the present application that the contribution of different steps to the sparse deconvolution of the synthetic image may vary with the degree of noise pollution;

[0030] Figures 15A-15F The exemplary results of some embodiments of the present application are shown to show that when processing real biological structures, the spatial resolution is actually extended by a sparse deconvolution algorithm;

[0031] Fig.16A and 16B It is an exemplary result showing that the optical transfer function is obtained by Fourier transform of fluorescent microbead visualization under different conditions according to some embodiments of the present application;

[0032] Fig.17A and 17B The exemplary results of prior reconstruction using only sparsity, only continuity, or both sparsity and continuity are shown in some embodiments of the present application;

[0033] Figures 18A-18E is an exemplary result of three-dimensional image stacking of fluorescent microbeads under SD-SIM and sparse SD-SIM according to some embodiments of the present application;

[0034] Figures 19A-19D are exemplary results showing dual-color live cell imaging of clathrin and actin using sparse SD-SIM according to some embodiments of the present application; and

[0035] Fig. 20A and 20B The figures are exemplary results showing the use of sparse SD-SIM to display endoplasmic reticulum-lysosome contacts according to some embodiments of the present application. DETAILED DESCRIPTION

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. However, it should be understood by those skilled in the art that the present application can be implemented without these details. In other cases, in order to avoid unnecessarily obscuring the various aspects of the present application, well-known methods, processes, systems, components and / or circuits have been described at a higher level. For those of ordinary skill in the art, it is obvious that various changes can be made to the disclosed embodiments, and without departing from the principles and scope of the present application, the general principles defined in the present application can be applied to other embodiments and application scenarios. Therefore, the present application is not limited to the embodiments shown, but conforms to the broadest scope consistent with the scope of the application.

[0037] The terms used in this application are only for the purpose of describing specific example embodiments and are not restrictive. The singular forms "one", "an" and "the" used in this application may also include plural forms, unless the context clearly indicates an exception. It should also be understood that the terms "including" and "comprising" used in this application specification only indicate the presence of the features, integers, steps, operations, components and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, parts and / or combinations thereof. It should be understood that the terms "object" and "target" can be used interchangeably to refer to things that undergo the imaging process of the present application.

[0038] It is to be understood that the terms "system", "engine", "unit", "module" and / or "block" used herein are methods for distinguishing different components, elements, parts, portions or assemblies at different levels in ascending order. However, these terms may also be replaced by other expressions if the same purpose can be achieved.

[0039] Generally, the terms "module", "unit" or "block" as used herein refer to logic embodied in hardware or firmware, or a collection of software instructions. The modules, units or blocks described herein may be implemented as software and / or hardware, and may be stored in any type of non-transitory computer-readable medium or other storage device. In some embodiments, software modules / units / blocks may be compiled and linked into an executable program. It should be understood that software modules may be called from other modules / units / blocks or themselves, and / or may be called in response to detected events or interrupts. For use in computing devices (e.g., Figure 2The software modules / units / blocks executed on the processor 210 shown in the figure can be provided on a computer-readable medium, such as a compact disc, a digital video disc, a flash drive, a disk, or any other tangible medium or as a digital download (and can be initially stored in a compressed or installable format that needs to be installed, decompressed or decrypted before execution). The software code here can be stored in part or in whole in the storage device of the computing device performing the operation and applied in the operation of the computing device. The software instructions can be embedded in firmware, such as an EPROM. It should also be understood that the hardware modules / units / blocks can be included in connected logical components, such as gates and triggers, and / or can include programmable units, such as programmable gate arrays or processors. The modules / units / blocks or computing device functions described here can be implemented as software modules / units / blocks, but can be represented in hardware or firmware. Generally, the modules / units / blocks described here refer to logical modules / units / blocks, which can be combined with other modules / units / blocks or divided into sub-modules / sub-units / sub-blocks, although they are physical organizations or storage devices. This description can be applied to the system, engine or part thereof.

[0040] It is understood that, unless the context clearly indicates otherwise, when a unit, engine, module, or block is referred to as being "on," "connected," or "coupled to" another unit, engine, module, or block, it may be directly on, connected, coupled, or in communication with the other unit, engine, module, or block, or there may be intermediate units, engines, modules, or blocks. In this application, the term "and / or" may include any one or more of the relevant listed items or combinations thereof.

[0041] It should be noted that when describing operations performed on an image, the term "image" used herein may refer to a data set (e.g., a matrix) containing values ​​of pixels in the image (e.g., pixel values). As used herein, for ease of description, a representation of an object in an image (e.g., a person, an organ, a cell, or a portion thereof) may be referred to as an object. For example, for ease of description, a representation of a cell or an organelle (e.g., a mitochondria, an endoplasmic reticulum, a centrosome, a Golgi apparatus, etc.) in an image may be referred to as a cell or an organelle. As used herein, for ease of description, operations on representations of objects in an image may be referred to as operations on objects. For example, for ease of description, segmentation of a portion of an image including representations of cells or organelles may be referred to as segmentation of cells or organelles.

[0042] It should be understood that the term "resolution" as used herein refers to a measure of the clarity of an image. The terms "super-resolution" or "super-resolution" or "SR" as used herein refer to enhanced (or increased) resolution, for example, by the process of combining a series of low-resolution images to generate a higher resolution image or sequence. The terms "fidelity" or "integrity" as used herein refer to the degree to which an electronic device (e.g., an image acquisition device) replicates its effect (e.g., an image). The term "continuity" as used herein refers to a feature that is continuous in time and / or space. The term "sparseness" as used herein refers to a feature in a vector or matrix that is mostly zero-valued, for example, the count of 0 values ​​in a vector or matrix is ​​much greater than the count of 1 values ​​in the vector or matrix.

[0043] These and other features, characteristics and functions and methods of operation of the related structural elements of the present application, as well as the combination of components and manufacturing economy, may become more apparent from the following description of the accompanying drawings, which constitute a part of the specification of this application. However, it should be understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of the present application. It should be understood that the drawings are not drawn to scale.

[0044] The flowcharts used in this application illustrate the operations performed by the systems shown in some embodiments disclosed in this application. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. Instead, the various steps may be processed in reverse order or simultaneously. At the same time, one or more other operations may also be added to these flowcharts. One or more operations may also be deleted from the flowcharts.

[0045] The present application relates to image processing systems and methods. The systems and methods can generate an initial image by filtering image data generated by an image acquisition device. The systems and methods can perform a first iterative operation on the initial image to generate an intermediate image based on a first objective function. The first objective function can include a first term, a second term, and a third term. The first term can be associated with a first difference between the intermediate image and the initial image. The second term can be associated with the continuity of the intermediate image. The third term can be associated with the sparsity of the intermediate image. The systems and methods can perform a second iterative operation on the intermediate image to generate a target image based on a second objective function. The second objective function can be associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image.

[0046] According to the system and method of the present application, the image data generated by the image acquisition device can be reconstructed, and the fidelity (and / or integrity), continuity and sparsity of the image information can be considered to generate a target image, thereby improving the reconstruction speed and reducing photobleaching. Compared with the reference image generated by only considering the fidelity and continuity, the processed (or reconstructed) target image can have an improved resolution. More importantly, the system and method can extend the resolution of the target image beyond the physical limitations of the image acquisition device, and can be expanded within the bandwidth limited by the image acquisition device.

[0047] In addition, although the systems and methods disclosed in the present application are mainly about the processing of images generated by structured light microscopes, it should be understood that the description is for illustrative purposes only and is not intended to limit the scope of the present application. The systems and methods of the present application can be applied to image processing of any other type of system including an image acquisition device. For example, the systems and methods of the present application can be applied to microscopes, telescopes, cameras (e.g., surveillance cameras, camera phones, web cameras), drones, medical imaging devices, etc. or any combination thereof.

[0048] Merely as an example, the methods disclosed in this application can be used to process images generated by structured light microscopy (SIM) and obtain reconstructed images with relatively high resolution. In some embodiments, modern live cell SR microscopes may ultimately be limited by the maximum collected photon flux in order to temporarily encode SR information through specific optical and fluorescent switch indicators. Exploiting the prior knowledge of the sparsity and continuity of fluorescent structures, the present application provides a mathematical deconvolution algorithm for extended resolution beyond the physical limitations imposed by the hardware. As a result, sparse structured light microscopy (Sparse-SIM) (e.g., Figure 5 SIM using sparse reconstruction (e.g., sparse deconvolution) has been shown to achieve approximately 60 nm resolution at approximately 2 ms temporal resolution, enabling resolution of complex structures (e.g., small vesicle fusion pores, relative motion between the inner and outer mitochondrial membranes in living cells, etc.). Similarly, sparse spinning disk-based confocal SIM can allow routine four-color and three-dimensional live-cell SR imaging (with approximately 90 nm resolution), demonstrating the general applicability of sparse deconvolution.

[0049] The advent of SR fluorescence microscopy has the potential to revolutionize biology and uncover previously unappreciated complex structures, such as periodic actin in neuronal dendrites, the complex structure of nuclear pores, and the organization of pericentriolar material around centrioles. However, many of these previous experiments were performed in fixed cells, which lose the dynamic structural and functional changes of cellular architecture.

[0050] Although theoretically all major SR techniques have infinite resolution, the spatial resolution of live-cell SR microscopy is limited. On the one hand, for a limited number of fluorophores within the cell volume, an n-fold increase in spatial resolution in the XYZ axis may result in a reduction of N voxels. 3 times, need to increase more than N 3 times the illumination intensity to maintain the same contrast. In addition, because multiple raw images are typically taken to reconstruct a single SR image, relatively small voxels are more likely to be affected by motion artifacts from moving molecules in living cells, which can reduce the achievable resolution. Therefore, increases in spatial resolution may need to be matched by increases in temporal resolution to maintain meaningful resolution. Given the saturation of fluorescence emission under excessive illumination, the current highest spatial resolution of intravital SR microscopy is limited to, for example, roughly 60 nm, regardless of the imaging modality used. To achieve resolution, excessive illumination power (e.g., kW-mw / mm2) and / or long exposures (e.g., >2 s) are often required, which can compromise the integrity of the overall fluorescent structure and the practical resolution of rapidly moving subcellular structures, such as tubular endoplasmic reticulum, lipid droplets, mitochondria, and lysosomes.

[0051] Compared to other types of SR imaging techniques, structured light microscopy (SIM) may have higher photon efficiency but may be subject to artifacts. By developing Hessian deconvolution for SIM reconstruction, high-fidelity SR imaging can be achieved using only one-tenth of the illumination dose required in conventional SIM, which enables ultra-fast, hour-long imaging in living cells. However, Hessian SIM may suffer from resolution limitations, e.g., 90–110 nm, imposed by linear spatial frequency mixing, which prevents structures such as caveolae or vesicle fusion pores from being well resolved. Although nonlinear SIM can be used to achieve resolutions of approximately 60 nm, such implementations may come at the expense of reduced temporal resolution and may require photoactivated or photoswitchable fluorescent proteins that are susceptible to photobleaching. In addition, because conventional SIM uses wide-field illumination, fluorescence emission and scattering from out-of-focus planes may affect the contrast and resolution of images deep within the cytoplasm and nucleus. Although total internal reflection illumination or grazing incidence illumination can be combined with SIM, high-contrast SR-SIM imaging may be largely limited to imaging depths, e.g., 0.1–1 μm. It can be challenging for SR techniques to achieve millisecond frame rates with spatiotemporal resolution of approximately 60 nm (e.g., in living cells), or to be able to perform multicolor, three-dimensional, and / or long-term SR imaging.

[0052] As proposed in the 1960s and 1970s, mathematical bandwidth extrapolation can be used to improve spatial resolution. Therefore, when the imaged object has a finite size, there may be a unique analysis function that coincides within the bandwidth-limited spectral band of the optical transfer function (OTF) of the image acquisition device, allowing the complete object to be reconstructed by extrapolating the observed spectrum. However, these bandwidth extrapolation operations may fail in practical applications because the reconstruction depends critically on the accuracy and availability of assumed prior knowledge. By replacing specific prior knowledge about the object itself with a more general sparse assumption, the compressed sensing paradigm can make SR possible in proof-of-principle experiments. However, unless the measurements are performed in the near field, such reconstructions may be unstable, and their resolution limit is inversely proportional to the SNR. Therefore, although theoretically possible, it may be challenging to demonstrate mathematical SR techniques in real biological experiments. More description of mathematical SR techniques can be found elsewhere in this application (e.g., Figure 4-6 More descriptions of biological experiments can be found elsewhere in this application (e.g., Examples 1-5, Figure 7-2 0 and its description).

[0053] By combining sparsity and continuity priors, we can provide a deconvolution algorithm that can transcend the physical limitations of the hardware of the image acquisition device and simultaneously improve the resolution and contrast of live-cell SR microscopy images. By applying the sparse deconvolution algorithm to widefield microscopy, SIM, and spinning disk-based confocal SIM (SD-SIM), we can demonstrate legitimate mathematical SR that allows complex subcellular structures and dynamics to be visualized in living cells.

[0054] It should be understood that the application scenarios of the systems and methods disclosed herein are only some exemplary embodiments provided for illustrative purposes, and are not intended to limit the scope of the present application. For those of ordinary skill in the art, various changes and modifications can be made based on the description of the present application.

[0055] Figure 1 is an exemplary schematic diagram of an application scenario of an image processing system according to some embodiments of the present application. Figure 1 As shown, the image processing system 100 may include an image acquisition device 110 , a network 120 , one or more terminals 130 , a processing device 140 , and a storage device 150 .

[0056] The components in the image processing system 100 may be connected in one or more ways. By way of example only, the image acquisition device 110 may be connected to the processing device 140 via the network 120. For another example, the image acquisition device 110 may be directly connected to the processing device 140, as indicated by the double-headed arrow in the dashed line connecting the image acquisition device 110 and the processing device 140. For another example, the storage device 150 may be connected to the processing device 140 directly or via the network 120. For another example, the terminal 130 may be connected to the processing device 140 directly (as indicated by the double-headed arrow in the dashed line connecting the terminal 130 and the processing device 140) or via the network 120.

[0057] The imaging processing system 100 may be configured to use sparse deconvolution techniques (e.g., Figure 5 The target image may have a relatively high resolution, which may exceed the physical limitations imposed by the image acquisition device 110. At least two target images in a time series may form a video stream with a relatively high resolution. For example, the image processing system 100 may obtain at least two raw cell images with a relatively low resolution generated by the image acquisition device 110 (e.g., SIM). The image processing system 100 may generate an SR cell image based on at least two raw cell images using a sparse deconvolution technique. For another example, the image processing system 100 may obtain one or more images captured by the image acquisition device 110 (e.g., a camera phone or a mobile phone with a camera). One or more images may be blurred and / or have a relatively low resolution due to factors such as shaking of the camera phone, movement of the imaged object, inaccurate focus, etc. during the capture and / or physical limitations imposed by the camera phone. The image processing system 100 may process the image using a sparse deconvolution technique to generate one or more target images with a relatively high resolution. The processed target image may be clearer than an image not processed by the sparse deconvolution technique. Therefore, the image processing system 100 may provide a user of the image processing system 100 with a target image having a relatively high resolution.

[0058] The image acquisition device 110 may be configured to obtain image data associated with a target within its detection area. In the present application, "object" and "target" are used interchangeably. The target may include one or more biological or non-biological objects. In some embodiments, the image acquisition device 110 may be an optical imaging device, a radiation-based imaging device (e.g., a computer tomography device), a nuclide-based imaging device (e.g., a positron emission tomography device), a magnetic resonance imaging device), etc. Exemplary optical imaging devices may include a microscope 111 (e.g., a fluorescence microscope), a monitoring device 112 (e.g., a security camera), a mobile terminal device 113 (e.g., a camera phone), a scanning device 114 (e.g., a flatbed scanner, a drum scanner, etc.), a telescope, a web camera, etc., or any combination thereof. In some embodiments, the optical imaging device may include a capture device (e.g., a detector or a camera) for collecting image data. For illustrative purposes, the present application takes a microscope 111 as an example to describe the exemplary functions of the image acquisition device 110. Exemplary microscopes may include structured light microscopy (SIM) (e.g., two-dimensional SIM (2D-SIM), three-dimensional SIM (3D-SIM), total internal reflection SIM (TIRF-SIM), spinning disk-based confocal SIM (SD-SIM), etc.), photoactivated localization microscopy (PAMP), stimulated emission depletion fluorescence microscopy (STED), stochastic optical reconstruction microscopy (STORM), etc. SIM may include detectors such as EMCCD cameras, SCMOS cameras, etc. The targets detected by SIM may include one or more objects of biological structures, biological tissues, proteins, cells, microorganisms, etc., or any combination thereof. Exemplary cells may include INS-1 cells, COS-7 cells, HELA cells, liver sinusoidal endothelial cells (LSECs), human umbilical vein endothelial cells (HUVECs), HEK293 cells, etc., or any combination thereof. In some embodiments, one or more objects may be fluorescent or fluorescently labeled. Fluorescent or fluorescently labeled objects may emit fluorescence for imaging. More description about an exemplary SIM device can be found in Chinese patent application No. 2010196491.X, filed on March 9, 2018, entitled “A method and system for image reconstruction”, the contents of which are incorporated herein by reference.

[0059] The network 120 may include any suitable network that may facilitate the image processing system 100 to exchange information and / or data. In some embodiments, one or more components of the image processing system 100 (e.g., the image acquisition device 110, the terminal 130, the processing device 140, the storage device 150, etc.) may communicate information and / or data with another component via the network 120. For example, the processing device 140 may acquire image data from the image acquisition device 110 via the network 120. For another example, the processing device 140 may obtain a user instruction from the terminal 130 via the network 120. The network 120 may be and / or include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi network, etc.), a cellular network (e.g., a long term evolution (LTE) network), an image relay network, a virtual private network (“VPN”), a satellite network, a telephone network, a router, a hub, a switch, a server computer, and / or a combination of one or more thereof. For example, the network 120 may include a cable network, a wired network, an optical fiber network, a telecommunication network, a local area network, a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication network (NFC), etc. or a combination thereof. In some embodiments, the network 120 may include one or more network access points. For example, the network 120 may include a wired and / or wireless network access point, such as a base station and / or a network switching point, through which one or more components of the image processing system 100 can access the network 120 for data and / or information exchange.

[0060] In some embodiments, the user can operate the image processing system 100 through the terminal 130. The terminal 130 may include a mobile device 131, a tablet computer 132, a laptop computer 133, etc. or a combination thereof. In some embodiments, the mobile device 131 may include a smart home device, a wearable device, a mobile device, a virtual reality device, an augmented reality device, etc. In some embodiments, the smart home device may include a smart lighting device, a smart electrical device, a smart monitoring device, a smart TV, a smart camera, a control device for an intercom, etc. or a combination thereof. In some embodiments, the wearable device may include a bracelet, a shoe, a pair of glasses, a helmet, a watch, a piece of clothing, a backpack, a smart accessory, etc. or a combination thereof. In some embodiments, the mobile device may include a mobile phone, a personal digital assistant (PDA), a gaming device, a navigation device, a point of sale (POS) device, a laptop computer, a tablet computer, a desktop computer, etc. or a combination thereof. In some embodiments, the virtual reality device and / or the augmented reality device may include a virtual reality helmet, virtual reality glasses, virtual reality goggles, an augmented reality helmet, augmented reality glasses, augmented reality goggles, etc. or a combination thereof. For example, the virtual reality device and / or the augmented reality device may include Google Glass. TM 、Oculus Rift TM 、Hololens TM or GearVR TM Etc. In some embodiments, the terminal 130 may be a part of the processing device 140 .

[0061] The processing device 140 can process data and / or information obtained from the image acquisition device 110, the terminal 130 and / or the storage device 150. For example, the processing device 140 can process the image data generated by the image acquisition device 110 to generate a target image with a relatively high resolution. In some embodiments, the processing device 140 can be a server or a server group. The server group can be centralized or distributed. In some embodiments, the processing device 140 can be local or remote. For example, the processing device 140 can access information and / or data stored in the image acquisition device 110, the terminal 130 and / or the storage device 150 via the network 120. For another example, the processing device 140 can be directly connected to the image acquisition device 110, the terminal 130 and / or the storage device 150 to access the stored information and / or data. In some embodiments, the processing device 140 can be implemented on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc. or a combination thereof. In some embodiments, the processing device 140 can be provided by a computer having a plurality of computer systems such as a processor, ... Figure 2 A computing device 200 implementation is shown having one or more components.

[0062] The storage device 150 may store data, instructions, and / or any other information. In some embodiments, the storage device 150 may store data obtained from the terminal 130, the image acquisition device 110, and / or the processing device 140. In some embodiments, the storage device 150 may store data and / or instructions, and the processing device 140 may execute or use it to execute the exemplary method described in the present application. In some embodiments, the storage device 150 may include a large-capacity storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), etc. Exemplary large-capacity storage devices may include magnetic disks, optical disks, solid-state hard disks, etc. Exemplary removable storage devices may include flash drives, floppy disks, optical disks, memory cards, compressed disks, tapes, etc. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAM may include dynamic random access memory (DRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), static random access memory (SRAM), thyristor random access memory (T-RAM), and zero capacitance random access memory (Z-RAM), etc. Exemplary ROMs may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EPROM),

[0063] In some embodiments, the storage device 150 may be executed on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or a combination thereof.

[0064] In some embodiments, the storage device 150 may be connected to the network 120 to communicate with one or more other components of the image processing system 100 (e.g., the processing device 140, the terminal 130, etc.). One or more components of the image processing system 100 may access data or instructions stored in the storage device 150 via the network 120. In some embodiments, the storage device 150 may be directly connected to one or more other components of the image processing system 100 (e.g., the processing device 140, the terminal 130, etc.). In some embodiments, the storage device 150 may be part of the processing device 140.

[0065] Figure 22 is an exemplary schematic diagram of a computing device according to some embodiments of the present application; the computing device 200 can be used to implement any component of the image processing system 100 as described herein. For example, the processing device 140 and / or the terminal 130 can be implemented on the computing device 200 through its hardware, software program, firmware or a combination thereof. Although only one such computing device is shown for convenience, multiple similar platforms can be used to implement computer functions related to the image processing system 100 described herein in a distributed manner to distribute the processing load.

[0066] like Figure 2 As shown, computing device 200 may include processor 210 , memory 220 , input / output (I / O) 230 , and communication port 240 .

[0067] The processor 210 can execute computer instructions (e.g., program code) and perform functions of the image processing system 100 (e.g., processing device 140) according to the techniques described herein. The computer instructions may include, for example, routines, programs, objects, components, data structures, processes, modules, and functions that perform specific functions described herein. For example, the processor 210 can process image data obtained from any component of the image processing system 100. In some embodiments, the processor 210 may include one or more hardware processors, such as microcontrollers, microprocessors, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc. or a combination thereof.

[0068] For illustration purposes only, only one processor is described in the computing device 200. However, it should be noted that the computing device 200 in the present application may also include multiple processors, and thus the operations and / or method operations performed by one processor as described in the present application may also be performed by the multiple processors together or separately. For example, if in the present application, the processor of the computing device 200 performs operation A and operation B, it should be understood that operation A and operation B may also be performed jointly or separately by two or more different processors in the computing device 200 (e.g., the first processor performs operation A and the second processor performs operation B, or the first and second processors perform operations A and B together).

[0069] The memory 220 may store data / information obtained from any component of the image processing system 100. In some embodiments, the memory 220 may include a mass storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. Exemplary mass storage devices may include magnetic disks, optical disks, solid-state hard disks, etc. Removable storage devices may include flash drives, floppy disks, optical disks, memory cards, compressed disks, tapes, etc. Volatile read-write memory may include random access memory (RAM). RAM may include dynamic RAM (DRAM), dual date rate synchronous dynamic RAM (DDRSDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero capacitance RAM (Z-RAM), etc. Among them, ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), and digital versatile disk ROM, etc.

[0070] In some embodiments, the memory 220 may store one or more programs and / or instructions to perform the exemplary methods described in this application. For example, the memory 220 may store a program for the processing device 140 to process the image generated by the image acquisition device 110.

[0071] The I / O 230 may input and / or output signals, data, information, etc. In some embodiments, the I / O 230 may enable a user to interact with the image processing system 100 (e.g., the processing device 140). In some embodiments, the I / O 230 may include an input device and an output device. Examples of input devices may include a keyboard, a mouse, a touch screen, a microphone, etc., or a combination thereof. Examples of output devices may include a display device, a speaker, a printer, a projector, etc., or a combination thereof. Examples of display devices may include a liquid crystal display (LCD), a light emitting diode (LED), a display based on a light emitting diode (LED), a flat display, a curved screen, a television device, a cathode ray tube (CRT), a touch screen, etc., or a combination thereof.

[0072] The communication port 240 can be connected to a network to facilitate data communication. The communication port 240 can establish a connection between the processing device 140 and the image acquisition device 11, the terminal 130 and / or the storage device 150. The connection can be a wired connection, a wireless connection, any other communication connection that can achieve data transmission and / or reception, and / or any combination of these connections. The wired connection can include, for example, an electrical cable, an optical cable, a telephone line, etc., or a combination thereof. The wireless connection can include Bluetooth TM Link, Wi-Fi TM Link, WiMAX TMLink, WLAN link, ZigBee TM In some embodiments, the communication port 240 may be a wireless communication port, such as a wireless communication port such as a 3G, 4G, or 5G wireless communication port, or a combination thereof. In some embodiments, the communication port 240 may be and / or include a standardized communication port, such as RS232, RS485, or the like. In some embodiments, the communication port 240 may be a specially designed communication port. For example, the communication port 240 may be designed according to the Digital Imaging and Communications in Medicine (DICOM) protocol.

[0073] Figure 3 This is an exemplary block diagram of a mobile device in which a terminal is located according to some embodiments of the present application.

[0074] like Figure 3 As shown, the mobile device 300 may include a communication unit 310, a display unit 320, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, an I / O 350, a memory 360, a storage unit 370, etc. In some embodiments, any other suitable components, including but not limited to a system bus or a controller (not shown), may also be included in the mobile device 300. In some embodiments, an operating system 361 (e.g., IOS TM 、Android TM 、Windows Phone TM 300) and one or more applications 362 may be loaded from storage unit 370 into memory 360 for execution by CPU 340. Application 362 may include a browser or any other suitable mobile application for receiving and presenting information related to imaging, image processing, or other information from image processing system 100 (e.g., processing device 140). User interaction with the information stream may be achieved through I / O 350 and provided to processing device 140 and / or other components of image processing system 100 via network 120. In some embodiments, a user may input parameters into image processing system 100 via mobile device 300.

[0075] In order to implement the above-mentioned various modules, units and their functions, a computer hardware platform can be used as one or more elements (for example, Figure 1 The computer with the user interface may be used as a personal computer (PC) or other type of workstation or terminal device. After being properly programmed, the computer with the user interface may be used as a server. It is contemplated that the person skilled in the art may also be familiar with the structure, procedures or general operation of this type of computing device.

[0076] Figure 4 yes Figure 4 is an exemplary schematic diagram of a processing device according to some embodiments of the present application. Figure 4 As shown, the processing device 140 may include an acquisition module 402 , an initial image generation module 404 , a background estimation module 406 , an upsampling module 408 , a sparse reconstruction module 410 , and a deconvolution module 412 .

[0077] The acquisition module 402 may be configured to obtain information and / or data from one or more components of the image processing system 100. In some embodiments, the acquisition module 402 may obtain image data from the storage device 150 or the image acquisition device 110. As used herein, image data may refer to raw data (e.g., one or more raw images) acquired by the image acquisition device 110. Further description of obtaining image data may be found elsewhere in this application (e.g., Figure 5 Operation 502 in ).

[0078] The initial image generation module 404 may be configured to generate an initial image. In some embodiments, the initial image generation module 404 may determine the initial image by performing a filtering operation on the image data. For example only, the initial image generation module 404 may determine the initial image by performing SIM on one or more original images. More descriptions on generating the initial image may be found elsewhere in this application (e.g., Figure 5 Operation 504 in ) is found.

[0079] Background estimation module 406 can be configured to generate an estimated background image based on the initial image. In some embodiments, background estimation module 406 can generate an estimated background image by performing an iterative wavelet transform operation on the initial image. For example, background estimation module 406 can use a wavelet filter to decompose the initial image into one or more frequency bands in the frequency domain. Exemplary wavelet filters may include Daubechies wavelet filter, biorthogonal wavelet filter, Morlet wavelet filter, Gaussian wavelet filter, Marr wavelet filter, Meyer wavelet filter, Shannon wavelet filter, Battle-Lemarie wavelet filter, etc. or any combination thereof. Background estimation module 406 can extract the lowest frequency band (for example, the lowest frequency wavelet band) of the frequency domain from one or more frequency bands in the frequency domain. Background estimation module 406 can generate an estimated image based on the lowest frequency wavelet band. If the estimated image meets the first termination condition, background estimation module 406 can determine the estimated image as the estimated background image. Or, if the estimated image does not meet the first termination condition, background estimation module 406 can extract the lowest frequency wavelet band from the estimated image to determine the updated estimated image until the first termination condition is met. Further description of generating an estimated background image can be found elsewhere in this application (e.g., Figure 5 Operation 506 in Figure 6 operations and their descriptions in ).

[0080] The upsampling module 408 may be configured to perform an upsampling operation. In some embodiments, the upsampling module 408 may determine a residual image corresponding to the initial image. For example, the upsampling module 408 may extract an estimated background image from the initial image to determine the residual image. The upsampling module 408 may generate the upsampled image by performing an upsampling operation on the residual image, for example, performing Fourier interpolation on the residual image. In some embodiments, the upsampled image may have a larger size than the residual image. More description of generating an upsampled image based on the initial image may be found elsewhere in this application (e.g., Figure 5 Operation 508 and its description in ).

[0081] The sparse reconstruction module 410 may be configured to perform a sparse reconstruction operation. In some embodiments, the sparse reconstruction module 410 may perform a first iterative operation on the initial image based on a first objective function to generate an intermediate image. The first objective function may include a first term (also referred to as a fidelity term), a second term (also referred to as a continuity term), and a third term (also referred to as a sparse term). The first term may be associated with a first difference between the intermediate image and the initial image. The second term may be associated with the continuity of the intermediate image. The third term may be associated with the sparsity of the intermediate image. In some embodiments, the sparse reconstruction module 410 may determine an initial estimated intermediate image based on the initial image. If the upsampling operation is omitted, the initial estimated intermediate image may refer to a residual image determined based on the initial image. Alternatively, if an upsampling operation is required, the initial estimated intermediate image may refer to an upsampled image determined based on the residual image. The sparse reconstruction module 410 may perform at least two iterations of the first objective function based on the initial estimated intermediate image to update the estimated intermediate image. In each of the at least two iterations, the sparse reconstruction module 410 may determine whether the second termination condition is met. In response to determining that the second termination condition is satisfied, the sparse reconstruction module 410 may determine an intermediate image and designate the estimated intermediate image in the iteration as the intermediate image. In response to determining that the second termination condition is not satisfied, the sparse reconstruction module 410 may determine an estimated intermediate image updated in the iteration. In addition, the sparse reconstruction module 410 may perform a next level iteration of the first objective function based on the updated estimated intermediate image. More description of generating an intermediate image based on the initial image may be found elsewhere in this application (e.g., Figure 5 Operation 510 and its description in FIG. 5 ).

[0082] The deconvolution module 412 can be configured to perform a deconvolution operation on the intermediate image to determine the target image. In some embodiments, the deconvolution module 412 can perform a second iterative operation on the intermediate image based on a second objective function to generate the target image. The second objective function can be associated with the system matrix of the image acquisition device 110 and the second difference between the intermediate image and the target image. The second difference may include a third difference between the intermediate image and the target image multiplied by the system matrix. In some embodiments, the second iterative operation may include using the Landweber (LW) deconvolution algorithm, the RichardsonLucy (RL) deconvolution algorithm, the like, or any combination thereof. More description of generating the target image can be found elsewhere in this application (e.g., Figure 5 Operation 512 and its description in FIG. 5 ).

[0083] It should be noted that the description of the modules of the above-mentioned processing device 140 is only for the purpose of illustration, not for the purpose of limiting the present application. For those of ordinary skill in the art, under the teaching of the present application, without departing from the principle of the present application. Modules can be combined in various ways or connected with other modules as a subsystem. In some embodiments, one or more modules can be added or omitted in the processing device 140. For example, the initial image generation module 404 can be omitted, and the acquisition module 402 can directly obtain the initial image from one or more components of the processing system 100 (e.g., image acquisition device 110, storage device 150, etc.). In some embodiments, one or more modules in the processing device 140 can be integrated into a single module to perform the functions of one or more modules. For example, modules 406-410 can be integrated into a single module.

[0084] In some embodiments, the imaging of the aberration-free optical imaging system can be controlled by a simple propagation model of electromagnetic waves interacting with the geometrically ideal surfaces of lenses and mirrors in the optical imaging system. In this case, the process can be described as the two-dimensional Fourier transform in equation (1):

[0085] G(u,v)=H(u,v)F(u,v), (1)

[0086] Where G represents the two-dimensional Fourier transform of the recorded image, F represents the transform of the intensity distribution of the object associated with the image, and H represents the spatial frequency transfer function (i.e., OTF) of the optical imaging system. Because the total amount of light energy entering any optical imaging system is limited by the physically limited pupil or aperture, the OTF can be controlled by the aperture function under aberration-free conditions. Therefore, the OTF H(u,v) can be calculated from the aperture function a(x,y)

[0087]

[0088] Thus, the OTF may be the autocorrelation of the aperture function, with x and y representing coordinates in the spatial domain, and u and v representing coordinates in the spatial frequency domain. The transmission of electromagnetic waves within the aperture may be perfect, with a(x, y) = 1, while outside the aperture no waves may propagate and a(x, y) = 0. Thus, the OTF may vary to zero outside the boundaries defined by the autocorrelation of the aperture function. Since no spatial frequency information of objects outside the OTF support is transmitted to the recording device, the optical imaging system itself may always be bandwidth limited.

[0089] In some embodiments, in order to recover the information of the original object from the recorded image, equation (1) can be rearranged into the following form:

[0090]

[0091] In the preceding equation, the retrieved object spatial frequency can only exist when H(u,v)≠0. Outside the OTF support region, the object spatial frequency may be ambiguous, and any finite value of F(u,v) can be consistent with equations (1) and (3). Therefore, there may be an infinite number of invisible small objects whose Fourier transform is zero inside the OTF of the optical imaging system and are therefore detected. This is why diffraction limits the resolution of images and leads to the common belief that information lost due to the cutoff of the system's OTF cannot be recovered by mathematical inversion or deconvolution processes.

[0092] In some embodiments, in optics, the spatial bandwidth of an optical imaging system may be traditionally considered to be constant. However, similar to communication systems, where the channel capacity may be constant, but the temporal bandwidth may not be constant. In some embodiments, it may be proposed that only the degrees of freedom (DOF) of the optical imaging system are constant. Acting as a communication channel to transmit information from the object plane to the image plane, the ultimate limit may be set by the DOF that the optical imaging system can transmit.

[0093] The DOF of any optical field can be the minimum number of independent variables required to describe the optical signal. According to the Huygens principle, if there is no absorption, the two-dimensional information carried by the light wave can remain unchanged during propagation. Therefore, the information capacity of a two-dimensional optical imaging system can be analyzed based on Shannon's information theory, and then the time-bandwidth term is added, and finally the z-direction, DC, noise, and polarization terms are added.

[0094] The total information capacity of an optical imaging system can be described as follows:

[0095]

[0096] Among them B x , B y and B z represents the spatial bandwidth, L x , L y , L z denote the dimensions of the field of view in the x, y, and z axes, respectively. For any given observation time T, B t represents the temporal bandwidth of the system. In equation (4), s represents the average power of the detection signal and n represents the additive noise power. The factor 2 represents the two polarization states.

[0097] Exemplary current hardware-based SR techniques, including SIM, STED, and PALM / STORM, can rely on the assumption that the object does not change during the SR image formation process. It is obvious from equation (4) that SR spatial information can be encoded within a temporal sequence of multiple exposures. In this sense, the information capacity of the optical imaging system can remain unchanged because the increased spatial resolution comes at the expense of reduced temporal resolution.

[0098] In some embodiments, mathematical bandwidth expansion can be performed and can operate under different principles than optics. In some embodiments, the basic principles can include that the two-dimensional Fourier transform of a spatially bounded function is an analytical function in the spatial frequency domain. If the analytical function in the spatial frequency domain is known on a finite interval, it can be shown that a unique entire function can be found by the method of analytical continuation. For optical imaging systems, if an object is of finite size, then there may be a unique analytical function within G(u,v). By inferring the observed spectrum using an algorithm such as the Gerchberg algorithm, the object can in principle be reconstructed to arbitrary precision.

[0099] Therefore, if the object is assumed to be analytic, it is possible to recover information from outside the diffraction limit. Therefore, analytic continuation can represent a key feature of mathematical SR (e.g., adding a priori information about the object for SR reconstruction). Given equation (4), adding more information about the object can increase the total information capacity of the system, which can theoretically be transferred to improve the spatial resolution.

[0100] The validity of this theory can be proved by iterative deconvolution operation under certain conditions. The basic algorithm calculation can be given by the following equation:

[0101]

[0102] For image data g(x,y) and point spread function (psf) h(x,y), (n+1) th The object of the iteration can be th The estimated calculation is performed in the space domain. Unlike the direct inversion of equation (3) in the spatial domain, this calculation can be performed in the spatial domain, avoiding the problem of division by zero outside the OTF support region. Therefore, expansion beyond the diffraction limit is possible and may occur for two reasons: n The product with the other quantities in square brackets is nonlinear and has prior information about the nonnegative value of the object.

[0103] Although mathematical SR is theoretically feasible, bandwidth expansion is often unstable due to small changes in the image data. The noise present in actual experiments may not be an analytical function, thus not allowing the solution of bandwidth expansion based on analytical continuation. In addition, even in cases where a solution may exist, there may be many analytical functions that can reproduce the data well within the OTF, but result in completely different frequency components outside the OTF.

[0104] From an information capacity perspective, bandwidth extension based on analytical continuation may extend the spatial bandwidth at the expense of signal-to-noise ratio. For object fields containing large spatial bandwidth products, even a small extension of the spatial bandwidth may lead to a large amplification of noise in the final reconstruction. For object domains containing small spatial bandwidth products, meaningful resolution improvements can be obtained by incorporating more object information for analytical continuation. Despite these theoretical possibilities, since bandwidth extension reconstruction is susceptible to noise and strongly relies on assumed prior knowledge, it can be concluded that the Rayleigh diffraction limit represents a practical boundary that cannot be overcome by conventional imaging systems.

[0105] In some embodiments, compressed sensing (CS) performed based on sparse features of discrete signals can be successfully applied in many different signal processing fields. CS can be a method that can achieve accurate signal reconstruction with fewer measurements than required by the Nyquist-Shannon sampling theorem. CS theory can assume that discrete signals have sparse representations on some basis. If measurements are made using a basis that is unrelated or incoherent to the signal basis, the signal can be reconstructed by sampling at a rate twice the maximum frequency of the signal. CS can be combined with a point scanning confocal microscope and can be used for single molecule localization (pointillist) super-resolution techniques such as PALM and STORM. In some embodiments, applying CS techniques to these imaging techniques can reduce image acquisition time and increase frame rate.

[0106] In addition to bandwidth interpolation using sparsity priors, bandwidth expansion and mathematical SR can also be performed using sparsity priors. Because the spatial and frequency domains are maximally uncorrelated, and the object is measured in the spatial domain, the band-limited frequency information is expanded under the assumption of a sparse signal in the spatial domain. It can be shown that the bandwidth can be expanded under the CS framework and a more accurate reconstruction can be generated from the original image with some noise than using the Gerchberg algorithm. This can provide a proof of principle when the object is very sparse and does not exhibit sub-wavelength resolution. In some embodiments, because the noise in the original image is significantly amplified during reconstruction, the sub-wavelength mode may generally be unstable unless measured in the near field. Similarly, the resolution can be inversely proportional to the SNR, which limits the application of CS techniques.

[0107] In some embodiments, the above SR techniques based on CS techniques (e.g., using sparsity) can be combined with continuity priors for image processing, which can maintain a balance between resolution and SNR to improve the resolution of image data. Details on the mathematical SR techniques combining continuity priors and sparsity priors can be described as follows (e.g., Figure 5-6 and its description).

[0108] Figure 5 500 is a flowchart of an exemplary process of image processing according to some embodiments of the present application. In some embodiments, process 500 may be performed by image processing system 100. For example, process 500 may be implemented as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, memory 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of computing device 200, CPU 340 of mobile device 300, and / or Figure 4 The one or more modules shown in the figure can execute the set of instructions and can be instructed to perform process 500 accordingly. The operations of the process shown below are for illustration purposes only. In some embodiments, the process can be completed with one or more additional operations not described and / or without one or more operations discussed. In addition, as shown in FIG. Figure 5 The order in which the process operations are illustrated and described below is not limiting.

[0109] In some embodiments, two criteria may need to be met to mathematically improve the resolution. First, the prior assumption can add information beyond the otherwise unchanged total information carried by the optical imaging system. In this sense, the sparsity prior is more general than the assumption of object compactness, and incorporating more available information about the object in the reconstruction algorithm may lead to greater bandwidth expansion. Second, mathematical SR image reconstruction can improve spatial resolution, but at the expense of decreasing the signal-to-noise ratio in a logarithmic relationship, suggesting that it is critical to start with a high signal-to-noise ratio image before the bandwidth expansion process.

[0110] To address these problems, the present application provides a three-step process. In the first step, the SR spatial frequency encoded by the structured illumination (grid or pinhole) is estimated, extracted and recombined in the spatial frequency domain, and an inverse filter (e.g., a traditional Wiener inverse filter) is used to obtain an SR image. The first step can improve the SNR of high-frequency components within the bandwidth determined by optics. The background in the SR image can also be estimated, and the SR image can be removed using background fluorescence generated by the out-of-focus plane from the SR image, which is crucial for high-fidelity reconstruction of intracellular objects. In the second step, continuity prior information is used to further enhance the image SNR, and sparsity prior information can be used to extend the bandwidth outside the region of the OTF. Applying strict requirements on sparsity to obtain more accurate reconstruction may result in the removal of weak real signals. Therefore, compared with the classical bandwidth expansion technique that attempts to reveal all the details of the object based on analytical continuation or CS prior information, image reconstruction can be framed as an iterative process constrained by prior information.

[0111] Due to the logarithmic relationship between resolution increase and image SNR, the out-spread spatial frequencies typically have low contrast. Therefore, in the third step, an iterative deconvolution process can be used to improve the contrast of the newly spread spatial frequencies outside the OTF. Combining the non-negative prior and the nonlinearity of the deconvolution procedure may also result in an increase in resolution to a certain extent. More description of the three-step process can be found in one or more operations (e.g., operations 506, 510, and 512) in process 500.

[0112] At 502 , the processing device 140 (eg, acquisition module 402 ) may obtain image data.

[0113] In some embodiments, the image data herein may refer to raw data (e.g., one or more raw images) acquired by the image acquisition device 110. As used herein, a raw image may refer to an image with a relatively low signal-to-noise ratio (SNR), or a partially damaged image, etc. As an example only, for a SIM system, the image data may include one or more raw images acquired by the SIM system. One or more groups of raw images may be arranged in a time series. Each group of raw images may include at least two raw images (e.g., 9 raw images, 15 raw images) with different phases and / or directions. That is, at least two raw images may be acquired by the SIM system at different phases and / or directions.

[0114] In some embodiments, the processing device 140 may obtain image data from one or more components of the image processing system 100. For example, the image acquisition device 110 may acquire and / or generate image data and store the image data in the storage device 150. The processing device 140 may retrieve and / or obtain image data from the storage device 150. For another example, the processing device 140 may obtain image data directly from the image acquisition device 110.

[0115] At 504 , the processing device 140 (eg, the initial image generation module 404 ) may generate an initial image based on the image data.

[0116] In some embodiments, the processing device 140 may determine an initial image (e.g., an SR image) by filtering the image data. Exemplary filtering operations may include Wiener filtering, inverse filtering, least squares filtering, etc. or any combination thereof. For example, for each group of original images of the image data, the processing device 140 may generate an image stack (i.e., an initial image) by performing Wiener filtering on at least two original images in a group of original images. Specifically, if each group of original images includes 9 original images, the processing device 140 may combine the 9 original images in the group of original images into the initial image. The initial image may include information of each of the 9 original images and have a higher resolution than each of the 9 original images. In some embodiments, the filtering operation may be omitted. For example, for a camera phone system, the image data may include only one original image, and the processing device 140 may designate a unique original image as the initial image.

[0117] In 506, the processing device 140 (e.g., background estimation module 406) can generate an estimated background image based on the initial image. As used herein, the estimated background image can refer to an image that includes an estimated background (e.g., low-frequency noise) of the initial image. For example, if the object in the initial image is a living cell, the liquid surrounding the living cell in the initial image can be regarded as noise in the initial image. The processing device 140 can generate an estimated background image by estimating the noise in the initial image. For another example, for a 2D-SIM system, the initial image can contain a strong background fluorescence signal originating from out-of-focus emission and cellular autofluorescence, which can be regarded as noise in the initial image. The processing device 140 can predict the noise in the estimated background image.

[0118] In some embodiments, the processing device 140 may generate an estimated background image by performing an iterative wavelet transform operation on the initial image. For example, the processing device 140 may use a wavelet filter to decompose the initial image into one or more frequency bands in the frequency domain. Exemplary wavelet filters may include Daubechies wavelet filters, biorthogonal wavelet filters, Morlet wavelet filters, Gaussian wavelet filters, Marr wavelet filters, Meyer wavelet filters, Shannon wavelet filters, Battle-Lemarie wavelet filters, etc. or any combination thereof. The processing device 140 may extract the lowest frequency band in the frequency domain (e.g., the lowest frequency wavelet band) from one or more frequency bands in the frequency domain. The processing device 140 may generate an estimated image based on the lowest frequency wavelet band. If the estimated image satisfies the first termination condition, the processing device 140 may determine the estimated image as the estimated background image. Alternatively, if the estimated image does not satisfy the first termination condition, the processing device 140 may extract the lowest frequency wavelet band from the estimated image to determine an updated estimated image until the first termination condition is met. More descriptions on generating an estimated background image may be found elsewhere in this application (e.g., Figure 6 and its description) found.

[0119] In some embodiments, under certain conditions where the background noise in the initial image is low (eg, under ultrafast or long-term 2D-SIM imaging conditions), the processing device 140 may directly assign pixel values ​​in the estimated background image to zero.

[0120] At 508 , the processing device 140 (eg, the upsampling module 408 ) may generate an upsampled image based on the initial image.

[0121] In some embodiments, the processing device 140 may determine a residual image corresponding to the initial image. For example, the processing device 140 may determine the residual image by subtracting the estimated background image from the initial image. For another example, when the initial image (e.g., the initial image in the SD-SIM system) exhibits a low and stable noise-like distribution, the processing device 140 may determine the average value of the total pixel values ​​in the initial image. The processing device 140 may determine the residual image by assigning the values ​​of the pixels in the initial image that are less than the average value to zero. Further, the processing device 140 may generate an upsampled image by performing an upsampling operation on the residual image, for example, performing Fourier interpolation (e.g., OTF external zero padding of the image acquisition device 110) on the residual image. For example, the processing device 140 may pad zeros near one or more pixel points of the residual image in the spatial domain (e.g., each pixel or pixel on the edge) to determine the upsampled image. For another example, the processing device 140 may pad the residual image with zeros in the frequency domain to determine the upsampled image. For example, for a 2D-SIM system, the processing device 140 may use Fourier interpolation (e.g., padding zeros outside the OTF of SIM) to upsample the residual image. As another example, for an SD-SIM system, the processing device 140 may select a spatial oversampling method (e.g., padding zeros around each pixel in the residual image), which is less sensitive to the signal-to-noise ratio and less prone to generating snow artifacts.

[0122] In some embodiments, the upsampled image may include a larger size than the original image. For example, the original image may include a size of 256×256, and the upsampled image may include a size of 512×512.

[0123] In some embodiments, the pixel size may be a factor that limits SR image reconstruction / processing. For example, the pixel size of a 2D-SIM system may be 32.5 nm, which can theoretically be extended to approximately 60 nm, which may introduce undersampling issues that limit spatial resolution. Similarly, when an EMCCD camera is used in an SD-SIM system, the pixel size may be 94 nm, which may limit the resolution achieved by the sparse deconvolution algorithm (approximately 90 nm). To address potential issues with undersampling, an oversampling operation (i.e., an upsampling operation) may be provided during the image reconstruction process. In some embodiments, the upsampling operation may be omitted, and the residual image may be used directly for image reconstruction.

[0124] In 510, the processing device 140 (e.g., the sparse reconstruction module 410) may perform a first iterative operation of the initial image based on the first objective function to generate an intermediate image. The intermediate image may be regarded as an image restored from the initial image, so that the intermediate image may also be referred to as a restored image. The first iterative operation may also be referred to as sparse reconstruction or sparse deconvolution.

[0125] In some embodiments, the processing device 140 may determine an initial estimated intermediate image based on the initial image. If the upsampling operation is omitted, the initial estimated intermediate image may refer to the residual image as described in operation 508. Alternatively, if the upsampling operation is required, the initial estimated intermediate image may refer to the upsampled image as described in operation 508. The processing device 140 may perform at least two iterations of the first objective function based on the initial estimated intermediate image to update the estimated intermediate image. In each of the at least two iterations, the processing device 140 may determine whether the second termination condition is met. In response to determining that the second termination condition is met, the processing device 140 may determine the intermediate image and designate the estimated intermediate image of the iteration as the intermediate image. In response to determining that the second termination condition is not met, the processing device 140 may determine the estimated intermediate image updated in the iteration. The processing device 140 may perform the next iteration of the first objective function based on the updated estimated intermediate image.

[0126] In some embodiments, the second termination condition may involve the value of the first objective function. For example, if the value of the first objective function is the smallest or less than a threshold value (e.g., a constant), the second termination condition may be satisfied. For example, if the value of the first objective function converges, the second termination condition may be satisfied. In some embodiments, if the change in the value of the first objective function in two or more consecutive iterations is equal to or less than a threshold value (e.g., a constant), convergence may be considered to have occurred. In some embodiments, if the difference between the value of the first objective function (e.g., the value of the first objective function) and the target value is equal to or less than a threshold value (e.g., a constant), convergence may be considered to have occurred. In some embodiments, the second termination condition may involve the number of iterations (count) of the first objective function. For example, when a specified number of iterations (or count) is performed in the first iteration operation, the second termination condition may be satisfied. In some embodiments, the second termination condition may involve the iteration time of the first objective function (e.g., the length of time to perform the first iteration operation). For example, if the iteration time of the first objective function is greater than a threshold value (e.g., a constant), the termination condition may be satisfied.

[0127] In some embodiments, the first objective function may include a first term (also referred to as a fidelity term), a second term (also referred to as a continuity term), and a third term (also referred to as a sparsity term). The first term may be associated with a first difference between the intermediate image and the initial image. The second term may be associated with the continuity of the intermediate image. The third term may be associated with the sparsity of the intermediate image. In some embodiments, for an initial image determined without an upsampling operation, the first objective function may be expressed as the loss function shown in equation (6):

[0128]

[0129] in, Represents the first term, R Hessian (g) represents the second term, λ L1 ‖g‖1 represents the third term. In the first term, f represents the initial image (also briefly referred to as image f), b represents the estimated background image (also briefly referred to as image b), g represents the intermediate image (also briefly referred to as image g or variable g), and ‖·‖1 represents norm (i.e., L-1 norm), λ represents the first weight factor associated with the initial image fidelity of the intermediate image. In the second term, R Hessian (g) represents the Hessian matrix of the intermediate image. In the third term, ‖·‖2 represents norm (i.e. L-2 norm), λ L1 represents the second weight factor associated with the sparsity of the intermediate image. λ and λ L1 It is possible to balance image fidelity and sparsity.

[0130] In some embodiments, the second term may be associated with structural continuity along the xy-t (z) axis of the intermediate image, i.e., continuity along the x-axis is associated with the dimension of the intermediate image width, continuity along the y-axis is associated with the dimension of the intermediate image height, and continuity along the t-axis (or z-axis) is associated with the temporal dimension of the intermediate image. Hessian (g)) can be defined as equation (7):

[0131]

[0132] where g xx represents the gray value of the second-order difference of image g pixels along the x-axis, g xy represents the grayscale value of the second-order difference of image g pixels along the x-axis and y-axis, g xt represents the gray value of the second-order difference of image pixels along the x-axis and t-axis, g yx represents the gray value of the second-order difference of image g pixels along the y-axis and x-axis, g yy represents the gray value of the second-order difference of image g pixel along the y-axis, g yt represents the gray value of the second-order difference of image g pixels along the y-axis and t-axis, g tx Represents the grayscale value of the second-order difference of image g pixels along the t-axis and the x-axis, g ty Represents the grayscale value of the second-order difference of image g pixels along the t-axis and y-axis, g tt represents the gray value of the second-order difference of the image g pixel along the t axis, λ t represents the regularization parameter for continuity along the t-axis (which may be turned off after imaging high-speed moving objects), and ‖·‖1 represents the L-1 norm.

[0133] In some embodiments, finding the best solution to the loss function in equation (6) (i.e., the termination condition of the loss function) can be converted into a convex optimization problem (COP). For example, due to its fast convergence speed, the split Bregman algorithm (also known as Bregman splitting) can be used. Using Bregman splitting, the variable g can be replaced by an intermediate variable u. Then, the loss function in equation (6) can be transformed into a constrained minimization problem, such as equation (8):

[0134]

[0135] in,

[0136] In some embodiments, by using the Lagrange multiplier method to weaken the implementation constraints, the unconstrained problem of equation (9) can be obtained:

[0137]

[0138] where μ represents the Lagrange multiplier. Finally, the constraint can be strictly enforced by applying the simplified Bregman iteration via equation (10):

[0139]

[0140] Wherein, v is used to reduce the computational complexity in the first iteration operation.

[0141] In some embodiments, the iterative minimization process may be as follows:

[0142]

[0143]

[0144]

[0145] Using equations (11-13), COP (smooth convex) can be divided into three steps. Then, the final first iteration operation can be expressed as follows:

[0146]

[0147] Among them, g k+1 represents the finalized intermediate image in the last iteration, represents the x-axis direction (for example, The second-order derivative operator in , and It can be similarly defined as u k+1 and g k +1The relationship between and can be expressed as equation (15):

[0148]

[0149] Among them, u k+1 represents the intermediate variable u that is finally determined in the last iteration. Using equations (13-15), the COP can be easily solved.

[0150] In some embodiments, for the initial image determined by the upsampling operation, the loss function (6) can be rewritten as:

[0151]

[0152] Where D represents the downsampling matrix.

[0153] At 512 , the processing device 140 (eg, the deconvolution module 412 ) may perform a second iterative operation on the intermediate image based on a second objective function to generate a target image.

[0154] In some embodiments, the intermediate image determined by the first iterative operation may have low contrast in the extended spatial frequencies, which is mainly due to the convolution of the real object with the PSF of the image acquisition device 110 (e.g., 2D-SIM / SD-SIM) and the bandwidth expansion due to the sparsity and continuity constraints. The second iterative operation can be used to further increase the bandwidth obtained by the intermediate image to exceed the bandwidth set by the image acquisition device 110, for example, to eliminate the blur of the intermediate image caused by the image acquisition device 110 itself.

[0155] In some embodiments, the processing device 140 may determine an initial estimated target image based on the intermediate image. The processing device 140 may perform at least two iterations of the second objective function based on the initial estimated target image to update the estimated target image. In each of the at least two iterations, the processing device 140 may determine whether a third termination condition is met. In response to determining that the third termination condition is met, the processing device 140 may determine a target image and designate the estimated target image in the iteration as the target image. In response to determining that the third termination condition is not met, the processing device 140 may determine an updated estimated target image in the iteration. The processing device 140 may perform the next iteration of the first objective function based on the updated estimated target image. The third condition may be similar to the second condition and is not repeated here.

[0156] In some embodiments, the second objective function may be associated with the system matrix of the image acquisition device 110 and the second difference between the intermediate image and the target image. The second difference may include a third difference between the intermediate image and the target image multiplied by the system matrix. For example, the second objective function may be expressed as a loss function as follows:

[0157]

[0158] Among them, g represents the intermediate image, A represents the system matrix, x represents the target image, Ax represents the target image multiplied by the system matrix, g-Ax represents the third difference, and ‖·‖2 represents the L-2 norm.

[0159] In some embodiments, the second iterative operation may include iterative deconvolution using a Landweber (LW) deconvolution algorithm, a RichardsonLucy (RL) deconvolution algorithm, or the like, or any combination thereof. For example, for an intermediate image with a relatively high SNR (e.g., an intermediate image in a 2D-SIM system), the processing device 140 may apply Landweber (LW) deconvolution to process the intermediate image, wherein a gradient descent algorithm with Nesterov momentum acceleration may be employed:

[0160]

[0161] Among them, g N represents the sparsely reconstructed image (i.e., the intermediate image), and x j+1 represents the target image after j+1 iterations. That is, in some embodiments, in order to obtain reliable image fidelity, the second iteration operation can be terminated at an early stage to constrain the target image. The number of deconvolution iterations (i.e., the number of iterations of the second objective function) can be listed in the parameter table in the software user manual.

[0162] For another example, for an intermediate image with a relatively low SNR (e.g., an intermediate image in an SD-SIM system), the processing device 140 may apply a RichardsonLucy (RL)-based deconvolution algorithm to process the intermediate image corrupted by excessive noise. The RL-based deconvolution algorithm may be accelerated, as expressed as:

[0163]

[0164] Among them, g N represents the image after sparse reconstruction (i.e., the intermediate image), x j+1 denotes the target image after j+1 iterations, and α denotes an adaptive acceleration factor representing the computational iteration step size, which can be directly estimated from experimental results. In some embodiments, with stable convergence performance, the number of iterations required for the accelerated RL algorithm may be significantly reduced by 10 times compared to the classical RL algorithm.

[0165] In some embodiments, since equation (17) does not consider the effect of Poisson distributed noise, the RL-deconvolution algorithm described in equation (19) is not an ideal model for sparse deconvolution; nevertheless, it can show excellent performance when processing real data (e.g., intermediate images). This may happen because the original data g, as seen in the first term of equation (19), constrains the iterative deconvolution process, allowing relatively high-fidelity recovery under low signal-to-noise ratio conditions without introducing unexpected artifacts.

[0166] In some embodiments, the target image has an improved spatial resolution compared to a reference image generated based on the third objective function. The third objective function may include the first and second terms of the first objective without the third term of the first objective function. For example, the processing device 140 may perform a fourth iterative operation (similar to the first iterative operation) on the initial image based on the third objective function to determine the second intermediate image. The processing device 140 may perform a fifth iterative operation (similar to the second iterative operation) on the second intermediate image based on the second objective function to generate a reference image. The first spatial resolution of the target image may be improved by 50%-100% relative to the second spatial resolution of the reference image.

[0167] It should be noted that the above description of process 500 is provided for illustrative purposes only and is not intended to limit the scope of the present application. It should be understood that after understanding the operating principle, a person with ordinary skills in the art can arbitrarily combine any operation, add or delete any operation, or apply the operating principle to other image processing processes without departing from the principle. In some embodiments, process 500 may include one or more additional operations. For example, additional operations may be added after operation 512 to display the target image. For another example, additional operations may be added after operation 504 to pre-process the initial image. In some embodiments, one or more operations in process 500 may be omitted. For example, operation 502, operation 506, and / or operation 508 may be omitted. In some embodiments, two or more operations in process 500 may be integrated into one operation. For example, operations 502 and 504 may be integrated into one operation. For another example, operations 506-510 may be integrated into one operation. In some embodiments, operations 510 and 512 may be collectively referred to as sparse deconvolution operations (or algorithms).

[0168] Figure 6is an exemplary flowchart of background estimation according to some embodiments of the present application. In some embodiments, process 600 can be performed by image processing system 100. For example, process 600 can be implemented as a set of instructions (e.g., application program) stored in a storage device (e.g., storage device 150, memory 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of computing device 200, CPU 340 of mobile device 300, and / or Figure 4 The one or more modules shown in the figure can execute a set of instructions and can be instructed accordingly to perform process 600. The operations of the process shown below are for illustration purposes only. In some embodiments, the process can be completed by one or more additional operations not described, and / or one or more operations not discussed. In addition, as shown in FIG. Figure 6 The order of the process operations shown and described below is not limiting. In some embodiments, one or more operations of process 600 may be performed to implement the following. Figure 5 Operation 506 is shown.

[0169] At 602 , the processing device 140 (eg, the background estimation module 406 ) may determine an input image based on an initial image.

[0170] In some embodiments, the processing device 140 may directly designate the initial image as the input image. In some embodiments, the processing device 140 may pre-process the initial image and designate the pre-processed initial image as the input image. For example, the processing device 140 may determine the input image by designating the pixel values ​​in the initial image that are less than the average value of the pixels in the initial image as zero. For another example, the processing device 140 may determine the input image by correcting the initial image.

[0171] In 604, the processing device 140 (eg, the background estimation module 406) may generate a decomposed image by performing a multi-scale wavelet decomposition operation on the input image. As used herein, a decomposed image may refer to an image in the frequency domain.

[0172] In some embodiments, processing device 140 may use Figure 5The one or more wavelet filters in operation 506 of the processing device 140 perform a multiscale wavelet decomposition operation on the input image. After the multiscale wavelet decomposition operation, the processing device 140 can decompose the input image into at least two levels of frequency bands in the frequency domain. The processing device 140 can extract the lowest frequency band from the at least two frequency bands. Further, the processing device 140 can generate a decomposed image based on the lowest frequency band. For example, for a 2D-SIM system, the processing device 140 can use a 2D Daubechies-6 wavelet filter to decompose the initial image into the 7th level in the frequency domain, that is, the initial image can be decomposed into 7-level frequency bands. The processing device 140 can determine the decomposed image based on the lowest frequency band between the 7-level frequency bands.

[0173] In 606, the processing device 140 (e.g., the background estimation module 406) may generate a transformed image by performing an inverse wavelet transform operation on the decomposed image. As used herein, a transformed image may refer to an image in a spatial domain. An inverse wavelet transform operation may refer to an operation that may convert from a frequency domain to a spatial domain, such as an inverse Fourier transform operation. In some embodiments, operation 606 may be used to remove high intensity pixels resulting from an inaccurate estimate.

[0174] In some embodiments, in order to prevent accidental removal of small useful information (or signals), an inverse wavelet transform operation may be performed on the decomposed image (i.e., the lowest frequency band) to generate a transformed image in the spatial domain. For example, the processing device 140 may perform an inverse wavelet transform operation on the decomposed image to generate an intermediate transformed image. The processing device 140 may combine the intermediate image and half of the square root of the input image by maintaining the minimum value at each pixel to determine the transformed image. That is, the processing device 140 may compare the value of each pixel in the intermediate transformed image with half of the square root of the value of the corresponding pixel in the input image. As used herein, the relative position of the corresponding pixel in the input image relative to the input image may be the same as the relative position of each pixel in the intermediate transformed image. If the value of a specific pixel in the intermediate transformed image is greater than half of the square root of the value of the corresponding pixel in the input image, the processing device 140 may determine the transformed image by replacing the value of the specific pixel in the intermediate transformed image with the square root of the value of the corresponding pixel in the input image. If the value of a specific pixel in the intermediate transformed image is less than half of the square root of the value of the corresponding pixel in the input image, the processing device 140 may determine the transformed image by maintaining the value of the specific pixel in the intermediate transformed image. Additionally or alternatively, if the value of a specific pixel in the intermediate transformed image is equal to half of the square root of the value of a corresponding pixel in the input image, the processing device 140 may determine the transformed image by maintaining the value of the specific pixel in the intermediate transformed image or replacing the value of the specific pixel in the intermediate transformed image by the square root of the value of the corresponding pixel in the input image.

[0175] In 608, the processing device 140 (eg, the background estimation module 406) may generate an estimated image by performing a cutoff operation on the transformed image. In some embodiments, the estimated image may also be referred to as a low peak background estimate.

[0176] After the cutoff operation, the amplitude of the transformed image may be truncated at a certain percentage in the frequency domain. That is, the amplitude of the estimated image in the frequency domain may be a certain percentage of the amplitude of the transformed image. For example, the processing device 140 may perform a 50% cutoff operation on the transformed image. The amplitude of the estimated image may be 50% of the amplitude of the transformed image. For another example, the processing device 140 may perform a 40% cutoff operation on the transformed image. The amplitude of the estimated image may be 40% of the amplitude of the transformed image.

[0177] At 610, the processing device 140 (eg, the background estimation module 406) may determine whether a termination condition (ie, Figure 5 In response to determining that the first termination condition is satisfied, the processing device 140 may proceed to operation 612. In response to determining that the first termination condition is not satisfied, the processing device 140 may designate the estimated image determined in operation 608 as a new input image for performing iterative operations 604-608.

[0178] In some embodiments, the first termination condition may be associated with the number of iterations (count) of operations 604-608. For example, if the number of iterations is equal to a threshold count (e.g., 3), the first termination condition may be satisfied. In some embodiments, the first termination condition may be associated with a difference between two estimated images determined in two consecutive iterations. For example, if the difference between two estimated images determined in two consecutive iterations is less than a threshold, the first termination condition may be satisfied. In some embodiments, the first termination condition may be associated with an iteration time of the total iterations. For example, if the iteration time of all iterations is less than a threshold, the first termination condition may be satisfied.

[0179] In 612, the processing device 140 (eg, the background estimation module 406) may determine the estimated image as an estimated background image for the initial image. In addition, as described in operations 508-512, the processing device 140 may extract the estimated background image from the initial image to generate a residual image for sparse deconvolution.

[0180] It should be noted that the description of the above process 600 is provided only for the purpose of illustration, and not for limiting the scope of the present application. It should be understood that after understanding the operating principle, a person skilled in the art can arbitrarily combine any operation, add or delete any operation, or apply the operating principle to other image processing processes without departing from the principle. In some embodiments, one or more operations can be added or omitted in the process 600. For example, other operations can be added to store the estimated background image.

[0181] In some embodiments, fluorescence microscopy can achieve high resolution and contrast by labeling designated structures such as actin, microtubules, and vesicles, which are usually sparse in the spatial domain due to the presence of a large number of zero values. However, the final captured fluorescence image (e.g., raw image or SR image) can be blurred by the point spread function (PSF) and exhibit an increase in non-zero values ​​relative to the high-frequency information filtered by the microscope's OTF. In this regard, enhanced spatial resolution may mean accepting sparsity in SR images. However, previous attempts to recover SR information using a priori knowledge of sparsity may suffer from unstable one-step reconstruction, which depends on the logarithm of the image signal-to-noise ratio (SNR).

[0182] In this application, a three-step method for improving resolution and maintaining the fidelity of SR information is provided, such as Figure 7 and Fig.12 First, the original image can be inversely filtered using a conventional Wiener algorithm to maximize the signal-to-noise ratio of the SR component within the assembled OTF determined by the image acquisition device 110 (e.g. Figure 5 ), and removing background fluorescence from out-of-focus planes (as described in operation 504 in FIG. Figure 5 Operation 506 or Figure 6 For images that do not meet the Nyquist sampling criteria (eg, residual images), an upsampling step may be manually added (eg, as described in operation 508).

[0183] Second, to improve the reconstruction fidelity, the Hessian penalty on the continuity along the xy-t axis can be combined with the sparse term (the sum of the absolute values ​​of each element) norm), the iterative approximation extends to SR information beyond OTF (such as Figure 5 The sparse reconstruction / deconvolution may use constrained optimization to generate an optimized image g (i.e., an intermediate image) from the Wiener filtered original SIM image f (i.e., the initial image). The image (i.e., the initial image) may first be corrected for the background b (the detailed process may be found in Figure 5 Operation 506 or Figure 6 Then, g can be generated by iteratively minimizing the following loss function:

[0184]

[0185] where the first term represents the regular squared deviation of the reconstructed image (i.e., the intermediate image) from the background-corrected filtered data (i.e., the residual image), the second term represents the continuity constraint as a penalty for high second-order spatial derivatives (Hessian curvature) of the result image (i.e., the intermediate image), and the third term represents the sparsity constraint as a penalty for too many non-zero values. And, ‖·‖1 and ‖·‖2 represent and Norm, λ, and λ L1 denote the weighting factors for balancing the fidelity and sparsity of the image, respectively. The loss function can be formulated as a convex optimization problem using the split Bregman algorithm (the detailed equations and procedures are in Figure 5 14, 15, 16, and 17). While the sparsity term (the sum of the absolute values ​​of all pixels in the image stack) allows for some extension beyond the bandwidth limit of the image acquisition device 110 (FIGS. 13, 14, 15, 16, and 17), the fidelity term (the Hessian penalty for continuity along the xy-t axis) ensures that the reconstruction has a robust signal-to-noise ratio that is resistant to artifacts (FIGS. 14 and 17).

[0186] Finally, another iterative deconvolution step (e.g., Figure 5 The contrast of high-frequency components beyond the OTF range determined by the image acquisition device 110 can be further enhanced based on the maximum a posteriori estimation (Figure 5) (see the specific steps for details). Figure 5 Operation 512 in ).

[0187] The present application is further described according to the following examples, which should not be construed as limiting the scope of the present application.

[0188] Example

[0189] Materials and methods

[0190] Cell maintenance and preparation.

[0191] INS-1 cells were cultured in RPMI1640 medium (Gibco, 11835-030) supplemented with 10% FBS (GIBCO), 1% 100 mM pyruvate solution and 0.1% 55 mM 2-thiol (GIBCO, 21985023) and cultured in an incubator at 37°C and 5% CO2 to about 75% confluence. COS-7 cells and HeLa cells were cultured in high-glucose DMEM (GIBCO, 21063029) supplemented with 10% fetal bovine serum (FBS) (GIBCO) and 1% 100 mM sodium pyruvate solution (Sigma-Aldrich, S8636) and cultured in an incubator at 37°C and 5% CO2 to about 75% confluence. For 2D-SIM imaging experiments, cells were seeded on coverslips (H-LAF10L glass, reflection index: 1.788, diameter: 26 mm, thickness: 0.15 mm, custom made). For SD-SIM imaging experiments, 25 mm coverslips (Fisherbrand, 12-545-102) were coated with 0.01% poly-L-lysine solution (SIGMA) approximately 24 hours before seeding the transfected cells.

[0192] In 2D-SIM experiments, to label LEs / LYs, COS-7 cells were incubated with 1× Lyso View TM488 (Biotium, 70067-T) in whole cell culture medium at 37°C for 15-30 minutes, protected from light, without washing and imaging. To label LES / Lys in SD-SIM experiments, COS-7 cells were incubated in 50nm Lyso Tracker Deep Red (Thermo Fisher Scientific, L12492) for 45 minutes and washed three times with PBS before imaging. To label LDs, COS-7 cells were incubated with 1× Lipidspot TM488 (Biotium, 70065-T) in whole cell culture medium at 37°C for 30 minutes and protected from light before washing and imaging. To label mitochondria, COS-7 cells were incubated with 250nm Mito Tracker TM Green FM (ThermoFisher Scientific, M7514) and 250nm Mito Deep Red FM (Thermo Fisher Scientific, M22426) was used in the presence of Ca 2+ and Mg 2+But incubated in HBSS without phenol red (Thermo Fisher Scientific, 14025076) at 37°C for 15 minutes and then washed three times before imaging. For nuclear staining, COS-7 cells were incubated with 10 μg / mL Hoechst (Thermo Fisher Scientific, H1399) in PBS for about 5 minutes and washed three times in PBS.

[0193] To label cells with genetic markers, Cos-7 cells were transfected with caveolin-EGFP / LifeAct-EGFP / LAMP1-EGFP / LAMP1-mChery / Tom20-mScarlet / Tom20-mCherry / Sec61β-Cherry / Sec61β-EGFP / clathrin-EGFP / clathrin-DsRed. Hela cells were transfected with Tubulin-EGFP / PEX11A-BFP / LAMP1-mChery. INS-1 cells were transfected with Vamp2-pHluorin. Lipofectamine was used according to the manufacturer's instructions. TM Transfection was performed using 2000 (Thermo Fisher Scientific, 11668019). After transfection, cells were plated on pre-coated coverslips. Live cells were imaged in full cell culture medium without phenol red at 37°C in a live cell imaging system. For experiments with INS-1 cells, cells were stimulated with a solution containing 70 mM KCl and 20 mM glucose, and vesicle fusion events were observed under a TIRF-SIM microscope.

[0194] Interference based SIM settings.

[0195] The schematic diagram of the system is based on a commercial inverted fluorescence microscope (IX83, Olympus) equipped with a TIRF objective (ApoN100× / 1.7HIOil, Olympus), a wide-field objective (100× / 1.45Oil, ApoN, Olympus) and a multi-band dichroic mirror (DM, ZT405 / 488 / 561 / 640-phase R; Chroma). Briefly, lasers with wavelengths of 488 nm (Sapphire488LP-200) and 561 nm (Sapphire 561LP-200, Coherent) and an acousto-optic tunable filter (AOTF, AA Opto-Electronic, France) were used to combine, switch and adjust the illumination power of the lasers. The lasers were coupled to a polarization-maintaining single-mode fiber (QPMJ-3AF3S, Oz Optics) using a collimating lens (focal length: 10 mm, Lightpath). The output laser is collimated by an objective lens (CFIPlan Apochromat Lambda 2×NA 0.10, Nikon) and diffracted by a pure phase grating composed of a polarizing beam splitter (PBS), a half-wave plate, and an SLM (3DM-SXGA, ForthDD). The diffracted beam is then focused by another achromatic lens (AC508-250, Thorlabs) onto the intermediate pupil plane, where a carefully designed light stop mask is placed to block the zero-order beam and other stray light, allowing only ±1 pairs of ordered beams to pass. In order to maximize the modulation of the illumination pattern while eliminating the switching time between different excitation polarizations, a homemade polarization rotator is placed after the light stop mask (x). Next, the light is focused by another lens (AC254-125, Thorlabs) and a tube lens (ITL200, Thorlabs) onto the objective rear focal plane, where it interferes with the image plane after passing through the objective. The emitted fluorescence is collected by the same objective through a dichroic mirror (DM), an emission filter, and another tube lens. Finally, the emitted fluorescence was split by an image splitter (W-View Gemini, Hamamatsu, Japan) and then captured by sCMOS (Flash 4.0V3, Hamamatsu, Japan).

[0196] SD-SIM settings.

[0197] SD-SIM is a commercial system based on an inverted fluorescence microscope (IX81, Olympus) equipped with a (100×, 1.3NA) oil immersion objective and a scanning confocal system (CSU-X1, Yokogawa). SD-SIM is combined with 4 lasers at 405 nm, 488 nm, 561 nm, and 647 nm. The Live-SR module (Gataca Systems, France) was equipped with SD-SIM. Images were acquired by sCMOS (C14440-20UP, Hamamatsu, Japan) or EMCCD (iXon3

[0198] 897, Andor) as shown in Figure 10 or Figure 11.

[0199] FRC resolution map.

[0200] To analyze the imaging capabilities of SR microscopes, FRC resolution can be implemented to describe the effective resolution. To visualize the FRC resolution more clearly, block-wise FRC resolution maps can be applied to evaluate (sparse) SIM and (sparse) SD-SIM images. More specifically, each image can be divided into independent blocks and local FRC values ​​can be calculated individually. If the FRC estimates in a block are sufficiently correlated, the area can be color-coded in the FRC resolution map. Otherwise, the area can be color-coded based on its neighbors interpolation. It should be noted that in the calculation of Fig. 10A , 10H Before obtaining the FRC resolution map of the SD-SIM original image in 11A, a wavelet denoising algorithm can be applied to the image in advance to avoid the ultra-low signal-to-noise ratio of the SD-SIM original image affecting the FRC calculation.

[0201] FWHM and Rayleigh criterion analysis.

[0202] All datasets extracted from the full width at half maximum (FWHM) and bimodal fits can be processed using multimodal fitting and Gaussian functions in OriginPro. To avoid the influence of large beads on the bead resolution estimate, a bead size correction factor can be included in the half width resolution calibration of (sparse) SD-SIM (Figure 18). Therefore, the exact resolution value can be calculated using the following equation:

[0203]

[0204] Among them, FWHM m represents the FWHM estimated directly from a single point in the image, FWHM r represents the corrected resolution of (sparse) SD-SIM and d represents the bead diameter, i.e. 100 nm in FIG18 .

[0205] The pore size of the actin network.

[0206] The pore size of the actin network can be analyzed ( Figure 8H ). Considering the low SNR conditions and high background of 2D-SIM and SD-SIM images, a pre-filter can be used to replace the pre-conversion. Specifically, a Gaussian filter and an unsharp mask filter can be used continuously to reduce the noise of the original image and remove its background. The holes can be extracted by Meyer watershed segmentation. Then, the hole size can be calculated by multiplying the number of pixels contained in the inverse binary image by the physical pixel size. In some embodiments, sparse-SIM and sparse SD-SIM images (i.e., target SIM and SD-SIM images obtained by sparse reconstruction) can be segmented directly using hard thresholding because their backgrounds are extremely clear.

[0207] CCP diameter.

[0208] In calculation Fig. 10A Before the CCP diameter in the Fig. 10A The image in the image is subjected to local adaptive threshold segmentation to segment the annular structure in the image. In the second step, the obtained binary image is subjected to a distance transform followed by a Meyer watershed process to separate the touching CCPs, and the goal is to generate a boundary as far as possible from the center of the overlapping CCP. Subsequently, the CCP diameter can be calculated in the same way as the mesh aperture.

[0209] Image rendering and processing.

[0210] The color map SQURREL-FRC associated with ImageJ can be used to represent Fig. 8E RFC diagram in . Color diagram Morgenstemning can be used to display Fig. 10H Lysosomes and Fig.11D The Fourier transform result in . Figure 10K The depth can be represented by 16-color projection, Figure 11K The depth and Figure 10G The time series can be represented by jet projection. Fig.11H The volumes in can be rendered by ClearVolume. All data processing can be achieved using MATLAB and ImageJ. All images can be prepared with Matlab, ImageJ\MicrosoftVisio and OriginPro, and videos can be made with a self-written Matlab library that can be found at https: / / github.com / weisongzhao / img2vid.

[0211] Adaptive filter of SD-SIM.

[0212] Confocal type images often show isolated pixels Extremely bright values ​​due to voltage instability or dead / hot camera pixels. The amplitude of these pixels is about 5 to 100 times higher than the normal intensity amplitude of biological structures. These isolated pixels may be unsuitable for sparse reconstruction. An adaptive median filter can be created to remove these unsuitable pixels. More specifically, instead of a normal median filter, which replaces each pixel in a window with the median of the neighboring pixels, a threshold can be set for the developed adaptive median filter. If the pixel intensity in the window is greater than a threshold × median, the pixel can be replaced with the median; otherwise, the window can be moved to the next pixel. By using this algorithm, isolated pixels can be passed without blurring the image. The relevant algorithm is written as an ImageJ plugin and can be found at https: / / github.com / imagej / imagej-plugin.

[0213] Found at / / github.com / weisongzhao / adaptivemedian.imagej.

[0214] Software and GPU acceleration.

[0215] Sparse-SIM can be implemented in MATLAB using a CPU (Intel i9-7900X, 3.3GHz, 10 cores, 20 threads, and 128GB of memory), the NVIDIA CUDA Fast Fourier Transform Library (cuFFT), and a GPU (Titan RTX, 4608 CUDA cores and 24GB of memory). Convolution operations on high-content and multi-dimensional matrices in Sparse-SIM can be accelerated using FFT. The impact of the acceleration becomes more pronounced as the size of the data increases, provided that the GPU memory is sufficient for the input data. In most cases, a 60x improvement can be achieved with GPU acceleration compared to CPU processing. For example, Fig. 8A For a 1536×1536×20 image dataset, the GPU takes approximately 90 seconds to process, which is much less than the 46 minutes required for CPU processing. The version of the Sparse-SIM software used in this application (with sample data and user manual) is available as supplementary software. Updated Sparse-SIM software versions can be found at https: / / github.com / weisongzhao / Sparse-SIM.

[0216] Example 1: Verifying resolution expansion using sparse deconvolution

[0217] First, the deconvolution algorithm was used to test the expansion of spatial resolution on synthetic single fibers with known true values. As shown in Figure 13, after Wiener inverse filtering, two single fibers about 110 nm apart can be separated, while single fibers less than about 100 nm apart are difficult to distinguish. Iterative image reconstruction based on sparse priors may only cause a slight difference in the fluorescence of the middle part of the two single fibers, while deconvolution can clearly separate two single fibers with a distance of up to about 81 nm ( Fig.13A and 13B ). However, the contrast between two single fibers at a distance of about 65 nm may be low (probably limited by the 32 nm pixel size), which can be further improved after pixel upsampling ( Fig. 13C and 13D ). Next, the functionality of the different steps in the algorithm can be tested on synthetic single fiber structures corrupted with different noise levels. Deconvolution without adding a sparsity prior may fail to retrieve the high-frequency information eliminated by SIMOTF, while deconvolution without adding a continuity prior may lead to reconstruction artifacts that are particularly evident in low signal-to-noise ratio original images (Figure 14). However, the combination of continuity and sparsity priors allows for a robust and high-fidelity extrapolation of high-frequency information, even in the presence of excessive noise (Figure 14, Tables S1 and S2).

[0218] Finally, the ability of sparse deconvolution to increase the resolution of widefield microscopy and the fidelity of newly identified biological structures can be tested. By convolving the image obtained with a 1.4NA objective with the resulting PSF of a 0.8NA objective, a blurred version of a single actin filament can be generated ( Fig.15A and 15B Interestingly, under a low NA objective, two blurry opposing actin filaments can be clearly separated after sparse deconvolution and extended Fourier frequency domain coupling ( Fig. 15C and 15D ). Similarly, rather than simply reducing the diameter of an actin filament, two adjacent filaments that are almost 120 nm apart (confirmed by 2D-SIM) can be resolved by sparse deconvolution of the wide-field image ( Fig.15E ). In addition, under TIRF-SIM, a CCP with a diameter of about 135 nm may appear as a blur in the wide-field image, which may have undergone conventional deconvolution or previous deconvolution with Hessian continuity. Only after sparse deconvolution may the ring structure appear ( Fig.15F ). All these data can verify the effectiveness of the expansion of spatial resolution, which is also proved by the extended OTF obtained under different conditions (Figure 16).

[0219] Example 2: Sparse-SIM achieves 60nm resolution within millisecond exposure in living cells

[0220] Actin is probably the main cytoskeletal building block that constitutes the cell membrane skeleton and defines the shape and general organization of the cytoplasm. We know that concentrated actin can form a gel-like network under the cell cortex and is dynamically regulated during cell division, migration, polarization formation, etc. Although actin has pores, the diameter that can be detected by electron microscopy is about 50 to 200 nm; however, due to the lack of sufficient spatiotemporal resolution, the dynamic remodeling of these pores is difficult to be captured by any live cell SR imaging algorithm. Under wide-field illumination, two actin single fibers are roughly 66 nm apart, which may not be distinguishable by 2D-SIM or Hessen SIM, but can be distinguished by the entire algorithm (such as Fig. 8A and 8B ) can be clearly distinguished after deconvolution, but cannot be distinguished when only a subset of the algorithm is applied (Figure 17). Fig.8D As shown in Figure 2 ) and improving spatial resolution, dense actin can be better separated, which can be obtained from the full width at half maximum (FWHM) analysis of actin single fibers in SR images and the Fourier ring correlation (FRC) map analysis (as shown in Figure 2 ). Fig. 8E and 8F In time-lapse SR imaging of actin single fiber dynamics, the resolution of sparse-SIM can be kept stable, as evidenced by the relatively consistent values ​​obtained by the FRC method (see Figure 8G The increased resolution is well documented by the more frequent observation of small holes within the actin meshwork. Based on the sparse-SIM data (e.g. Figure 8I The average diameter of the cortical actin mesh is approximately 160 nm. The improved spatial resolution may also help to resolve the annular caveolae (about 60 nm in diameter) in living COS-7 cells under TIRF illumination (see Figure 8J-8L ). Previously, this requirement for nonlinear SIM may have come at the expense of additional reconstruction artifacts and limited imaging time. For fluorescently labeled vesicles such as lysosomes and lipid droplets under wide-field illumination, sparse deconvolution can eliminate background and reconstruction artifacts of low signal-to-noise ratio. These data could show that contrast and resolution deep in the cytoplasm can be maintained and SR images may be clearer than those obtained with conventional SIM or Hessen SIM (e.g. Figure 8M and 8N shown).

[0221] Finally, sparse deconvolution can provide a clearer picture of vesicle fusion events marked by VAMP2-pHluorin in INS-1 cells from raw images reconstructed with a scrolling frame rate of 1692 Hz and a temporal resolution of 564 Hz. Sparse-SIM can also distinguish open fusion pores with a diameter of approximately 60 nm (e.g. Fig.8O and 8P), which is much smaller than the pores detected by conventional or Hessen SIM (whose average diameter is approximately 130 nm, as shown in Figure 8Q Similarly, compared with the fusion holes detected by conventional SIM, the fusion holes detected by sparse-SIM occur earlier (approximately 40 ms vs. 69 ms after the initial opening of the fusion hole) and the image lasts longer throughout the fusion process (approximately 182 ms vs. 82 ms, as shown in Figure 2). Figure 8Q These data can support a spatiotemporal resolution of approximately 60 nm and 564 Hz for sparse-SIM, which may be currently unachievable using other existing SR algorithms.

[0222] Example 3: Two-color sparse-SIM observation of relative motion between structures under organelles

[0223] As a purely mathematical SR technique, sparse-SIM can be easily used in dual-color imaging paradigms to improve the resolution of two wavelengths. For example, in cells with outer and inner mitochondrial membranes (OMM and IMM) labeled with Tom20-mCherry and Mito-Tracker Green, sparse-SIM can show clearer cristae structures (e.g., Fig.9A In addition, the improved resolution may also allow the detection of subtle differences between the OMM and IMM in living mitochondria. At a resolution of 1.3 Å, sparse-SIM can easily detect two types of OMMs: IMM configurations, e.g., long ridges extending from one side of the OMM to about 142 nm from the other side, and short ridges extending only about 44 nm to the other side of the OMM (e.g., Figures 9B-9D Live cell SR imaging can also reveal rare events, such as IMM extensions that are not enclosed in Tom20-labeled structures in a small number of frames (e.g. Fig.9E This result may be due to the uneven distribution of Tom20 protein in the OMM.

[0224] Two-color sparse-SIM imaging can also resolve more details about organelle contacts, including those formed between mitochondria and the endoplasmic reticulum. With further refinement of mitochondrial cristae identification, it was found that endoplasmic reticulum tubules (labeled by Sec61β-mCherry) can randomly contact mitochondria with equal probability in cristae regions and between cristae (e.g. Fig.9F In addition, the contact between the ER tubules and the top of the mitochondria is not only related to the directional movement of the top of the mitochondria, but also to the subsequent coordinated rearrangement of the cristae (as shown in Figure 9G Whether this type of cristae remodeling is causally related to ER-mitochondrial contacts remains to be investigated in the future, but the ability to visualize these subtle structural changes may lead to new insights into the regulation of mitochondrial function.

[0225] Example 4: Sparse deconvolution can also be used to improve the 3D resolution of spinning disk-based confocal SIM (SD-SIM)

[0226] Since the continuity and sparsity assumptions are general features of SR fluorescence microscopy, the algorithm can be tested on SR image data obtained by SD-SIM, based on pinhole SIM with commercial names such as Airyscan, Live-SR, SpinSR10, Re-Scan confocal microscope, etc. Although the spatial resolution of SD-SIM is improved by 1.5-1.7 times compared with the diffraction limit, the photon consumption and signal-to-noise ratio (SNR) of SD-SIM are low in living cells due to the influence of microlenses and pinhole arrays on the excitation and emission efficiency. By performing sparse deconvolution on the three-dimensional image stack of fluorescent microbeads (100 nm) obtained by SD-SIM and correcting for the size effect of the fluorescent microbeads, a lateral resolution of approximately 82 nm and an axial resolution of approximately 266 nm can be obtained (as shown in Figure 18), which is an almost two-fold improvement in spatial resolution in all three axes compared with SD-SIM.

[0227] In Clathrin-EGFP-labeled cos7 living cells, sparse SD-SIM can resolve fuzzy fluorescent dots into ring structures with a diameter of 89 nm (e.g. Fig. 10A and 10B ), which is consistent with the resolution measured by the RFC graph algorithm (as shown in Fig. 10C The average diameter of all the annular CCPs is about 154 nm (as shown in Fig. 10D Some events, such as the disappearance of a ring-shaped CCP and the splitting of another CCP into two smaller rings nearby, can be observed (see Fig.10E and 10F These abundant CCP dynamics can be better explained by simultaneously labeling and imaging other molecules, such as actin filaments (labeled by LifeAct-EGFP). Because sparse SD-SIM allows photon budgets of less than about 0.9 W / cm 2 (Table S3), so at 5 s intervals, single actin fibers and CCPs (e.g. Figure 10G and 19). Under these conditions, many relative movements between CCP and filaments can be observed. For example, the CCP can be observed to reappear and stably dock at the intersection of two actin filaments, and then disappear from the focal plane as the adjacent filaments close and connect together (see Fig.19D), which may be consistent with the endocytosis of actin by CCP. Similarly, two-color sparse SD-SIM can also reveal the dynamic interactions between ER tubules and lysosomes, for example, the loading behavior (as shown in Figure 20).

[0228] Sparse SD-SIM is not susceptible to chromatic aberration and is easily adapted to four-color SR imaging, allowing monitoring of the dynamics of lysosomes, mitochondria, microtubules, and nuclei in living cells (e.g. Figure 8H and 8I As shown), the FRC resolution is as small as 79 to 90 nm (as shown Fig.10J Benefiting from the absence of out-of-focus fluorescence and improved axial resolution, sparse SD-SIM allows visualization of similar structures with similar contrast levels throughout the cell, with FWHMs of approximately 280 nm axially (e.g. Figure 8L The mitochondrial outer membrane structure is maintained in living cells with a thickness of approximately 7 μm (as shown in Figure 8K shown).

[0229] Example 5: Upsampling helps sparse deconvolution improve resolution in the case of Nyquist undersampling

[0230] EMCCD cameras have excellent quantum efficiency and electron multiplication capabilities and are often used in SD-SIM systems, but their large pixel size may limit the resolution of the system. For example, when using sparse deconvolution algorithms to image endoplasmic reticulum tubules under SD-SIM, RFC map analysis can produce a resolution of approximately 195nm, mainly due to the Nyquist-limited pixel size of approximately 94nm (e.g. Figures 11A-11B By artificially upsampling the data before deconvolution, it is possible to distinguish previously obscure circular ER tubules (e.g. Fig. 11C The OTF of the microscope can be extended after upsampling and deconvolution (as shown in Fig.11D This is consistent with the fact that the resolution of FRC measurements can be improved to 102 nm (as shown in Fig. 11B shown).

[0231] The upsampling before sparse deconvolution (sparse × 2) can also be extended to multicolor sparse SD-SIM imaging to follow the dynamics between different organelles. Using HeLa cells co-expressing LAMP1-mCherry, Pex11a-BFP, and Tubulin-EGFP, the structure and dynamics of lysosomes, peroxisomes, and microtubules can be obtained (e.g. Fig.11E Many peroxisomes may come into contact with lysosomes on microtubules, as evidenced by lysosomes moving along microtubules and colliding with peroxisomes that are tightly attached to the intersection of two tubulin filaments (e.g. Fig.11F), or the co-migration of lysosomes and peroxisomes at the front of separation and departure (as shown in Fig.11G As mentioned above, these contacts can mediate lipid and cholesterol transport.

[0232] Finally, it could be shown that the XYZ resolution of SD-SIM can also be improved by the sparsity ×2 process. In COS-7 cells labeled with Hoechst, Mito Tracker Deep Red, and Tubulin-EGFP, it was possible to visualize nuclei, mitochondria, and microtubules in a 3D volume with an axial span of 6 μm in living cells (Fig. Fig.11H Interestingly, after the sparsity ×2 treatment, the axial FWHM of the microtubule single fiber can be greatly reduced from about 465nm in the original dataset to about 228nm, which again indicates a two-fold increase in the Z axis (as shown in Fig.11I From the volume reconstruction, it is evident that the continuous, convex nuclear structure may bend inwards and become concave in some axial regions, which are invaded by extensive microtubule filaments and mitochondria (e.g. Fig.11J (shown). This reciprocal change may indicate that the tubulin network may influence nuclear assembly and morphology. Overall, the combination of sparse deconvolution and SD-SIM enables multicolor, volumetric SR imaging with minimal phototoxicity and aberrations; these images can provide a more dynamic understanding of cellular processes.

[0233] Figure 7 The following are exemplary results of a sparse deconvolution algorithm flow chart according to some embodiments of the present application. The original image of the 2D-SIM or SD-SIM microscope is subjected to Wiener inverse filtering to generate an SR image. Before the final iterative deconvolution, the sparsity and continuity prior information on the xy-t (z) axis can be used for background subtraction, upsampling and reconstruction. Scale bar: 1 μm.

[0234] Figures 8A-8Q These are exemplary results showing that sparse-SIM can achieve a temporal and spatial resolution of approximately 60 nanoseconds and milliseconds in living cells according to some embodiments of the present application. Fig. 8A Representative COS-7 cells labeled with LifeAct-EGFP observed under 2D-SIM (bottom), Hessen-SIM (top), and Sparse-SIM (middle) are shown. Figure 8B Shows Fig. 8A The enlarged area surrounded by the middle frame 810 (left: 2D-SIM, middle: Hessian-SIM, right: sparse-SIM). Figure 8C for Figure 8B Cross-sectional view corresponding to the center line 830-850. Fig.8Dis the average contrast of two peaks at different distances under 2D-SIM and sparse-SIM. The contrast can be calculated according to the equation (Imax-Imin) / (Imax+Imin). Fig. 8E RFC maps of actin single fibers under 2D-SIM and sparse-SIM are shown. Figure 8F The resolution of 2D-SIM and sparse-SIM is shown, i.e. the FWHM of the narrowest actin filaments measured by the FRC technique. Figure 8G Shown is the evolution of the minimum FRC resolution over time for 2D-SIM images (black hollow circles), Hessian-SIM pictures (grey hollow circles) and sparse-SIM images (grey hollow squares) relative to the theoretical limit of 2D-SIM (black hollow triangles). Figure 8H The left side shows Fig. 8A The middle box 810 shows an enlarged view of an actin filament, while the right side shows a segmented filament covered by a relatively thin line. Figure 8I Shows Figure 8H Cumulative distribution of pore sizes within the actin mesh measured by 2D-SIM (black) and sparse-SIM (grey) images. Figure 8J A typical example of COS-7 cells labeled with caveolin-EGFP under TIRF-SIM (left, FRC average resolution of approximately 98 nm) and sparse-SIM (right, FRC average resolution of approximately 58 nm) is shown. Figure 8K As shown, from top to bottom, TIRF-SIM, sparse-SIM (pixel size is 32nm), and 2× upsampled sparse-SIM (pixel size is 16nm) reconstructions are shown. Figure 8J Enlarged view of the middle white box. Figure 8L for Figure 8K Fluorescence spectrum of the midline 860-880. In COS-7 living cells, e.g. Figure 8M As shown, cytoplasmic lysosomes can be labeled with LAMP1-EGFP (left, yellow) or LysoView (middle, cyan), while lipid droplets can be labeled with Lipidpot (right, magenta). Top: Comparison of the same structure under 2D-SIM (left) and sparse-SIM (right). Bottom: Zoomed-in views reconstructed by 2D-SIM (left), Hessen-SIM (middle), and sparse-SIM (right). Figure 8N The average diameters of caveolin-EGFP, LAMP1-EGFP, and LipidSpot-tagged structures detected by sparse-SIM are shown. Fig.8ORepresentative clips of vesicle fusion events observed in TIRF-SIM (top) and sparse-SIM (bottom) configurations are shown. INS-1 cells can be labeled with VAMP2-pHluorin and SR imaged at a frame rate of 564 Hz. Figure 8P Shows Fig.8O Waveforms in TIRF-SIM (top) and sparse-SIM (bottom) images. Figure 8Q Mean pore size (left), pore open time (middle), and pore duration (right) measured in TIRF-SIM (red) and sparse-SIM (cyan) configurations are shown. Center line, median; Limits, 75% and 25%; Whiskers, maximum and minimum; Error bars, sem; Cumul.Freq., cumulative frequency; Scale bar: ( Fig. 8A , 8E , 8M) 5μm;

[0235] ( Figure 8B and 8J )500nm;( Figure 8H and 8M 1μm; ( Figure 8K and 8O )100nm;( Figure 8P )500ms.

[0236] Figures 9A-9G are exemplary results showing that the complex dynamics within mitochondria and between the ER and mitochondria are visualized by two-color sparse-SIM according to some embodiments of the present application. Fig.9A A typical example of a COS-7 cell is shown, in which both the outer (magenta) and inner (cyan) mitochondrial membranes (OMM and IMM) were labeled with Tom20-mScarlet and MitoTrackerGreen and visualized under 2D-SIM (left) and sparse-SIM (right). Fig. 9B Shows Fig.9A Magnified view of the IMM (left) and OMM (right) at center frame 910. Fig. 9C Tom20-mScarlet (magenta) and MitoTrackerGreen (cyan) are shown along Fig. 9B The fluorescence intensity distribution diagram of the continuous dashed lines corresponds to the two mitochondrial structures shown in the schematic diagram on the right (indicated by black arrows). Fig.9D Shows Fig.9A Average minimum resolution of IMM (cyan) and OMM (magenta) in 2D-SIM and sparse-SIM. 20 images along the t-axis can be used for FRC analysis. Fig.9E As shown, Fig.9AThe white dashed box 920 in FIG. 1 is magnified and shown at three time points, revealing the non-uniform distribution of Tom20 (right) relative to Mito Tracker Green (left). Fig.9F A typical example of a COS-7 cell is shown, where both the IMM (cyan) and ER (magenta) were labeled with Mito Tracker Green and Sec61β-mCherry and visualized under 2D-SIM (upper left) and sparse-SIM (lower right). The inset on the lower left shows ER-mitochondria contact sites at the mitochondrial cristae or matrix region. Figure 9G Shown under Sparse-SIM Fig.9F Magnified image of white box 930, showing the orientation of the internal cristae structure rearranged due to dynamic contact with the endoplasmic reticulum (shown on the left) (indicated by arrows 940-970), while the rearranged cristae region in the white box is magnified (shown on the right). Center line, median; limits, 75% and 25%; whiskers, maximum and minimum; error bars, sem; scale bar: ( Fig.9A and F) 1 μm; ( Fig. 9C )Axial: 0.2 arbitrary unit (au); Transverse: 100 nm; ( Fig. 9B , 9E and 9G)500nm.

[0237] Figures 10A-10L These are exemplary results showing that sparse SD-SIM can achieve three-dimensional, multi-color, and sub-90 nm resolution SR imaging of living cells according to some embodiments of the present application. Fig. 10A Representative snapshots of COS-7 cells expressing Clathrin-EGFP observed with a sCMOS camera under SD-SIM (left triangle) and sparse SD-SIM (right triangle) are shown (third frame, 0.2 s exposure per frame). Fig. 10B This is the fluorescence intensity curve corresponding to the central resolution ring of CCP (upper left). Fig. 10C The average minimum resolution of CCPs observed under SD-SIM and sparse SD-SIM evaluated by the FRC technique is shown. Fig. 10D Diameter histogram of CCPs in living cells. Fig.10E Shows Fig. 10A A magnified view of the middle frame area 1010 shows the continuous disappearance of CCPs (indicated by yellow arrows 1030-1060). The top image is the original SD-SIM image; the rest are sparse-SD-SIM images. Fig.10F Shows Fig. 10A The enlarged view of the middle frame area 1020 shows that the CCP splits into two CCPs. Figure 10G Shown is a temporal projection of CCP colors during 16.75 min of live-cell SR imaging. Fig. 10HA typical example of four-color live-cell SR imaging under sparse SD-SIM is shown. COS-7 cells can be simultaneously labeled with LAMP1-mCherry (yellow), Mito Tracker (green), Hoechst (blue), and Tubulin-EGFP (magenta). Fig.10I As shown, Fig. 10H The enlarged view of the middle frame area 1070 can be divided into two images, highlighting mitochondria and lysosomes (left) and nuclei and tubulin (right). Views under SD-SIM and sparse SD-SIM can be shown on the left and right of each picture, respectively. Fig.10J Shown are the average resolutions of differently colored labeled organelles observed under SD-SIM and sparse SD-SIM and assessed using the FRC technique. Figure 10K The 3D distribution of all mitochondria (labeled with TOM20-mCherry) within a 7 μm volume along the axis of a live COS-7 cell is shown using sparse SD-SIM. Different colors can be used to encode different distances from the cell surface to the imaging plane. Fig.10L for Figure 10K Color coding of the cross-section (left) and longitudinal section (right) of the middle frame area 1080, where the images obtained by SD-SIM and sparse SD-SIM can be shown above and below, respectively. Center line, median; limits, 75% and 25%; whiskers, maximum and minimum; error bars, sem; scale bar: ( Fig. 10A and 10I )3μm;( Fig.10E and 10F )300nm;( Figure 10G and 10L )1μm;( Fig. 10H and 10K )5μm.

[0238] Figures 11A-11K These are exemplary results showing that upsampling can enable sparse SD-SIM to overcome the Nyquist sampling limitation to achieve multi-color 3D SR imaging in living cells according to some embodiments of the present invention. Fig.11A Shown (from left to right) are ER tubules at the periphery of COS-7 cells (labeled with Sec61β-EGFP) in SD-SIM, Hessen SD-SIM, Magnitude SD-SIM, and Magnitude SD-SIM × 2 configurations. The SD-SIM system can be equipped with an EMCCD camera, and the physical pixel size in each SR image can be approximately 94 nm. Fig. 11B Shows the FRC technology evaluation Fig.11A The average minimum resolution of the images obtained under different configurations. The theoretical resolution according to the Nyquist criterion can be plotted as a dashed line 1120. Fig. 11C for Fig.11AMagnified images of ER tubules in different configurations (from left to right). The ER tubule structures indicated by arrows can only be resolved in the sparse SD-SIM × 2 configuration (right inset). Fig.11D Shown are the Fourier transforms of images obtained in SD-SIM (left), Hessian SD-SIM (center left), sparse SD-SIM (center right), and sparse SD-SIM×2 (right) configurations. Fig.11E Shown are snapshots of HeLa cells labeled with Tubulin-EGFP (magenta), Pex11a-BFP (cyan), and Lamp1-mCherry (yellow) in SD-SIM (left) and sparse SD-SIM ×2 (right) configurations. Fig.11F Shows Fig.11E A magnified view of the middle frame area 1130, where the mask at different times (above) shows that the lysosome moves along the microtubule to contact two peroxisomes. The contact between the lysosome and the peroxisome can be further magnified and shown below. Fig.11G Another example is shown, where lysosomes and peroxisomes co-move along microtubules in a sparse SD-SIM ×2 configuration. The dotted lines may represent microtubules. Fig.11H Live cell three-color 3D imaging of the sparse SD-SIM × 2 method is shown. COS-7 cells can be labeled with tubulin-EGFP (green), Hoechst (cyan), and Mito Tracker Deep Red (magenta). Fig.11I Recorded for SD-SIM (top) and sparse SD-SIM × 2 (bottom) Fig.11H The fluorescence spectra of lines 1140 (SD-SIM) and 1150 (Sparse SD-SIM×2) may pass through the microtubule single fiber in the z-axis, and Gaussian fits can be used to obtain FWHM resolutions of approximately 465 nm and approximately 228 nm for SD-SIM and sparse SD-SIM×2. Fig.11J Three horizontal sections of the cell nucleus (top) and mitochondria incorporating microtubules (bottom) are shown, where the invaginated region of the nucleus may be invaded by extensive microtubule filaments and mitochondria. Figure 11K From left to right, the volumes of the nucleus, mitochondria, and microtubules are shown. Different depths of the imaging plane can be coded in different colors. Scale bar: ( Fig.11A , 11D ,11E,11H,11I,11J,11K)5μm;( Fig. 11B and 11F Above, and 11G) 3μm; ( Fig. 11C )1μm;( Fig.11F Below) 500nm.

[0239] Fig.12This is an exemplary result showing the detailed workflow of sparse deconvolution according to some embodiments of the present application. I-II, III-IV, and V-VI can be captured by 2D-SIM using an sCMOS camera, SD-SIM using an sCMOS camera, and SD-SIM using an EMCCD camera, respectively. Then, the SR image can be reconstructed using inverse Wiener filtering. Thereafter, overall similar but slightly different strategies can be used to process these different types of data sets.

[0240] Figures 13A-13D The following are exemplary results showing the spatial resolution benchmark of a synthetic image at different steps of sparse deconvolution according to some embodiments of the present application. Fig.13A The resolution plate has a pixel size of 16.25 nm and contains five pairs of lines with distances of 48.75 nm, 65.00 nm, 81.25 nm, 97.50 nm and 113.75 nm. The composite image (512×512 pixels) can be first illuminated by pattern excitation and then warped with the PSF under the microscope (1.4NA, 488 nm excitation). The signal can be recorded with a sCMOS camera with a pixel size of 65 nm, which means a 4× upsampling of the original image (128×128 pixels). Gaussian noise with a peak intensity of 5% can also be included in the original image. Then (from left to right) a conventional SIM image (256×256 pixels) is obtained using Wiener inverse filtering, followed by reconstruction under continuity and sparsity prior information constraints and finally deconvolution. By the equation λ / 2(NA i +NA d ) The theoretical limiting resolution of Wiener-SIM can be calculated to be 97.6 nm, where i and d represent the illumination and detection NA, respectively. Fig. 13B Shows Fig.13A The intensity corresponding to different line pairs in the image. Only when the original image is fully sparsely deconvolved can the two lines separated by 65nm be distinguished. Fig. 13C As shown in FIG. 1 , the SIM image of 512×512 pixels obtained after Wiener inverse filtering can also be upsampled and processed in other steps in the subsequent sparse deconvolution pipeline. Fig.13D Shows Fig. 13C The intensities corresponding to different line pairs in . Note that in the upsampled image, two lines separated by 48.75 nm can be separated by sparse deconvolution. Scale bar: 1 μm.

[0241] Figures 14A-14D The following are exemplary results showing that the contributions of different steps to the sparse deconvolution of the synthesized image may vary with the degree of noise pollution according to some embodiments of the present application. Fig.14AAs shown, synthetic single fiber structures can be created using the RandomWalk procedure, and the maximum curvature is used in the program to generate single fiber structures. To simulate time-lapse sequences of single fiber structures, random shifts can be generated in these single fibers based on the Markov chain on the t axis using a subpixel shift operation, thereby generating image stacks of 128×128×5 pixels. To simulate SIM acquisition, these real-value formations can be illuminated by pattern excitation and blurred with the microscope's PSF (1.4NA, 488nm excitation, 32.5nm pixels) to produce raw images of wide fields of view. In addition to the single fibers, background signals may also be included that combine cell autofluorescence and fluorescence effects from out-of-focus planes (simulated by convolution of synthetic single fibers with an out-of-focus PSF of 1μm from the focal plane). In addition, Gaussian noise of different variances (0%, 11%, 25%, 50%, 80%) may be incorporated into the fiber peak fluorescence intensity. The raw images can be acquired by a sCMOS camera with a pixel size of 65nm and a pixel amplitude of 16 bits. Fig. 14B The performance comparison between five different types of SR reconstruction for 0%, 11% and 80% noise levels (from top to bottom) is shown. Specifically, the original image can be reconstructed first with Wiener inverse filtering (column 1), then reconstructed with only continuous constraints (column 2), reconstructed with continuous constraints and deconvolution (column 3), reconstructed with only sparse constraints and deconvolution (column 4), or reconstructed with sparse plus continuous constraints and deconvolution (column 5). Fig. 14C for Fig. 14B The fluorescence intensity curves of two opposing synthetic single fibers under different conditions are shown in Figure 1. Deconvolution after continuous reconstruction may not be able to separate these single fibers, and deconvolution after sparse reconstruction may cause artifacts in the original image containing 80% noise (an example shown by the pointer). Fig.14D As shown in Figure 2, the average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values ​​of SR images reconstructed by different techniques may vary with different levels of noise (0%, 11%, 25%, 50%, 80%) compared to the synthetic ground truth. More details can be found in Tables S1 and S2. Scale bar: Fig.14A Middle 1μm; Axial: Fig. 14C Center: 0.1 arbitrary unit (au); lateral: 100 nm.

[0242] Figures 15A-15F These are exemplary results showing the actual extended spatial resolution achieved by the sparse deconvolution algorithm when processing real biological structures according to some embodiments of the present application. Figures 15A-15D A comparison between wide field of view high NA (1.4) and low NA (0.8) imaging after sparse deconvolution in both spatial and Fourier space is shown. Fig.15AAs shown, from left to right, they can be the high Na (1.4) image after sparse deconvolution, the simulated low NA (0.8) image, and the NA (0.8) image (data and Fig. 8A Here, the formula IFFT{FFT(IMG1.4)×

[0243] (OTF0.8 / OTF1.4) to generate a low NA image. In this formula, FFT (IFFT) represents (inverse) Fast Fourier Transform, and IMG1.4 represents a high NA image. Fig. 15B The optical transfer functions of Na=1.4 (OTF1.4) and Na=0.8 (OTF0.8) are shown. Fig. 15C Shows Fig.15A The corresponding Gaussian fitting cross section of the inset. It can be seen that at low NA, after sparse deconvolution, two peaks with a distance of 259 nm can be resolved, which is similar to 265 nm in the original high NA image. This shows that sparse deconvolution can achieve high NA resolution with better contrast. Fig.15D As shown in , the corresponding Fourier space images can be presented from left to right. Fig.15E and 15F As shown in Figure 2, using SIM images as ground truth and performing sparse deconvolution on wide field (WF) images can actually improve spatial resolution. Fig.15E ) or Clathrin-EGFP( Fig.15F ) of COS-7 living cells were imaged using 2D-SIM and TIRF-SIM, respectively. For the original data set, conventional deconvolution, continuous reconstruction followed by deconvolution, upsampling plus sparse and continuous reconstruction, or upsampling plus sparse and continuous reconstruction followed by deconvolution can be used. Obviously, only the fully sparse deconvolution algorithm can distinguish two opposing actin fibers ( Fig.15E ) and the ring structure of CCP ( Fig.15F ), whose fidelity is confirmed by the SIM image on the right. Scale bar: ( Fig.15A , 15B , 15D and 15F) 500 nm; (A, inset) 200 nm.

[0244] Fig.16A and 16B The following are exemplary results showing that the optical transfer function is obtained by Fourier transform of fluorescent microbead visualization under different circumstances according to some embodiments of the present application. Fig.16AImages of fluorescent microbeads (48 nm in diameter) imaged in wide field of view: (upper left), wide field of view plus deconvolution (upper right), wide field of view plus sparse and continuous reconstruction (lower left), and wide field of view plus sparse and continuous reconstruction followed by deconvolution (lower right). Upper inset: a single microbead observed under different conditions; lower inset: the corresponding Fourier transform of the image on the left. Fig. 16B Shows from Fig.16A The lower inset shows the amplitude in the frequency domain along the white line 1610-1640, while the two vertical dashed lines 1650 and 1660 indicate the spatial frequency limit of widefield microscopy. Scale bar: 5 μm; inset: 500 nm.

[0245] Fig.17A and 17B These are exemplary results of reconstruction using only sparsity, only continuity, or both sparsity and continuity priors according to some embodiments of the present application. Fig.17A It is shown that under 2D-SIM Fig. 8A Actin filaments in living COS-7 cells labeled with LifeAct-EGFP (left) followed by reconstruction and deconvolution with sparsity constraints (center left), reconstruction and deconvolution with continuity constraints (center right), or reconstruction and deconvolution that includes both sparsity and continuity constraints (right). Fig. 17B Clathrin-EGFP labeled CCP under 2D-SIM (left), followed by reconstruction and deconvolution with sparsity constraints (middle left), reconstruction and deconvolution with continuity constraints (middle right), or reconstruction and deconvolution with both sparsity and continuity constraints (right). Scale bar: ( Fig.17A )1μm;( Fig. 17B )300nm.

[0246] Figures 18A-18E 1 is an exemplary result of a three-dimensional image stack showing fluorescent microbeads under SD-SIM and sparse SD-SIM according to some embodiments of the present application. Figures 18A-18B The SD-SIM records are shown respectively ( Fig.18A ) and after sparse deconvolution ( Fig.18B ) after the fluorescent beads (diameter 100nm) ( Fig.18A ) of a fluorescent microbead (left) and a horizontal cross-section (right). The inset in the lower left corner shows a magnified view of the same fluorescent microbead under different reconstruction algorithms. Fig. 18C It shows ( Fig.18A From the corresponding Gaussian fitting curve in the lower left corner, it can be seen that the lateral resolutions of SD-SIM and sparse SD-SIM are 165nm and 82nm, respectively. Fig.18D On the left and right sides are shown Fig.18A and 18BThe horizontal cross sections enlarged by boxes 1810 and 1820 in FIG. 1 and 1830 , and the SD-SIM images can be processed with a median filter to avoid non-convergence of the fitting results. Fig.18E As shown, a Gaussian function can be used to fit Fig.18D The intensity curve of fluorescent microbeads along the axial direction. The axial resolution of SD-SIM and sparse SD-SIM reaches 484nm and 266nm respectively. Scale bar: ( Fig.18B )4μm, ( Fig.18B , inset) 100nm, ( Fig.18D )200nm.

[0247] Figures 19A-19D . The following are exemplary results showing dual-color live cell imaging of clathrin and actin using sparse SD-SIM according to some embodiments of the present application. Fig.19A and 19B CCPs recorded at 5-second intervals for 16.75 minutes for SD-SIM (left) and sparse SD-SIM (right) are shown. Fig.19A ) and actin single fiber network ( Fig.19B ). Fig.19C Shown are CCPs (cyan) and the cortical actin cytoskeleton (magenta) in COS-7 cells captured by Sparse SD-SIM. Fig.19D Shown after enlargement Fig.19C Frame area 1910 ( Fig.19C ) Clip at 5 time points. The first image observed under SD-SIM may appear in the upper left corner for comparison. It can be observed that CCP can stably dock under the conditions of connecting two actin filaments and then disappear from the focal plane as these adjacent filaments merge. Scale bar: ( Figures 19A-19C )4μm; ( Fig.19D )500nm.

[0248] Fig. 20A and 20B The figures are exemplary results showing the use of sparse SD-SIM to display endoplasmic reticulum-lysosome contacts according to some embodiments of the present application. Fig. 20A Shown are contacts between ER tubules (labeled with Sec61-EGFP, green) and lysosomes (labeled with Lyso tracker Deep Red, magenta) in living COS-7 cells visualized by Sparse SD-SIM. Fig. 20B Time-lapse images showing typical lysosome-endoplasmic reticulum contact dynamics, from Fig. 20AThe dashed box area of ​​2010 is enlarged by SD-SIM (upper) and sparse SD-SIM (lower). The ER tubules may move with the lysosomes, and then the tubular structure shrinks into a polygon around the lysosomes (indicated by the white arrows). Finally, due to the shrinkage of the ER tubules, the polygons disappear. Scale bar: ( Fig. 20A )3μm;( Fig. 20B )2μm.

[0249] Table S1: Images with different noise levels reconstructed by different algorithms relative to the synthetic true value (G represents Gaussian noise) Fig.13A )

[0250]

[0251] Table S2: Relative to the synthetic true value (G represents Gaussian noise), images with different noise levels were reconstructed using different algorithms ( Fig.13A )

[0252]

[0253] Table S3: Imaging conditions used in different figures

[0254]

[0255] The long-standing goal of applying mathematical SR to image data from different fluorescence microscopes can be achieved by developing sparse deconvolution algorithms. In this application, evidence is presented to prove the fidelity of mathematical SR. First, using synthetic structures in simulation experiments, it can be shown that sparse deconvolution can provide many more complex details close to the true value than traditional SIM deconvolution (Figures 13 and 14). Second, by performing sparse deconvolution on the wide-field image reconstructed from the original SIM image, complex structures that cannot be observed with wide-field microscopy, such as branched single fibers or ring-shaped CCPs ( Fig.15E and 15F ). The fidelity of these structures can be fully confirmed by the corresponding SR-SIM reconstructions. Third, for real biological structures for which ground-truth images are not available, the improvement in spatial resolution can be quantitatively assessed by the FRC technique, and the claimed resolution can be confirmed by its ability to distinguish adjacent structures. The resolution of SD-SIM almost doubled from about 150nm to about 80nm, enabling it to resolve most annular CCPs with an average diameter of around 150nm (Figure 9). Fourth, using sparse TIRF-SIM, it may be possible to resolve many annular caveolae with a diameter of approximately 60nm. Similarly, the size of the cortical actin mesh measured with sparse-SIM is approximately 160nm ( Figure 8I). Finally, when sparse deconvolution is used for individuals, the 2D images obtained by SD-SIM can improve the axial resolution by a factor of two, while manual upsampling combined with the sparse deconvolution algorithm can recover SR information that was previously limited by the Nyquist sampling criterion. Taken together, these data suggest that the spatial resolution extension of the sparse deconvolution algorithm is reasonable and real, and can be combined with other hardware-based SR techniques to reduce the photon budget to improve resolution (Table S3).

[0256] For microscopists, it has long been believed that the bandwidth of the microscope is constant. However, another view is that the entire information carried by the microscope can be constant, which can be demonstrated by the examples in this application. In this sense, the introduction of additional prior knowledge of the imaging object can be used to achieve mathematical SR. The real expansion of the bandwidth frequency limited by the image acquisition device 110 can be attributed to the deconvolution algorithm in this application. The addition of a Hessian penalty on object continuity can improve the signal-to-noise ratio of the image, which is crucial for any band-limited frequency expansion beyond the OTF. In the absence of continuity, sparse deconvolution may lead to over-sharpening or random artifacts (Figures 14 and 17). In contrast, the purpose of the bandwidth expansion technique is to reconstruct all object details, and the sparsity prior can be used as an adjustable parameter to constrain the iterative approximation of SR information. By adjusting the parameters to impose only mild requirements on the sparsity of the object in the final reconstruction, the removal of weak fluorescence signals can be avoided, making the algorithm universally applicable to different data sets. The final iterative deconvolution step is also essential to enhance high-frequency information beyond the OTF and to achieve visual separation of closely packed structures (Figures 13, 14, 16, and 18). The fact that the resolution of at least three microscopes can be extended by the sparse deconvolution algorithm suggests that the sparse deconvolution algorithm may be generally applicable to other SR microscopes.

[0257] Among the different possible applications of sparse deconvolution, Sparse-SIM can achieve an unprecedented combination of spatiotemporal resolution. At roughly 60 nm resolution and 564 Hz frame rate, the opening of vesicle fusion pores smaller than 90 nm can be seen earlier, and longer pore durations can be easily observed ( Fig.8O and 8P ), which helps to elucidate the molecular mechanisms of vesicle fusion and other ultrafast processes. The different conformations of the outer and inner mitochondrial membranes observed by sparse-SIM suggest that it can also be used to reveal details about organelles and organelle contacts in living cells. For multi-color 3D SR imaging, sparse SD-SIM may be a better choice. At the expense of reduced temporal resolution, sparse SD-SIM can better preserve the resolution and contrast of different planes within living cells, and reducing the light dose can also achieve long-term 3D SR imaging.

[0258] In summary, this demonstrates the universal applicability of the sparse deconvolution algorithm, which has the ability to mathematically improve the resolution of any existing fluorescence microscopy image. Its combination with hardware-based SR microscopy enables a resolution of about 60nm to an unprecedented time resolution of about milliseconds (ms), and provides biologists with routine multi-color, sub-90nm SR imaging, which will help biologists better resolve different structures and dynamics in three-dimensional space and time.

[0259] The basic concepts have been described above. Obviously, for those of ordinary skill in the art who have read this application, the above invention disclosure is only for example and does not constitute a limitation of this application. Although not explicitly stated here, those of ordinary skill in the art may make various modifications, improvements and amendments to this application. Such modifications, improvements and amendments are suggested in this application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of this application.

[0260] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0261] In addition, it will be appreciated by those skilled in the art that the various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful combination of processes, machines, products or substances, or any new and useful improvements thereto. Therefore, the various aspects of the present application may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "units", "modules" or "systems". In addition, the various aspects of the present application may take the form of a computer program product embodied in one or more computer-readable media, wherein the computer-readable program code is contained therein.

[0262] A computer readable signal medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. Such propagated signals may be in a variety of forms, including electromagnetic, optical, etc., or any suitable combination. A computer readable signal medium may be any computer readable medium other than a computer readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on a computer readable signal medium may be propagated via any suitable medium, including radio, cable, fiber optic cable, RF, etc., or any combination of the above.

[0263] The computer program code required for the operation of each part of the application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., such as C programming language, VisualBasic, Fortran2103, Perl, COBOL2102, PHP, ABAP, such as Python, Ruby and Groovy dynamic programming language or other programming languages. The program code can be executed completely on the user's computer, can be executed partially on the user's computer as an independent software package, can be executed partially on the user's computer and partially on the remote computer, or can be executed completely on the remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0264] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this application, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a pure software solution, for example, installation on an existing server or mobile device.

[0265] Similarly, it should be noted that in order to simplify the description disclosed in this application and thus facilitate the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes grouped into one embodiment, figure, or description thereof. However, this method of the present application should not be interpreted as reflecting the intention that the claimed object material to be scanned requires more features than those expressly stated in each claim. On the contrary, the subject matter of the invention should have fewer features than the single embodiment described above.

[0266] In some embodiments, the numbers representing quantities or properties used to describe and claim certain embodiments of the present application should be understood to be modified by the terms "approximately", "approximately" or "roughly" in some cases. For example, unless otherwise stated, "approximately", "approximately" or "roughly" can indicate a ±20% variation of the value described. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which can change according to the required characteristics of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0267] All patents, patent applications, patent application publications, and other materials (such as papers, books, specifications, publications, records, things, and / or the like) referred to herein are hereby incorporated by reference in their entirety for all purposes, except any prosecution record related to such documents, any such documents that are inconsistent or conflicting with this document, or any such documents that have the effect of limiting the broad scope of the claims that may or may not be related to this document. For example, if there is any inconsistency or conflict between the description, definition, and / or use of a term related to any incorporated material and a term related to this document, the description, definition, and / or use of that term in this document shall control.

[0268] Finally, it should be understood that the embodiments described in this application are only used to illustrate the principles of the embodiments of the present application. Other variations may also fall within the scope of the present application. Therefore, as an example and not a limitation, the alternative configurations of the embodiments of the present application may be considered to be consistent with the teachings of the present application. Accordingly, the embodiments of the present application are not limited to the embodiments explicitly introduced and described in the present application.

Claims

1. An image processing method implemented on at least one machine, each machine having at least one processor and at least one storage device, characterized in that: include: generating an initial image by filtering image data generated by an image acquisition device; Based on a first objective function, performing a first iterative operation on the initial image to generate an intermediate image, the first objective function being associated with image fidelity of the intermediate image, continuity of the intermediate image, and sparsity of the intermediate image; as well as A second iterative operation is performed on the intermediate image to generate a target image based on a second objective function associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image.

2. The method according to claim 1, characterized in that The filtering of the image data generated by the image acquisition device comprises: The image data is filtered by performing an inverse Wiener filter on the image data.

3. The method according to claim 1, characterized in that The first iterative operation includes sparse deconvolution of the initial image.

4. The method according to claim 3, characterized in that The first objective function includes a first weighting factor associated with image fidelity of the intermediate image or an L-2 norm of a first difference between the intermediate image and the initial image.

5. The method according to claim 3, characterized in that Also includes: An estimated background image is generated by performing an iterative wavelet transform operation on the initial image, and the first objective function is also associated with the estimated background image.

6. The method according to claim 5, characterized in that The first objective function also includes an L-2 norm of a first difference between the intermediate image and the initial image and a fourth difference between the estimated background image.

7. The method according to claim 5, characterized in that The iterative wavelet transform operation includes one or more iterations, and each current iteration includes: Determine an input image based on the initial image or an estimated image generated in a previous iteration; Generate a decomposed image by performing a multi-scale wavelet decomposition operation on the input image; Generating a transformed image by performing an inverse wavelet transform operation on the decomposed image; generating an updated transformed image based on the transformed image and the input image; generating an estimated image of a current iteration by performing a cutoff operation on the updated transformed image; and In response to determining that a termination condition is satisfied, the estimated image is designated as a background image.

8. The method according to claim 3, characterized in that The first iterative operation comprises: determining an initial estimated intermediate image based on the initial image; and At least two iterations of the first objective function are performed based on the initial estimated intermediate image to update the estimated intermediate image, wherein, in each of the at least two iterations, in response to determining that a termination condition is satisfied, the intermediate image is determined.

9. The method according to claim 1, characterized in that The second iterative operation comprises an iterative deconvolution, the second objective function being associated with a third difference between the intermediate image and the target image multiplied by the system matrix.

10. An image processing system, characterized in that: include: an initial image generation module configured to generate an initial image by filtering image data generated by the image acquisition device; a sparse reconstruction module configured to perform a first iterative operation on the initial image to generate an intermediate image based on a first objective function, wherein the first objective function is associated with image fidelity of the intermediate image, continuity of the intermediate image, and sparsity of the intermediate image; as well as A deconvolution module is configured to perform a second iterative operation on the intermediate image to generate a target image based on a second objective function, wherein the second objective function is associated with a system matrix of the image acquisition device and a second difference between the intermediate image and the target image.

Images