Deep learning based single frame super-resolution microscopy image processing

Abstract

Methods, apparatus, systems, and non-transitory computer-readable media for image processing are provided. In an aspect, a computer-implemented method of image processing is provided, the method comprising: receiving a low resolution image of an object; generating, by an edge extractor, an edge map of the low resolution image; inputting the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object, wherein the neural network is trained using a multi-component loss function.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 252,181, filed October 5, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This specification relates broadly, but not exclusively, to methods, apparatuses, systems, and computer-readable media for image processing, and more specifically, to methods, apparatuses, systems, and computer-readable media for processing single-frame super-resolution microscopy images. Background Technology

[0004] Fluorescence microscopy has become an indispensable tool in biological research, but its spatial resolution is limited by the diffraction effect of light waves. Therefore, traditional fluorescence microscopy cannot resolve subcellular structures smaller than 200 nm.

[0005] Over the past two decades, numerous types of super-resolution microscopy have emerged that bypass the diffraction limit of light, such as structured illumination microscopy (SIM), stimulated emission depletion microscopy (STED), and single-molecule localization microscopy (SMLM). These super-resolution microscopes can push the achievable resolution of fluorescence microscopy to 20 nm to 150 nm; however, this comes at the cost of increased phototoxicity and reduced imaging speed, posing a challenge for live-cell applications that require both high spatial and temporal resolution. For example, SMLM can be used to resolve subcellular structures and achieves a tenfold improvement in spatial resolution compared to conventional fluorescence microscopy. However, the separation of single-molecule fluorescence events across thousands of frames significantly increases image acquisition time and phototoxicity, hindering the observation of transient intracellular dynamics. Therefore, despite the many advanced fluorescent probes, optical imaging systems, and image reconstruction algorithms proposed for temporally resolved and non-invasive super-resolution imaging, an inherent trade-off must be struck between spatial and temporal resolution, achievable signal intensity, and cytotoxicity due to the physical limitations of optical systems.

[0006] To overcome the aforementioned technical limitations, deep learning networks have been combined with various super-resolution microscopes to surpass the hardware limitations of fluorescence microscopy. Nevertheless, limited resolution improvements have been achieved. To achieve single-molecule accuracy in image resolution, multiple frames with single-molecule fluorescence events are still required for single-shot super-resolution image reconstruction. Therefore, fundamental problems of multi-frame super-resolution imaging, such as long acquisition times during microscopy and phototoxicity induced by photobleaching, continue to hinder its application in microscopic imaging—e.g., imaging of live cell dynamics.

[0007] Therefore, there is a need for efficient and compatible image processing methods to overcome the aforementioned technical limitations, particularly single-frame super-resolution microscopy image processing schemes that reconstruct super-resolution images from single frames (without requiring multiple frames with single-molecule fluorescence events). Summary of the Invention

[0008] According to one aspect, a computer-implemented image processing method is provided, the method comprising: receiving a low-resolution image of an object; generating an edge map of the low-resolution image using an edge extractor; and inputting the edge map and the low-resolution image into a neural network to reconstruct a super-resolution image of the object, wherein the neural network is trained using a multi-component loss function.

[0009] According to another aspect, an apparatus for image processing is provided, the apparatus comprising: at least one processor; and a memory including computer program code for execution by the at least one processor, the computer program code instructing the at least one processor to: receive a low-resolution image of an object; generate an edge map of the low-resolution image by an edge extractor; and input the edge map and the low-resolution image into a neural network to reconstruct a super-resolution image of the object, wherein the neural network is trained using a multi-component loss function.

[0010] According to another aspect, a microscope imaging system is provided, comprising: a fluorescence microscope for generating a low-resolution image of an object, and a microscope image processing apparatus for the low-resolution image as described herein, wherein the apparatus is coupled to the fluorescence microscope.

[0011] According to another aspect, a non-transitory computer-readable storage medium is provided, having instructions encoded thereon that, when executed by a processor, cause the processor to perform one or more steps in the method for image processing described herein. Attached Figure Description

[0012] The following embodiments and implementations are provided by way of example only, and will be better understood and readily comprehended by those skilled in the art from the following written description, which is read in conjunction with the accompanying drawings, wherein:

[0013] Figure 1 This is a schematic diagram of an image processing apparatus 100 according to an embodiment.

[0014] Figure 2 This is a flowchart illustrating a method 200 for image processing according to an embodiment.

[0015] Figure 3 A schematic diagram of embodiment 300 of the apparatus 100 is shown, in which an embodiment of method 200 is implemented.

[0016] Figure 4A An example of the perceptual loss function 400 is described.

[0017] Figure 4B An example of the adversarial loss function 410 is described.

[0018] Figure 4C An example of the frequency loss function 420 is described.

[0019] Figure 5 A diagram is shown of a method 500 for generating an edge map according to an embodiment.

[0020] Figure 6A An embodiment of a low-resolution (LR) image 600 of an object is depicted. Figure 6D A ground-based image 630 depicting the object is shown. In this embodiment, the object includes a microtube (MT).

[0021] Figure 6B Edge map 610 of the low-resolution image 600 extracted by the Canny operator is depicted. In contrast, Figure 6C Depicting as Figure 5 The method 500 of this application describes the edge map 620 of the low-resolution image 600 extracted.

[0022] Figure 7A A low-resolution (LR) image 700 of an object is displayed, along with a magnified image 702 of the region selected by the white squares in the LR image 700.

[0023] Figure 7B A ground reality (GT) image 704 of the object and a magnified image 706 of the area selected by the white square in the GT image 704 are displayed.

[0024] Figure 7C The image 708 of the reconstructed object is shown, generated by a commonly used deep learning network trained with a conventional multi-scale similarity (MS-SSIM) loss function using a single input. Figure 7C Also shown is a magnified image 710 of the area selected by the white square in the reconstructed image 708.

[0025] Figure 7D A reconstructed image 712 of the object is shown, generated by an embodiment of the image processing method 200 of this application. Figure 7D Also shown is a magnified image 714 of the region selected by the white square in the reconstructed image 712.

[0026] Figures 8A to 8HAn example is described in which SR images reconstructed using the SFSRM image processing method based on this application are used to study intracellular dynamics.

[0027] Figure 8A A microscope image 800 of an object is shown. In this embodiment, the object comprises microtubules from cells expressing mEmerald-ensconsin.

[0028] Figure 8B Multiple SR images of microtubules reconstructed based on the SFSRM image processing method described in this application are shown.

[0029] Figure 8C Describing as Figure 8B The radii of curvature are shown for the microtube bending dynamics at frequencies of 2 Hz and 100 Hz, respectively.

[0030] Figure 8D Multiple SR images of microtubules reconstructed based on the SFSRM image processing method described in this application are shown.

[0031] Figure 8E Describing as Figure 8D The terminal displacements of the microtubule terminal growth and contraction dynamics are shown at frequencies of 2 Hz and 100 Hz, respectively.

[0032] Figure 8F The image shows an LR image 810 of microtubules in living cells, an SR image 820 of microtubules reconstructed based on the SFSRM image processing method described in this application, a graph 830 showing the fluctuation of microtubules over time recorded at a frequency of 100 Hz, and a histogram 840 showing the lateral displacement of microtubules at 10 ms intervals.

[0033] Figure 8G The image shows an LR image 850 of microtubules in a living cell and multiple SR images 852, 854, 856, 858, 860, 862, 864, 866, and 868 of microtubules reconstructed based on the SFSRM image processing method described in this application.

[0034] Figure 8H The image shows an LR image 870 of microtubules in a living cell and multiple SR images 872, 874, 876, 878, 880, 882, and 884 of microtubules reconstructed based on the SFSRM image processing method described in this application.

[0035] Figures 9A to 9G An embodiment is described in which an SR image reconstructed using the SFSRM image processing method based on this application is used to study organelle interactions.

[0036] Figure 9ATwo-channel SR images 900 depicting microtubules and vesicles from cells expressing mEmerald-ensconsin and endocytogenic QDots655-streptavidin-tagged epidermal growth factor (EGF) proteins were reconstructed based on the SFSRM image processing method of this application.

[0037] Figure 9B(I) depicts multiple SR images of microtubules and vesicles reconstructed based on the SFSRM image processing method as described in this application.

[0038] Figure 9B(II) depicts multiple SR images of microtubules and vesicles reconstructed based on the SFSRM image processing method as described in this application.

[0039] Figure 9B(III) depicts multiple SR images of microtubules and vesicles reconstructed based on the SFSRM image processing method as described in this application.

[0040] Figure 9C(I) shows an LR image 910 of microtubules and vesicles and an SR image 920 of microtubules and vesicles reconstructed based on the SFSRM image processing method as described in this application.

[0041] Figure 9C(II) shows the trajectory diagrams of vesicles from an LR image sequence (not shown) recorded at a frequency of 2 Hz and an SR image sequence (not shown) recorded at a frequency of 100 Hz. The trajectories show the millisecond-level diffusion motion of vesicles along microtubules.

[0042] Figure 9C(III) shows the mean square displacement (MSD) analysis plot of the trajectory in Figure 9C(II).

[0043] Figure 9C(IV) shows a histogram that provides a statistical comparison of the instantaneous vesicle velocities recorded at 2 Hz and 100 Hz, respectively.

[0044] Figure 9D(I) and 9D(II) An embodiment is described in which SR images reconstructed using the SFSRM image processing method based on this application are used to study microtubule dynamics leading to non-directional vesicle transport.

[0045] Figure 9E A box-and-whisker diagram was drawn, showing the instantaneous vesicle vesicle transport of directional and non-directional vessels and a statistical comparison of their representative trajectories.

[0046] Figure 9F(I) , 9F(II) 9F(III) depicts an embodiment in which SR images reconstructed using the SFSRM image processing method based on this application are used to study vesicle transport dynamics at the intersection of different types of microtubules in cells.

[0047] Figure 9G A pie chart depicts a statistical comparison of the intersections of three types of microtubules in a cell.

[0048] Figures 10A to 10D An example is described in which SR images reconstructed based on the SFSRM image processing method of this application are used for in situ genome sequencing in the nuclei of interphase human fibroblasts.

[0049] Figures 11A to 11D Examples of using SR images reconstructed based on the SFSRM image processing method of this application for image-based phenotypic analysis or drug screening are described.

[0050] Figure 12 A block diagram of a computer system 1200 suitable for use as an image processing device is shown.

[0051] Figure 13 An example of a subject comprising a DNA nanometer is shown.

[0052] Figure 14 The image shows an LR image 1402 of microtubules in fixed cells, a ground truth (GT) image 1404 of the microtubules, and an SR image 1406 of the microtubules reconstructed from the LR image 1402 using the SFSRM image processing method of this application.

[0053] Figure 15 One embodiment is shown, wherein the objects include mitochondria labeled with the mitochondrial membrane protein Tomm20, endoplasmic reticulum (ER) labeled with the ER membrane protein Sec61β, EGFR protein after EGF internalization, clathrin-coated pits after EGF internalization, and enlarged nuclear pore complex protein Nup133.

[0054] Those skilled in the art will understand that the various elements in the figures are shown for simplicity and clarity and are not necessarily depicted to scale. For example, the dimensions of some elements in the illustrations, block diagrams, or flowcharts may be exaggerated relative to other elements to aid in understanding the present embodiment. Detailed Implementation

[0055] Embodiments will be described by way of example only with reference to the accompanying drawings. The same reference numerals and characters in the drawings refer to the same elements or equivalents.

[0056] Certain parts of the following description are presented, explicitly or implicitly, in the form of algorithms and functional or symbolic representations of operations on data in computer memory. These algorithmic descriptions and functional or symbolic representations are means by which those skilled in the art of data processing most effectively communicate their work to others skilled in the art. An algorithm herein and generally is considered to be a self-consistent sequence of steps that leads to a desired result. These steps are steps that require physical operations on physical quantities, such as electrical, magnetic, or optical signals that can be stored, transmitted, combined, compared, and otherwise manipulated.

[0057] Unless otherwise specified or apparent from the following, it is understood that throughout this specification, discussions using terms such as “receive,” “generate,” “input,” “reconstruct,” “train,” “extract,” “quantify,” “produce,” etc., refer to the actions and processes of a computer system or similar electronic device that manipulate and convert data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system or other information storage, transmission, or display device.

[0058] This specification also discloses apparatus for performing the method operations. Such apparatus may be specifically constructed for the desired purpose, or may include a computer or other means selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various machines may be used with the programs taught herein. Alternatively, it may be appropriate to construct more specialized apparatus to perform the desired method steps. The architecture of a computer suitable for performing the various methods / processes described herein will be presented in the following description.

[0059] Furthermore, this specification implicitly discloses a computer program, as it will be apparent to those skilled in the art that the various steps of the methods described herein can be implemented using computer code. The computer program is not intended to be limited to any particular programming language or its implementation. It is understood that the teachings of this specification can be implemented using a variety of programming languages ​​and their encodings. Moreover, the computer program is not intended to be limited to any particular control flow. Many other variations of the computer program are possible, which may use different control flows without departing from the spirit or scope of the invention.

[0060] Furthermore, one or more steps of the computer program may be executed in parallel rather than sequentially. Such a computer program can be stored on any computer-readable medium. The computer-readable medium may include storage devices such as disks or optical discs, memory chips, or other storage devices suitable for interfacing with a computer. The computer-readable medium may also include, for example, hardwired media exemplified in Internet systems, or wireless media exemplified in GSM mobile phone systems. When loaded and executed on such a computer, the computer program effectively implements a device for carrying out the steps of a preferred method.

[0061] This specification uses the term "configured as" in relation to systems, devices, and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action. For a dedicated logic circuit to be configured to perform a specific operation or action, this means that the circuit has electronic logic for performing the operation or action.

[0062] Embodiments of this application provide an image processing method that utilizes a deep learning network optimized through a joint optimization strategy to reconstruct a super-resolution (SR) image based on a single low-resolution image of an object. This joint optimization strategy combines prior information conditioning and a multi-component loss function to optimize the reconstruction, achieving significant resolution improvements and high detail accuracy in the reconstructed SR image.

[0063] When the low-resolution image is a fluorescence microscopy image (e.g., a wide-field fluorescence microscopy image), this image processing method provides a single-frame super-resolution microscopy (SFSRM) method, which reconstructs a single-molecule resolution SR image from a single fluorescence microscopy image without requiring multiple frames with single-molecule fluorescence events. In this way, this application advantageously improves the speed of super-resolution microscopy and avoids phototoxicity caused by the high illumination required to generate single-molecule excitations, thereby pushing the limits of fluorescence microscopy to unprecedented spatiotemporal resolution with a finite photon budget and avoiding possible trade-offs between spatial resolution, imaging speed, and light dose.

[0064] In addition to wide-field fluorescence microscopy imaging systems, well-trained deep learning networks enable image processing methods (such as SFSRM) to adapt to various microscopy imaging systems, including echo-plane imaging (EPI), total internal reflection fluorescence (TIRF) microscopy, confocal microscopy, and light-sheet microscopy. This makes super-resolution imaging possible for laboratories lacking advanced optical systems. SFSRM methods hold great potential in studying subcellular processes requiring explanation of time-dynamics within the context of ultrastructural information, which in turn opens doors to new discoveries in live-cell imaging involving organelle dynamics and interactions, viral infection, cellular uptake, and intracellular delivery of nanoparticles.

[0065] Furthermore, the SFSRM method can advantageously reconcile the conflict between high resolution and high throughput in any existing microscope, making it suitable for large-scale, high-workload super-resolution imaging, such as whole-genome imaging, imaging-based cell analysis heterogeneity, phenotypic analysis, and drug screening.

[0066] Figure 1 A schematic diagram of an apparatus 100 for image processing is shown. The apparatus 100 includes at least one or more processors 102 and a memory 104. The at least one processor 102 and the memory 104 are interconnected. The memory 104 includes computer program code (…). Figure 1 (not shown), which is used by the at least one processor 102 to perform according to an image processing method—for example, such as Figure 2 The steps are performed using the method illustrated in Embodiment 200 and the method described in this application.

[0067] In step 202, computer program code instructs the at least one processor 102 of the device 100 to receive a low-resolution image of the object. In some embodiments, low resolution refers to a resolution in the range of 200 nm to 300 nm, while high resolution refers to a resolution in the range of 30 nm to 50 nm.

[0068] In some embodiments, the device 100 is a standalone device specifically configured for image processing. In these embodiments, at least one processor 102 of the device 100 can receive low-resolution images from a memory 104 or storage unit (not shown) within the device 100. These low-resolution images may have been previously acquired and transmitted to the device 100 by various imaging systems, or acquired on-the-fly by various imaging systems and transmitted to the device 100 for real-time processing. In this way, the device 100 is advantageously compatible with processing various types of images acquired by various microscope imaging systems—e.g., wide-field fluorescence microscopy, echo-plane imaging (EPI), total internal reflection fluorescence (TIRF), confocal microscopy, light sheet microscopy, etc.—to reconstruct super-resolution images based on various low-resolution images.

[0069] On the other hand, those skilled in the art will understand that device 100 can be integrated into a microscope imaging system and implemented as an integral image processing component of the microscope imaging system. Therefore, in some alternative embodiments, device 100 can be coupled to and receive low-resolution microscope images from a microscope component or a memory or storage component of the microscope imaging system. Depending on the type of microscope imaging system, device 100 is specified to reconstruct a super-resolution image based on a specific type of low-resolution image.

[0070] In any of the above embodiments, the low-resolution image is a fluorescence microscope image of the object. Depending on the type of microscope imaging system from which the device 100 receives images or the type of microscope assembly to which the device 100 is coupled, the fluorescence microscope image may be a wide-field fluorescence microscope image, a confocal microscope image, a total internal reflection fluorescence (TIRF) microscope image, or a light sheet microscope image.

[0071] The object includes intracellular structures of one or more cells. The intracellular structures include one or more microtubules and / or organelles of one or more cells. The organelles include one or more subcellular structures of the one or more cells, such as the nucleus, mitochondria, endoplasmic reticulum, Golgi apparatus, vesicles, vacuoles, etc.

[0072] In some embodiments, the subcellular structure includes one or more DNA nanometers, mitochondria labeled with the mitochondrial membrane protein Tomm20, endoplasmic reticulum (ER) labeled with the ER membrane protein Sec61β, EGFR protein internalized by EGF, clathrin-coated pits internalized by EGF, and enlarged nuclear pore complex protein Nup133. These embodiments are described in detail below. Figure 13 and 15 It is depicted in the text and described in the corresponding paragraphs of the instruction manual.

[0073] In some embodiments, the one or more cells include living cells. Examples where the one or more cells are living cells are described in... Figure 8F-8H It is depicted in the text and described in the corresponding paragraphs of the instruction manual.

[0074] In some other embodiments, the one or more cells may be fixed cells. Embodiments where the one or more cells are fixed cells are described below. Figure 14 It is described in the relevant paragraphs of the instruction manual.

[0075] In some embodiments, the one or more cells include human fibroblasts. The organelles include genomic segments of the nucleus of a human fibroblast.

[0076] Subsequently, in step 204, computer program code instructs at least one processor 102 of the device 100 to generate an edge map of the low-resolution image by an edge extractor.

[0077] Figure 5 The document describes an embodiment of method 500 for step 204 of generating edge map 502. (See reference...) Figure 5 In step 204, the edge extractor (not shown) of device 100 extracts an edge map 502 at the sub-pixel level based on the radial symmetry 504 of fluorophores in the fluorescence microscope image, wherein each sub-pixel (x c y c The edge intensity of a pixel is defined by the degree to which the surrounding intensity gradients converge to the sub-pixel, denoted as . And the edge intensity mentioned therein is determined by sub-pixels (x) c y c The edge map 502 is weighted by the pixel intensity I. Therefore, the edge map 502 can be represented as...

[0078] By determining the edges of intracellular structures in fluorescence microscopy images, the features of the intracellular structures are extracted in edge map 502.

[0079] Subsequently, in step 206, computer program code instructs at least one processor 102 of the device 100 to input edge maps and low-resolution images into a neural network to reconstruct a super-resolution image of the object, wherein the neural network is trained using a multi-component loss function.

[0080] In one embodiment, the neural network is an enhanced super-resolution generative adversarial network (ESRGAN). Those skilled in the art will understand that this neural network can be implemented using other deep learning neural networks.

[0081] In some embodiments, the multi-component loss function includes one or more of a pixel-oriented loss function, a perceptual loss function, an adversarial loss function, and / or a frequency loss function. In one embodiment, the pixel-oriented loss function includes a multi-scale structural similarity (MS-SSIM) and L1 norm loss function, i.e., an MS-SSIM-L1 loss function. In this regard, Figure 4A , 4B Examples of the perceptual loss function 400, the adversarial loss function 410, and the frequency loss function 420 are illustrated in Figures 4C and 4C, respectively, with details provided in the following paragraphs concerning these figures.

[0082] Subsequently, computer program code instructs at least one processor 102 of the device 100 to input the reconstructed super-resolution image of the object and the ground reality image of the object into a multi-component loss function to quantify the differences between the reconstructed super-resolution image and the ground reality image. This step is in Figure 3 The steps described are 330 and 332, the details of which are in the section on... Figure 3 Provided in the following paragraphs.

[0083] Subsequently, computer program code instructs at least one processor 102 of the device 100 to input the quantized difference between the reconstructed super-resolution image and the ground reality image into a neural network for subsequent training to optimize the neural network. This step is in Figure 3 This is described as step 334, the details of which are in the section on... Figure 3 Provided in the following paragraphs.

[0084] As mentioned above, Figure 3 An embodiment 300 of the apparatus 100 is illustrated, in which an embodiment of method 200 is implemented.

[0085] For implicit purposes, embodiment 300 of device 100 is depicted as including neural network 306 and multi-component loss function component 312. Those skilled in the art will understand that device 100 may include... Figure 3 Other components not shown. In some embodiments, the neural network 306 and the multi-component loss function component 312 may be implemented as separate components in the device 100. Alternatively, the neural network 306 and the multi-component loss function component 312 may be integrated into a deep network training component in the device 100 based on actual needs and requirements.

[0086] like Figure 3 As shown, the device 100 receives a low-resolution image 302 of the object according to step 202 of method 200. As described above, the low-resolution image 302 can be received from the memory or storage unit of the device 100, or from the microscope component or memory or storage component of the microscope imaging system.

[0087] An embodiment of the low-resolution image 302 of the object is shown in Figure 6A The image is shown in low resolution (LR) image 600. Figure 6D A ground-based (GT) image 630 depicting the object is provided. In this embodiment, the LR image is degraded from the GT image 630 by blurring with a 200 nm dot spread function and then downsampling by a factor of 10. In this embodiment, the object comprises a microtube (MT).

[0088] Another embodiment of the low-resolution image 302 of the object is... Figure 7A The image is shown in low resolution (LR) image 700. Figure 6B A ground-based (GT) image 704 depicting the object is shown. In this embodiment, the LR image is degraded from the GT image 704 by blurring with a 200 nm dot spread function and then downsampling by a factor of 10. In this embodiment, the object includes a microtube (MT).

[0089] Subsequently, the edge map 304 of the low-resolution image 302 is generated 322 by an edge extractor (not shown) according to step 204 of method 200.

[0090] Unlike conventional edge extraction operators used to define edges from the intensity gradient of an image, such as the Prewitt, Canny, Roberts, and Sobel operators, the edge extractor of device 100 (not shown) extracts the edge map 302 at the subpixel level based on the radial symmetry 504 of fluorophores in a fluorescence microscope image, as described above. This can advantageously generate sharper edge maps with more accurate edge information of intracellular structures (e.g., microtubules).

[0091] Comparison of edge extraction in Figure 6B and 6C As shown in the image. Figure 6B Edge map 610 of a low-resolution image 600 extracted by the conventional Canny operator is depicted. In contrast, Figure 6C Depicting through, as Figure 5 The method 500 of this application shows the edge map 620 of the low-resolution image 600 extracted. Figure 6C It is clearly shown that, compared with the edge map 610 generated by the conventional Canny operator, the edge map 620 generated by this application provides more accurate edge information of the microtube.

[0092] Since features of intracellular structures can be extracted in edge map 620 by determining the edges of intracellular structures in low-resolution image 600, edge map 620 with more accurate microtubule edge information generated by this application can advantageously and inherently produce super-resolution images reconstructed with high detail accuracy.

[0093] Subsequently, according to step 206 of method 200, edge map 304 and low-resolution image 302 are input (324, 326) into neural network 306 to reconstruct super-resolution image 308 of the object, wherein the neural network is trained using multi-component loss function 312.

[0094] Subsequently, in steps 330 and 332, the reconstructed super-resolution image 308 and the ground reality image 310 of the object are input into the multi-component loss function 312 to quantify the difference between the reconstructed super-resolution image 308 and the ground reality image 310.

[0095] like Figure 3 As shown, the multi-component loss function 312 includes a pixel-oriented loss function 314, a perceptual loss function 316, an adversarial loss function 318, and a frequency loss function 320. In this embodiment, the pixel-oriented loss function includes the MS-SSIM-L1 loss function, which measures the pixel-wise difference between the super-resolution image 308 and the ground reality image 310.

[0096] Figure 4A An embodiment 400 of the perceptual loss function is described. In this embodiment, the perceptual loss function 400 receives the reconstructed super-resolution image 308 and the ground reality image 310 in steps 330 and 332, and measures the difference between the feature maps extracted from the reconstructed super-resolution image 308 and the ground reality image 310 using a visual geometric group (VGG) neural network 402. Those skilled in the art will understand that other neural networks can also be used as alternatives to or supplements to the VGG neural network 402.

[0097] Figure 4B An embodiment 410 of the adversarial loss function is described. In this embodiment, the adversarial loss function 410 receives the reconstructed super-resolution image 308 and the ground reality image 310 in steps 330 and 332, and uses a U-net neural network 412 to distinguish the blurriness of the ground reality image 310 and the blurriness of the reconstructed super-resolution image 308. Those skilled in the art will understand that other neural networks can also be used as alternatives to or supplements to the U-net neural network 402.

[0098] Figure 4CAn embodiment 420 of the frequency loss function is described. In this embodiment, the frequency loss function 420 receives the reconstructed super-resolution image 308 and the ground reality image 310 in steps 330 and 332, and compares the spectral difference 424 between the reconstructed super-resolution image 308 and the ground reality image 310 in a specified frequency region using a fast Fourier transform function 422. In some embodiments, the specified region for frequency comparison includes the entire spectrum for the noise-free image and 75% of the spectrum for the noisy image.

[0099] Subsequently, in step 334, the quantization difference between the reconstructed super-resolution image 308 and the ground reality image 310 is input into the neural network 306 for subsequent training to optimize the neural network 306.

[0100] like Figure 3 and Figures 4A to 4C As shown, the multi-component loss function 312 combines pixel-oriented loss with perceptual loss to improve the fidelity / accuracy of details in the reconstructed super-resolution image 308, and combines adversarial loss with frequency loss to penalize blur and suppress high-frequency artifacts in the reconstructed super-resolution image 308. In this way, the present application advantageously allows for fine-structure reconstruction from a 10x degraded low-resolution image, resulting in a reconstructed super-resolution image 308 that achieves high detail accuracy relative to the ground reality image 310 with an MS-SSIM exponent of 0.96. Figures 7A to 7D As shown in the image.

[0101] Figures 7A to 7D The image shows a comparison of detail accuracy among a low-resolution (LR) image 700, a ground truth (GT) image 704, a reconstructed image 708, and a reconstructed super-resolution image 712. The reconstructed image 708 was generated by a commonly used deep learning network trained with a single input using a conventional multi-scale similarity (MS-SSIM) loss function. The reconstructed super-resolution image 712 was reconstructed using the image processing method 200 of this application.

[0102] Figure 7A A low-resolution (LR) image 700 of the object and a magnified image 702 of the region selected by the white squares in the LR image 700 are shown. As described above, the object comprises microtubules (MTs). The LR image 700 is degraded from a ground-based image 704 by blurring with a 200 nm point spread function and then downsampling by a factor of 10. Detailed structures in the selected region of the LR image 700 are indicated by white arrows in the magnified image 702. It can be seen that almost no details are identifiable in the magnified image 702 of the LR image 700.

[0103] Figure 7B The image displays a ground truth (GT) image 704 of the object and a magnified image 706 of the area selected by the white square in the GT image 704. This selected area is the same as the selected area in the LR image 700. The detailed structure of the selected area in the GT image 704 is indicated by the white arrows in the magnified image 706.

[0104] Figure 7C The image 708 of the reconstructed object is shown, generated by a commonly used deep learning network trained with a conventional multi-scale similarity (MS-SSIM) loss function using a single input. Figure 7C Also shown is a magnified image 710 of the region selected by the white square in reconstructed image 708. This selected region is the same as the selected region in LR image 700. The detailed structure in the selected region of reconstructed image 708 is indicated by white arrows in magnified image 710. It can be seen that the detailed structure is identifiable in magnified image 710. However, these identifiable detailed structures deviate from the detailed structure in magnified image 706 of GT image 704.

[0105] and Figure 7C compared to, Figure 7D A reconstructed image 712 of the object reconstructed by the image processing method 200 of this application is shown. Figure 7D Also shown is a magnified image 714 of the region selected by the white square in reconstructed image 712. This selected region is the same as the selected region in LR image 700. The detailed structures in the selected region of reconstructed image 712 are indicated by white arrows in magnified image 714. It can be seen that the detailed structures in magnified image 714 are identifiable. Advantageously, these identifiable detailed structures are very similar to the detailed structures in magnified image 706 of GT image 704, indicating a satisfactory image reconstruction with high detail accuracy.

[0106] like Figures 7A to 7D As shown, the reconstructed image 712 generated by the image processing method 200 of this application correctly reconstructs the fine structure of the selected regions that are highly similar to those shown in the GT image 704 and the magnified image 706, while the reconstructed image 708 generated by a commonly used deep learning network trained using the MS-SSIM loss function with a single input fails to reconstruct the correct structure.

[0107] pass Figures 2 to 5 The image processing method 200 described herein achieves the aforementioned accurate high-resolution image reconstruction based on a single low-resolution image, i.e., a single frame. This demonstrates that the present application advantageously provides a single-frame super-resolution microscopy (SFSRM) image processing method that reconstructs a super-resolution image with high detail accuracy from a single frame.

[0108] In other words, the image processing method of this application provides a single-frame super-resolution (SR) microscopy (SFSRM) method that reconstructs a single-molecule resolution SR image from a single fluorescence microscopy image without requiring multiple frames with single-molecule fluorescence events. In this way, this application advantageously improves the speed of super-resolution microscopy and avoids phototoxicity caused by the high illumination required to generate single-molecule excitations, thereby pushing the limits of fluorescence microscopy to unprecedented spatiotemporal resolution with a limited photon budget, and circumventing possible trade-offs between spatial resolution, imaging speed, and light dose.

[0109] Furthermore, a well-trained deep learning network 306 enables the image processing methods (e.g., the SFSRM method) to adapt to various microscopy imaging systems, such as wide-field fluorescence microscopy, echo-plane imaging (EPI), total internal reflection fluorescence (TIRF) microscopy, confocal microscopy, and light-sheet microscopy, making super-resolution imaging possible in laboratories lacking advanced optical systems. The SFSRM method holds great potential in studying subcellular processes where time-dynamics need to be interpreted within the context of ultrastructural information, which in turn opens doors to new discoveries in live-cell imaging, involving organelle dynamics and interactions, viral infection, cellular uptake, and intracellular delivery of nanoparticles.

[0110] Furthermore, the SFSRM method can advantageously reconcile the conflict between high resolution and high throughput in any existing microscope, making it suitable for large-scale, high-workload super-resolution imaging, such as whole-genome imaging, imaging-based cellular heterogeneity analysis, phenotypic analysis, and drug screening. SFSRM image processing methods are used to study intracellular dynamics at ultra-high spatiotemporal resolution.

[0111] Figures 8A to 8H An example is described in which SR images reconstructed using the SFSRM image processing method based on this application are used to study intracellular dynamics. Details are as follows.

[0112] Microtubules (MTs) are considered high-speed transport systems within cells. MT dynamics are involved in a variety of cellular functions, such as organizing and maintaining cell shape, promoting ciliary movement, and mediating cargo transport. Due to the resolution limitations of conventional fluorescence microscopy, in vivo MT fluctuations in routine techniques can only be inferred from the movement of bound motor proteins. Figures 8A to 8H In this paper, the SR image reconstructed based on the SFSRM image processing method of this application successfully visualizes the dynamics of MT, which has not been explored in the prior art.

[0113] Figure 8AA microscopic image 800 of the object is shown. In this embodiment, the object comprises microtubules from cells expressing mEmerald-ensconsin. Microscopic image 800 includes a portion of a low-resolution (LR) image of the microtubules in the lower left of image 800. In one embodiment, the LR image is a wide-field (WF) fluorescence microscopy image. Microscopic image 800 also includes a super-resolution (SR) image of the microtubules reconstructed on the remainder of image 800 based on the SFSRM image processing method as described in this application. Utilizing the unprecedented resolution improvement provided by SFSRM image processing, the entangled MT network is clearly resolved, and in Figure 8A Various MT morphologies can be observed in the SR images, such as bending, crossing, and bundled.

[0114] Figure 8B Multiple SR images of the microtubules reconstructed using the SFSRM image processing method described in this application are shown. The reconstructed SR images correspond to LR images of the microtubules obtained at 0 seconds, 6 seconds, 13 seconds, 19 seconds, 26 seconds, and 32 seconds, respectively. Figure 8B In the image, the multiple SR images indicate the microtubule bending dynamics of the microtubule.

[0115] In this application Figures 8B to 8H and the following Figures 9A to 9G The figure depicts multiple SR images of microtubules or other intracellular structures reconstructed based on the SFSRM image processing method. Those skilled in the art will understand that various numbers of LR images can be obtained at different intervals (i.e., different imaging speeds) based on actual needs, and corresponding SR images can be reconstructed accordingly.

[0116] Figure 8C Describing as Figure 8B The radii of curvature for the microtube bending dynamics shown are 2 Hz and 100 Hz, respectively. Figure 8C As shown, even at an imaging speed of 100 Hz, the SR image reconstructed based on the SFSRM image processing method of this application can clearly capture the motion details of the microtube.

[0117] Figure 8D Multiple SR images of microtubules reconstructed using the SFSRM image processing method described in this application are shown. The reconstructed SR images correspond to LR images of the microtubules obtained at 0 seconds, 15 seconds, 30 seconds, and 45 seconds, respectively. Figure 8D In the images, the multiple SR images indicate the microtubule tip growth and contraction dynamics.

[0118] Figure 8E Describing as Figure 8DThe diagram shows the terminal displacement of the microtubule terminal growth and contraction dynamics at frequencies of 2 Hz and 100 Hz. Figure 8E As shown, even at an imaging speed of 100 Hz, the SR image reconstructed based on the SFSRM image processing method of this application clearly captures the details of the microtubule's distal displacement.

[0119] Figure 8F The image shows an LR image 810 of microtubules in a living cell, an SR image 820 of the microtubules reconstructed based on the SFSRM image processing method described in this application, a graph 830 showing the microtubules fluctuating over time at a frequency of 100 Hz, and a histogram 840 showing the lateral displacement of the microtubules at 10 ms intervals.

[0120] Figure 8G The image shows an LR image 850 of microtubules in a living cell and multiple SR images 852, 854, 856, 858, 860, 862, 864, 866, and 868 of the microtubules reconstructed based on the SFSRM image processing method described in this application. Due to the above... Figures 8E to 8F The paper demonstrates that even at an imaging speed of 100 Hz, the SR image reconstructed based on the SFSRM image processing method of this application can clearly capture the structural details of the microtubules. Figure 8G The reconstructed SR image corresponds to the LR image of the microtube obtained at an imaging speed of 100 Hz (i.e., a time interval of 0.01 seconds). Figure 8G In the images, the multiple SR images indicate microtubule bundle instability caused by uncoordinated vibrations of the microtubules.

[0121] Figure 8H LR image 870 of microtubules in living cells and multiple SR images 872, 874, 876, 878, 880, 882, and 884 of the microtubules reconstructed based on the SFSRM image processing method described in this application are shown. Figure 8G Similarly, in Figure 8H The reconstructed SR image corresponds to the LR image of the microtube acquired at an imaging speed of 100 Hz (i.e., a time interval of 0.01 seconds). Figure 8H In the images, these multiple SR images indicate that the local microtubule topology changes within a short period of time.

[0122] Figures 8B to 8E This illustrates the deformation dynamics of MT, such as MT bending, recorded at ultrafast imaging speeds (see [link]). Figure 8B This allows in Figure 8C High-frequency fluctuations, including the time-varying MT bending radius, were captured in (100Hz), as well as in... Figure 8EThe randomly walking MT growth trajectory was captured at (100 Hz). These results demonstrate that, due to hardware limitations of conventional fluorescence microscopy, it is difficult to capture MT growth trajectories at low sampling frequencies (see [reference]). Figure 8C and 8E At 2 Hz, the intracellular dynamics observed in milliseconds by conventional fluorescence microscopy may be significantly underestimated. This limitation is overcome by reconstructing SR images using the SFSRM image processing method based on this application.

[0123] also, Figures 8F to 8H For the first time, intracellular MT transverse vibrations at high frequencies in the approximately 100 nm range have been revealed (see [link]). Figure 8F This is undetectable in low-resolution wide-field (WF) microscopy images. Statistical analysis of the lateral MT displacement over a 10 ms interval (see...) Figure 8F The histogram (840) shows that the typical displacement is approximately 30 nm, which is much larger than the system drift (less than 10 nm over 50 seconds), thus confirming the observations regarding real MT vibrations in this application. Figure 8G and 8H It can be seen that inconsistent vibrations of the MT within the bundle can lead to bundle instability (see...). Figure 8G Furthermore, the local MT network morphology can change rapidly due to the random vibrations of the MT (see...). Figure 8H ).exist Figures 8F to 8H The direct visualization of MT vibrations studied in this study can help advance our understanding of intracellular force fluctuations and provide evidence for different microtubule vibration models.

[0124] SFSRM image processing method for studying organelle interactions at ultra-high spatiotemporal resolution

[0125] exist Figures 9A to 9G In the depicted embodiments, SR images reconstructed using the SFSRM image processing method based on this application were used to study organelle interactions. Details are as follows.

[0126] The intracellular environment is highly ambiguous and dynamic. Within the cell, organelles cooperate to perform cellular functions. For example, cargo proteins endocytosed are carried in vesicles and travel along the microtubules (MT) to the nucleus. Studying cargo transport processes provides guidance for drug delivery or viral infection processes, and is therefore of great research interest. Due to the small size of vesicles (50 nm to 200 nm) and their dense microtubules, super-resolution microscopy is required to clearly reveal their spatial relationships. Unfortunately, super-resolution microscopy cannot provide high temporal resolution, which is also necessary to capture dynamic interactions.

[0127] The above technical limitations are overcome by reconstructing SR images based on the SFSRM image processing method of this application. Figures 9A to 9G The details of vesicle delivery are clearly shown at a spatial resolution of up to 20 nm and a temporal resolution of 100 Hz.

[0128] Figure 9A A dual-channel SR image 900 depicting microtubules and vesicles from cells expressing mEmerald-ensconsin and endocytogenic QDots655-streptavidin-tagged epidermal growth factor (EGF) proteins is provided, reconstructed based on the SFSRM image processing method of this application. With the high detail accuracy achieved by this application, both microtubules and vesicles can be identified from the reconstructed SR image 900.

[0129] Because the drawings in patent application documents are not required to be in color, Figure 9A In the image, vesicles are depicted with bright tones, while microtubules are depicted with relatively darker tones. In some embodiments, the dual-channel SR image 900 is generated by using... Figure 3 The image depicts a color-merged image of the reconstructed SR images of microtubules and vesicles obtained from SR reconstruction.

[0130] Two reconstructed SR images of microtubules and vesicles can be based on two LR images of microtubules and vesicles captured by single-channel or multi-channel microscopy.

[0131] Figure 9B(I) depicts multiple SR images of microtubules and vesicles reconstructed using the SFSRM image processing method described in this application. The reconstructed SR images correspond to LR images of microtubules and vesicles acquired at 0 sec, 0.06 sec, 0.15 sec, 0.45 sec, 0.86 sec, 0.94 sec, and 1.20 sec, respectively. In Figure 9B(I), these multiple SR images indicate the vesicle transport dynamics as vesicles move back and forth along the microtubules.

[0132] Figure 9B(II) depicts multiple SR images of microtubules and vesicles reconstructed based on the SFSRM image processing method described in this application. The reconstructed SR images correspond to LR images of microtubules and vesicles acquired at 0 seconds, 0.02 seconds, 0.03 seconds, 0.04 seconds, 0.05 seconds, 0.06 seconds, and 0.07 seconds, respectively. In Figure 9B(II), these multiple SR images indicate the vesicle transport dynamics as the vesicles rotate around the microtubules.

[0133] Figure 9B(III) depicts multiple SR images of microtubules and vesicles reconstructed using the SFSRM image processing method described in this application. The reconstructed SR images correspond to LR images of microtubules and vesicles acquired at 0 sec, 0.03 sec, 0.04 sec, 0.11 sec, 0.21 sec, 0.22 sec, and 0.26 sec, respectively. In Figure 9B(III), these multiple SR images indicate the vesicle transport dynamics when one vesicle collides with another vesicle.

[0134] Figure 9C(I) shows an LR image 910 and an SR image 920 of microtubules and vesicles reconstructed based on the SFSRM image processing method as described in this application.

[0135] Figure 9C(II) shows a plot of vesicle trajectories from an LR image sequence (not shown) recorded at a frequency of 2 Hz and an SR image sequence (not shown) recorded at a frequency of 100 Hz. The trajectories show the diffusion motion of vesicles along microtubules on the order of milliseconds.

[0136] Figure 9C(III) shows the mean square displacement (MSD) analysis of the trajectory in Figure 9C(II). The MSD reflects the mean square distance <Δr2(τ)> traveled by the vesicle over a certain lag time τ, which usually follows a power law trend <Δr2(τ)>∝τα, where α represents the motion characteristics. The smaller α is, the more random or diffuse the motion; while the larger α is, the more directional the motion.

[0137] Figure 9C(IV) shows a histogram that provides a statistical comparison of the instantaneous vesicle velocities recorded at 2 Hz and 100 Hz, respectively.

[0138] As described above, Figures 9B(I) through 9B(IV) illustrate three examples of vesicle transport dynamics: (I) moving back and forth along the MT, (II) rotating around the MT, and (III) colliding with other vesicles and then changing direction. These subtle and rapid random walks are undetectable at low spatial resolution (see LR image 910 of Figure 9C(I)) and low temporal resolution (see Figure 9C(II), 2 Hz), indicating that vesicle motion is scale-dependent. At the millisecond scale, thermal diffusion dominates vesicle motion (see Figure 9C(III), 100 Hz, α = 0.25), while at the second scale, directional transport dominates vesicle motion (see Figure 9C(III), 100 Hz, α = 1.2). In cases of insufficient imaging speed, the diffusion motion of vesicles is missed, as evidenced by the different trajectories obtained from images acquired at 2 Hz and 100 Hz, as shown in the MSD plot (see Figure 9C(III); 2 Hz vs. 100 Hz). Therefore, the actual instantaneous velocity of the vesicle during transport—which is approximately 4 μm / s (see Figure 9C(IV), 100 Hz)—will be significantly underestimated (estimated at 0.5 μm / s at 2 Hz in Figure 9C(IV) according to previous reports).

[0139] In addition to the subtle diffusion motion of vesicles, Figures 9D(I)-9D(II) The non-directional transport of vesicles was also investigated, which had been reported in previous studies using single-particle tracking but was not fully explained.

[0140] exist Figure 9D(I) and 9D(II) In the depicted embodiments, SR images reconstructed using the SFSRM image processing method based on this application are used to study microtubule dynamics that lead to non-directional vesicle transport.

[0141] Figure 9D(I) shows multiple SR images of microtubules and vesicles reconstructed based on LR images of microtubules and vesicles acquired at 0 sec, 0.03 sec, 0.06 sec, 0.21 sec, 0.26 sec, 0.35 sec, and 0.72 sec, respectively. In Figure 9D(I), these multiple SR images show the lateral movement of microtubules, which transport vesicles attached to them to nearby microtubules, resulting in non-directional transport.

[0142] Figure 9D(II) shows multiple SR images of microtubules and vesicles reconstructed based on LR images of microtubules and vesicles acquired at 0 sec, 0.09 sec, 0.57 sec, 0.78 sec, 1.02 sec, 1.10 sec, and 1.15 sec, respectively. In Figure 9D(II), these multiple SR images demonstrate that random oscillations of the surrounding microtubules facilitate vesicle switching to different microtubules, thereby enabling non-directional transport.

[0143] Figure 9E A box-and-whisker diagram is depicted, showing, as Figure 9D(I) and 9D(II) A statistical comparison of the instantaneous vesicle velocities and their representative trajectories for directional and non-directional vesicle delivery is shown. Figure 9E The study showed that, compared to directional motion, non-directional motion has approximately twice the average instantaneous velocity and four times the widest distribution range, suggesting that these displacements may be related to MT fluctuations rather than motor-driven motion.

[0144] Due to the dense distribution of mediators (MTs), they provide vesicles with pathways. Furthermore, intersections form and influence vesicle transport. Previous studies have shown that vesicles can pass through, pause, switch, or reverse at intersections. In this application, it is noted that all vesicles ultimately pass through the investigated intersections; however, dwell times vary considerably and depend largely on the complexity of the intersection. Accordingly, intersections are divided into three groups / classes based on the number of MTs at each intersection.

[0145] therefore, Figure 9F(I) , 9F(II) 9F(III) depicts an embodiment in which SR images reconstructed using the SFSRM image processing method based on this application are used to study vesicle transport dynamics at different types of microtubule intersections in cells. Based on the number of microtubules involved, microtubule intersections are classified into three categories: (I) 2 microtubules, (II) 3 to 5 microtubules, and (III) more than 5 microtubules.

[0146] Figure 9F(I) shows that for the simplest intersection of two MTs, a vesicle can easily pass through one MT by crawling over it, usually within two seconds; and MT vibration is unlikely to interrupt vesicle transmission.

[0147] Figure 9F(II) shows that for intersections with 3–5 medians, vesicles tend to interfere with the dynamics of nearby medians. Therefore, if the surrounding medians fluctuate violently, the vesicles will be blocked, and the time required to pass through one of these intersections ranges from a few seconds to ten seconds.

[0148] Figure 9F(III) shows that for intersections involving more than 5 MTs tethered together, vesicles are most likely to become trapped at the intersection until the fluctuations of the surrounding MTs become coordinated and the star-shaped intersection loosens.

[0149] However, the coordination of fluctuations and loosening at intersections is highly uncertain, and such a process can take anywhere from tens of seconds to several minutes. In this regard, Figure 9G A pie chart depicts a statistical comparison of vesicle residence time and percentage of vesicles in cells at the intersection of three types of microtubules.

[0150] Figure 9GThe box-and-whisker diagram shows that, generally, the more complex the intersection, the longer the dwell time. For intersections involving more than 5 medians, the dwell time can exceed one minute. Fortunately, such intersections account for only about 10% of all intersections in a cell, and more than half of the intersections contain only 3-5 medians, such as... Figure 9G As shown in the pie chart.

[0151] SFSRM Image Processing Methods for High-Throughput Whole Genome Sequencing

[0152] Figures 10A to 10D An example is described in which an SR image reconstructed based on the SFSRM image processing method of this application is used for in situ genome sequencing in the nucleus of interphase human fibroblasts.

[0153] The ability to view the genome in situ at high genomic resolution and as a whole is becoming increasingly important. Previous methods combined fluorescence in situ hybridization (FISH) with stochastic optical reconstruction microscopy (STORM) to provide super-resolution images of genomic segments in human fibroblasts. However, imaging times could take several hours to image tens to hundreds of cells per experiment. In contrast, the SFSRM image processing method advantageously achieves imaging speeds up to 100 Hz, allowing it to image hundreds to thousands of cells per experiment with relatively negligible imaging times.

[0154] in this regard, Figure 10A The genome of interphase human fibroblast nuclei was shown to be chromosomally marked by fluorescent in situ hybridization (FISH). Figure 10B A wide-field LR image of the cell nucleus was depicted. Figure 10C An SR image of a cell nucleus reconstructed using the SFSRM image processing method of this application is depicted. Figure 10D Chromosome sequence listings were depicted using different probe combinations.

[0155] SFSRM Image Processing Method for Phenotypic Analysis and Subcellular Morphology-Based Drug Screening

[0156] Figures 11A to 11D Examples of using SR images reconstructed based on the SFSRM image processing method of this application for image-based phenotypic analysis or drug screening are described.

[0157] Figure 11A A schematic diagram of multiple organelles in a cell is shown. Figure 11B The paper depicts SR images of some organelles among several organelles reconstructed using the SFSRM image processing method of this application. Figure 11C A schematic diagram depicts how SR images are analyzed to extract features of multiple organelles. Figure 11DA schematic diagram depicts the relationships between the features to obtain the outlines of the multiple organelles.

[0158] Image-based profiling holds immense potential for identifying disease-associated screenable phenotypes, understanding disease mechanisms, and predicting drug activity, toxicity, or mechanisms of action. As the lowest-cost high-dimensional profiling technique, and inherently offering single-cell resolution—capturing important heterogeneous cellular behaviors—image-based profiling is gaining increasing popularity. However, meaningful profiling requires hundreds of trials. Traditional single-molecule localization microscopy takes over ten minutes to acquire a single super-resolution image. Therefore, applying it to image-based profiling is prohibitively time-consuming. Benefiting from the high temporal resolution provided by the SFSRM image processing method, this application enables high-throughput super-resolution imaging, thereby advancing the development of image-based profiling.

[0159] Figure 12 A block diagram of a computer system 1200 is shown, which is suitable for use as an apparatus 100 for image processing as described herein.

[0160] The following description of the computer system / computing device 1200 is provided by way of example only and is not intended to be limiting.

[0161] like Figure 12 As shown, the example computing device 1200 includes a processor 1204 for executing software routines. Although a single processor is shown for clarity, the computing device 1200 may also include a multiprocessor system. The processor 1204 is connected to a communication infrastructure 1206 for communicating with other components of the computing device 1200. The communication infrastructure 1206 may include, for example, a communication bus, a crossbar switch, or a network.

[0162] The computing device 1200 also includes a main memory 1208 such as random access memory (RAM) and an auxiliary memory 1210. The auxiliary memory 1210 may include, for example, a hard disk drive 1212 and / or a removable storage drive 1214, which may include a magnetic tape drive, an optical disc drive, etc. The removable storage drive 1214 reads from and / or writes to the removable storage unit 1218 in a well-known manner. The removable storage unit 1218 may include magnetic tapes, optical discs, etc., read from and written to by the removable storage drive 1214. As will be understood by those skilled in the art, the removable storage unit 1218 includes a computer-readable storage medium storing computer-executable program code instructions and / or data.

[0163] In an alternative embodiment, auxiliary memory 1210 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into computing device 1200. Such means may include, for example, removable memory cell 1222 and interface 1220. Examples of removable memory cell 1222 and interface 1220 include removable memory chips (e.g., EPROM or PROM) and associated sockets, as well as other removable memory cells 1222 and interfaces 1220 that allow software and data to be transferred from removable memory cells 1222 to computer system 1200.

[0164] The computing device 1200 also includes at least one communication interface 1224. The communication interface 1224 allows software and data to be transferred between the computing device 1200 and external devices via a communication path 1226. In various embodiments, the communication interface 1224 allows data to be transferred between the computing device 1200 and a data communication network (e.g., a public or private data communication network). The communication interface 1224 can be used to exchange data between different computing devices 1200 that form part of an interconnected computer network. Examples of the communication interface 1224 may include a modem, a network interface (e.g., an Ethernet card), a communication port, an antenna with associated circuitry, and so on. The communication interface 1224 can be wired or wireless. The software and data transmitted via the communication interface 1224 are in the form of signals, which can be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 1224. These signals are provided to the communication interface via the communication path 1226.

[0165] Optionally, the computing device 1200 also includes a display interface 1202 and an audio interface 1232, the display interface 1202 performing operations for rendering an image to an associated display 1230, and the audio interface 1232 performing operations for playing audio content via an associated speaker(s) 1234.

[0166] As used herein, the term "computer program product" may refer in part to removable storage unit 1218, removable storage unit 1222, hard disk installed in hard disk drive 1212, or carrier wave that carries software to communication interface 1224 via communication path 1226 (wireless link or cable). Computer-readable storage medium means any non-transitory tangible storage medium that provides recorded instructions and / or data to computing device 1200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-rays. TMDisks (Blu-ray discs), hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical discs, or computer-readable cards, such as PCMCIA cards, etc., whether these devices are internal or external to the computing device 1200. Examples of temporary or intangible computer-readable transmission media that may also participate in providing software, applications, instructions, and / or data to the computing device 1200 include radio or infrared transmission channels and network connections to another computer or networked device, as well as the Internet or intranets including email transmissions and information recorded on websites, etc.

[0167] The computer program (also referred to as computer program code) is stored in main memory 1208 and / or auxiliary memory 1210. The computer program may also be received via communication interface 1224. When executed, such a computer program enables computing device 1200 to perform one or more features of the embodiments discussed herein. In various embodiments, when executed, the computer program enables processor 1204 to perform the features of the embodiments described above. Therefore, such a computer program represents a controller of computer system 1200.

[0168] The software may be stored in a computer program product and loaded into a computing device 1200 using a removable storage drive 1214, a hard disk drive 1212, or an interface 1220. Alternatively, the computer program product may be downloaded to the computer system 1200 via communication path 1226. When executed by the processor 1204, the software causes the computing device 1200 to perform the functions of the embodiments described herein.

[0169] Understandable. Figure 12 The embodiments described are presented by way of example only. Therefore, in some embodiments, one or more features of the computing device 1200 may be omitted. Furthermore, in some embodiments, one or more features of the computing device 1200 may be combined together. Additionally, in some embodiments, one or more features of the computing device 1200 may be divided into one or more components.

[0170] Figure 13 An embodiment is shown, wherein the object comprises a DNA nanometer. As illustrated... Figure 13 The image depicts: a portion of an LR image 1302 showing the entire field of view of multiple DNA nanometers; a portion of an SR image 1304 reconstructed from the LR image 1302 using the SFSRM image processing method of this application; an LR image 1306 showing a magnified view of a single DNA nanometer among the multiple DNA nanometers; a ground reality (GT) image 1308 of the single DNA nanometer; and an SR image 1310 reconstructed from the LR image 1306 using the SFSRM image processing method of this application.

[0171] In this embodiment, LR image 1306 is a WF fluorescence microscopy image of the entire field of view of multiple DNA nanometers obtained from a Zeiss Elyra7 microscope in HILO mode. GT image 1308 is a STORM image reconstructed from 20,000 frames of a single-molecule image of the individual DNA nanometer.

[0172] Figures 1312, 1314, and 1316 depict intensity distribution curves along the dashed lines shown in images 1306, 1308, and 1310, respectively. The measured FWHM PSF size in WF image 1306 is approximately 300 nm, while the measurement distance between two points in STORM image 1308 and SR image 1310 is 30 nm. The scale bar used is 1 μm across the entire field of view and 200 nm in the magnified view.

[0173] Figure 14 Displayed are: an LR image 1402 of microtubules in fixed cells, a ground truth (GT) image 1404 of the microtubules, and an SR image 1406 of the microtubules reconstructed from the LR image 1402 using the SFSRM image processing method of this application.

[0174] In this embodiment, LR image 1402 is a WF fluorescence microscopy image. The fixed cells are Beas2B cells immunostained with Alexa Fluor 647. GT image 1404 is a STORM image reconstructed from 20,000 frames of single-molecule images of the fixed cells.

[0175] Figure 15 One embodiment is shown, wherein the objects include mitochondria labeled with the mitochondrial membrane protein Tomm20, endoplasmic reticulum (ER) labeled with the ER membrane protein Sec61β, EGFR protein after EGF internalization, clathrin-coated pits after EGF internalization, and enlarged nuclear pore complex protein Nup133.

[0176] As shown in the figure Figure 15 Line 1502 depicts LR images of mitochondria labeled with mitochondrial membrane protein Tomm20, endoplasmic reticulum (ER) labeled with ER membrane protein Sec61β, EGFR protein after EGF endocytosis, clathrin-coated pits after EGF endocytosis, and enlarged nuclear pore complex protein Nup133 (the specimen was magnified 2.5 times with a magnifying microscope after immunostaining).

[0177] In this embodiment, the LR image in line 1502 is a WF fluorescence microscope image. This image was acquired using HILO mode from a Zeiss Elyra 7 microscope.

[0178] Figure 15 Line 1504 depicts a GT image of mitochondria, endoplasmic reticulum (ER), EGFR protein, clathrin-coated pits, and enlarged nuclear pores. Line 1506 depicts an SR image of mitochondria, endoplasmic reticulum (ER), EGFR protein, clathrin-coated pits, and enlarged nuclear pores reconstructed from LR images using the SFSRM image processing method of this application. GT images are STORM images reconstructed from 20,000 frames of single-molecule images of mitochondria, endoplasmic reticulum (ER), EGFR protein, clathrin-coated pits, and enlarged nuclear pores.

[0179] The techniques described in this specification produce one or more technical effects. As described above, embodiments of this application provide an image processing method for reconstructing a super-resolution image with high detail accuracy from a single frame.

[0180] More specifically, the image processing method of this application provides a single-frame super-resolution (SR) microscopy (SFSRM) method, which reconstructs a single-molecule resolution SR image from a single fluorescence microscopy image without requiring multiple frames with single-molecule fluorescence events. In this way, this application advantageously improves the speed of super-resolution microscopy and avoids phototoxicity caused by the high illumination required to generate single-molecule excitations, thereby pushing the limits of fluorescence microscopy to unprecedented spatiotemporal resolution with a limited photon budget, and avoiding possible trade-offs between spatial resolution, imaging speed, and light dose.

[0181] Furthermore, this image processing method (e.g., the SFSRM method) can be advantageously adapted to various microscopy imaging systems, such as wide-field fluorescence microscopy, echo-plane imaging (EPI), total internal reflection fluorescence (TIRF) microscopy, confocal microscopy, and light-sheet microscopy, enabling laboratories lacking advanced optical systems to perform super-resolution imaging. The SFSRM method holds great potential in studying subcellular processes where time-dynamics need to be interpreted within the context of ultrastructural information, which in turn opens the door to new discoveries in live-cell imaging involving organelle dynamics and interactions, viral infection, cellular uptake, and intracellular delivery of nanoparticles.

[0182] Furthermore, this SFSRM method can advantageously reconcile the conflict between high resolution and high throughput in any existing microscope, making it suitable for large-scale, high-workload super-resolution imaging, such as whole-genome imaging, imaging-based cellular heterogeneity analysis, phenotypic analysis, and drug screening.

[0183] Those skilled in the art will understand that various changes and / or modifications can be made to the invention as illustrated in the specific embodiments without departing from the spirit or scope of the invention as broadly described. Therefore, the embodiments are to be considered illustrative rather than restrictive in all respects.

Claims

1. A computer-implemented image processing method, the method comprising: receiving a low resolution image of an object, wherein the low resolution image is a fluorescence microscope image of the object; generating an edge map of the low resolution image by an edge extractor, wherein the generating of the edge map comprises extracting, by the edge extractor, the edge map at a sub-pixel level based on radial symmetry of a fluorophore in the fluorescence microscope image, an edge intensity at each sub-pixel is defined by a degree to which surrounding intensity gradients converge to the sub-pixel, and the edge intensity is weighted by a pixel intensity of the sub-pixel; and inputting the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object, wherein the neural network is trained using a multi-component loss function.

2. The method of claim 1, further comprising: inputting the reconstructed super resolution image of the object and a ground truth image of the object to the multi-component loss function to quantify a difference between the reconstructed super resolution image and the ground truth image, wherein the multi-component loss function comprises one or more of a pixel-wise loss function, a perceptual loss function, an adversarial loss function, and / or a frequency loss function.

3. The method of claim 2, further comprising: inputting the quantified difference between the reconstructed super resolution image and the ground truth image to the neural network for subsequent training to optimize the neural network.

4. The method of claim 2, wherein, the pixel-wise loss function comprises a multi-scale structural similarity and L1 norm loss function.

5. The method of claim 1, wherein, the fluorescence microscope image is a wide-field fluorescence microscope image, a confocal microscope image, a total internal reflection fluorescence microscope image, or a light-sheet microscope image.

6. The method of claim 1, wherein, the object comprises intracellular structures of one or more cells.

7. The method of claim 6, wherein, the intracellular structures comprise one or more microtubules and / or organelles of the one or more cells.

8. The method of claim 6, wherein, the one or more cells comprise live cells.

9. An apparatus for image processing, the apparatus comprising: at least one processor; and a memory including computer program code for execution by the at least one processor, the computer program code instructing the at least one processor to: receive a low resolution image of an object, wherein the low resolution image is a fluorescence microscope image of the object; generate an edge map of the low resolution image by an edge extractor, wherein in generating the edge map, the edge map is extracted by the edge extractor at a sub-pixel level based on radial symmetry of a fluorophore in the fluorescence microscope image, an edge intensity at each sub-pixel is defined by a degree to which surrounding intensity gradients converge to the sub-pixel, and the edge intensity is weighted by a pixel intensity of the sub-pixel; and input the edge map and the low resolution image to a neural network to reconstruct a super resolution image of the object, wherein the neural network is trained using a multi-component loss function.

10. The apparatus of claim 9, wherein, the computer program code further instructs the at least one processor to: inputting the reconstructed super-resolution image of the object and the ground truth image of the object to the multi-component loss function to quantify a difference between the reconstructed super-resolution image and the ground truth image, wherein the multi-component loss function comprises one or more of a pixel-wise loss function, a perceptual loss function, an adversarial loss function, and / or a frequency loss function.

11. The apparatus of claim 10, wherein, The computer program code further instructs the at least one processor to: input the quantified difference between the reconstructed super-resolution image and the ground truth image to the neural network for subsequent training to optimize the neural network.

12. The apparatus of claim 10, wherein, The pixel-wise loss function comprises a multi-scale structural similarity and L1 norm loss function.

13. The apparatus of claim 9, wherein, The fluorescence microscope image is a wide-field fluorescence microscope image, a confocal microscope image, a total internal reflection fluorescence microscope image, or a light-sheet microscope image.

14. The apparatus of claim 9, wherein, The object comprises one or more intracellular structures of one or more cells.

15. The apparatus of claim 14, wherein, The intracellular structure comprises one or more microtubules and / or organelles of the one or more cells.

16. The apparatus of claim 14, wherein, The one or more cells comprise living cells.

17. A microscope imaging system, comprising: a fluorescence microscope for producing a low-resolution image of an object, and an apparatus for microscope image processing of a low-resolution image according to any one of claims 9 to 16, wherein the apparatus is coupled to the fluorescence microscope.

18. A non-transitory computer-readable storage medium having encoded thereon instructions that, when executed by a processor, cause the processor to perform one or more steps in the method for image processing according to any one of claims 1-8.

Images