System and method for image processing

Abstract

To provide systems and methods for image processing.SOLUTION: A system may: obtain a first image and a second image associated with a same object; determine a plurality of first blocks of the first image and a plurality of second blocks of the second image, the plurality of second blocks and the plurality of first blocks being in one-to-one correspondence; determine a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks; and / or generate a first target map associated with the first image and the second image based on the plurality of first characteristic values.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more particularly, to a system and method for determining errors or artifacts that occur during image reconstruction.

Background Art

[0002] Original images generated or collected by an image acquisition device (e.g., a microscope, a telescope, a camera, a webcam) are typically reconstructed using reconstruction techniques (e.g., physics-based reconstruction, learning-based reconstruction). However, the reconstruction process may introduce image errors or artifacts, which may cause misinterpretation of information in the image, especially when the image quality is relatively low. Therefore, it is desirable to provide a system and method that can efficiently perform image processing for more accurate and effective determination of errors / artifacts.

Summary of the Invention

Means for Solving the Problems

[0003] In one aspect of the present disclosure, a method for image processing is provided. The method may include steps of acquiring a first image and a second image associated with the same object, determining a plurality of first blocks of the first image and a plurality of second blocks of the second image that correspond one-to-one, determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks, and / or generating a first target map associated with the first image and the second image based on the plurality of first characteristic values.

[0004] In another aspect of the present disclosure, a system for image processing is provided. The system may include at least one storage device that stores executable instructions, and at least one processor communicable with the at least one storage device. When executing the set of instructions, the at least one processor may cause the system to perform operations including: obtaining a first image and a second image associated with the same object; determining a plurality of first blocks of the first image and a plurality of second blocks of the second image that correspond one-to-one; determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks; and / or generating a first target map associated with the first image and the second image based on the plurality of first characteristic values.

[0005] In another aspect of the present disclosure, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include at least one set of instructions for image processing, and when the at least one set of instructions is executed by one or more processors of a computing device, the computing device may be caused to perform a method for image processing.

[0006] In the following description, some of the other features are described, and some of these features may become apparent to those skilled in the art by the following considerations and the accompanying drawings, or may be understood by the manufacture or operation of the examples. The features of the present disclosure may be realized and achieved by practicing or using various aspects of the methodologies, apparatuses, and combinations described in the detailed examples below.

[0007] Hereinafter, the present disclosure will be described in more detail by way of exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These embodiments are not limiting, and the same numbers in these embodiments indicate the same configurations.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 11C

Figure 12A

Figure 12B

Figure 12C

Figure 12

Figure 12H

Figure 13

Figure 14

Figure 14C

Figure 14D

Figure 15

Figure 16

Figure 17A

Figure 17B

Figure 17C

Figure 17D

Figure 17E

Figure 17F

Figure 17G

Figure 17H

Figure 17I

Figure 17J

Figure 17K

Figure 17L

Figure 17M

Figure 17N

Figure 17O

Figure 17P

Figure 17Q

Figure 17R

Figure 17S

Figure 17T

Figure 17U

Figure 17V

Figure 17W

Figure 17X

Figure 17Y

Figure 18A

Figure 18B

Figure 18C

Figure 18D

Figure 18E

Figure 18F

Figure 18G

Figure 18H

Figure 18I

Figure 1�J

Figure 18K

Figure 18L

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 19E

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 20E

Figure 20F

Figure 20G

Figure 20H

Figure 20I

Figure 20J

Figure 20K

Figure 20L

Figure 20M

Figure 20N

Figure 21A

Figure 21B

Figure 21C

Figure 21D

Figure 21E

Figure 21F

Figure 21G

Figure 21H

Figure 21I

Figure 21J

Figure 22A

Figure 22B

Figure 22C

Figure 23A

Figure 23B

Figure 23C

Figure 23D

Figure 23E

Figure 23F

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 24E

Figure 24F

Figure 25A

Figure 25B

Figure 25C

Figure 25D

Figure 25E

Figure 25F

Figure 25G

Figure 25H

Figure 25I

Figure 26

Figure 27A

Figure 27B

Figure 27C

Figure 27D

Figure 27E

Figure 27F

Figure 27G

Figure 27H

Figure 27I

Figure 27J

Figure 27K

Figure 27L

Figure 28A

Figure 28B

Figure 28C

Figure 28D

Figure 28E

Figure 28F

Figure 28G

Figure 28H

Figure 28I

Figure 28J

Figure 28K

Figure 28L

Figure 28M

Figure 28N

Figure 28O

Figure 28P

Figure 29A

Figure 29B

Figure 29C

Figure 29D

Figure 29E

Figure 32A

Figure 32B

Figure 32C

Figure 32D

Figure 32E

Figure 32F

Figure 32G

Figure 32H

Figure 32I

Figure 32J

Figure 33A

Figure 33B

Figure 33C

Figure 33D

Figure 33E

Figure 33F

Figure 33G

Figure 33H

Figure 33I

Figure 34A

Figure 34B

Figure 34C

Figure 34D

Figure 34E

Figure 34F

Figure 35A

Figure 35B

Figure 35C

Figure 35D

Figure 35E

Figure 36A

Figure 36B

Figure 36C

Figure 36D

Figure 36E

Figure 36F

Figure 37A

Figure 37B

Figure 37C

Figure 37D

Figure 37E

Figure 37F

Figure 37G

Figure 37H

Figure 37I

Figure 38A

Figure 38B

Figure 38C

Figure 38D

Figure 38E

Figure 38F

Figure 39A

Figure 39B

Figure 39C

Figure 39D

Figure 39E

Figure 39F

Figure 40A

Figure 40B

Figure 40C

Figure 41

Figure 42

Figure 43

[0009] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without such details. In other instances, well-known methods, procedures, systems, components, and / or circuits, etc. are described at a relatively high level in order to avoid unnecessarily obscuring aspects of the present disclosure. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Accordingly, the present disclosure is not limited to the embodiments shown, but should be accorded the widest scope consistent with the claims.

[0010] Note that the terms used in this specification are for the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used in this specification, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. Further, as used in this specification, the terms "comprise", "comprises", and / or "comprising", "include", "includes", and / or "including" identify the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Note that the terms "object" and "subject" are used synonymously based on the target for executing the imaging procedure of the present disclosure.

[0011] Note that the terms "system", "engine", "unit", "module", and / or "block" used in this specification are one way to distinguish different components, elements, parts, sections, or assemblies at different levels in ascending order. However, the terms may be replaced with other expressions if they achieve the same purpose.

[0012] As used herein, the words "module," "unit," or "block" generally refer to logic implemented in hardware or firmware, or a collection of software instructions. The modules, units, or blocks described herein may be implemented as software and / or hardware, and may be stored on any type of non-transitory computer-readable medium or other storage device. In some embodiments, the software modules / units / blocks may be compiled and linked into an executable program. The software modules may be called from other modules / units / blocks or itself, and / or may be called in response to detected events or interrupts. The software modules / units / blocks executed on an arithmetic device (e.g., the processor 210 shown in FIG. 2) may be provided on a computer-readable medium such as a compact disk, digital video disk, flash drive, magnetic disk, or other tangible medium, or may be provided as a digital download (originally stored as a file in a compressed or installable format that requires installation, decompression, or decoding before execution). Such software code may be stored, in whole or in part, in the storage device of the arithmetic device for execution by the arithmetic device. The software instructions may be incorporated into firmware such as EPROM. Further, the hardware modules / units / blocks may be included in connected logic components such as gates and flip-flops, and / or may be included in programmable units such as programmable gate arrays or processors. The modules / units / blocks or arithmetic device functions described herein may be implemented as software modules / units / blocks, but may also be represented in hardware or firmware. The modules / units / blocks described herein generally refer to logical modules / units / blocks that may be combined with other modules / units / blocks and may be divided into sub-modules / sub-units / sub-blocks regardless of physical configuration or storage. The description may be applied to a system, an engine, or a part thereof.

[0013] When a unit, engine, module, or block is described as being "on," "connected to," or "coupled to" another unit, engine, module, or block, unless the context specifically indicates otherwise, it may be directly on the other unit, engine, module, or block, may be directly connected or coupled to the other unit, engine, module, or block, may communicate directly with the other unit, engine, module, or block, or intervening units, engines, modules, or blocks may be present. As used herein, "and / or" includes any and all combinations of one or more of the associated listed items.

[0014] Note that when describing operations performed on an image, the term "image" as used herein may be a dataset (e.g., a matrix) that includes the values of the pixels (pixel values) within the image. As used herein, for convenience of explanation, the representation of an object (e.g., a person, an organ, a cell, or a part thereof) within an image may be referred to as an object. For example, for convenience of explanation, the representation of a cell or an organelle (e.g., mitochondria, endoplasmic reticulum, centrosome, Golgi apparatus, etc.) within an image may be referred to as a cell or an organelle. As used herein, for convenience of explanation, an operation on the representation of an object within an image may be referred to as an operation on the object. For example, for convenience of explanation, the segmentation of a portion of an image that includes the representation of a cell or an organelle from the image may be referred to as the segmentation of the cell or the organelle.

[0015] It should be understood that the term "resolution" as used herein is a measure indicating the sharpness of an image. The terms "super-resolution," "super-resolved," or "SR" as used herein mean an improved (or increased) resolution that can be obtained, for example, by a process of synthesizing a series of low-resolution images to generate a higher-resolution image or sequence.

[0016] These and other features and characteristics of the present disclosure, methods of operation, functions of related elements and components of the structure, and economies of manufacture will become more apparent upon consideration of the following description with reference to the accompanying drawings, which form a part of the present disclosure. However, it should be clearly understood that the drawings are for purposes of illustration and description only and are not intended to limit the scope of the present disclosure. It should be understood that the drawings are not to scale.

[0017] The flowcharts used in the present disclosure illustrate the operations performed by a system according to some embodiments of the present disclosure. It should be clearly understood that the operations of the flowchart may be performed in any order. Conversely, the operations may be performed in reverse order or simultaneously. Further, one or more other operations may be added to the flowchart. One or more operations may be deleted from the flowchart.

[0018] With the development of technology, a series of super-resolution approaches have the potential to break through the diffraction limit (e.g., 200 - 300 nm) by using computational processes. Such computational operations may generally be performed on images generated based on several categories of techniques, for example, including: (i) molecular localization such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM); (ii) molecular intensity fluctuations such as super-resolution fluctuation imaging (SOFI) and super-resolution radial fluctuation (SRRF); (iii) high-frequency information mixing such as structured illumination microscopy (SIM); and (iv) deconvolution, etc., or other techniques that rely on high computational reconstruction to extend the spatial resolution of the image. In addition to the original physical configuration, multiple learning-based algorithms may be actively developed to reduce the photon / hardware budget in image generation, reconstruction, and / or processing. For example, artificial neural network-accelerated PALM (ANNA-PALM) may be applied to reduce the total number of frames (or images, or image frames) without trading off the spatial resolution of PALM. As another example, a conditional adversarial generative network (cGAN) can be used to directly convert total internal reflection fluorescence (TIRF) microscopy images to match the results obtained by TIRF-SIM. As a further example, the content-aware image restoration (CARE) network framework can perform noise removal and deconvolution applications more effectively.

[0019] Such computational modalities are considered to be promising imaging techniques. In some embodiments, image errors or artifacts may occur during image reconstruction, and relatively low image quality may lead to misinterpretation of biological information. In some embodiments, the achieved image quality and resolution may be related to one or more comprehensive factors including, for example, the photophysics of the phosphor used for imaging, the chemical environment of the sample being imaged, the optical setup conditions, the analytical approach for generating the image (e.g., super-resolution image), the network learning procedure, etc., or combinations thereof. For the quantitative mapping and minimization of image defects, most existing quality assessments may rely on subjective comparison of the reconstructed image against a reference standard structure, benchmarking of the reconstructed image data against other high-resolution imaging techniques such as electron microscopy, or corresponding specific design analysis algorithms. As a mere example, in the case of STORM / PLAM, considering the input of a precise imaging model and noise statistics, the reliability of individual localizations within the STORM / PLAM dataset may be accessible based on the Wasserstein-induced flux calculation process even without the knowledge of the ground truth. In some embodiments, SIM checks that require complex pipelines and specialized usage may be used to identify the causes of errors and artifacts in the SIM system. In some embodiments, modifications may be required to the existing learning procedures of learning-based approaches (e.g., Bayesian neural network (BNN) framework, learning of the distribution over weights, etc.) to obtain the uncertainty of the learning-based approach. In some embodiments, such modifications may be complex to apply and may have high computational costs compared to standard techniques (e.g., general learning-based approaches).

[0020] In some embodiments, to generally and accurately evaluate local errors / artifacts, a resolution scale error map (RSM) (e.g., a reference map) may be used as an evaluation tool for the super-resolution image. In some embodiments, a wide-field reference image may be used to generate the RSM. In some embodiments, the RSM may be generated based on one or more hypotheses including, for example: (i) the wide-field reference image may have a relatively high SNR, (ii) the Gaussian kernel convolution process performed in generating the RSM map may be spatially invariant, and (iii) the illumination used in generating the raw data of the super-resolution image may be uniform. According to these hypotheses, in some embodiments, the RSM may have a significant potential to induce false negatives in the estimated error map. On the other hand, when converting the super-resolution image to its low-resolution region, the detectable scale of the RSM error may be limited by Abbe diffraction. In some embodiments, a reference free process (e.g., pixel-level analysis of error positions (PANEL)) may be used to detect errors / artifacts at the super-resolution scale. In some embodiments, the PANEL technique may be built on rolling Fourier ring correlation (rFRC) that uses one or more individually reconstructed super-resolution frames, and / or may further cooperate with a modified RSM. In some embodiments, one or more strategies for implementing single-frame rFRC operations may be involved in the PANEL framework of physical-based and / or learning-based applications.

[0021] In some embodiments, in the PANEL framework, the RSM and the rFRC map are integrated together, and the PANEL framework may be developed as a multifunctional and model-free metric. Thus, the PANEL framework may have the ability to infer errors contained in multi-dimensional signals up to the super-resolution scale and mitigate possible false-negative defects. The PANEL framework can be used in various scenarios, such as physical super-resolution imaging approaches (2D-STORM or 3D-STORM, TIRF-SIM or 3D-SIM, SRRF, Richardson-Lucy deconvolution, etc.), various learning-based image restoration techniques (ANNA-PALM, TIRF~TIRF-SIM, CARE, etc.), and / or non-super-resolution applications such as noise removal tasks. In some embodiments, one or more tiny quantitative error maps generated using the PANEL framework can be utilized to advance the design of multiple computational processes (e.g., STORM image fusion, adaptive low-pass filtering, automatic deconvolution iterative determination, etc.). In some embodiments, the PANEL framework is used as a tool for robust and detailed evaluation of the image quality of physical-based or learning-based super-resolution imaging approaches and can facilitate the optimization of super-resolution imaging and network learning procedures.

[0022] Extensive super-resolution microscopy methods may rely on corresponding arithmetic operations that can form reconstruction errors at one or more scales. The reconstruction error of an image may cause misinterpretation of biological information. The quantitative mapping of such errors in super-resolution reconstruction may be limited to identifying minute errors resulting from necessarily using the diffraction-limited image as a reference. According to some embodiments of the present disclosure, the rolling Fourier ring correlation (rFRC) operation may be used to trace pixels within a reconstructed image and provide a relatively precise model-free metric that enables pixel-level analysis (PANEL) of error positions. In some embodiments, a target image or map (e.g., an rFRC map) can be generated according to the pixel-level analysis (PANEL) of error positions. The target image or map can illustrate the defects of the reconstructed image up to the super-resolution scale. In some embodiments, the reliability assessment of the PANEL approach may be performed on an image reconstructed based on one or more arithmetic modalities (e.g., physical super-resolution microscopy, deep learning super-resolution microscopy, etc.).

[0023] The present disclosure relates to a system and method for image processing. The system and method may acquire a first image and a second image associated with the same object. The system and method may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image. The plurality of first blocks and the plurality of second blocks may correspond one-to-one. The system and method may determine a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks. The system and method may generate a target map (e.g., a first target map) associated with the first image and the second image based on the plurality of first characteristic values. The first target map may quantitatively map errors in the first image and / or the second image generated by an image acquisition device without using a reference (e.g., ground truth) and / or any prior information of the image acquisition device. According to some embodiments of the present disclosure, the first target map may be used as a tool for quantitative assessment of image quality.

[0024] Furthermore, although the systems and methods disclosed in the present disclosure are mainly described with respect to the generation of rFRC maps that reflect the errors of images reconstructed by physical super-resolution microscopy and / or deep learning super-resolution microscopy, it should be understood that the description is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. The systems and methods of the present disclosure may be applied to any other type of system including an image acquisition device for image processing. For example, the systems and methods of the present disclosure may be applied to microscopes, telescopes, cameras (e.g., surveillance cameras, mobile phones with cameras, webcams), unmanned aerial vehicles, medical imaging devices, etc., or any combination thereof.

[0025] As a mere example, the method disclosed in the present disclosure may be used to generate an rFRC map based on a dual frame (e.g., sequential first image and second image) or a single frame (e.g., initial image) generated by physical super-resolution imaging and / or deep learning super-resolution reconstruction.

[0026] It should be understood that the application scenarios of the systems and methods disclosed in this specification are merely some exemplary embodiments provided for illustrative purposes and are not intended to limit the scope of the present disclosure. Those skilled in the art can make multiple modifications and variations based on the teachings of the present disclosure.

[0027] FIG. 1 is a schematic diagram showing an exemplary application scenario of an image processing system according to some embodiments of the present disclosure. As shown in FIG. 1, the image processing system 100 may include an image acquisition device 110, a network 120, one or more terminals 130, a processing device 140, and a storage device 150.

[0028] The components of the image processing system 100 may be connected in one or more ways out of various ways. As a mere example, the image acquisition device 110 may be connected to the processing device 140 via the network 120. As another example, the image acquisition device 110 may be directly connected to the processing device 140 as shown by the dotted two-way arrow connecting the image acquisition device 110 and the processing device 140. As yet another example, the storage device 150 may be directly connected to the processing device 140 or via the network 120. As a further example, the terminal 130 may be directly connected to the processing device 140 (as shown by the dotted two-way arrow connecting the terminal 130 and the processing device 140) or via the network 120.

[0029] The image processing system 100 may be configured to generate a first target map (e.g., an rFRC map) based on a first image and a second image using an rFRC process (e.g., one or more operations shown in FIG. 5). The first target map may illustrate or display errors and / or artifacts of the first image and / or the second image generated by the image acquisition device 110 and / or may reflect image quality (e.g., the degree of error). The image processing system 100 may be configured to determine a reference map associated with the first image. In some embodiments, the reference map may be used to identify errors not reported by the first target map. The image processing system 100 may be configured to determine a fourth target map by fusing the first target map and the reference map. The fourth target map may be configured to comprehensively illustrate or display errors and / or artifacts of the first image and / or the second image. In some embodiments, the errors and / or artifacts displayed by the target map (e.g., the first target map, the fourth target map) may be related to errors generated in the reconstruction of the first image and / or the second image or may include errors generated in the reconstruction of the first image and / or the second image.

[0030] The image acquisition device 110 may be configured to acquire one or more images (e.g., a first image, a second image, an initial image, etc.) associated with an object within its detection area. The object may include one or more biological objects or non-biological objects. In some embodiments, the image acquisition device 110 may be an optical imaging device, a radiation-based imaging device (e.g., a computed tomography device), a nuclide-based imaging device (e.g., a positron emission tomography device, a nuclear magnetic resonance imaging device), etc. Exemplary optical imaging devices may include a microscope 111 (e.g., a fluorescence microscope), a surveillance device 112 (e.g., a security camera), a portable terminal device 113 (e.g., a mobile phone with a camera), a scanning device 114 (e.g., a flatbed scanner, a drum scanner, etc.), a telescope, a webcam, etc., or any combination thereof. In some embodiments, the optical imaging device may include an acquisition device (e.g., a detector or a camera) that collects image data. For the sake of explanation, in the present disclosure, the microscope 111 may be used as an example to illustrate the exemplary functions of the image acquisition device 110. Exemplary microscopes may include a structured illumination microscope (SIM) (e.g., two-dimensional SIM (2D-SIM), three-dimensional SIM (3D-SIM), total internal reflection SIM (TIRF-SIM), spinning disk confocal-based SIM (SD-SIM), etc.), a photoactivated localization microscope (PALM), a stimulated emission depletion microscope (STED), a stochastic optical reconstruction microscope (STORM), etc.). The SIM may include a detector such as an EMCCD camera, an sCMOS camera, etc. The object detected by the SIM may include one or more objects such as a biological structure, a biological subject, a protein, a cell, a microorganism, etc., or any combination thereof. Exemplary cells may include INS-1 cells, COS-7 cells, Hela cells, liver sinusoidal endothelial cells (LSEC), human umbilical vein endothelial cells (HUVEC), HEK293 cells, etc., or any combination thereof. In some embodiments, one or more objects may be fluorescent or fluorescently labeled. An object that is fluorescent or fluorescently labeled may be excited to emit fluorescence for imaging.

[0031] Network 120 may include any suitable network that can facilitate the exchange of information and / or data by the image processing system 100. In some embodiments, one or more of the components of the image processing system 100 (e.g., the image acquisition device 110, the terminal 130, the processing device 140, the storage device 150, etc.) may communicate information and / or data with each other via the network 120. For example, the processing device 140 may obtain image data (e.g., the first image, the second image) from the image acquisition device 110 via the network 120. As another example, the processing device 140 may obtain user instructions from the terminal 130 via the network 120. Network 120 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi network, etc.), a cellular network (e.g., a long term evolution (LTE) network), an image relay network, a virtual private network (“VPN”), a satellite network, a telephone network, a router, a hub, a switch, a server computer, and / or a combination of one or more of them. For example, network 120 may include a cable network, a wired network, a fiber network, a telecommunications network, a local area network, a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth (registered trademark) network, a ZigBee (registered trademark) network, a near field communication network (NFC), etc., or a combination of them. In some embodiments, network 120 may include one or more network access points.For example, network 120 may include a wired network access point such as a base station and / or a network switching point, and / or a wireless network access point. Through these access points, one or more components of image processing system 100 may access network 120 for data and / or information exchange.

[0032] In some embodiments, the user can operate image processing system 100 via terminal 130. Terminal 130 may include a mobile device 131, a tablet computer 132, a laptop computer 133, etc., or a combination thereof. In some embodiments, mobile device 131 may include a smart home device, a wearable device, a mobile device, a virtual reality device, an augmented reality device, etc. In some embodiments, the smart home device may include a smart lighting fixture, a control device for intelligent electrical equipment, a smart monitoring device, a smart TV, a smart video camera, an intercom, etc., or a combination thereof. In some embodiments, the wearable device may include a bracelet, footwear, glasses, a helmet, a watch, clothing, a backpack, a smart accessory, etc., or a combination thereof. In some embodiments, the mobile device may include a mobile phone, a personal digital assistant (PDA), a gaming device, a navigation device, a point of sale (POS) device, a laptop, a tablet computer, a desktop, etc., or a combination thereof. In some embodiments, the virtual reality device and / or the augmented reality device may include a virtual reality helmet, virtual reality glasses, virtual reality eyewear, an augmented reality helmet, augmented reality glasses, augmented reality eyewear, etc., or a combination thereof. For example, the virtual reality device and / or the augmented reality device may include Google Glass (registered trademark), Oculus Rift (registered trademark), HoloLens (registered trademark), GearVR (registered trademark), etc. In some embodiments, terminal 130 may be part of processing device 140.

[0033] The processing device 140 may process data and / or information acquired from the image acquisition device 110, the terminal 130, and / or the storage device 150. For example, the processing device 140 may process one or more images (e.g., the first image, the second image, etc.) generated by the image acquisition device 110 to generate an rFRC map (e.g., the first target map, the third target map, etc.). In some embodiments, the processing device 140 may be a server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 140 may be local or remote. For example, the processing device 140 may access information and / or data stored in the image acquisition device 110, the terminal 130, and / or the storage device 150 via the network 120. As another example, the processing device 140 may be directly connected to the image acquisition device 110, the terminal 130, and / or the storage device 150 to access the stored information and / or data. In some embodiments, the processing device 140 may be implemented on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an interconnected cloud, a multi-cloud, etc., or a combination thereof. In some embodiments, the processing device 140 may be implemented by an arithmetic device 200 having one or more components as shown in FIG. 2.

[0034] The memory device 150 may store data, instructions, and / or any other information. In some embodiments, the memory device 150 may store data obtained from the terminal 130, the image acquisition device 110, and / or the processing device 140. In some embodiments, the memory device 150 may store data and / or instructions that the processing device 140 may execute or use to execute the exemplary methods described in this disclosure. In some embodiments, the memory device 150 may include a mass storage device, a removable storage device, a volatile read / write memory, a read-only memory (ROM), etc. Exemplary mass storage devices may include magnetic disks, optical disks, solid state drives, etc. Exemplary removable storage devices may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tapes, etc. Exemplary volatile read / write memories may include random access memory (RAM). Exemplary RAM may include dynamic RAM (DRAM), double data rate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero capacitor RAM (Z-RAM), etc. Exemplary ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), and digital versatile disk ROM, etc. In some embodiments, the memory device 150 may be executed on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an interconnected cloud, a multi-cloud, etc., or a combination thereof.

[0035] In some embodiments, the storage device 150 may be connected to the network 120 and communicate with one or more other components of the image processing system 100 (e.g., the processing device 140, the terminal 130, etc.). One or more components of the image processing system 100 may access the data or instructions stored in the storage device 150 via the network 120. In some embodiments, the storage device 150 may be directly connected to and communicate with one or more other components of the image processing system 100 (e.g., the processing device 140, the terminal 130, etc.). In some embodiments, the storage device 150 may be part of the processing device 140.

[0036] FIG. 2 is a schematic diagram showing an exemplary arithmetic unit according to some embodiments of the present disclosure. The arithmetic unit 200 may be used to implement any component of the image processing system 100 described herein. For example, the processing device 140 and / or the terminal 130 may be respectively implemented on the arithmetic unit 200 via its hardware, software program, firmware, or a combination thereof. Although only one such arithmetic unit is shown, for convenience, the computer functions related to the image processing system 100 described herein may be implemented in a distributed manner on a plurality of similar platforms to distribute the processing load.

[0037] As shown in FIG. 2, the arithmetic unit 200 may include a processor 210, a storage 220, an input / output (I / O) unit 230, and a communication port 240.

[0038] Processor 210 may execute computer instructions (e.g., program code) and perform the functions of the image processing system 100 (e.g., processing device 140) according to the techniques described herein. The computer instructions may include, for example, routines, programs, objects, components, data structures, procedures, modules, and functions for performing the specific functions described herein. For example, processor 210 may process image data obtained from any component of the image processing system 100. In some embodiments, processor 210 may include one or more hardware processors such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or a combination thereof.

[0039] For the sake of simplicity, only one processor is described in the arithmetic unit 200. However, it should be noted that the arithmetic unit 200 in the present disclosure may also include a plurality of processors, whereby the operations and / or method operations executed by one processor as described in the present disclosure may also be executed jointly or individually by a plurality of processors. For example, in the present disclosure, when the processor of the arithmetic unit 200 executes both operation A and operation B, operation A and operation B may also be executed jointly or individually by two or more different processors in the arithmetic unit 200 (e.g., the first processor executes operation A, the second processor executes operation B, or the first processor and the second processor jointly execute operation A and operation B).

[0040] Storage 220 may store data / information obtained from any component of the image processing system 100. In some embodiments, storage 220 may include a mass storage device, a removable storage device, a volatile read / write memory, a read-only memory (ROM), etc., or any combination thereof. Exemplary mass storage devices may include magnetic disks, optical disks, solid state drives, etc. Removable storage devices may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tapes, etc. Volatile read / write memory may include random access memory (RAM). RAM may include dynamic RAM (DRAM), double data rate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero capacitor RAM (Z-RAM), etc. ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), and digital versatile disk ROM, etc.

[0041] In some embodiments, storage 220 may store one or more programs and / or instructions for executing the exemplary methods described in this disclosure. For example, storage 220 may store a program for the processing device 140 to process an image generated by the image acquisition device 110.

[0042] The I / O unit 230 may input and / or output signals, data, information, etc. In some embodiments, the I / O unit 230 may enable user interaction with the image processing system 100 (for example, the processing device 140). In some embodiments, the I / O unit 230 may include an input device and an output device. Examples of the input device may include a keyboard, a mouse, a touch screen, a microphone, etc., or a combination thereof. Examples of the output device may include a display device, a speaker, a printer, a projector, etc., or a combination thereof. Examples of the display device may include a liquid crystal display (LCD), a light-emitting diode (LED)-based display, a thin display, a curved screen, a television device, a cathode ray tube (CRT), a touch screen, etc., or a combination thereof.

[0043] The communication port 240 may be connected to a network to facilitate data communication. The communication port 240 may establish a connection between the processing device 140 and the image acquisition device 110, the terminal 130, and / or the storage device 150. The connection may be a wired connection, a wireless connection, any other communication connection capable of enabling data transfer and / or data reception, and / or any combination of those connections. The wired connection may include, for example, an electrical cable, an optical cable, a telephone line, etc., or a combination thereof. The wireless connection may include a Bluetooth (registered trademark) link, a Wi-Fi (registered trademark) link, a WiMAX (registered trademark) link, a WLAN link, a ZigBee (registered trademark) link, a mobile network link (for example, 3G, 4G, 5G), etc., or a combination thereof. In some embodiments, the communication port 240 may be a standardized communication port such as RS232, RS485, etc., and / or may include it. In some embodiments, the communication port 240 may be a specially designed communication port. For example, the communication port 240 may be designed according to the Digital Imaging and Communications in Medicine (DICOM) protocol.

[0044] FIG. 3 is a block diagram showing an exemplary mobile device on which the terminal 130 may be implemented according to some embodiments of the present disclosure.

[0045] As shown in FIG. 3, the mobile device 300 may include a communication unit 310, a display unit 320, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, an I / O unit 350, a memory 360, a storage unit 370, and the like. In some embodiments, any other suitable components, including but not limited to a system bus or a controller (not shown), may also be included in the mobile device 300. In some embodiments, an operating system 361 (e.g., iOS®, Android®, Windows® Phone, etc.) and one or more applications (apps) 362 may be loaded from the storage unit 370 into the memory 360 to be executed by the CPU 340. The application 362 may include a browser or any other suitable mobile app for receiving and rendering information regarding imaging, image processing, or other information from the image processing system 100 (e.g., the processing device 140). User interaction with the information stream may be realized via the I / O unit 350 and provided to the processing device 140 and / or other components of the image processing system 100 via the network 120. In some embodiments, the user may input parameters to the image processing system 100 via the mobile device 300.

[0046] To implement the various modules, units, and their functions described above, a computer hardware platform of one or more elements (e.g., the processing device 140 and / or other components of the image processing system 100 shown in FIG. 1) may be used as the hardware platform. Since these hardware elements, operating systems, and programming languages are common, those skilled in the art may be proficient in these technologies and may be able to provide the information required for imaging and evaluation according to the technologies described in the present disclosure. A computer having a user interface may be used as a personal computer (PC), or other types of workstations or terminal devices. After being appropriately programmed, a computer having a user interface may be used as a server. Those skilled in the art may also be considered proficient in such structures, programs, or general operations of this type of computing device.

[0047] FIG. 4 is a schematic diagram showing an exemplary processing device according to some embodiments of the present disclosure. As shown in FIG. 4, the processing device 140 may include an acquisition module 402, an expansion module 404, a rolling module 406, a map generation module 408, a filtering module 410, a display module 412, an image quality determination module 414, a decomposition module 416, and a neural network processing module 418.

[0048] The acquisition module 402 may be configured to acquire one or more images (e.g., a first image and / or a second image, an initial image associated with the same object).

[0049] The expansion module 404 may be configured to expand one or more images (e.g., a first image, a second image, a third image, and / or a fourth image).

[0050] The rolling module 406 may be configured to generate one or more image blocks (e.g., a plurality of first blocks of the first image, a plurality of second blocks of the second image, a plurality of third blocks of the third image, a plurality of fourth blocks of the fourth image, etc.).

[0051] The map generation module 408 is configured to determine a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks, generate a first target map associated with the first image and the second image based on the plurality of first characteristic values, determine a reference map associated with the first image or the second image, and determine a fourth target map by fusing the first target map and the reference map. In some embodiments, the map generation module 408 may determine a plurality of second characteristic values based on the plurality of third blocks and the plurality of fourth blocks, generate a second target map associated with the third image and the fourth image based on the plurality of second characteristic values, and generate a third target map based on the first target map and the second target map.

[0052] The filtering module 410 may be configured to filter one or more images (e.g., the first target map, the second target map, the third target map, the fourth target map, etc.).

[0053] The display module 412 may be configured to display one or more images (e.g., the first target map, the second target map, the third target map, the fourth target map, the reference map, etc., or any other image generated by or used in image processing). In some embodiments, the display module 412 may display one or more images using a displacement jet color map.

[0054] The image quality determination module 414 may be configured to determine the image quality of one or more images. For example, the image quality determination module 414 may determine a global image quality metric of the first image and / or the second image based on the first target map.

[0055] The decomposition module 416 may be configured to decompose one or more images. For example, the decomposition module 416 may decompose an initial image into a first image, a second image, a third image, and a fourth image.

[0056] The neural network processing module 418 may be configured to generate one or more images by processing one or more initial images using a neural network.

[0057] A more detailed description of the modules of the processing device 140 may be found elsewhere in the present disclosure (for example, FIGS. 5 to 8 and their descriptions).

[0058] Note that the above description of the modules of the processing device 140 is provided for illustrative purposes only and is not intended to limit the present disclosure. A person skilled in the art may combine the modules in various ways and connect them to other modules as subsystems without departing from the principles of the present disclosure under the teachings of the present disclosure. In some embodiments, one or more modules may be added or omitted in the processing device 140. For example, the extension module 404 and the rolling module 406 may be integrated into a single module.

[0059] FIG. 5 is a flowchart showing an exemplary process for determining a target map according to some embodiments of the present disclosure. In some embodiments, process 500 may be executed by image processing system 100. For example, process 500 may be configured as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, storage 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of arithmetic device 200, CPU 340 of portable device 300, and / or one or more modules illustrated in FIG. 4) may execute the set of instructions and thereby execute process 500. The operations of the processes shown below are intended to be exemplary. In some embodiments, process 500 may be accomplished with one or more additional operations not described and / or without one or more of the operations described. Further, the order of the operations of process 500 shown in FIG. 5 and described below is not intended to be limiting.

[0060] In some embodiments, an imaging system (e.g., image processing system 100) may observe one or more objects using an optical device mathematically expressed as a transfer function, and collect corresponding signals using one or more sensors. In some embodiments, the signals may be sampled by the sensors and / or mixed with noise. In some embodiments, artificial observations may deviate from real-world objects in a relatively high-dimensional space (e.g., as shown in the left panel of FIG. 15). In such situations, restoring the hidden real signal from the measurement data may be regarded as an ill-posed problem, and the reconstruction from these observations may deviate from real-world objects. When two individually measured signals on the same object are captured, these two signals may be related to the same object. It may be considered that the greater the distance between the reconstructions of these two signals (e.g., as shown in the right panel of FIG. 15), the more deviation the reconstruction may show from the real object. This distance (e.g., the distance between reconstruction 1 and reconstruction 2) is related to the reconstruction error (e.g., the deviation of the reconstruction from the real object). If the above intuition is followed, an important hypothesis holds that the distance between two reconstructions has a positive correlation with the magnitude of the reconstruction error.

[0061] To verify this hypothesis convincingly, the derivation may be limited to a specific general situation for fluorescence imaging by a wide-field (WF) microscope, in which case the corresponding image may be processed, for example, by the minimalist form of Wiener deconvolution.

[0062] The lemma may be explained as follows. Under ideal conditions, an object may be observed using an optical system without aberration, and such observations may be sampled at an infinite sampling rate without noise. As expressed by Equation (1), the object may be restored using the minimalist form of Wiener deconvolution.

[0063]

Equation

[0064] Here, ω represents spatial coordinates, and I, o, and h represent the illumination, object, and point spread function (PSF) of the microscope, respectively. F and F% are the operators of the Fourier transform and its inverse Fourier transform. However, considering the combined effects of sampling and noise, a realistic imaging model may be expressed as follows.

[0065]

Equation

[0066] Here, s and n represent the sampling model and the noise model, and the image 広視野 represents the final image collected by the wide-field microscope. This imaging model may be considered closer to the actual physical world state as shown in Equation (2) and may be different from that based on the Wiener model.

[0067]

Equation

[0068] The minimalist form of Wiener deconvolution with a PSF without aberration (unbiased estimation) may be used to process the following image 広視野

[0069]

Equation

[0070] The image 逆畳込 which is the corresponding result of Equation (4) may be clearly different from the real object.

[0071] ​The above-mentioned lemma may suggest that the distance between the real object and the result of reconstruction is mainly caused by the combined effect of the sampling rate and the mixed noise. According to this lemma, when variables are controlled so as to image the same real object and statistically capture independent image pairs, and when an appropriate reconstruction model is used, such a distance between the reconstruction and the object may be emphasized by the difference between the reconstructions from the image pairs.

[0072] The explanation of the theorem may be described as follows. If there are two wide-field images measured individually for the same sample (or object), these two images may show the same object. The greater the distance remaining between the corresponding two reconstruction results, the more it may seem that the reconstruction results deviate from the real object (see Fig. 15).

[0073] This distance (e.g., Wiener result 1 vs. Wiener result 2) may be related to the reconstruction error (e.g., the difference between Wiener result 1 and Wiener result 2 and the real object). In some embodiments, the distance may be expressed as follows.

[0074]

Equation

[0075] Here, U represents the union operation, and ||Wiener result 1, Wiener result 2|| p represents the distance between Wiener result 1 and Wiener result 2. Wiener result 1 and Wiener result 2 may be related to the reconstruction results generated based on the Wiener model respectively. n1, n2, s1, and s2 represent two separate noise models (n1 and n2) and sampling models (s1 and s2) respectively. In some embodiments, Equation (5) may be changed by taking the Fourier transform of both sides.

[0076]

Equation

[0077] In some embodiments, the following two direct combinations of the distances of "Wiener result 1 and Wiener result 2" from the real object (i.e., combinations using addition of Euclidean distances or multiplication of cross-correlation distances instead of union) may not be as accurate as the union operation of Equation (6). In some embodiments, the actual model of the union operation in the real world may be very complex, and therefore, the combination of the two distances (reconstruction and real object) may be difficult to express explicitly. In some embodiments, the following two simple examples may be utilized to explain the relationship between the distances of the two individual reconstructions and the reconstruction error.

[0078] The first example relates to the Euclidean distance. The Euclidean distance may be used to define the distance between two reconstructions. Assuming that the image is captured at an infinite sampling rate and is corrupted only by additive noise denoted as n, the addition operation may be utilized to combine the distances.

[0079]

Number

[0080] Here, ||Wiener result 1, Wiener result 2||2 represents the Euclidean distance between Wiener result 1 and Wiener result 2. n1 and n2 represent the individual additive noises in two observations. In some embodiments, Equation (7) may be simplified to obtain the following equation.

[0081]

Number

[0082] The Fourier transform of the microscope and the PSF may be removed to simplify Equation (8) as follows.

[0083]

Number

[0084] Equation (9) shows a close relationship between the Euclidean distances of Wiener result 1 and Wiener result 2, the Euclidean distance of Wiener result 1 and the real object, and the Euclidean distance of Wiener result 2 and the real object.

[0085] The second example relates to the cross-correlation distance. Alternatively, when using cross-correlation as the distance, it is necessary to change the addition operation in the Euclidean distance calculation to multiplication.

[0086]

Number

[0087] Here,

Number

[0088]

Number

[0089] Here, O, n1, and n2 represent I(ω)×o(ω), n1 / h(ω,0), and n2 / h(ω,0), respectively. In some embodiments, Equation (11) may be further simplified by removing the Fourier transform.

[0090]

Number

[0091] Similar to the Euclidean distance, Equation (12) shows a close relationship between the cross-correlation (e.g., the cross-correlation between Wiener result 1 and Wiener result 2) and the multiplication of cross-correlations (e.g., the cross-correlation between Wiener result 1 and the real object, the cross-correlation between Wiener result 2 and the real object, etc.).

[0092] As described above, Euclidean distance and / or cross-correlation may be used to estimate the distance between the real object and the reconstruction. In some embodiments, one of Euclidean distance and / or cross-correlation may be selected to quantitatively map such distance in multiple dimensions (2D image or 3D volume). In some embodiments, a distance map may be generated to describe the distance. Conventional spatial processing algorithms such as root mean square error in RSM may generally be sensitive to intensity and micromovements during measurement. Additionally, many of the processing algorithms may be used to generate "absolute differences", which may lead to relatively high false negative results that are misleading and may overwhelm true negatives in the distance map.

[0093] Based on the above considerations, in some embodiments, the algorithm may be introduced to measure the distance between two signals in the Fourier domain, i.e., the Fourier ring correlation (FRC) or the spectral signal-to-noise ratio (SSNR) that may describe the highest acceptable frequency component between the two signals. In some embodiments, the FRC metric may be utilized as an effective resolution criterion for super-resolution fluorescence microscopy and electron microscopy. In some embodiments, the FRC metric may be used to quantify the similarity (or distance) between two signals. The FRC metric may be insensitive to intensity changes and small movements, may quantify the "relative error" or "error based on significance" (e.g., the most reliable frequency component), and may significantly reduce false negatives induced by the quantification of the "absolute distance". Also, during such two independent observations, it may be assumed that the aberration of the system (i.e., the system used to generate the signal of the object) does not change. The error due to aberration (i.e., the biased estimation) may be difficult to estimate using a simple spatial or frequency algorithm. Since the FRC defines the most reliable frequency component, these errors may be visible. In some embodiments, the FRC may be used to quantify the distance between two signals. A more detailed description of the FRC may be found elsewhere in this disclosure (e.g., operation 508 in FIG. 5, the operations in FIGS. 6A - 6D, and their descriptions).

[0094] In some embodiments, when considering the FRC as a global similarity estimation between two images, the FRC metric may be extended to a rolling FRC (rFRC) map to provide a local distance measurement at the pixel level. In some embodiments, even without ground truth, the error of the multi-dimensional reconstruction signal may be quantitatively mapped. In some embodiments, if the same region of both images is blank (however, the ground truth has content), such information may be lost in the same region of the images, leading to a small FRC value. In other words, if two reconstruction signals lose the same component (even if this situation is rare), the rFRC may introduce false positives. Therefore, in some embodiments, to mitigate such incompleteness of the rFRC map, a modified resolution scale error map (RSM) may be introduced as an additional error map.

[0095] In some embodiments, the RSM may be constructed based on one or more of three hypotheses.

[0096] (i) Since the RSM needs to refer to the corresponding diffraction-limited wide-field image, the wide-field image needs to have a sufficiently high SNR. Otherwise, the noise contained in the wide-field image may induce a relatively high false negative result.

[0097] (ii) The conversion from the super-resolution (SR) scale to the low-resolution (LR) scale may be approximated as a spatially invariant global Gaussian kernel convolution (across the entire field of view). If the conversion from the super-resolution image to the low-resolution image is not limited to such a spatially invariant Gaussian model, it may generate relatively high false negatives.

[0098] (iii) The irradiation intensity and the emitted fluorescence intensity need to be uniform. The generated RSM may be weighted by the irradiation intensity and the emitted fluorescence intensity of the corresponding label.

[0099] According to the above hypothesis, the RSM itself may introduce false negatives (i.e., model errors) into the estimated error map. In the configuration of single molecule localization microscopy (SMLM), the hypotheses of (i) and (ii) may be satisfied. In the first hypothesis, the wide-field image of SMLM may be created by averaging thousands of blinking frames, whereby noise is removed and a reference wide-field image with a relatively high SNR may be generated. Regarding the second hypothesis, when the imaging field of view is a relatively small field of view, it may usually be treated as being uniformly illuminated. Furthermore, since such error estimation of RSM is calculated at a low-resolution scale, RSM may be able to find errors at the low-resolution scale (usually large error components), such as misrepresentation or disappearance of structures. The errors at the super-resolution scale (small error components) estimated by RSM may generally be false negatives. To reduce such potential false negatives, the normalized RSM may be segmented (e.g., set to 0 when less than 0.5), the corresponding small error components may be removed, the large components may be left, and the lost information of the reconstruction that may be ignored by the rFRC map may be visualized. Overall, the rFRC map and the segmented normalized RSM may be integrated (e.g., in the green and red channels respectively) to generate a complete PANEL map (see FIG. 16), whereby the errors of the reconstructed results may be accurately and comprehensively mapped and visualized. A more detailed description regarding the determination of RSM may be found elsewhere in the present disclosure (e.g., operation 518 in FIG. 5, operations in FIG. 7, and their descriptions). A more detailed description regarding the determination of the rFRC map may be found elsewhere in the present disclosure (e.g., operations 502 to 516 in FIG. 5, operations in FIG. 8, and their descriptions). A more detailed description regarding the determination of the PANEL map may be found elsewhere in the present disclosure (e.g., operation 520 in FIG. 5 and its description).

[0100] In 502, the processing device 140 (e.g., the acquisition module 402) may acquire a first image and a second image associated with the same object. In some embodiments, the processing device 140 may acquire the first image and / or the second image from one or more components of the image processing system 100. For example, the image acquisition device 110 may collect and / or generate the first image and / or the second image and store the images in the storage device 150 or any other storage associated with the image processing system 100. The processing device 140 may search for and / or acquire the images from the storage device 150 or any other storage. As another example, the processing device 140 may directly acquire the images from the image acquisition device 110. In some embodiments, the processing device 140 may acquire one or more initial images, process the initial images, and / or generate the first image and / or the second image. In the present disclosure, "image", "image frame", and "frame" are used synonymously. In some embodiments, the first image and the second image may be 2D images or 3D images. Note that the description of the case where the image (e.g., the first image, the second image) is a 2D image is provided merely for the purpose of explanation and is not intended to limit the scope of the present disclosure.

[0101] The object may include a non-biological object or a biological object. Exemplary non-biological objects may include phantoms, industrial materials, etc., or any combination thereof. Exemplary biological objects may include biological structures, biological subjects, proteins, cells, microorganisms, etc., or any combination thereof. In some embodiments, the object may be fluorescent or fluorescently labeled. The fluorescent or fluorescently labeled object may be excited to emit fluorescence for imaging.

[0102] In some embodiments, the first image and the second image may be generated from physical imaging. In some embodiments, the first image and the second image may be two consecutive images (e.g., Frame 1 and Frame 2 shown in FIG. 9A) obtained from two image captures of the same object by the image acquisition device 110. In some embodiments, the first image and the second image may be obtained from one image capture of the same object by the image acquisition device 110. For example, the first image and the second image may be obtained by decomposing an initial image captured by the image acquisition device 110. As shown in FIG. 12A, Frame 1 represents the initial image and may be decomposed into two or more images (e.g., the first image, the second image, the third image, and the fourth image, etc.). A more detailed description regarding generating a target map (e.g., the third target map) based on a single image may be found elsewhere in the present disclosure (e.g., FIG. 8 and its description). In some embodiments, the first image and / or the second image may have a relatively high resolution (e.g., super resolution).

[0103] In some embodiments, the first image and the second image may be obtained from two initial images based on a neural network. The processing device 140 may obtain a first initial image and a second initial image associated with the same object from one or more components of the image processing system 100 (e.g., the image acquisition device 110, the storage device 150). In some embodiments, the first initial image and / or the second initial image may have a relatively low resolution, and the first image and / or the second image may have a relatively high resolution. In some embodiments, the processing device 140 (e.g., the neural network processing module 418) may generate the first image and the second image by processing the first initial image and the second initial image respectively using a neural network (or an artificial neural network (ANN)). Exemplary neural networks may include a convolutional neural network (CNN), a deep learning network, a support vector machine (SVM), a k-nearest neighbor method (KNN), a backpropagation neural network (BPNN), a deep neural network (DNN), a Bayesian neural network (BNN), etc., or any combination thereof. In some embodiments, the processing device 140 may generate the first image and the second image by processing the first initial image and the second initial image respectively based on a neural network model. For example, the first initial image and / or the second initial image may be input into the neural network model. The neural network model may output the first image and / or the second image. The first image and the second image may be two frame images used to generate a first target map (e.g., an rFRC map) as shown in 510. In some embodiments, the rFRC map may reflect an error (e.g., at the pixel level) generated during processing (e.g., reconstruction) using a neural network. As shown in FIG. 13, observer 1 and observer 2 represent the first initial image and the second initial image respectively, and r1 and r2 represent the first image and the second image output by the neural network respectively.

[0104] In some embodiments, the first image and / or the second image may be generated from a single initial image using a neural network. In some embodiments, the processing device 140 (e.g., the neural network processing module 418) may generate the first image and the second image by processing a single initial image using a neural network. The neural network reconstruction technique may be different from the physical imaging reconstruction technique. For example, adjacent pixels in an image output from a neural network (e.g., a DNN) may be considered independent of noise. Therefore, the determination of the rFRC map based on a single frame used in physical imaging may not be directly applicable to such neural network reconstruction. Different single-frame strategies may be used to determine the rFRC map of a neural network (e.g., a DNN). In some embodiments, noise (e.g., Gaussian noise) may be added to the initial image. Exemplary noise may include Gaussian noise, impulse noise, Rayleigh noise, gamma noise, exponential noise, uniform noise, etc., or any combination thereof. For example, a single initial image may be obtained as the first initial image, and the second initial image may be generated by adding noise (e.g., the magnitude of the noise is 0.5%, 1.5%, 2%, etc.) to the first initial image. Therefore, the two images (i.e., the first initial image and the second initial image) may be used as inputs to the neural network. After reconstructing these two input images (i.e., the first initial image, the second initial image) using the neural network respectively, the two resulting output images may be obtained as the first image and the second image. The two output images may be further used to determine the rFRC map. In some embodiments, two different noises may be added to the initial image to obtain two different images. For example, the first initial image (see the image represented by "Observer + n1" in FIG. 13) may be generated by adding the first noise to the initial image, and the second initial image (see the image represented by "Observer + n2" in FIG. 13) may be generated by adding the second noise to the initial image.In some embodiments, the type of the first noise may be the same as the type of the second noise, but the magnitudes of the two noises may be different. In some embodiments, the type of the first noise may be different from the type of the second noise, but the magnitudes of the two noises may be the same. In some embodiments, the type of the first noise and the type of the second noise may be different, and the magnitude of the first noise and the magnitude of the second noise may be different. Accordingly, two images (i.e., the first initial image and the second initial image) may be used as inputs to the neural network. Then, after reconstructing each of the two input images (i.e., the first initial image and the second initial image) using the neural network, the two images output as a result may be obtained as the first image and the second image. The two output images may be further used to determine the rFRC map. In some embodiments, the magnitude of the noise added to the initial image may be determined according to a specific situation, and this specification is not intended to limit the scope of the present disclosure.

[0105] In 504, the processing device 140 (e.g., the expansion module 404) may expand the first image and / or the second image.

[0106] In some embodiments, the processing device 140 may expand the first image and / or the second image by performing a padding operation around the first image and / or the second image respectively. Performing a padding operation around an image means that data may be padded in one or more regions adjacent to the edges of the image. In some embodiments, the padded regions may have one or more predetermined sizes. For example, the padded region may have a width that is half of the width of a block (e.g., the block described in 506). In some embodiments, the padded data may have one or more predetermined values, such as 0, 1, 2, or any other value. In some embodiments, the padded data may be determined based on the pixel values of the image to be expanded. In some embodiments, the data may be padded symmetrically around the image (see FIG. 9A).

[0107] In 506, the processing device 140 (e.g., the rolling module 406) may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image.

[0108] In some embodiments, the processing device 140 may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image by performing a rolling operation on the first image and the second image, respectively. In some embodiments, the rolling operation may be performed by sliding a window over an image (e.g., the (extended) first image and the (extended) second image). In the rolling operation, the window may slide row by row and / or column by column over the image, and the designated center of the window may traverse one or more pixels (e.g., each pixel) of the image. In some embodiments, the window may have a predetermined size (e.g., 32×32, 64×64, 128×128, etc.). The predetermined size of the window may be determined by the image processing system 100 or preset by a user or an operator via the terminal 130. In some embodiments, the size of the window may be associated with the size of the first image and / or the second image. For example, the window may have a size smaller than the size of the first image and / or the second image. The window may have various shapes. For example, the window may be square, rectangular, circular, oval, etc. In some embodiments, the window used for the first image may be the same as the window used for the second image.

[0109] In some embodiments, a window may be slid pixel-by-pixel on the (extended) first image to generate a plurality of first blocks of the first image. As used herein, a block of an image may mean a part of the image. When the window slides on the image (e.g., the (extended) first image), a part of the image (e.g., the (extended) first image) may be surrounded by the window, and the part of the image (e.g., the (extended) first image) surrounded by the window may be designated as a block of the image (e.g., the first block of the first image). Thereby, a plurality of first blocks of the first image may be determined by sliding the window on the (extended) first image. Similarly, a plurality of second blocks of the second image may be determined by sliding the window on the (extended) second image. In some embodiments, the plurality of second blocks and the plurality of first blocks may correspond one-to-one. In some embodiments, the number of the first blocks (or the second blocks) may be the same as the number of pixels of the first image (or the second image) obtained at 502.

[0110] The rFRC is an error map that enables evaluation at the pixel level. However, the finest scale of detectable errors can only reach the highest resolution of the corresponding image (e.g., the first image and the second image). Therefore, if the image (e.g., the first image, the second image) satisfies the Nyquist-Shannon sampling theorem, per-pixel rolling operations may not be necessary. In some embodiments, the minimum error may be larger than ~3×3 pixels. Thus, in order to obtain a 4-fold to 16-fold acceleration for the rFRC mapping operation, 2 to 4 pixels may be skipped for each rolling operation. In some embodiments, a plurality of first blocks (or a plurality of second blocks) may be determined by sliding a window over the (extended) first image (or the (extended) second image) and skipping a predetermined number of pixels over the (extended) first image (or the (extended) second image). For example, when the window slides to the position of the (extended) first image (or the (extended) second image), the first block of the first image (or the second block of the second image) may be determined. Thereafter, the window skips a predetermined number of pixels over the (extended) first image (or the (extended) second image) and slides to the next position of the (extended) first image (or the (extended) second image), and another first block of the first image (or another second block of the second image) may be determined. Thus, the number of first blocks (or second blocks) may be less than the number of pixels of the first image (or the second image) obtained at 502. The predetermined number of pixels may be determined by the image processing system 100 or may be preset by the user or operator via the terminal 130. In some embodiments, the predetermined number of pixels may be associated with the size of the window. For example, the predetermined number of pixels (e.g., 2, 3, or 4) may be close to the width of the window (e.g., 3). In some embodiments, the predetermined number of pixels may be random.

[0111] At 508, the processing device 140 (e.g., the map generation module 408) may determine a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks.

[0112] The first characteristic value may refer to a value associated with a pixel value of the first target map (i.e., the rFRC map). For example, the first characteristic value corresponding to a pixel of the first target map may be associated with the correlation between the first block and the second block corresponding to the pixel of the first target map where the central pixel is. In some embodiments, at least one of the first characteristic values may be determined based on the correlation between one of the plurality of first blocks and the corresponding second block. The correlation between the first block and the corresponding second block may be represented by a correlation value (or called an FRC value). In some embodiments, the correlation between the first block and the corresponding second block may be determined by performing a Fourier transform on the first block and the second block respectively. For example, the processing device 140 may determine a first intermediate image in the frequency domain based on the first block (e.g., by performing a Fourier transform on the first block). The processing device 140 may determine a second intermediate image in the frequency domain based on the second block (e.g., by performing a Fourier transform on the second block). In some embodiments, the processing device 140 may determine a target frequency (also called a cut-off frequency) based on the first intermediate image and the second intermediate image. The processing device 140 may determine the first characteristic value based on the target frequency. For example, the processing device 140 may determine, as the first characteristic value, the target frequency, the reciprocal of the target frequency, or the product of the reciprocal of the target frequency and the pixel size of the first image (and / or the second image) (e.g., the physical dimension of the camera pixel divided by the magnification of the image (the first image, the second image, etc.)). A more detailed description regarding determining a plurality of first characteristic values may be found elsewhere in the present disclosure (e.g., FIGS. 6A-6D).

[0113] In some embodiments, there may be a background region in the first image and / or the second image. In some embodiments, the information of the background region may remain unchanged or may change slightly in the first image and the second image. In some embodiments, the FRC operation for the background region may be avoided to reduce the calculation burden. The FRC operation may refer to an operation of determining an FRC value. In this case, the first characteristic value for the background region may be determined based on one or more predetermined values. In some embodiments, for the first block and the corresponding second block, the processing device 140 may determine whether to perform the FRC operation (or whether the first block and / or the corresponding second block is a background region) before determining the first characteristic value. A more detailed description regarding the determination of the first characteristic value may be found in other parts of the present disclosure (for example, FIG. 6A).

[0114] In 510, the processing device 140 (for example, the map generation module 408) may generate a first target map associated with the first image and the second image based on a plurality of first characteristic values.

[0115] As used herein, the first target map (for example, the rFRC map) may refer to an image indicating the reconstruction error and / or artifacts of the first image and / or the second image acquired in 502. The first target map may include a plurality of pixels, and the value of each pixel may be determined based on the first characteristic value. The first target map may have the same size as the first image and / or the second image. In some embodiments, the pixel value determined based on the first characteristic value may be assigned to the pixel (of the target map) corresponding to the central pixel of the first block or the second block (used to determine the first characteristic value). That is, the pixels of the first target map may correspond to the pixels of the first image and / or the second image.

[0116] In some embodiments, since the pixels of the first target map correspond one-to-one with the pixels of the first image (and / or the second image), the first target map may reflect the image quality of the image (e.g., the first image and / or the second image) at the pixel level. In some embodiments, when the pixel value of the pixel of the first target map is determined as the reciprocal of the target frequency, i.e., meaning the resolution, the smaller the pixel value (i.e., the resolution), the higher the accuracy of the pixels of the first image and / or the second image, and the larger the pixel value (i.e., the resolution), the lower the accuracy of the pixels of the first image and / or the second image. The relatively low accuracy of the pixels here may indicate an error. In some embodiments, when the pixel value of the pixel of the first target map is determined as the target frequency meaning the spatial frequency, the smaller the pixel value (i.e., the spatial frequency), the lower the accuracy of the pixels of the first image and / or the second image, and the larger the pixel value (i.e., the spatial frequency), the higher the accuracy of the pixels of the first image and / or the second image. In some embodiments, when the pixel value of the pixel of the first target map is determined as the product of the reciprocal of the target frequency and the pixel size of the first image and / or the second image, the smaller the pixel value, the higher the accuracy of the pixels of the first image and / or the second image, and the larger the pixel value, the lower the accuracy of the pixels of the first image and / or the second image.

[0117] In some embodiments, the first target map may quantitatively present the resolution of each pixel of the first image and / or the second image, thereby indicating the error (and / or artifact) present in the first image and / or the second image. Therefore, using the target map (e.g., the first target map), the error (and / or artifact) may be visualized more intuitively. As a mere example, if the first image and / or the second image has a resolution of 100 nm and the resolution of the pixel is determined to be 10 nm, it can be shown that the pixel is accurate and there is no error in the pixel reconstruction. If the first image and / or the second image has a resolution of 100 nm and the resolution of the pixel is determined to be 200 nm, it can be shown that the pixel is not accurate and there is an error in the pixel reconstruction.

[0118] In some embodiments, the first target map may be represented in the form of a grayscale image. In some embodiments, the shading value of each pixel of the first target map may have a positive correlation with the first characteristic value, and the shading degree of the first target map may reflect the image quality (or error) of the first image and / or the second image.

[0119] In some embodiments, when the skip operation is executed at 506, the first target map may be resized (e.g., by bilinear interpolation) to the original image size of the first image and / or the second image for better visualization.

[0120] At 512, the processing device 140 (e.g., the filtering module 410) may filter the first target map. In some embodiments, the processing device 140 may filter the first target map using an adaptive median filter.

[0121] In some embodiments, according to the Nyquist sampling theorem, the minimum error of the first image and / or the second image in which the operations of the present disclosure can be detected may be greater than, for example, 3×3 pixels. Therefore, the rolling motion in a small area (e.g., 3×3 pixels) may change smoothly without any isolated pixels (e.g., ultra-high value pixels or ultra-low value pixels compared to neighboring pixels) occurring. However, due to inappropriate determination of the target frequency, the FRC operation may become unstable and abnormal characteristic values may be determined. In some embodiments, an adaptive median filter may be used to remove these inappropriate characteristic values in these pixels. In some embodiments, the adaptive median filter may replace each pixel with the median value of the adjacent pixel values within the filtering window of the first target map. In some embodiments, a threshold value may be used in the adaptive median filter. For example, if the pixel value (e.g., grayscale value) of a pixel is greater than the product of the threshold value and the median value of the adjacent pixel values within the filtering window, the pixel value may be replaced with the median value; otherwise, the filtering window may move to the next pixel. By using the adaptive median filter, isolated pixels may be filtered without blurring the rFRC map. In some embodiments, the threshold value (e.g., 2) may be determined by the image processing system 100 or preset by the user or operator via the terminal 130.

[0122] In 514, the processing device 140 (e.g., the image quality determination module 414) may determine a global image quality metric of the first image and / or the second image based on the first target map.

[0123] The rFRC map generated at 510 may be used to quantitatively visualize the errors included in images (e.g., the first image and the second image). In addition to such local quality evaluation, a global image quality metric for the first image and / or the second image may be determined. In some embodiments, the global image quality metric may have dimensions as resolution. In some embodiments, the global image quality metric may be normalized without considering dimensions.

[0124] In some embodiments, the global image quality metric with dimensions as resolution may be expressed as follows.

[0125]

Equation

[0126] Here, ||FV||0 is the l0 norm representing the number of non-zeros in the rFRC map, and FV represents the rFRC map.

[0127] In some embodiments, the global image quality metric normalized without considering dimensions may be expressed as follows.

[0128]

Equation

[0129] Here, ||FV||0 is the l0 norm representing the number of non-zeros in the rFRC map, and FV represents the rFRC map.

[0130] Note that both of these two metrics shown in Equation (13) and Equation (14) may be extended to three dimensions, and the two-dimensional coordinates (x, y) in Equation (13) and Equation (14) may be directly raised to three-dimensional coordinates (x, y, z).

[0131] At 516, the processing device 140 (e.g., the display module 412) may display a first target map using a displacement jet color map.

[0132] In some embodiments, the first target map may be displayed using one or more color maps. The color map may include one or more existing common color maps such as a jet color map. The jet color map may use colors from blue to red that index different error magnitudes to emphasize all error regions. In some embodiments, such a jet color map may be counterintuitive or contrary to human intuition or habit in some way. The human visual system may be intended to define black (dark color) as a small grade and bright (light color) as a large grade, which is the same as the logic of the grayscale color map. In some embodiments, human vision may be insensitive to light or dark grayscale levels and more sensitive to different colors. Therefore, a color map (e.g., a displacement jet color map) including a color index that is more compatible with human intuition may be used.

[0133] In some embodiments, to create a displaced jet (sJet) color map (see FIG. 14B) in which the red component at relatively low levels and the green component at relatively high levels are removed and the blue component is extended across the entire color index, the jet color map (see FIG. 14A) may be displaced. In some embodiments, considering that human intuition may be more sensitive to green, green may be moved as a relatively large level to emphasize large errors. In some embodiments, humans may be accustomed to treating bright colors (e.g., white) as relatively large levels and dark colors (e.g., black) as relatively small levels, and accordingly, the black zone (0,0,0) and the white zone (1,1,1) may be involved in the displaced jet (sJet) color map. In some embodiments, with the extension of the blue component, the white zone may be created to represent the maximum error. In some embodiments, the background of the rFRC map may mean positive (i.e., error-free), and accordingly, to display such a background, the black zone (instead of the blue of the jet color map) may be used, which is more suitable for human intuition. As shown in FIGS. 14C-14D, compared to the original jet color map (see FIG. 14C), the sJet color map (see FIG. 14D) will be clearly more intuitive for the visualization of the error map.

[0134] 518, the processing device 140 (e.g., the map generation module 408) may determine a reference map associated with the first image or the second image.

[0135] In some embodiments, the reference map may include a modified resolution scale error map (RSM). The reference map may reflect errors and / or artifacts in the first image and / or the second image. In some embodiments, the reference map may be in the form of a grayscale map.

[0136] In some embodiments, the reference map may be determined based on the difference between the intermediate image and the reference image (e.g., an image with relatively low resolution). The intermediate image may be determined based on the first image (or the second image) and the convolution kernel. The convolution kernel may be determined based on the first image (or the second image) and the reference image. A more detailed description regarding the determination of the reference map may be found elsewhere in the present disclosure (e.g., FIG. 7 and its description).

[0137] Note that in some embodiments, when the first image (or the second image) is three-dimensional, or when it is under a non-Gaussian convolution relationship (e.g., between a low-resolution scale and a high-resolution scale), the reference map may not be determined. That is, operation 518 may be omitted. Accordingly, for such a first image or second image, the RSM may not be involved in the fourth target map, and operation 520 may also be omitted.

[0138] In operation 520, the processing device 140 (e.g., the map generation module 408) may determine the fourth target map by fusing the first target map and the reference map.

[0139] In some embodiments, the first target map (e.g., rFRC map) may illustrate minute and subtle errors in the reconstruction of the first image (and / or the second image). In some embodiments, the first target map may be impossible when structures are simultaneously missing in both the first image and the second image. As a mere example, a 2D SMLM dataset may be used to explain this possible false negative (see FIGS. 17A - 17C). As indicated by the bright arrows in FIG. 17A, small portions of the filaments in the same region of the two reconstructions may be artificially removed. As shown in FIG. 17B, the rFRC map may normally detect most of the error components in the image, except for the artificially removed regions. In some embodiments, such disappearance of structures may be detected by a reference map (e.g., RSM (see FIG. 17C)). In some embodiments, the reference map (e.g., RSM) may be introduced as an additional error map for the first target map (e.g., rFRC map).

[0140] The fourth target map may reflect the overall error and / or artifacts of the first image (and / or the second image) indicated by the first target map and the reference map. The fourth target map may also be referred to as the PANEL map.

[0141] In some embodiments, the processing device 140 may process the first target map and / or the reference map in the form of a grayscale map. In some embodiments, the processing device 140 may convert the first target map and the reference map into an RGB image. For example, the first target map may be displayed in green (as shown in the left panel of FIG. 11A), and the reference map may be displayed in red (as shown in the right panel of FIG. 11A). In some embodiments, the processing device 140 may normalize the first target map and / or the reference map, for example, by dispersion normalization, mean normalization, etc. In some embodiments, the processing device 140 may determine the fourth target map based on the weighted sum of the normalized first target map and the reference map.

[0142] In some embodiments, the processing device 140 may perform a threshold processing operation on the first target map and / or the reference map to filter the background regions therein, thereby highlighting important information (e.g., critical errors in the reconstruction of the first image and / or the second image) (see FIG. 11B). In some embodiments, the threshold of the first target map may be the same as or different from the threshold of the reference map. In some embodiments, the threshold of the first target map and / or the reference map may be in the range of 0 to 1.

[0143] In some embodiments, the threshold of the reference map may be determined by the image processing system 100 or preset by the user or operator via the terminal 130. For example, the threshold of the reference map may be 0.5. In some embodiments, the same lost information (e.g., misrepresentation or disappearance of structure) in both the first image and the second image may have a relatively large grade and may be present in a relatively large area of the reference map. In some embodiments, the small-grade components included in the reference map may induce a relatively high false negative. In some embodiments, the reference map may be segmented based on the threshold by the following formula before involving the reference map in the fourth target map.

[0144]

Equation

[0145] Here, R(x, y) represents the normalized RSM value at the coordinates (x, y), and R% represents the segmented RSM. In some embodiments, a low false negative (e.g., less than 0.5) may be filtered according to Equation (15) so as to leave only relatively strong low-resolution errors and focus on the true negatives detected by the reference map.

[0146] In some embodiments, the rFRC map may indicate the degree of error, and the minimum FRC value in the map may not indicate an acceptable error. In some embodiments, the threshold of the first target map may be determined by Otsu's binarization method (also abbreviated as Otsu) used to perform automatic image thresholding. Using Otsu's method, the threshold may be automatically determined by maximizing the between-class variance. Image thresholding may be performed to filter the background of the rFRC map, thereby highlighting the critical errors in the reconstruction of the first and second images. An exemplary threshold for the first target map may be 0.4. Those pixels whose pixel values are less than the two thresholds may be filtered from the first target map and the reference map, respectively. For example, the pixel values of such pixels may be designated as 0.

[0147] In some embodiments, the processing device 140 may determine a fourth target map (e.g., PANEL map) by fusing the first target map and the reference map to comprehensively indicate the errors and / or artifacts in the reconstruction of the first image and / or the second image (see FIG. 11C). In some embodiments, the fourth target map may be determined by fusing the third target map determined in process 800 and the reference map.

[0148] Note that the above description of process 500 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. Those skilled in the art will understand, upon understanding the principle of operation, that they can arbitrarily combine any operations, add or delete any operations, or apply the principle of operation to other image processing processes without departing from the principle. In some embodiments, one or more operations in process 500 may be omitted. For example, operation 504, operation 512, operation 514, operation 516, operation 518, and / or operation 520 may be omitted. In some embodiments, two or more operations of process 500 may be integrated into one operation. For example, operations 506 to 510 may be integrated into one operation. As another example, operations 518 to 520 may be integrated into one operation. As a further example, operations 506 to 510 and operations 518 to 520 may be integrated into one operation.

[0149] In some embodiments, the first image and the second image may be 3D images. The procedure for processing a 3D image may be the same as the procedure for processing a 2D image. The rolling operation in 506 may be operable in three dimensions. For example, the processing device 140 may perform the rolling operation in such a form as to execute a 3D filter. Since the window used for the three-dimensional rolling operation may be cube-shaped, the first block and the second block may be cube-shaped. In some embodiments, the FRC calculation may be extended to a 3D version called Fourier-Schell correlation (FSC). Thereby, the first ring image and the second ring image (see FIG. 6D) generated from the first block and the second block may be spherical. In some embodiments, in order to avoid a relatively heavy computational burden, the three-dimensional rFRC calculation may be directly generated by the 2D rFRC calculation for each plane.

[0150] FIG. 6A is a flowchart showing an exemplary process for determining whether to perform an FRC operation on a first block or a second block, according to some embodiments of the present disclosure. In some embodiments, process 600 may be performed by image processing system 100. For example, process 600 may be configured as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, storage 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of arithmetic device 200, CPU 340 of portable device 300, and / or one or more modules shown in FIG. 4) may execute the set of instructions and, accordingly, may be configured to perform process 600. The illustrated operations of the processes presented below are intended to be exemplary. In some embodiments, process 600 may be accomplished with one or more additional operations not described and / or without one or more of the operations described. Further, the order of operations of process 600 as shown in FIG. 6A and described below is not intended to be limiting. In some embodiments, operation 508 shown in FIG. 5 may be performed in accordance with one or more operations of process 600.

[0151] As described above, the FRC may measure the global similarity between two images (e.g., a first image and a second image), and the FRC value may be extended in the form of a rolling FRC (rFRC) map to provide local distance measurements at the pixel level. In some embodiments, the form of the FRC operation may be considered as a filter in which the image is processed block by block and a corresponding FRC value is assigned to each block. In some embodiments, as shown in 504, the image (e.g., the first image and / or the second image) may be symmetrically padded with half the size of the block to facilitate the FRC operation for the image boundary. In some embodiments, the background threshold may be set for the central pixel of the block (e.g., a 1×1 pixel to 3×3 pixel) to determine whether to determine the FRC value in this block, thereby avoiding the FRC operation for the background region. In some embodiments, if the average of the central pixels of the block is greater than the background threshold, the FRC operation may be executed and the FRC value may be assigned to the central pixel of the block. In some embodiments, if the average is less than the background threshold, a zero value may be set for the central pixel of the block. This procedure block may be executed block by block until the entire image is processed.

[0152] In 601, the processing device 140 (e.g., the map generation module 408) may determine the average value of the pixel values of the pixels in the central region of each first block and / or the average value of the pixel values of the pixels in the central region of each second block. In some embodiments, the central region may be located within or near the center of each first block or each second block. The central region may be of any shape, such as a square, a circle, etc. In some embodiments, the size of the central region may be smaller than or equal to the size of each first block or each second block.

[0153] The average value of the pixel values of a pixel may be a value obtained by dividing the sum of the pixel values by the total number of pixels in the central region. In some embodiments, the processing device 140 may determine the average value of the pixel values of the pixels within the central region of each first block and the pixels within the central region of each second block. In some embodiments, the processing device 140 may determine a plurality of average values corresponding to a plurality of first blocks and / or a plurality of average values corresponding to a plurality of second blocks. In some embodiments, the processing device 140 may determine a plurality of average values corresponding to a plurality of first blocks and a plurality of second blocks.

[0154] In 602, the processing device 140 (for example, the map generation module 408) may compare the average value with a threshold value (for example, a background threshold value).

[0155] The first image and / or the second image may have a background region. In some embodiments, the background region may have noise (for example, background noise, readout noise). In some embodiments, such a background region may result in a relatively large FRC value, and as a result, may introduce false negatives during the generation of the rFRC map, thereby preventing the highlighting of true negatives (for example, errors generated in image reconstruction). Therefore, the background threshold value may be set to avoid such FRC calculations for these background regions. In the present disclosure, a true negative refers to a true error determined by the processing device 140, a false negative refers to a false error determined by the processing device 140, and a false positive refers to the determination by the processing device 140 that an error is not an error.

[0156] In some embodiments, the threshold value may be determined by the image processing system 100 or preset by a user or an operator via the terminal 130. In some embodiments, a fixed threshold value may be used for the entire first image and / or the entire second image (i.e., a plurality of first blocks of the first image and / or a plurality of second blocks of the second image). In some embodiments, an adaptive threshold value may be used. In some embodiments, the adaptive threshold value may be determined according to each first block and / or each second block. In some embodiments, one or more threshold values may be adaptively determined for a small local area of the first image (or the first block) or the second image (or the second block). That is, different threshold values may be used for different small local areas. In some embodiments, an iterative wavelet transform may be performed to determine the (adaptive) threshold value. As a mere example, the local background intensity distribution of the corresponding image (e.g., the first image, the second image, the first block, and the second block, etc.) may be estimated. Specifically, the background may be iteratively estimated from the lowest frequency wavelet band of the corresponding image, and in each iteration, one or more (e.g., all) values of the corresponding image that exceed the current estimated background level may be clipped.

[0157] In 603, the processing device 140 (e.g., the map generation module 408) may determine whether the average value is greater than the threshold value. In some embodiments, in response to determining that the average value is greater than or equal to the threshold value, the process 600 may proceed to operation 605. In some embodiments, in response to determining that the average value is not greater than or equal to the threshold value, the process 600 may proceed to operation 604.

[0158] In some embodiments, the processing device 140 may compare the average value for each first block, the average value for each second block, and / or the average value for each first block and each second block with a threshold value. In some embodiments, in response to determining that both the average value for each first block and the average value for each second block are greater than or equal to the threshold value, the process may proceed to operation 605; otherwise, the process 600 may proceed to 604. In some embodiments, in response to determining that the average value of each first block and each second block is greater than or equal to the threshold value, the process may proceed to operation 605; otherwise, the process 600 may proceed to 604.

[0159] At 604, the processing device 140 (e.g., the map generation module 408) may determine a characteristic value (e.g., a first characteristic value corresponding to each first block and / or each second block) based on a predetermined value. For example, the processing device 140 may assign the predetermined value to the characteristic value. The predetermined value may be, for example, 0. Thus, if the average value of the first block and / or the second block is not greater than the threshold value, the characteristic value corresponding to the first block and the second block may be 0, thereby avoiding unnecessary FRC calculations for the background area.

[0160] At 605, the processing device 140 (e.g., the map generation module 408) may determine a characteristic value based on each first block and each second block.

[0161] If the average value is greater than the threshold value, the processing device 140 may determine the characteristic value by performing an FRC calculation based on each first block and each second block. In some embodiments, the processing device 140 may determine the characteristic value based on a first intermediate image and a second intermediate image obtained by performing a Fourier transform on each first block and each second block. A more detailed description regarding determining the characteristic value by performing an FRC calculation may be found elsewhere in the present disclosure (e.g., FIG. 6B and its description).

[0162] The processing device 140 may execute process 600 for all the first blocks and / or the second blocks to determine a plurality of characteristic values.

[0163] FIG. 6B is a flowchart showing an exemplary process for determining characteristic values according to some embodiments of the present disclosure. In some embodiments, process 610 may be executed by the image processing system 100. For example, process 610 may be configured as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, storage 220, and / or storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the arithmetic device 200, the CPU 340 of the portable device 300, and / or one or more modules shown in FIG. 4) may execute the set of instructions and accordingly be configured to execute process 610. The operations of the illustrated processes presented below are intended to be exemplary. In some embodiments, process 610 may be accomplished with one or more additional operations not described and / or without one or more of the operations described. Further, the order of the operations of process 610 as shown in FIG. 6B and described below is not intended to be limiting. In some embodiments, operation 605 shown in FIG. 6A may be executed according to one or more operations of process 610.

[0164]

[0165] In some embodiments, the processing device 140 may determine a first intermediate image in the frequency domain by performing a Fourier transform on each first block. In some embodiments, the processing device 140 may determine a second intermediate image in the frequency domain by performing a Fourier transform on each second block. As used herein, the first intermediate image may be regarded as a first transformed image (in the frequency domain) from the first block, and the second intermediate image may be regarded as a second transformed image (in the frequency domain) from the second block. The Fourier transform may be used to convert the information of an image or a block (e.g., the first block and the second block) from the spatial domain to the frequency domain. In some embodiments, the Fourier transform may be a discrete Fourier transform (DFT). In some embodiments, each first block may correspond to the first intermediate image, and each second block may correspond to the second intermediate image. Accordingly, a plurality of first intermediate images corresponding to the plurality of first blocks may be determined, and a plurality of second intermediate images corresponding to the plurality of second blocks may be determined.

[0166] 612, the processing device 140 (e.g., the map generation module 408) may determine a target frequency based on the first intermediate image and the second intermediate image. In some embodiments, each first intermediate image and the corresponding second intermediate image may be used to determine the target frequency. Accordingly, a plurality of target frequencies may be determined based on the plurality of first intermediate images and the plurality of second intermediate images.

[0167] In some embodiments, in order to quantitatively determine the characteristic value, a certain criterion may be used to obtain the target frequency. For example, when the FRC curve falls below a threshold at a certain point, the frequency corresponding to that point may be defined as the target frequency. The threshold of the FRC may indicate the spatial frequency position of meaningful information above the random noise level. In some embodiments, a threshold that is a fixed value or a sigma factor curve (see curve b in FIG. 9C) may be used to determine the threshold. For example, a fixed value of 1 / 7 of the FRC curve may be used as a hard threshold.

[0168] In some embodiments, the target frequency may be determined based on a graph (see FIG. 9C) with the spatial frequency on the horizontal axis and the correlation value on the vertical axis. The target frequency may correspond to the intersection of a first curve (i.e., curve a in FIG. 9C) and a second curve (i.e., curve b in FIG. 9C). The first curve may represent a first relationship between a plurality of correlation values based on the first intermediate image and the second intermediate image and a plurality of frequencies. The second curve may represent a second relationship between a plurality of correlation values and a plurality of frequencies based on a predetermined function. A more detailed description of the determination of the target frequency may be found elsewhere in the present disclosure (e.g., FIGS. 6C and 6D and their descriptions).

[0169] At 613, the processing device 140 (e.g., the map generation module 408) may determine a characteristic value based on the target frequency. In some embodiments, the processing device 140 may specify each target frequency as the characteristic value. In some embodiments, the processing device 140 may determine the characteristic value based on the reciprocal of the target frequency. For example, the processing device 140 may specify the reciprocal of the target frequency as the characteristic value. In some embodiments, the processing device 140 may determine the characteristic value based on the reciprocal of the target frequency and the pixel size of the first image and / or the second image acquired at 502. For example, the processing device 140 may specify the product of the reciprocal of the target frequency and the pixel size as the characteristic value. As shown in FIG. 9D, FV represents a characteristic value (e.g., the rFRC value) determined based on the target frequency determined based on the first block and the second block.

[0170] FIG. 6C is a flowchart illustrating an exemplary process for determining a target frequency according to some embodiments of the present disclosure. In some embodiments, process 620 may be executed by image processing system 100. For example, process 620 may be configured as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, storage 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of arithmetic device 200, CPU 340 of mobile device 300, and / or one or more modules shown in FIG. 4) may execute the set of instructions and thereby be configured to execute process 620. The illustrated operations of the processes presented below are intended to be exemplary. In some embodiments, process 620 may be accomplished with one or more additional operations not described and / or without one or more of the operations described. Further, the order of operations of process 620 as shown in FIG. 6C and described below is not intended to be limiting. In some embodiments, operation 612 shown in FIG. 6B may be executed according to one or more operations of process 620.

[0171] At 621, processing device 140 (e.g., map generation module 408) may determine a first relationship between a plurality of correlation values and a plurality of frequencies based on the first intermediate image and the second intermediate image.

[0172] In some embodiments, processing device 140 may determine a plurality of correlation values between each first intermediate image and a corresponding second intermediate image. In some embodiments, the correlation values may be determined based on a first ring image associated with the first intermediate image and a corresponding second ring image associated with the second intermediate image. Accordingly, the plurality of correlation values may be determined based on the plurality of first ring images associated with the first intermediate image and the plurality of second ring images associated with the second intermediate image. A more detailed description regarding determining the correlation values may be found elsewhere in the present disclosure (e.g., FIG. 6D and its description).

[0173] In some embodiments, each of the correlation values may correspond to one of a plurality of frequencies. The correlation value determined based on the first ring image and the corresponding second ring image may correspond to the frequency of the first ring image and / or the second ring image. In some embodiments, the first curve (also referred to as the FRC curve) may indicate a first relationship between a plurality of correlation values and a plurality of frequencies. For example, the first curve may be represented by curve a in the graph shown in FIG. 9C.

[0174] At 622, the processing device 140 (e.g., the map generation module 408) may determine a second relationship between a plurality of correlation values and a plurality of frequencies based on a predetermined function.

[0175] In some embodiments, the second relationship may be indicated by a second curve. In some embodiments, the second curve may be a threshold curve. In some embodiments, the second relationship may be determined based on a predetermined function described as Equation (16). The threshold curve may also be referred to as a sigma factor curve or a 3σ curve, as shown by curve b in FIG. 9C.

[0176]

Equation

[0177] Here, N i represents the number of pixels of the first ring image and / or the second ring image having a radius q i , and σ 因子 may be 3. Since the calculation of the FRC sum includes all Hermite pairs in the Fourier domain, an extra factor of "2" may be required. If the two images (e.g., the first block, the second block) completely contain noise instead of signals, the second curve is FRC i = 1 / √N iIt is represented by. Therefore, the corresponding 3σ curve means that a standard deviation three times the expected random noise fluctuation may be selected as a threshold for determining the target frequency. In some embodiments, the second curve represented by Equation (16) may be more suitable for the PENEL to identify the category (whether it is noise or not) of the components in the image (e.g., the first image, the second image). Note that σ in Equation (16) 因子 may be other numerical values, such as 5, 7, 9, etc., and is not intended to be limiting.

[0178] In some embodiments, since the threshold for determining the target frequency may be a fixed value, the second curve may be a straight line. The threshold may be determined, for example, as 1 / 7 of the FRC curve. In some embodiments, the second curve may indicate the frequency of meaningful information above the random noise level. For example, the information corresponding to a frequency smaller than the second curve may not be confident, while the information corresponding to a frequency larger than the second curve may be confident.

[0179] In some embodiments, the 3σ curve may be suitable for small images in terms of its robustness and accuracy, and may be adapted to the determination of local resolution (corresponding to the cut-off frequency). The threshold, which is a fixed value, may be suitable for large images and may be adapted to the conventional global resolution determination. A more detailed description regarding the comparison between the two second curves may be found in other parts of the present disclosure (e.g., Example 11).

[0180] In 623, the processing device 140 (e.g., the map generation module 408) may determine the target frequency based on the first relationship and the second relationship. In some embodiments, the target frequency may correspond to the intersection of the first curve indicating the first relationship and the second curve indicating the second relationship. The target frequency may be the frequency at the intersection. For a plurality of first intermediate images and the corresponding second intermediate images, a plurality of target frequencies may be determined.

[0181] FIG. 6D is a flowchart illustrating an exemplary process for determining a correlation value according to some embodiments of the present disclosure. In some embodiments, process 630 may be executed by image processing system 100. For example, process 630 may be configured as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, storage 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of arithmetic device 200, CPU 340 of mobile device 300, and / or one or more modules shown in FIG. 4) may execute the set of instructions and may be configured to execute process 630 accordingly. The operations of the illustrated processes presented below are intended to be exemplary. In some embodiments, process 630 may be accomplished with one or more additional operations not described and / or without one or more of the operations described. Further, the order of the operations of process 630 as shown in FIG. 6D and described below is not intended to be limiting. In some embodiments, operation 621 shown in FIG. 6C may be executed according to one or more of the operations of process 630.

[0182] At 631, processing device 140 (e.g., map generation module 408) may determine a first ring image from the first intermediate image based on one of a plurality of frequencies.

[0183] The first ring image refers to a part of the first intermediate image (in the frequency domain) that is in the shape of a ring having a diameter equal to the frequency. The first ring image may include pixels within a ring (see the light gray circle in FIG. 9B) having a diameter equal to the frequency. The frequency may range from 0 to half of the reciprocal of the pixel size of the first image or the second image. The first ring image may correspond to the frequency, and accordingly, a plurality of first ring images may be determined based on a plurality of frequencies.

[0184] At 632, the processing device 140 (e.g., the map generation module 408) may determine a second ring image from the second intermediate image based on frequency. The second ring image may include pixels within a ring having a diameter equal to the frequency (see the light gray circle in FIG. 9B). The second ring image may be determined in a manner similar to the first ring image. The second ring image may correspond to the frequency, and accordingly, a plurality of second ring images may be determined based on a plurality of frequencies. The plurality of second ring images and the plurality of first ring images may correspond one-to-one. The second ring image and the corresponding first ring image may correspond to the same frequency.

[0185] At 633, the processing device 140 (e.g., the map generation module 408) may determine a correlation value based on the first ring image and the second ring image. In some embodiments, the processing device 140 may determine a plurality of correlation values based on a plurality of first ring images and a plurality of second ring images.

[0186] In some embodiments, the processing device 140 may determine a correlation value through a Fourier ring correlation (FRC) (or so-called spectral signal-to-noise ratio) operation. The FRC operation may measure the statistical correlation between two images (e.g., the first block, the second block) over a series of concentric rings (i.e., ring images) in the Fourier domain. In some embodiments, the FRC may be proposed as a method for measuring the image resolution in super-resolution microscopy. The FRC operation may be executed to determine a correlation value based on frequency. The correlation value (or FRC value) may be a function having the corresponding spatial frequency q i as a variable and may be determined based on Equation (17).

[0187]

[0188] Here, F1 and F2 represent the first intermediate image and the second intermediate image after Fourier transform (e.g., DFT) is performed.​

Number

[0189] Also, in the FRC operation, a Hamming window may be used to suppress the edge effect and other pseudo-correlations caused by the DFT operation. An exemplary Hamming window may be defined as follows.

[0190]

Number

[0191] Here, N represents the number of elements of the mask (e.g., the image size of the first intermediate image or the second intermediate image). The coefficients α and β may be set to 0.5.

[0192] In some embodiments, in the FRC operation, or in the determination of the first ring image and / or the second ring image at 631-632, it is necessary to discretize the spatial frequencies of the first intermediate image and the second intermediate image. In some embodiments, it is necessary to define the discretization of the spatial frequency of the FRC curve to determine the discrete values of the corresponding spatial frequencies. In some embodiments, the maximum frequency f max is half of the reciprocal of the pixel size (p s ) of the first image or the second image, i.e., f max = 1 / (2p s ). In some embodiments, a non-square image (i.e., a ring image) may be padded (e.g., zero-padding) to a square image for calculating the FRC curve. In some embodiments, the FRC curve may include N / 2 points, and the discretization step (Δf) may be obtained for the spatial frequency by Equation (19).

[0193] [Number]

[0194] Here, N represents the number of elements of the mask (e.g., the image size of the first intermediate image or the second intermediate image). f max represents the maximum frequency, and p s represents the pixel size.

[0195] In some embodiments, an average filter with a half-width of the average window (e.g., equal to 3 frequency bins) may be applied to smooth the FRC curve.

[0196] Note that the descriptions of the above processes 600, 610, 620, and 630 are provided for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that, based on an understanding of the operating principles, they can arbitrarily combine any operations, add or delete any operations, or apply the operating principles to other image processing processes without departing from the principles. In some embodiments, one or more operations in process 600 may be omitted. For example, operations 601 to 604 may be omitted. That is, the processing device 140 may perform FRC calculations on each first block and / or each second block without filtering the background region. In some embodiments, process 610 may be integrated into process 600. In some embodiments, process 620 may be integrated into process 610. In some embodiments, one or more operations may be integrated into one operation. For example, operations 631 and 632 may be integrated into one operation.

[0197] An exemplary FRC calculation process may be shown in FIGS. 9A-9D. FIGS. 9A-9D show an exemplary process for determining a target map according to some embodiments of the present disclosure. FIGS. 9A-9D are provided for illustrative purposes only and are not intended to limit the scope of the present disclosure. As shown in FIG. 9A, the first image (Frame 1) and the second image (Frame 2) are of size 4×4. Frame 1 (and Frame 2) is extended by symmetrically padding the periphery of the frame using some pixel values of Frame 1 (and Frame 2). For example, the pixel value (indicated by number 2) is symmetrically padded about the left boundary of Frame 1, the pixel value (indicated by number 6) is symmetrically padded about the left boundary and the upper boundary of Frame 1, and the pixel value (indicated by number 5) is symmetrically padded about the upper boundary of Frame 1. Similarly, other portions of Frame 1 (and Frame 2) may be symmetrically padded. To explain the rolling operation in relation to Frame 1 of FIG. 9A, the window (3×3) is represented by the solid-line box. The 3×3-sized image enclosed by the window represents the first block. Initially, the window is located at the initial position of the extended first image, and the first block has a center represented by "c". Next, the window slides to the next position of the extended first image, and the next first block is obtained, and the center (represented by "c") of the next first block may coincide with the pixel (represented by number 2) of the first image. The window may slide over the entire extended first image to generate a plurality of first blocks. The upper black solid-line box in FIG. 9B represents the first intermediate image corresponding to the upper first block in FIG. 9A, and the lower black solid-line box in FIG. 9B represents the second intermediate image corresponding to the second block in the upper part of FIG. 9B. As shown in FIG. 9B, the circle with the radius indicated by the dotted arrow represents a ring (or ring image). A plurality of first ring images associated with the first intermediate image and a plurality of second ring images associated with the second intermediate image may be determined. A plurality of correlation values (also called FRC curves, as shown by curve a in FIG. 9C) may be determined based on the first ring image and the second ring image. The intersection of each FRC curve and the 3σ curve (shown by curve b in FIG. 9C) may be determined.The characteristic value may be determined based on the target frequency corresponding to each intersection point. The rFRC map may be generated based on a plurality of characteristic values (see FIG. 9D).

[0198] FIG. 7 is a flowchart showing an exemplary process for determining a reference map according to some embodiments of the present disclosure. In some embodiments, process 700 may be executed by image processing system 100. For example, process 700 may be configured as a set of instructions (eg, an application) stored in a storage device (eg, storage device 150, storage 220, and / or storage unit 370). In some embodiments, processing device 140 (eg, processor 210 of arithmetic device 200, CPU 340 of portable device 300, and / or one or more modules shown in FIG. 4) may execute the set of instructions and accordingly be configured to execute process 700. The operations of the illustrated processes presented below are intended to be illustrative. In some embodiments, process 700 may be accomplished with one or more additional operations not described and / or without one or more of the operations described. Further, the order of operations of process 700 as shown in FIG. 7 and described below is not intended to be limiting. In some embodiments, operation 518 shown in FIG. 5 may be executed in accordance with one or more operations of process 700.

[0199] At 702, processing device 140 (eg, map generation module 408) may obtain a reference image with a resolution lower than that of the first image.

[0200] In some embodiments, processing device 140 may obtain a reference image from one or more components of image processing system 100. For example, image acquisition device 110 may collect and / or generate a reference image and store it in storage device 150. Processing device 140 may search for and / or obtain the reference image from storage device 150. As another example, processing device 140 may directly obtain the reference image from image acquisition device 110.

[0201] In some embodiments, the reference image may be generated by the same measurement of the same object as the first image. In some embodiments, the reference image may be obtained by capturing an object using the image acquisition device 110. In some embodiments, the reference image and the first image may be reconstructed based on the same measurement signal, but different reconstruction algorithms are used. The reference image may have a lower resolution than the first image. The reference image in this specification may be regarded as a low-resolution image, and the first image may be regarded as a high-resolution (or super-resolution) image. For example, the first image may be a super-resolution (SR) image, and the reference image may be a low-resolution (LR) image corresponding to the first image. As shown in FIG. 10, the SR image represents the first image, and the LR image represents the reference image.

[0202] In some embodiments, the reference image may be used to reconstruct the first image. In some embodiments, some errors (e.g., information loss) may be introduced in the reconstruction. In some embodiments, the reference image may be used as a ground truth for determining the error.

[0203] At 704, the processing device 140 (e.g., the map generation module 408) may determine a convolution kernel based on the first image and the reference image.

[0204] In some embodiments, the convolution kernel may include a resolution scaling function (RSF), a point spread function (PSF), etc. In some embodiments, the convolution kernel may be an RSF. In some embodiments, the RSF may be used to convert a high-resolution image to a low-resolution scale. In some embodiments, the RSF may be defined by a 2D Gaussian function having a parameter σ. Since the RSF is usually anisotropic in the x and y directions, σ may be set as a vector including two elements, i.e., σ x and σ y and may be set as a vector including two elements, i.e., σx and σy. Thus, the function of the RSF may be expressed as follows.

[0205] [Mathematics]

[0206] To discretely calculate the RSF, since the 2D Gaussian function may be integrated over finite pixels, the Gaussian error function (erf) may be given as follows.

[0207] [Mathematics]

[0208] In some embodiments, the discrete mathematical formula for the RSF may be determined as follows.

[0209] [Mathematics]

[0210] At 706, the processing device 140 (e.g., the map generation module 408) may determine an intermediate image based on the first image and the convolution kernel.

[0211] The intermediate image may be regarded as an image with a resolution scaled back from the first image. The intermediate image (e.g., the intermediate image I HL ) may be determined by converting the first image using the convolution kernel. For example, the intermediate image may be determined based on the convolution of the first image and the convolution kernel.

[0212] In some embodiments, four parameters (e.g., μ, θ, σ x , σ y ) may be optimized. μ and θ may be parameters for rescaling the image intensity, and σ x and σ yIt may be a parameter for parameterization of RSF. Rescaling of image intensity may refer to linearly rescaling the intensity of the first image.

[0213] In some embodiments, to globally equalize the intensity between a reference image (e.g., LR image) and a first image (e.g., high-resolution image) and maximize the similarity therebetween, the intensity of the first image I H needs to be linearly rescaled by Equation (23).

[0214]

Equation

[0215] Here, I HS represents the first image after being linearly rescaled, μ represents an intensity parameter, θ represents a parameter associated with the background region, and I H represents the first image. μ and θ in Equation (23) may be selected to maximize the similarity between the reference image I L and the rescaled first image I HS after being convolved with RSF.

[0216]

Equation

[0217] Here, I L represents the reference image, I HS represents the first image after being linearly rescaled, μ represents an intensity parameter, θ represents a parameter associated with the background region,

Equation

[0218] In some embodiments, μ and θ for rescaling of image intensity, and σ x and σy To estimate these four variables (i.e., μ, θ, σ x , σ y ), they may be jointly optimized to minimize the following function.

[0219]

Equation

[0220] In some embodiments, since it may be difficult to calculate the gradient of Equation (25), a derivative-free optimizer may be used to search for the four optimal parameters. Different from Particle Swarm Optimization (PSO), the Pattern Search Method (PSM) may be used for the optimization of Equation (25). PSO may search a relatively large space of candidate solutions, which may not be necessary for optimizing such four parameters. Compared with the unstable and slow metaheuristic optimization approach PSO, PSM may be a stable and computationally efficient way to directly search for solutions and is generally used for small-scale parameter optimization. In some embodiments, PSM may be more suitable for determining the RSM.

[0221] In some embodiments, μ and θ for rescaling the image intensity, and σ x and σ y for RSF parameterization, after being optimized, the high-resolution image (i.e., the first image) I H may be converted to its low-resolution scale (i.e., the intermediate image) I HL by convolving with the estimated RSF. In some embodiments, the intermediate image I HL may be determined by Equation (26).

[0222]

Equation

[0223] In some embodiments, the global quality of the intermediate image may be estimated. In some embodiments, the original low-resolution image (i.e., the reference image) I L For the resolution-scaled back image (i.e., the intermediate image) I HL To evaluate the global quality of, commonly used metrics (e.g., root mean square error (referred to as RSE for resolution-scaling error), and Pearson correlation coefficient (referred to as RSP for resolution-scaling Pearson coefficient)) may be utilized in this operation. RSE and RSP may be expressed as follows.

[0224]

Equation

[0225] In 708, the processing device 140 (e.g., the map generation module 408) may determine a reference map based on the difference between the intermediate image and the reference image.

[0226] In some embodiments, the reference map may include a resolution-scaling error map (RSM). In some embodiments, the reference map may be determined based on the difference (e.g., absolute difference) between the intermediate image and the reference image (see FIG. 10). Thus, the reference map may display at least a portion of the error and / or artifacts in the reconstruction process of the first image. To visualize the absolute difference in pixel units, I HL and I L The reference map RSM between and may be calculated by the following equation.

[0227]

Equation

[0228] Note that the above description of process 700 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. Those skilled in the art will, upon understanding the principle of operation, be able to arbitrarily combine any operations, add or delete any operations, or apply the principle of operation to other image processing processes without departing from the principle. In some embodiments, one or more operations may be integrated into one operation. For example, operations 704 to 708 may be integrated into one operation. In some embodiments, the first image in process 700 may be replaced with a second image.

[0229] FIG. 8 is a flowchart showing another exemplary process for determining a target map according to some embodiments of the present disclosure. In some embodiments, process 800 may be executed by image processing system 100. For example, process 800 may be configured as a set of instructions (e.g., an application) stored in a storage device (e.g., storage device 150, storage 220, and / or storage unit 370). In some embodiments, processing device 140 (e.g., processor 210 of arithmetic device 200, CPU 340 of mobile device 300, and / or one or more modules illustrated in FIG. 4) may be configured to execute the set of instructions and thereby execute process 800 accordingly. The operations of the processes shown below are intended to be illustrative. In some embodiments, process 800 may be achieved with one or more additional operations not described and / or without one or more of the operations described. Further, the order of the operations of process 800 shown in FIG. 8 and described below is not intended to be limiting. In some embodiments, process 800 may be a process for generating an rFRC map (e.g., a second target map) based on a single initial image (e.g., a single frame).

[0230] At 802, the processing device 140 (e.g., the acquisition module 402) may acquire an initial image. As shown in FIG. 12A, Frame 1 represents an initial image with a size of 8×8. In some embodiments, the initial image may be generated by physical imaging. In some embodiments, the initial image may be a single-frame image captured by the image acquisition device 110. The initial image may be generated based on a single measurement of an object. The object may be a biological object or a non-biological object as described elsewhere in the present disclosure.

[0231] In some embodiments, the processing device 140 may acquire the initial image from one or more components of the image processing system 100. For example, the image acquisition device 110 may collect and / or generate the initial image and store it in the storage device 150. The processing device 140 may search for and / or acquire the initial image from the storage device 150. As another example, the processing device 140 may directly acquire the initial image from the image acquisition device 110.

[0232] At 804, the processing device 140 (e.g., the decomposition module 416) may decompose the initial image into a first image, a second image, a third image, and a fourth image. In some embodiments, the first image and the second image at 502 may be acquired according to 804. In some embodiments, the FRC operation may require statistically independent image pairs that share the same details but different noise realizations. In some physical imaging applications such as SMLM, SRRF, SOFI, etc., these modalities may preferably generate statistically independent images by splitting an input image sequence (e.g., the initial image) into two subsets and reconstructing them independently.

[0233] In some embodiments, the processing device 140 may decompose the initial image using a wavelet filter. Exemplary wavelet filters may include, but are not limited to, the Daubechies wavelet filter, the biorthogonal wavelet filter, the Morlet wavelet filter, the Gaussian wavelet filter, the Marr wavelet filter, the Meyer wavelet filter, the Shannon wavelet filter, the Battle-Lemarie wavelet filter, or any combination thereof.

[0234] In some embodiments, since each pixel of the initial image may be sampled independently of the physical imaging, the processing device 140 may divide the initial image into four sub-images (e.g., the first image, the second image, the third image, and the fourth image). In some embodiments, pixels at the (even, even), (odd, odd), (even, odd), and (odd, even) row / column indices of the initial image may be extracted to form four sub-images, respectively. As shown in FIG. 12B, the pixel numbered 1 is at the (odd, odd) row / column index of frame 1 shown in FIG. 12A, the pixel numbered 2 is at the (odd, even) row / column index, the pixel numbered 3 is at the (even, odd) row / column index, and the pixel numbered 4 is at the (even, even) row / column index. As shown in FIG. 12C, the pixels numbered 1, 2, 3, and 4 may be extracted to form four sub-images numbered 1, 2, 3, and 4, respectively. As shown in FIG. 12C, the four dotted boxes may represent the first image (e.g., the dotted box numbered 1), the second image (e.g., the dotted box numbered 4), the third image (e.g., the dotted box numbered 3), and the fourth image (e.g., the dotted box numbered 2), each having a size of 4×4. In some embodiments, the four sub-images may be of the same size. The four sub-images may be used to create two image pairs having the same detailed content but different noise realizations. In some embodiments, the two image pairs may be randomly created from the four sub-images. For example, the first image and the second image may be randomly combined to form an image pair, and the other two sub-images may be combined to form another image pair. In some embodiments, the two image pairs may be formed according to the positions of the sub-image pixels in the initial image.For example, as shown in FIG. 12C, the sub-image numbered 1 having pixels at (odd, odd) row / column indices and the sub-image numbered 4 having pixels at (even, even) row / column indices may form an image pair, and the sub-image numbered 2 having pixels at (odd, even) row / column indices and the sub-image numbered 3 having pixels at (even, odd) row / column indices may form another image pair.

[0235] At 806, the processing device 140 (e.g., the rolling module 406) may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image (see FIG. 12D similar to FIG. 9A). Operation 806 may be similar to operation 506, and the related description will not be repeated here. In some embodiments, the first block and the second block may correspond one-to-one.

[0236] At 808, the processing device 140 (e.g., the map generation module 408) may determine a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks (see FIGS. 12E - 12G similar to FIGS. 9B - 9D). Operation 808 may be similar to operation 508, and the related description will not be repeated here.

[0237] In some embodiments, the first characteristic value may be modified or corrected. In some embodiments, during the FRC operation of a single-frame image (i.e., the initial image), a single pixel displacement in both the x-direction and the y-direction may be generated in the image pair. This spatial displacement may result in frequency-phase modulation, where s = √x0 -i2πsr / N + y0 2 + y0 2 represents the total length of the spatial displacement, and r represents the radius in the FRC operation. Additionally or alternatively, the calibration procedure may be used to correct the determined characteristic values using the following formula:

[0238]

Equation

[0239] Here, r tf represents the corrected characteristic value, and r sf represents the target frequency corresponding to the image pair (i.e., the first image and the second image), and the four parameters a, b, c, d are experimentally adapted (for example, a = 0.9599, b = 0.9798, c = 13.9044, d = 0.5515). In some embodiments, the characteristic value may be corrected to correct such a bias caused by spatial displacement, making the characteristic value more accurate.

[0240] At 810, the processing device 140 (for example, the map generation module 408) may generate a first target map associated with the first image and the second image based on a plurality of first characteristic values. Operation 810 may be similar to operation 510, and the related description will not be repeated here.

[0241] At 812, the processing device 140 (for example, the rolling module 406) may determine a plurality of third blocks of the third image and a plurality of fourth blocks of the fourth image. In some embodiments, similar to the expansion of the first image and / or the second image, the processing device 140 (for example, the expansion module 404) may expand the third image and / or the fourth image before the third block and the fourth block are generated. The processing device 140 may determine a plurality of third blocks of the third image and a plurality of fourth blocks of the fourth image in a manner similar to the determination of the first block and the second block, and the related description will not be repeated here. The plurality of third blocks and the plurality of fourth blocks may correspond one-to-one.

[0242] At 814, the processing device 140 (for example, the map generation module 408) may determine a plurality of second characteristic values based on the plurality of third blocks and the plurality of fourth blocks. Operation 814 may be executed in the same manner as 808, and the related description will not be repeated here.

[0243] At 816, the processing device 140 (e.g., the map generation module 408) may generate a second target map associated with the third image and the fourth image based on a plurality of second characteristic values. The processing device 140 may determine the second target map associated with the third image and the fourth image in a manner similar to the determination of the first target map at 810 or 510, and the related description will not be repeated here.

[0244] At 818, the processing device 140 (e.g., the map generation module 408) may generate a third target map based on the first target map and the second target map.

[0245] The third target map may be regarded as an rFRC map generated from the initial image. In some embodiments, the processing device 140 may generate the third target map based on a weighted sum of the first target map and the second target map. In some embodiments, the processing device 140 may generate the third target map by directly combining the first target map and the second target map. In some embodiments, the processing device 140 may average the first target map and the second target map to obtain the third target map.

[0246] In some embodiments, through the image decomposition at 804, the horizontal dimensions of the four sub-images may be the same and may be half the size of the original image (i.e., the initial image at 802). In some embodiments, the third target map may be resized to the size of the original image (i.e., the size of the initial image) (see FIG. 12H). Alternatively, the first target map and the second target map may be resized to the size of the original image, and the third target map may be generated based on the resized first target map and the resized second target map. In some embodiments, the resizing may be performed using bilinear interpolation.

[0247] Note that the above description of process 800 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. Those skilled in the art, upon understanding the principle of operation, will be able to arbitrarily combine any operations, add or delete any operations, or apply the principle of operation to other image processing processes without departing from the principle. In some embodiments, one or more operations may be integrated into one operation. For example, operations 806 to 814 may be integrated into one operation. As another example, operations 816 and 818 may be integrated into one operation. As a further example, a processing device (e.g., filtering module 410) may filter the first target map, the second target map, and / or the third target map. In some embodiments, one or more operations may be added. For example, an operation of filtering the first target map and an operation of filtering the second target map may be added. As another example, an operation of filtering the third target map may be added. As a further example, an operation of displaying the third target map using a displacement jet color map and / or an operation of determining a global image quality metric of the initial image based on the third target map may be added.

[0248] The present disclosure will be further described according to the following examples, but the examples should not be construed as limiting the scope of the present disclosure.

[0249] Examples Method

[0250] STORM imaging The microscope setup will be described below. After washing with phosphate buffered saline (PBS), in Tris-HCl (pH: 7.5), 5 w / v% glucose, 100×10 -3 M cysteamine, 0.8 mg·mL -1 of glucose oxidase, and 40 μg·mL -1The sample was placed on a slide glass using a standard STORM imaging buffer consisting of catalase. Next, data was collected by 3D-STORM performed in a custom setup using an oil immersion objective lens (100× / 1.45 NA, CFI Plan Apochromat λ, manufactured by Nikon Corporation) with a modified commercially available inverted fluorescence microscope (Eclipse Ti-E, manufactured by Nikon Corporation). Lasers of 405 nm and 647 nm were introduced from the rear focal plane of the objective lens to the cell sample, displaced toward the end of the objective lens, and irradiated for approximately 1 μm within the glass-water interface. The strong (approx. 2 kW·cm -2 ) excitation laser at 647 nm photo-switched most of the labeled dye molecules to the dark state, and simultaneously excited the fluorescence from the remaining sparsely distributed emitting dye molecules to perform single molecule localization. When a weak (typical range: 0 - 1 W·cm -2 ) laser at 405 nm was used in combination with the 647 nm laser to reactivate the phosphor to the emitting state, only a small optically resolvable phosphor was in the emitting state at any given moment. A cylindrical lens was placed in the imaging path to introduce astigmatism, and the depth (z) position was encoded into the ellipticity of the single molecule image. The EMCCD (iXon Ultra 897, manufactured by Andor) camera recorded images at a frame rate of 110 for a frame size of 256×256 pixels, and typically about 50,000 frames were recorded in each experiment. Also, to form 2D-STORM imaging, the cylindrical lens in the optical layout was removed.

[0251] STORM reconstruction is described below. The open-source software package Thunder-STORM and the customized 3D-STORM software were used for STORM image reconstruction. Images labeled as "ME-MLE" and "SE-MLE" were reconstructed by Thunder-STORM with maximum likelihood estimation (integrated PSF method) and multi-emitter fitting possible ("ME-MLE") or multi-emitter fitting impossible ("SE-MLE"). Images labeled as "SE-Gaussian" were reconstructed using the customized 3D-STORM software by fitting the extrema with an (elliptical) Gaussian function. Drift correction was performed after localization, and the images were rendered using a normalized Gaussian function (σ = 2 pixels).

[0252] Cell culture, fixation, and immunofluorescence are described below. COS-7 cells were cultured in DMEM supplemented with 10% fetal bovine serum (FBS) at 37 °C with 5% CO2 in a humidified CO2 incubator according to standard tissue culture protocols. Next, the cells were seeded onto 12 mm glass coverslips in 24-well plates at a ratio of approximately 2×104 cells per well and cultured for 12 hours. For STORM of actin filaments, a previously established fixation protocol was used: the samples were first fixed and extracted for 1 minute in cytoskeleton buffer (CB, 10×10 -3 M MES, pH: 6.1, 150×10 -3 M NaCl, 5×10 -3 M EGTA, 5×10 -3 M glucose, 5×10 -3 M MgCl2) using 0.3 v / v% glutaraldehyde and 0.25 v / v% Triton® X-100, then post-fixed for 15 minutes in 2 (v / v)% glutaraldehyde in CB, and reduced with a freshly prepared 0.1% sodium borohydride solution in PBS. Alexa Fluor 647-conjugated phalloidin at approximately 0.4×10 -6It was applied for 1 hour at the concentration of M. After the sample was briefly washed 2 - 3 times with PBS, it was immediately placed for imaging. For imaging of other targets, the sample was fixed with 3 w / v% paraformaldehyde and 0.1 w / v% glutaraldehyde in PBS for 20 minutes. After reduction with a freshly prepared 0.1% sodium borohydride solution in PBS for 5 minutes, the sample was permeabilized and blocked with blocking buffer (3 w / v% BSA, 0.5 v / v% Triton X - 100 in PBS) for 20 minutes. Then, the cells were incubated with the primary antibody (described above) in the blocking buffer for 1 hour. After washing 3 times with washing buffer (0.2 w / v% BSA and 0.1 v / v% Triton X - 100 in PBS), the cells were incubated with the secondary antibody at room temperature for 1 hour. Then, the sample was washed 3 times with the washing buffer and placed for imaging.

[0253] SIM imaging TIRF-SIM will be described below. A commercially available inverted fluorescence microscope (IX83, manufactured by Olympus) equipped with a TIRF objective lens (100x / 1.7NA, Apo N, HI Oil, Olympus) and a multi-band dichroic mirror (DM, ZT405 / 488 / 561 / 640 phase R, Chroma) was used to construct the SIM system as described above. That is, laser light with wavelengths of 488 nm (Sapphire 488LP-200) and 561 nm (Sapphire 561LP-200, manufactured by Coherent) and an acousto-optic tunable filter (AOTF, AA Opto-Electronic, France) were used to perform laser combination, switching, and adjustment of the irradiation power. A collimating lens (focal length: 10 mm, manufactured by LightPath) was used to couple the laser to a polarization-maintaining single-mode fiber (QPMJ-3AF3S, manufactured by Oz Optics). The output laser was then collimated by an objective lens (CFI Plan Apochromat Lambda 2x NA 0.10, manufactured by Nikon) and diffracted by a pure phase grating composed of a polarization beam splitter (PBS), a half-wave plate, and an SLM (3DM-SXGA, manufactured by ForthDD). The diffracted beam was then focused onto the intermediate pupil plane by another achromatic lens (AC508-250, manufactured by Thorlabs), and a stop mask was placed on the intermediate pupil plane, which was densely designed to block the 0th-order light and other stray light and allow only the ±1st-order light pair to pass through. A homemade polarization rotator was placed behind the stop mask to maximize the modulation of the irradiation pattern while eliminating the switching time between different excitation polarizations. Next, the light passed through another lens (AC254-125, manufactured by Thorlabs) and a tube lens (ITL200, manufactured by Thorlabs) and was focused onto the rear focal plane of the objective lens, interfering with the image plane after passing through the objective lens. The emitted fluorescence collected by the same objective lens passed through a dichroic mirror (DM), an emission filter, and another tube lens. Finally, the emitted fluorescence was split by an image splitter (W-VIEW GEMINI, manufactured by Hamamatsu Japan) and then captured by an sCMOS (Flash 4.0 V3, manufactured by Hamamatsu Japan) camera.

[0254] The Hessian SIM will be described below. A Hessian noise removal algorithm without the t - continuity constraint condition for the result of Wiener SIM reconstruction was applied to obtain the Hessian SIM image shown in FIG. 21E.

[0255] The 3D - SIM will be described below. Using a Nikon 3D - SIM microscope equipped with a TIRF objective lens (100× / 1.49NA, CFI Apochromat, oil, manufactured by Nikon), the 3D - SIM data sets of FIGS. 25A - 25I were obtained.

[0256] The maintenance and preparation of cells will be described below. Human umbilical vein endothelial cells were isolated and cultured in M199 medium (Thermo Fisher Scientific, 31100035) supplemented with fibroblast growth factor, heparin, and 20% fetal bovine serum (FBS), or in endothelial cell medium (ECM) (ScienCell, 1001) supplemented with endothelial cell growth supplement (ECGS) and 10% FBS. To express LifeAct - EGFP, cells were infected with a retroviral system. The transfected cells were cultured for 24 hours, detached using trypsin - EDTA, and seeded onto a cover slip (H - LAF10L glass, reflectance: 1.788, thickness: 0.15 mm, customized) coated with poly - L - lysine, and further cultured in an incubator at 37°C with 5% CO2 for 20 to 28 hours before the experiment.

[0257] LSECs were isolated, plated onto a cover slip coated with 100 μg / ml of collagen, and cultured in high - glucose DMEM supplemented with 10% FBS, 1% L - glutamine, 50 U / ml of penicillin, and 50 μg / ml of streptomycin in an incubator at 37°C with 5% CO2 for 6 hours before imaging. Live cells were incubated with DiI (100 μg / ml, Biotium, 60010) at 37°C for 15 minutes, and fixed cells were fixed with 4% formaldehyde at room temperature for 15 minutes before labeling with DiI.

[0258] STED imaging Image acquisition by STED was performed using a gated STED (gSTED) microscope (Leica TCS SP8 STED 3X, Leica Microsystems, Germany) equipped with a wide-field objective lens (100× / 1.40 NA, HCX PL APO, oil, manufactured by Leica). The excitation wavelength was 647 nm, and the depletion wavelength was 775 nm. All images were acquired using LAS AF software (manufactured by Leica). To label the microtubules of living cells shown in FIGS. 40A to 40C, COS-7 cells were incubated with SiR-Tubulin (Cytoskeleton, CY-SC002) for 20 minutes without washing before imaging.

[0259] PANEL framework In principle, the imaging process can be described as the measurement system observing an object with a transfer function and the corresponding signal being collected by a sensor. Since the signal is actually sampled by a sensor that follows various types of noise, such artificial observations are generally recognized to always deviate from real-world objects in a high-dimensional space (the left panel of FIG. 15). The errors / artifacts included in the reconstruction are mainly caused by such combined effects of the sampling rate and mixed noise when an appropriate reconstruction model is used. In other words, the reconstruction from such inevitably deviated observations may also deviate from real-world objects in the target / super-resolution region (the right panel of FIG. 15). When variables are controlled so that the same object is imaged and independent image pairs are statistically captured, the distance between the ground truth and its reconstruction can be emphasized by the differences between such individual reconstructions (FIGS. 9A to 9D). To quantify such a distance between the recorded image and its ground truth, conventional evaluation methods in the spatial domain, such as the relative error and absolute error used in RSM, are overly sensitive to the intensity and minute movements during measurement. These quantization algorithms can be regarded as "absolute differences" and may induce high false negatives that overwhelm true negatives in the distance map, potentially greatly misleading biologists' recognition of existing errors / artifacts.

[0260] Based on the above analysis, a reference-free approach for measuring such an unmeasurable distance between two signals in the Fourier domain, i.e., Fourier ring correlation (FRC), was introduced. FRC describes the highest allowable frequency component of the distance between two signals. Conventionally, the FRC metric has been widely used as an effective resolution criterion for super-resolution fluorescence microscopy and electron microscopy. It can also be applied to quantify the similarity or distance between two images. Since FRC is calculated as the "relative error" or "error based on saliency" for an image, it has the inherent advantage of being insensitive to intensity changes and small movements. Furthermore, to estimate the most reliable frequency components, FRC is a more quantitative and understandable metric that emphasizes only the saliency error. As a result, FRC becomes an excellent option for quantifying the distance between two signals, while significantly reducing the potential false-negative problem. Notably, considering FRC as the global similarity estimation between two images and aiming to provide more accurate local distance measurements down to the pixel level, the conventional FRC framework was extended into the form of a rolling FRC (rFRC) map (see FIGS. 9A to 9D), enabling quantitative evaluation of image quality at the super-resolution scale. The rolling FRC operation may be like a moving filter on the image plane, and a window that slides for each block of the image is used, and the corresponding FRC value is assigned to each block. First, the input image is symmetrically padded with half the block size to ensure that the FRC operation is performed at the image boundary (see FIG. 9A). Second, an appropriate background threshold may be used for the central pixel (1×1 to 3×3) to determine whether to calculate the FRC value of this block and avoid the FRC operation for the background region. Third, if the average value of the central pixel is greater than the threshold, the FRC operation is executed, and the FRC value is assigned to the central pixel of each block. Otherwise, zero may be set for the central pixel to avoid unnecessary operations for the background region. Then, this procedure may be executed for each block until the entire image is completely scanned.

[0261] Also, to visualize the error map, corresponding metrics, rFRC values, and color maps (displacement jet, or sJet) more suitable for human intuition were developed. The rFRC map can quantitatively map the error of the multidimensional reconstruction signal without prior information of the ground truth and the imaging system. However, it should be noted that (even though such situations are rare in practical experiments) the rFRC method fails and may report false positives when two reconstructed signals lose the same components. If the same area of both images is blank (however, there is content in the ground truth), such information may be lost in the same area, leading to incorrect small FRC values. Therefore, to remove such possible false positives, a modified resolution scale error map (see RSM in Figure 10) is introduced as an additional error map and integrated into the PANEL framework. The complete RSM may introduce high false negatives at a small level due to the aforementioned three hypotheses. To reduce such false negatives included in the RSM, the RSM may be segmented at a 0.5 hard threshold to highlight common large-scale artifacts such as misrepresentation or disappearance of structures before integrating into the final PANEL map (Figures 11A - 11C). On the other hand, the rFRC map indicates the degree of error, and as a result, the minimum FRC value in the map does not necessarily represent the error. To address this situation, a segmentation method called Otsu that automatically determines the threshold by maximizing the inter-class variance is introduced, and image thresholding is performed to filter the background of the original rFRC map to highlight the decisive error of the reconstruction (Figure 16).

[0262] Open source dataset

[0263] In addition to the custom collected dataset, freely available simulation / experimental datasets were used to illustrate the wide applicability of PANEL.

[0264] The 2D-SMLM simulation dataset is described below. As shown in Fig. 17A, as high-density and low-density 2D-SMLM simulation datasets, the "Bundle Tube High Density" (361 frames) dataset and the "Bundle Tube Long Sequence" (12,000 frames) dataset in the "Localization Microscopy Challenge Dataset" on the EPFL website were used. The NA of the optical system was 1.4 (oil immersion objective lens), and the fluorescence wavelength was 723 nm.

[0265] The 3D-SMLM simulation dataset is described below. As shown in Fig. 17C, as a low-density 3D-SMLM simulation dataset, the "MT1.N1.LD" (19,996 frames, 3D astigmatic PSF) dataset in the "Localization Microscopy Challenge Dataset" on the EPFL website was used. The NA of the optical system was 1.49 (oil immersion objective lens), and the fluorescence wavelength was 660 nm. All images were 64×64 pixels (pixel size was 100 nm). Next, every 20 frames from this low-density 3D-SMLM simulation dataset were averaged into 1 frame to generate the corresponding high-density 3D-SMLM simulation dataset (leading to 998 frames).

[0266] The 2D-SMLM experimental dataset is described below. The "Localization Microscopy Challenge Dataset" also includes experimental data. 500 high-density images of tubulin were obtained from the EPFL website (Figs. 18A - 18H). The NA of the optical system was 1.3 (oil immersion objective lens), and the fluorescence wavelength was 690 nm. The images had a frame size of 64×64 pixels (pixel size was 100 nm) and were recorded with a camera at a frame rate of 25 frames per second.

[0267] The live cell SRRF dataset will be described below. GFP-tagged microtubules in live HeLa cells were imaged in TIRF mode using a TIRF objective lens (100× / 1.46 NA, Plan Apochromat, oil, Zeiss), and further irradiated with a 488 nm laser at a magnification of 1.6× (a total of 200 frames). The super-resolution SRRF results were reconstructed using an open-source ImageJ plugin (Figs. 18I–18L).

[0268] Simulation when gridding a sample imaged by SMLM A regular grid with a pixel size of 10 nm (Fig. 17M) was created, and the density of randomly activated molecules was set to gradually increase from the center towards the side. Next, the acquired image sequence was convolved with a Gaussian kernel with a full width at half maximum (FWHM) of 280 nm and downsampled by a factor of 10 (pixel size: 100 nm). As a result, Poisson noise and 20% Gaussian noise were included in the image sequence. Finally, the image sequence was reconstructed by maximum likelihood estimation (integrated PSF method) and Thunder-STORM, which has multi-emitter fitting capabilities.

[0269] Simulation for FPM The United States Air Force (USAF) resolution target was used as the ground truth sample for FPM (Figure 24A). Both the intensity and phase of the imaged sample were set as a USAF target of 240×240 pixels (pixel size: 406.3 nm). It was irradiated from different angles by a 7×7 LED matrix with an emission wavelength of 532 nm and a distance to the sample of 90 mm. The sample was irradiated with each LED unit, filtered with an objective lens (4× / 0.1 NA), and sampled with a camera (the acquired image size was 60×60 and the pixel size was 1.625 μm). After each LED irradiated the sample one by one, a total of 49 low-resolution images were finally acquired. The image illuminated by the central LED was used as the initial image. Next, for each iteration of FPM, the amplitude and phase of the corresponding aperture were updated in sequence. After 10 rounds of iteration, a final high-resolution complex amplitude image (240×240) was acquired, which had a size 4 times larger than the corresponding low-resolution image.

[0270] Data generation process for learning-based applications Sparse sampling will be described below. A deep neural network (DNN) was trained with a sparsely sampled geometric structure and its corresponding intact geometric structure as the ground truth. Four simple and common geometric structures, namely triangles, circles, rectangles, and squares for simulation, were selected. Table 1 shows the spatial size of one structure and the number of structures in one input image. After obtaining the structures, the images were randomly sampled at a sampling rate of 8%. The rectangular structure was selected, and 5000 images were used as the training dataset. For each geometric structure, 200 images were generated as the test dataset.

[0271] The super-resolution of pixels will be described below. The DNN was trained with the RAW image as the ground truth and the corresponding downsampled RAW image as the input. As the training dataset, Drosophila expressing the membrane marker Ecad::GFP that was imaged was selected using a commercially available spinning disk microscope (camera exposure / laser output: 240 ms / 20%). First, PreMosa was used to obtain the surface-projected 2D ground truth. Second, images containing strong backgrounds were discarded. Third, after repeatedly performing wavelet transform to estimate the background, it was subtracted to generate a background-free image as the ground truth. Finally, an image downsampled 4 times with 4×4 pixels as 1 pixel was used as the input. Before inputting to the DNN, the image was upsampled 4 times by bilinear interpolation.

[0272] The deconvolution will be described below. The DNN was trained with the synthetic sub-diffraction image as the ground truth and the corresponding diffraction-limited wide-field image as the input. Specifically, a synthetic tubulin structure was created using a random walk process to simulate a two-dimensional trajectory with randomly changing orientations. On the other hand, the maximum curvature was set as a limited value with respect to the known physical rigidity of tubulin. These images were used as the ground truth with a pixel size of 65 nm and processed to create the input images. These images were convolved with a Gaussian kernel (FWHM: 300 nm) to obtain a blurred image. To simulate a realistic fluorescence background, the blurred image was convolved with a larger Gaussian kernel (FWHM: 1.2 μm) and added to the blurred image. Then, Poisson noise and 2% Gaussian noise were added to the image to create the final input image for DNN training. In the same procedure, 781 more images were generated as the test dataset.

[0273] Noise2Noise is described below. Noise2Noise is an unsupervised learning procedure for removing noise from noisy images without a clear image. During DNN training, only pairs of noisy images (two images with independent noise sharing the same detailed content), i.e., one as the input target and the other as the output target, are looked at. The Fluorescence Microscopy Denoising (FMD) dataset is used for this Noise2Noise task. Fixed zebrafish embryos [EGFP-labeled Tg(sox10:megfp) zebrafish at 2 days post-fertilization] were selected as a dataset imaged with a very low excitation output using a commercially available Nikon A1R-MP confocal laser scanning microscope. There are five noise levels in this image configuration. The RAW image has the highest noise level, and images of other noise levels were generated by averaging RAW images with multiple frames (2, 4, 8, 16) using the circular averaging method. To test extreme conditions, the RAW image with the highest noise level was selected as the input for the training set (two RAW images per pair). For each field of view (a total of 20) with 50 different noise realizations, 200 noise-to-noise data pairs were randomly selected. On the other hand, RAW images of size 512×512 were cut into four non-overlapping patches of size 256×256. Finally, 20×200×4 = 16000 images were obtained as the training dataset. The ground truth reference for evaluating the prediction accuracy of Noise2Noise was generated by averaging 50 noisy RAW images.

[0274] Network Architecture and Training Procedure The network architecture will be described below. This network architecture includes a contraction path and an expansion path and is a so-called U-shaped architecture (Unet). In the contraction path, following the input layer, downsampling blocks consisting of 4×4 kernel convolutions with a stride step of 2, batch normalization (BN), and leaky rectified linear unit (LeakyReLU) functions are consecutive. The convolutional layer is at the bottom of this U-shaped structure connecting the downsampling block and the upsampling block. The expansion pathway combines the feature and spatial information from the contraction path by a series of upsampling blocks (upsampling 2D operation with a stride step of 1 + BN + ReLU + 4×4 kernel convolution) and concatenation with high-resolution features. The last layer is another convolutional layer that maps a 32-channel image to a 1-channel image. In different tasks, two types of U-shaped network architectures (Unet 1 and Unet 2 ) were used (see Fig. 43). The difference between Unet 1 and Unet 2 is that Unet 1 has seven downsampling blocks and seven upsampling blocks, while Unet 2 has four downsampling blocks and four upsampling blocks.

[0275] The learning procedure will be described below. All networks were trained using stochastic gradient descent by adaptive moment estimation (Adam). The detailed input patch size, number of epochs, batch size, number of training images, learning rate, network architecture, number of parameters, and loss function for each task are shown in Fig. 42 and Table 2. Fig. 42 shows the network architecture, learning parameter configuration, and data overview used for different applications. All learning procedures were executed on a local workstation equipped with an NVIDIA Titan Xp GPU card. The relevant learning framework was implemented with TensorFlow framework version 1.8.0 and Python version 3.6.

[0276] Use of Open-Source Deep Learning Models ANNA-PALM will be described below. ANNA-PALM reconstructs super-resolution images from sparsely localized data captured at high speed. ANNA-PALM was trained using high-density sampled PALM images (long sequences) as ground truth and corresponding sparsely sampled PALM images (short sequences) as input. ANNA-PALM is based on cGAN 1 when using Unet as the generator 1 Its performance was tested using 500 high-density images of tubulin from the EPFL website. Phosphors in frames 1-25 and 26-50 were localized using the ME-MLE estimator to construct two sparse super-resolution inputs, after which ANNA-PALM predicted the corresponding high-density sampled images.

[0277] CARE will be described below. The CARE framework is a computational approach that can extend the spatial resolution of microscopes using the Unet 3 architecture. The first and second RAW images from the open-source SRRF dataset were filled into the open-source pre-trained model of CARE to generate the corresponding two super-resolution images.

[0278] Cross-modal super-resolution will be described below. The cross-modal imaging ability of DNNs was previously demonstrated by mapping TIFR to the TIRF-SIM modality (TIRF2SIM) using the cGAN approach. cGAN in TIRF2SIM 2 is based on Unet with residual convolutional blocks (Res-Unet) as the generator. It was trained and tested using the AP2-eGFP-tagged classlin of gene-edited SUM159 cells. The results were directly reproduced using the provided ImageJ plugin and sample data.

[0279] Rendering and processing of images A custom-developed color map, the displacement jet (sJet) color map, was used to visualize the rFRC map. The color map SQUIRREL-Error was used to present the RSM in Fig. 17P, Fig. 28F, the lower right of Figs. 17A - 17B, Fig. 18G, and Fig. 33H. The jet projection was used to show depth in Figs. 23D and 25A. All data processing was achieved using MATLAB® and ImageJ. All figures were created using MATLAB®, ImageJ, Microsoft Visio, and OriginPro.

[0280] Data availability All data supporting the results of this study are available from the corresponding author upon request.

[0281] Code availability The latest versions of the custom-described MATLAB® libraries (with user manuals and sample data), custom-designed color maps, and corresponding ImageJ plugins used in this disclosure are available as software. The updated version of the PANEL in MATLAB® library can be found at https: / / github.com / WeisongZhao / PANELM. The updated version of the ImageJ plugin and its source code can be found at https: / / github.com / WeisongZhao / PANELJ. (Example 1)

[0282] Verification of PANEL by SMLM simulation To evaluate the ability of the PANEL to identify fine defects contained in the super-resolution image, a 2D single molecule localization microscopy (SMLM) simulation dataset from the EPFL challenge was used. The 2D-SMLM simulation datasets (e.g., high density (HD) 2D-SMLM simulation dataset, and low density (LD) 2D-SMLM simulation dataset) each contained an HD emitter and an LD emitter (see FIGS. 17A-17F). FIGS. 17A-17F show simulations of 2D-SMLM using a high density (HD) emitter and a low density (LD) emitter. As shown in FIGS. 17A-17D, the integrated ground truth structure was represented by a dark gray channel labeled HD-GT or LD-GT, and the maximum likelihood estimation (MLE) reconstruction was represented by a light gray channel labeled MLE. FIGS. 17B-17E show the rFRC maps of the MLE, respectively. FIGS. 17C-17F show the complete PANEL of the MLE. The RSM was represented as the channel indicated by the white arrow, and the rFRC map was represented as the other channels. The white arrow and the light gray arrow indicated the errors found by the RSM or the rFRC map, respectively. In this quantitative mapping, one set of images was split into two statistically independent subsets to obtain two independent super-resolution images. After using the maximum likelihood estimation (MLE) reconstruction, the corresponding two super-resolution images of the HD 2D-SMLM simulation dataset were obtained. Next, a portion of the filament was artificially removed to create known structural artifacts as indicated by the white arrows in FIGS. 17A-17F, and the potential false positives induced by considering only the rFRC map were visualized. As indicated by the light gray arrow, it was found that the rFRC map could detect all subtle errors (rFRC value: 0.66), but could not detect the missing structures in both super-resolution frames. Therefore, as described above, to verify all errors in the reconstruction, the RSM map and the rFRC map were integrated and represented as the channel indicated by the white arrow and the other channels, respectively (FIGS. 17A-17F), and the final full PANEL map was created.On the one hand, when compared with the ground truth, it was found that the MLE results of the LD 2D-SMLM simulation dataset had almost no error (rFRC value: 0.16), and the corresponding rFRC maps also realistically described such appearances during the calculation process (Figs. 17A - 17F).

[0283] As a result of comparing each LD and HD 2D-SMLM simulation dataset, it became clear that the performance of the SMLM results was related to the density of the luminescent phosphor, which may greatly depend on the induced irradiation intensity. To further quantitatively evaluate the influence of the luminescence density in the MLE reconstruction within a single-frame image, a regular grid with high luminance in the center and decreasing irradiation intensity towards the ends was created, and it was shown that the blinking in the center was temporally separated from that at the ends (Fig. 17G). Fig. 17G is a diagram showing the MLE results of the 2D-SMLM simulation using inhomogeneous irradiation (high intensity in the center and decreasing irradiation towards the ends). Fig. 17H shows the rFRC map generated based on Fig. 17G. As shown in Fig. 17H, it is clearly shown that the rFRC map accurately describes the reconstruction performance induced by the blinking density transition. In contrast, since the hypothesis of RSM is not satisfied, the estimated RSM cannot appropriately present the error (even if it is opposite to the reference) (Figs. 17M - 17P). Fig. 17M shows the simulated ground truth. Fig. 17N shows the wide-field image of Fig. 17M with high luminance in the center and decreasing irradiation towards the ends. Fig. 17O shows the image after convolving the reconstructed MLE image of Fig. 17G with the estimated RSF. Fig. 17P shows the RSM of the reconstructed MLE image of Fig. 17G.

[0284] In the final step, the RSM was removed from the PANEL for 3D super-resolution imaging (Figs. 17I - 17L), and the rFRC map was directly extended to the 3D version by applying per-plane operations as a limitation in 3D model evaluation. Fig. 17I shows an image depicting the integrated ground truth structure (red channel labeled "LD-GT") and MLE reconstruction (green channel labeled "3D-MLE"). Fig. 17J shows the rFRC map of the low-density 3D-MLE with an rFRC value of 2.2. Fig. 17K shows an image depicting the integrated ground truth structure (red channel labeled HD-GT) and MLE reconstruction (green channel labeled 3D-MLE). Fig. 17L shows the rFRC map of the high-density "3D-MLE" with an rFRC value of 4.5. The 3D simulation dataset used for the verification of the PANEL was obtained from the EPFL SMLM challenge, including both LD and HD cases (averaged over 20 frames from the LD dataset). Similar to the 2D case, the rFRC results indicate that the global performance of 3D-MLE reconstruction is significantly affected by the density of the emitting phosphors, which is in good agreement with all actual physical experiences (see the cases of rFRC values of 2.2 vs. 4.5 for 3D, LD, and HD in Figs. 17I - 17L, Fig. 17Q, and Figs. 17X, 17Y). Fig. 17Q shows, from left to right, respectively, the ground truth of the low-density (LD) and high-density (HD) datasets and the color-coded diagrams of 3D-MLE reconstruction in dark colors. Fig. 17X shows, from left to right, respectively, the horizontal cross-sections (z position 0 nm) of the 3D-MLE reconstructions (LD and HD) and a representative frame (frame 19) of the LD dataset. Fig. 17Y shows, from left to right, respectively, the rFRC map of the horizontal cross-section of Fig. 17X and a representative frame (frame 99) of the HD dataset.

[0285] Also, although the rFRC map can detect all minute and subtle errors, it has been found that this method is impossible when the structures are missing simultaneously in both adjacent super-resolution frames. To clarify, an example is created using the 2D SMLM challenge dataset to explain this possible false negative (see FIGS. 17A-17C and FIGS. 17R-17T). FIG. 17R shows the rFRC map after Otsu threshold filtering for FIG. 17B. FIG. 17S shows the RSM. FIG. 17T shows the RSM after 0.5 threshold filtering. As shown in FIG. 17B, the rFRC map successfully detects most of the error components throughout the image, except for the artificially removed regions. However, the disappearance of such structures can be detected normally by the RSM (see FIG. 17R). Therefore, to address this possible false positive, the RSM was introduced as an error map added to the rFRC map. Also, it should be noted that such lost identical information in the two reconstructions includes large grades and may exist in large regions of the RSM. Furthermore, according to three hypotheses of the RSM, small-grade components included in the RSM may introduce high false negatives. Therefore, the RSM was segmented using 0.5 threshold filtering before involving the RSM in the PANEL. After this operation, low false negatives are accurately filtered, leaving only strongly low-resolution error components and focusing on the true negatives detected by the RSM.

[0286] On the one hand, in practice, since the rFRC map indicates the degree of error, the minimum FRC value in the map may not indicate the tolerance error. Similarly, a segmentation method called Otsu is introduced, which automatically determines the threshold by maximizing the between-class variance, performs thresholding of the image to filter the background of the rFRC map, and emphasizes the decisive error of reconstruction (Figs. 17U to 17W). Subsequently, the segmented rFRC map is integrated as a light gray channel, and the RSM is segmented as a dark gray channel to create a complete PANEL, and the corresponding integrated error map of the reconstruction is visualized (Figs. 17A to 17C, Figs. 17R to 17T, and Figs. 17U to 17W). It should be noted that when the dataset is in a three-dimensional or non-Gaussian convolution relationship (between the low-resolution scale and the high-resolution scale), the corresponding RSM cannot be estimated. As a result, in such a dataset, the RSM may not be involved in the PANEL. Fig. 17U shows the rFRC map of the same SRRF dataset as Figs. 18I to 18L, displayed as a displacement jet color map. Figs. 17V and 17W show the full rFRC map and the rFRC map after Otsu thresholding to emphasize the decisive error in the SRRF reconstruction shown by the green channel of the PANEL. (Example 2)

[0287] Minimization of analysis error by PANEL After synthesizing a ground truth image that includes a grid structure and a basic simulation, an experimental 2D-SMLM dataset from the EPFL challenge was evaluated using the proposed PANEL (Figs. 18A - 18H). Fig. 18A shows an image depicting the MLE localization results of 500 high-density images of tubulin from the EPFL website. Fig. 18B shows the rFRC map of "MLE". Fig. 18C shows the complete PANEL of "MLE" with an rFRC value of 1.2. Fig. 18D shows an image depicting the PANEL after Otsu threshold segmentation. Fig. 18E shows the corresponding "wide field" image. Fig. 18F shows the "MLE" image deconvolved back to the original low-resolution scale. Fig. 18G shows an image depicting the RSM of "MLE". Fig. 18H shows the FRC map of "MLE" with an FRC value of 584 nm. Based on the obtained corresponding rFRC maps, it was found that large FRC values tend to appear in the intersection regions of filaments, which is consistent with the SRRF results revealed in Figs. 18I - 18L. Fig. 18I shows a diffraction-limited TIRF image. Fig. 18J shows an image depicting the SRRF reconstruction results of 100 noisy images (see method for GFP-tagged microtubules in living HeLa cells). Fig. 18K shows the rFRC map of "SRRF" with an rFRC value of 2.25. Fig. 18L shows an image depicting the PANEL after Otsu threshold segmentation. The main reason considered is the influence of the emission density in 2D-SMLM simulations, and the intersection regions contain a relatively large emitter density, leading to a decrease in localization performance. In the actual experimental process, it has been experimentally recognized that the emission density (related to the complexity of the structure) varies significantly within the field of view, but there is no practical quantification method to identify the tiny errors at the super-resolution scale. So far, most existing algorithms have to accept such a performance trade-off by designing to consider only a uniform density instead of HD or LD.

[0288] The high-resolution error mapping function can integrate the advantages from existing HD or LD focusing algorithms respectively, thereby minimizing the errors included in all selected methods. To verify this description, a 2D-STORM dataset of immunolabeled α-tubulin in fixed COS-7 cells was analyzed using two different algorithms, namely multi-emitter MLE (MEM-LE) and single-emitter Gaussian fitting (SE-Gaussian) (see the methods in FIGS. 20A and 19A-19E). FIG. 19A shows an image depicting the reconstruction of multi-emitter MLE (ME-MLE). FIG. 19B shows an image depicting single-emitter Gaussian fitting (SE-MLE). FIG. 19C shows an image depicting the fusion result of ME-MLE and SE-MLE by the rFRC map. FIG. 19D shows the corresponding rFRC maps of FIGS. 19A-19C. FIG. 19E shows an enlarged view of the white box in FIG. 19A. FIG. 20A shows an image depicting the fused STORM result (COS-7 cells, α-tubulin labeled with Alexa Fluor 647) from the reconstructions by multi-emitter MLE (ME-MLE) and single-emitter Gaussian fitting (SE-MLE). FIG. 20B shows the rFRC map of FIG. 20A. FIGS. 20C-20E show enlarged images of the region enclosed by box 2001 in FIG. 20B, among which FIG. 20C shows an image depicting the result of the rFRC map, FIG. 20D shows an image depicting the result of the fused STORM, and FIG. 20E shows an image depicting the result of the RSM. FIGS. 20F-20H show enlarged images of the box 2002 in FIG. 20A, and FIGS. 20I-20K show enlarged images of the box 2003 in FIG. 20B, among which FIG. 20F shows an image depicting the result of ME-MLE, FIG. 20G shows an image depicting the result of SE-MLE, FIG. 20H shows an image depicting the result of the fused STORM, FIG. 20I shows the rFRC map of FIG. 20F where the rFRC value is 1.01, FIG. 20J shows the rFRC map of FIG. 20G where the rFRC value is 4.51, and FIG. 20K shows the rFRC map of FIG. 20H where the rFRC value is 0.87. FIG. 20L shows an enlarged view of the dashed circle in FIGS. 20F-20H.SE-Gaussian is more suitable for the reconstruction of simple structures (LD emitters), while in contrast, ME-MLE is more suitable for complex structures (HD emitters) as shown in FIGS. 20C - 20K. By utilizing the rFRC error map at the super-resolution scale, the spatial details regarding the local accuracy of each algorithm can be mapped, which can be converted into fusion weights (FIGS. 20B - 20E), and then the lowest error features of each reconstruction can be used to generate a new synthetic image with minimized defects. As expected, as shown in the two selected regions of interest (FIGS. 20F - 20K), it can be seen that the individual algorithms (ME-MLE or SE-Gaussian only) cannot achieve stable performance for all structures across the entire field of view. By combining the information generated by ME-MLE and SE-Gaussian using the proposed rFRC map, the advantages of both algorithms can be exploited to achieve excellent performance across the entire field of view (FIGS. 20C - 20K, FIG. 20M). Also, such a fusion approach is further applied to the 2D-STORM dataset of pits (CCP) coated with the heavy chain class of clathrin in COS-7 cells, where the average resolution is significantly improved (FIG. 20N). FIG. 20M shows the rFRC map of ME-MLE, the superiority map of fusion, and the TIRF image from the upper left to the lower left, and the rFRC map of SE-Gaussian, the fusion result, and the fusion result (Fused) from the upper right to the lower right. It was found that the ME-MLE method achieved excellent performance in regions with strong backgrounds, while in comparison, the SE-Gaussian method obtained better reconstruction quality in regions with weak backgrounds. FIG. 20N shows the enlarged results of a single CCP of ME-MLE, SE-Gaussian, and fusion from the upper left to the lower left, and the corresponding rFRC maps on the right side. The average resolution is described at the upper left of the rFRC map. As emphasized in FIG. 20M, in addition to stable fusion performance across the entire field of view, the rFRC map also helps in the fusion of microstructures such as such a single ring-shaped CCP, achieving a higher average resolution. (Example 3)

[0289] Diverse physics-based imaging applications assisted by rFRC maps The designed rFRC was identified and operated on a typical 3D-STORM reconstruction (Figure 21A), and the resulting rFRC map (Figure 21B) and its PANEL map (Figure 21C) showed that large errors occurred mainly between intertwined filament voxels (Figures 22A - 22C). Figure 21A shows an image depicting a 3D-MLE reconstruction (COS-7 cells, α-tubulin labeled with Alexa Fluor 647). Figure 21B shows the 3D rFRC map of Figure 21A. Figure 21C shows an image depicting the PANEL after Otsu thresholding of Figure 21B. Figure 21D shows an image depicting a curve of rFRC values along the axial position. Figure 21E shows representative images of live human umbilical vein endothelial cells (HUVEC) labeled with LifeAct-EGFP under Wiener SIM (top), Hessian SIM (middle), and TIRF (bottom) imaging. Figure 21F shows the rFRC map of Hessian SIM, and for the rFRC values, RSP values, and RSE values of Wiener SIM and Hessian SIM, rFRC = 1.24, RSP = 0.98, and RSE = 0.27. Figure 21G shows an image depicting representative results of fixed liver sinusoidal endothelial cells (LSEC) labeled with DiI under RL deconvolution (top) and TIRF (bottom) imaging. Figure 21H shows the rFRC map of the RL deconvolution result. Figure 21I shows an enlarged view of the box in Figure 21G, including the original TIRF image (upper left), images depicting the RL deconvolution results for 80 iterations and 200 iterations (upper right and lower left), and an image depicting the TIRF-SIM result (lower right). Figure 21J shows an image depicting curves of PSNR (versus TIRF-SIM), RSP (versus TIRF), and rFRC values along the iterations. Figure 22A shows an image depicting the maximum intensity projection (MIP) image of the 3D-MLE reconstruction. Figure 22B shows an image depicting the MIP image of the rFRC volume of Figure 22A. Figure 22C shows an image depicting the corresponding enlarged horizontal cross-sections of 3D-MLE (left) and rFRC volume (right) from the white box in Figure 22A. Also, in the experimental setup, it was found that the most accurate plane at the reconstruction depth of microtubules (Figure 21D) and actin filaments (Figures 23A - 23F) was located at the focal position.FIG. 23A shows an image showing a maximum intensity projection (MIP) image of TIRF (COS-7 cells labeled with Alexa Fluor 647-phalloidin). FIG. 23B shows an image showing a color-coded diagram of a 3D-MLE reconstruction in dark colors. FIG. 23C shows an image showing a horizontal cross-section of a 3D-MLE reconstruction at a z-position of -50 nm. FIG. 23D shows the corresponding rFRC map of FIG. 23C. FIG. 23E shows an image showing a horizontal cross-section of a 3D-MLE reconstruction at a z-position of +300 nm. FIG. 23F shows the corresponding rFRC map of FIG. 23E.

[0290] The Hessian noise removal algorithm (Hessian SIM) can be used for conventional Wiener SIM to separate random discontinuous artifacts from the actual structure (see FIGS. 21E and 29A-29I). FIG. 29A shows an image of Wiener SIM. FIG. 29B shows the corresponding rFRC map. FIG. 29C shows the corresponding RSM. FIG. 29D shows an image of Hessian SIM. FIG. 29E shows the corresponding rFRC map. FIG. 29F shows the corresponding RSM. FIG. 29G shows TIRF images of Wiener SIM and Hessian SIM. FIG. 29H shows a TIRF image convolved with a large Gaussian kernel and encoded with an inverted sJet color map. FIG. 29I shows an integrated image of PANEL (light gray channel) and Wiener SIM (dark gray channel). However, since conversion to the TIRF mode is required, the RSM may miss such artifacts, as can be seen from FIG. 21E where the RSE value remains 0.27. Conversely, when using the rFRC map, the improvement of Hessian SIM over conventional Wiener SIM can be detected, as can be seen from FIG. 21F where the corresponding rFRC value is significantly improved from 1.36 to 1.24.

[0291] Richardson-Lucy (RL) deconvolution has been actively studied for many reasons, including its potential to improve the resolution and contrast of RAW images. However, conventional RL algorithms may generate artifacts when performing excessive iterations, severely limiting their applications. In general usage of RL, tedious visual inspection is required to determine the optimal number of iterations. Here, to confirm the readout of the rFRC value that leads to such a decision for RL iteration count, after RL is applied to process TIRF images (FIGS. 21G, 30A-30I), the associated rFRC values are calculated for each iteration case (the right panels of FIGS. 21H and 21J). FIG. 30A shows the results of TIRF. FIGS. 30B and 30C show the results of RL deconvolution with 80 iterations and 200 iterations, respectively. FIG. 30D shows the corresponding TIRF-SIM image. FIGS. 30E and 30F show the rFRC maps of FIGS. 30B and 30C. FIG. 30G shows a TIRF image convolved with a large Gaussian kernel and color-coded with an inverted sJet color map. The inverted illumination intensity map (FIG. 30G) is proportional to the rFRC map (FIG. 30E), indicating that the degree of error in the results of RLD has a high correlation with the SNR. FIGS. 30H and 30I show the panels of FIGS. 30B and 30C. Interestingly, when the curve distribution of the peak signal-to-noise ratio (PSNR in the left panel of FIG. 21J) for which the corresponding TIRF-SIM image is used as the ground truth is compared, the curve of the rFRC value shows a similar distribution (quadratic form, with the minimum value appearing at 80 iterations). In contrast, the curve of the resolution-scaled Pearson coefficient (RSP) clearly cannot represent such a beneficial distribution (the middle panel of FIG. 21J). As shown in FIG. 21I, RL deconvolution with 200 iterations generated artifacts such as snow crystals as indicated by the white arrow, which was verified to be absent in the referenced TIRF-SIM image. Through comprehensive comparison, it was demonstrated that RL with 80 iterations optimally improves the contrast of the image while minimizing artifacts due to noise amplification.

[0292] Not only typical fluorescence modalities, but also coherent computational imaging techniques, such as Fourier ptychography microscopy (FPM), have been developed using PANEL and can be effectively evaluated by fully achieving their practical potential (see the methods in FIGS. 24A - 24F). FIG. 24A shows an image depicting a simulated ground truth. FIG. 24B shows the wide - field image of FIG. 24A. FIG. 24C shows an image depicting the corresponding FPM reconstruction. FIG. 24D shows the rFRC map of FPM. FIG. 24E shows an image depicting the RSM of FPM. FIG. 24F shows an integrated image of PANEL (light gray channel) and FPM (dark gray channel). Also, the proposed PANEL framework has been extended to a single - frame version for the purpose of accessing the image quality of modalities when restricted to forming statistically independent image subsets, such as 3D - SIM (see FIG. 25). FIG. 25A shows an image depicting the color - coded volume of HUVEC transfected with LifeAct - EGFP and imaged with a Nikon 3D - SIM microscope. FIGS. 25B - 25C show the 3D - rendered rFRC map displayed with an sJet jet color map (FIG. 25B) and after Otsu thresholding in the green channel (FIG. 25C). FIGS. 25D - 25F show the 3D - SIM image (FIG. 25D), the corresponding rFRC map (FIG. 25E), and its PANEL result (FIG. 25F) at an axial position of 500 nm. FIGS. 25G - 25I show the 3D - SIM image (FIG. 25G), the corresponding rFRC map (FIG. 25H), and its PANEL result (FIG. 25I) at an axial position of 700 nm. As indicated by the white arrow (FIG. 25I), it was found that there may be large errors in regions containing thick actin stress fibers (or strong stress fibers) that scatter and distort the structured incident light. (Example 4)

[0293] Verification of PANEL by simulation of a learning - based approach Deep learning algorithms can learn effective representations that map high-dimensional data to an array of outputs. However, these mappings often lack a reasonable explanation and are thus blindly assumed to be accurate. Therefore, characterizing the uncertainty of the corresponding network predictions and representations is extremely important for further profiling, especially in applications where safety is of utmost importance. In general, to obtain uncertainty for such learning-based approaches, in the framework of traditional Bayesian neural networks, it is necessary to make significant changes to the existing learning procedure by learning the distribution over the weights. However, a potential drawback of BNNs is that their applications are considerably more complex and computationally costly compared to the original learning-based techniques. To alleviate this conflict, the concept of PANEL is transplanted into the learning-based model with the intention of being able to identify the subtle and irregular errors generated by the black-box deep neural network.

[0294] First, four types of simple structures are created and sampled in a sparse form (see Fig. 26). Fig. 26 shows an image depicting a complete sparse sampling simulation, where the rectangle (top) is used as the training dataset, and other geometric shapes (square, circle, triangle) (from top to bottom) are used as the test datasets. "Input" means a representative sparsely sampled input of the corresponding geometric shape, "Prediction 1" means the network prediction of the "Input", "Ground Truth" means before sparse sampling, "GT + Prediction" means the integrated image of the ground truth in the green channel and the prediction result in the red channel, "Prediction 1 + Prediction 2" means the integrated image of the two prediction results, and "Integration" means the integrated image of the PANEL in the green channel and the prediction result in the white channel. The corresponding input images for "Prediction 1" and "Prediction 2" are sampled independently. To investigate the generalization effect, the obtained structures that are either sparsely sampled (used as model inputs) or not sparsely sampled (used as ground truth) are used as data pairs to verify the applicability of PANEL in a learning-based approach. The rectangular shape is used as the training dataset, and the square, triangle, and circle are used as the test datasets. As expected, the square can be considered a subset of the rectangle, and thus the network trained with the rectangular shape showed good results for the square dataset. Interestingly, as can be seen from Fig. 27A, when the network is faced with data not presented in the learning set (out-of-distribution data), such as triangular and circular shapes, the returned results still resemble a rectangular-like shape. Fig. 27A shows an image depicting representative integrated images (Prediction 1: cyan channel, Prediction 2: magenta channel) of a rectangular structure, a square structure, a triangular structure, and a circular structure, as shown from left to right. Fig. 27B shows an image depicting the average IoU values by prediction against the ground truth. Fig. 27C shows an image depicting the average IoU values between the two predictions. Fig. 27D shows an image depicting the average rFRC values between the two predictions. Fig. 27E shows an image depicting the corresponding ones of the representative results (top left) after pixel super-resolution ("PSR") and the undersampled input (bottom right).Figure 27F shows an image indicating the ground truth (「GT」) standard of Figure 27E. Figure 27G shows the rFRC map of the PSR result. Figure 27H shows the integrated image of PANEL (green channel) and the PSR result (gray channel). Figure 27I shows an image indicating a typical result of the deconvolution network. Figure 27J shows an image indicating the corresponding ground truth (「GT」) of Figure 27I. Figure 27K shows an image indicating an enlarged view from the white boxes of Figures 27I and 27J. Figure 27L shows the integrated image of PANEL (green channel) and the deconvolution result (gray channel).

[0295] To verify the ability of PANEL to detect such structural artifacts (model errors) introduced by the deep learning framework, the same structure was sampled individually twice to generate two predictions (denoted as prediction 1 and prediction 2 in Figure 27A). To directly evaluate the difference between the two predictions, the metric common set was selected instead of the set operation (IoU, also known as the Jaccard index). As can be seen from Figures 27B and 27C, the average IoU value between prediction 1 and prediction 2 showed the same distribution as in the case between the ground truth and prediction 1. Furthermore, Figure 27D clearly shows that the pattern of the averaged rFRC values is strikingly similar to one of the IoU values, verifying that the proposed concept and the rFRC metric have the ability to quantitatively analyze the learning-based approach.

[0296] Next, to verify the PANEL in more detail, another simulation verification, namely pixel super-resolution (PSR in FIGS. 27E and 31A - 31G), is also performed to evaluate the usability for learning - based applications. FIG. 31A shows the result after pixel super - resolution (PSR) (upper left) and the undersampled input (lower right). FIG. 31B shows an enlarged view of the undersampled input in the white box of FIG. 31A. FIG. 31C shows the network prediction of FIG. 31B. FIG. 31D shows the corresponding ground truth ("GT") of FIG. 31B. FIG. 31E shows the rFRC map of FIG. 31C. FIG. 31F shows the estimated RSM of FIG. 31C. FIG. 31G shows the integrated image of the PANEL (light gray channel) and the PSR result (dark gray channel). In this case, the corresponding deep neural network was trained by data pairs when the RAW image was used as the ground truth and the corresponding downsampled one was regarded as the input image. Compared with the ground truth (FIG. 27F), as indicated by the white arrow for the target artifact area, it is fully verified that the rFRC map (FIG. 27G) can accurately detect the error included in the network prediction (FIG. 27H) and provide a quantitative evaluation at the pixel level.

[0297] Also, for convenient applications, a special single-frame version of the rFRC operation strategy is also involved in learning-based applications. Another learning-based task, i.e., deconvolution, is used to evaluate such a strategy (Figs. 27I, 32A - 32J). Fig. 32A shows an input image. Fig. 32B shows the DNN result. Fig. 32C shows the ground truth (GT) of Fig. 32A. Fig. 32D shows an enlarged view from the white-boxes of Figs. 32A - 32C. Fig. 32E shows the RSM of Fig. 32A. Fig. 32F shows an integrated image of PANEL (white) and the DNN result (gray). Fig. 32G shows the inverted SSIM map of Figs. 32B and 32C. Fig. 32H shows the deconvolution of Fig. 32B. Figs. 32I and 32J show the rFRC maps of the DNN results from the input images with 1% and 1.5% additive Gaussian noise. TN is a true negative, FN is a false negative, and FP is a false positive. Intuitively, in contrast to the ground truth (Figs. 27J - 27K), the single-frame strategy also showed excellent quantitative performance for accessing image quality, as indicated by the white arrow (Fig. 27L). (Example 5)

[0298] Reliability of PANEL in various learning-based imaging methods After analysis and consideration, the usability of the PANEL was then applied to evaluate the performance of three typical open-source learning-based models, including the artificial neural network-accelerated PALM17 (ANNA-PALM in FIG. 28A), content recognition image restoration (CARE in FIG. 28E), and TIRF conversion to TIRF-SIM (TIRF2SIM in FIG. 28I). FIG. 28A shows an image showing the ANNA-PALM output (MLE result by 25 frames of tubulin as input). FIG. 28B shows an image showing the MLE reconstruction by full 500 frames. FIG. 28C shows the rFRC map of FIG. 28A. FIG. 28D shows an integrated image of the PANEL (green channel) and the ANNA-PALM result (gray channel). FIG. 28E shows an image (raw TIRF image as input) showing the CARE output of GFP-tagged microtubules in live HeLa cells. FIG. 28F shows an image showing CARE deconvolved inversely into the original low-resolution scale (top) and its RSM (bottom). FIG. 28G shows the corresponding TIRF image. FIG. 28H shows an integrated image of the PANEL (green channel) and the CARE result (gray channel). FIG. 28I shows an image showing the TIRF2SIM result of CCP (gene-edited SUM159 cells expressing AP2-eGFP). FIG. 28J shows the TIRF-SIM image. FIG. 28K shows an image showing the TIRF input. FIG. 28L shows an integrated image of the PANEL (green channel) and the TIRF2SIM result (gray channel). FIG. 28M shows an image showing the Noise2Noise result of EGFP-labeled Tg(sox10:megfp) zebrafish at 2 days post-fertilization. FIG. 28N shows a ground-truth reference image generated by averaging 50 noise images with the same content. FIG. 28O shows an image showing a representative noise input. FIG. 28P shows an integrated image of the PANEL (green channel) and the Noise2Noise result (gray channel).

[0299] In the case of ANNA-PALM, the sparse MLE reconstruction (including 25 frames) is used as the input image to the network (Figs. 28A, 33A - 33I), and rFRC is utilized in combination with the RSM for complementarity. Figs. 33A - 33C show the MLE reconstruction by 25 frames (Fig. 33A), the MLE reconstruction by full 500 frames (Fig. 33B), and the corresponding ANNA-PALM output (MLE result by 25 frames as input) (Fig. 33C). Figs. 33D - 33I show the integrated results of MLE (gray) and ANNA-PALM (white) (Fig. 33D), the PANEL result (Fig. 33E), the TIRF image (Fig. 33F), the deconvolved ANNA-PALM (Fig. 33G), the RSM (Fig. 33H), and the rFRC map (Fig. 33I). When compared with the ground truth high-density MLE reconstruction result (including full 500 frames in Fig. 28B), the ability of the rFRC map (the white channel in Figs. 28C and 28D) to normally divide minute errors is indicated by the cyan arrow. On the other hand, as indicated by the white arrow, the RSM (the white channel in Fig. 28D) found large missing structures. Based on these practical results, the integrated PANEL map can effectively detect all types of errors at different scales included in the ANNA-PALM method.

[0300] In the inverse convolution application of CARE20, considering that the output (Figure 28E) and input (Figure 28G) of the network do not satisfy the Gaussian convolution hypothesis, in this case, the RSM cannot accurately convert the output to its low-resolution scale (Figures 28F, 34A - 34F), and possible false negatives are indicated by purple circles. Figure 34A shows the input of the TIRF image. Figure 34B shows the CARE prediction of Figure 34A. Figure 34C shows the rFRC result color-coded in full color. Figure 34D shows the integrated image of the PANEL (light gray channel) and the CARE result (white channel). Figure 34E shows the CARE prediction deconvolved by the estimated RSF. Figure 34F shows the result of the RSM of Figure 34B. The RSF is the resolution scaling function. The RSM is the resolution scale error map. Conversely, as indicated by cyan circles, obvious non-biological structures occurring near the field of view boundary are effectively detected by the rFRC map (green channel in Figure 28H). Applying the single-frame rFRC calculation strategy, it can be seen that the TIRF2SIM19 application (Figures 28I - 28L) can also be evaluated by the PANEL. As shown in the insets of Figures 28I - 28K, two adjacent CCPs of TIRF-SIM (indicated by two green circles in Figure 28J) are incorrectly reconstructed by the network into a single large CCP (indicated by one white circle in Figure 28I), and the rFRC map (white channel in Figure 28L) approach accurately separates such refined errors.

[0301] The performance of the PANEL is further tested to access the performance of non-super-resolution learning-based applications, such as noise removal tasks. Noise2Noise38 is a widely used teacherless learning-based approach that has the advantage of removing noise from noisy images without a clear image. In such a teacherless learning model, during learning, the network only sees pairs of noisy images, and two images with independent noise share the same detailed content. In other words, one image is used as the input target and the other image is used as the output target. In this section, 59 fluorescence microscopy noise removal (FMD) datasets are involved in the learning of the Noise2Noise network (see the methods in FIGS. 28M to 28P and FIGS. 35A to 35E). Since the RSM does not satisfy the hypothesis, the RSM is removed from the PANEL framework in this evaluation example. According to the reconstruction and evaluation in FIGS. 28M to 28P, it is clear that Noise2Noise functions effectively except for the regions highlighted in the insets of FIGS. 28M to 28P (the suspicious regions are indicated by white arrows), and it is shown that the rFRC map (white channel in FIG. 28P) can accurately detect such types of errors. FIG. 35A shows the result after Noise2Noise (N2N) (upper left) and the noisy input (lower right, noise 1). FIG. 35B shows the result of Noise2Noise from the white box in FIG. 35A. FIG. 35C shows a reference image obtained by averaging 50 noisy images with the same content. FIG. 35D shows an integrated image of PANEL (light gray channel) and the result of Noise2Noise (dark gray channel). FIG. 35E shows the rFRC map of FIG. 35B.

[0302] As described above, a comprehensive, reliable, and universal quantitative mapping of errors in the super-resolution scale may be decisive by not depending on any reference information and by the emergence of computation- and learning-based super-resolution techniques in biomedical imaging science. By tracing the quantitative error map up to the super-resolution scale, it becomes possible to evaluate the reconstruction quality at the pixel level, and further design of computational processes such as STORM image fusion, adaptive low-pass filtering, and automatic iterative determination becomes possible. Furthermore, the proposed rFRC map may also be an excellent option for accessing the quality metric against the ground truth and estimating the local resolution map of an image.

[0303] In principle, as major classifications of reconstruction errors in imaging by a computational microscope, there are two types: model error and data error. The main cause of model error is the distance existing between an artificially created estimation model and its actual model in the physical world. In the case of a learning-based microscope, such a distance may result from ignoring the network for out-of-distribution data. The main cause of data error is due to the combined effect of the noise conditions and sampling capabilities of hardware devices such as sensors and cameras in the microscope system. Based on the theory of the corresponding model, errors related to the model may be discovered and reduced by carefully performing system calibration in a physical imaging system, and may be explained by specifically designed strategies and sufficient learning data in a learning-based application. On the other hand, data error may basically be model-free, cannot be avoided by artificial system calibration methods, and is difficult to explain with more learning data sets. Summarizing the above, it is suggested that in biological analysis, the estimation of such data error may be more decisive. In some embodiments, for the purpose of developing an absolutely model-free quantitative method, PANEL may be based only on one hypothesis, i.e., that the model for reconstruction is unbiased. This means that the proposed PANEL method can accurately detect data errors existing in various methods, but it should also be noted that it is limited to estimating model errors.

[0304] In particular, considering learning-based applications, the model error can be simply estimated by the disagreement of an ensemble that independently and repeatedly learns the model through multiple random initialization processes and optimization processes for the same dataset. Furthermore, since it is a pure data-driven approach that learns the representation of the training data, the model error and data error in such learning-based applications are not necessarily mutually exclusive. As shown in FIG. 27A, the PANEL framework shows the possibility of detecting both model error and data error. The model error from the prediction of out-of-distribution test samples (rectangles: training data, triangles / circles: test data) can be effectively detected by the PANEL concept and rFRC metric (FIGS. 27B to 27D). Alternatively, the uncertainty of the data and the uncertainty of the model can also be estimated by applying the rFRC map to the twin predictions from two inputs (from two data samplings) and two models (from two network trainings), respectively.

[0305] The PANEL method, as a model-agnostic and reference-free metric, can divide the local reconstruction quality of images from various modalities without requiring extra prior knowledge. PANEL may focus more on detecting data error than model error. However, considering a data-driven approach, the data error may generally be mixed with part of the model error, leading to PANEL evaluating the model error to some extent. PANEL may provide a universal and reliable local reconstruction quality evaluation framework after fully considering the capabilities and boundaries of the corresponding approach, which not only helps in image-based biological profiling but also promotes further progress in the rapidly developing field of computational microscopy imaging.

[0306] In the following Examples 6 to 11, new findings and extended applications of PANEL will be described. (Example 6)

[0307] Errors in SMLM, SIM, and deconvolution Errors in SMLM: The accuracy of SMLM was found to be highly related to the activation density of molecules in each frame. In the case of a small field of view, it can be regarded as uniform irradiation (uniform activation density), and errors are likely to occur in complex structures, such as the common set of filaments (Figures 18A - 18L). In the case of a large field of view, since the activation density of molecules is related to the irradiation intensity (the higher the intensity, the lower the activation density), errors are likely to occur in the regions of low - intensity irradiation (Figures 17G, 17H, 17P).

[0308] Errors in SIM and deconvolution: In SIM and deconvolution, high - frequency errors (smaller than the resolution), such as artifacts like snow crystals, cannot be detected. It was discovered that the magnitude of the errors in SIM and deconvolution is related to the emission intensity of the fluorescence signal (Figures 29A - 29I, Figures 30A - 30I), and the quality of the reconstruction results of SIM and deconvolution is related to the SNR of the image, and it is reasonable that the SNR is proportional to the signal magnitude in some form. (Example 7)

[0309] Adaptive low - pass filter based on rFRC FRC determines the high - reliability cut - off frequency (COF) of an image, indicating that noise and errors concentrate in frequency components greater than the cut - off frequency. Since rFRC calculates the local cut - off frequencies in different regions of the image, here, the local cut - off frequencies for adaptively applying a low - pass filter are applied to different block - box regions of the entire image.

[0310]

Number

[0311] Here, I x,y represents a subset image of the input image with the central pixel at the spatial position (x, y), and OTF(F x,y ) represents the cut - off frequency F x,yIt is an optical transfer function (OTF) that is 0 outside the cut-off frequency and 1 inside the cut-off frequency. Considering Richardson-Lucy deconvolution (RLD), since the image quality of the reconstruction result is highly related to the corresponding local SNR, such a reconstruction result usually has a spatially varying cut-off frequency. FIG. 36A shows a simulated wide-field image. FIG. 36B shows the image after performing RLD 500 times on the simulated wide-field (WF) image of FIG. 36A. FIG. 36C shows the workflow of an adaptive filter for the image, where the block size of the filter is set to 64×64 pixels and the overlap between adjacent blocks is set to 4 pixels. FIG. 36D shows the image after performing a global estimated cut-off frequency filter on the image of FIG. 36C. FIG. 36E shows the image after performing an adaptive local filter on the image of FIG. 36A. FIG. 36F shows the ground truth (GT) image of FIG. 36A. The ground truth image is convolved with a PSF (FWHM = 240 nm) and is involved in Poisson noise and 10% Gaussian noise. The global FRC filter does not have to achieve the optimal result (see SSIM = 0.32, PSNR = 19.30 in FIG. 36D). In comparison, the designed adaptive rFRC filter is an excellent choice for filtering the image after Richardson-Lucy deconvolution (see SSIM = 0.42, PSNR = 21.90 in FIG. 36E).

[0312] The global FRC value is used to estimate the OTF of the entire image, and such an estimated OTF is applied to blind Richardson-Lucy deconvolution. However, FRC can estimate the effective OTF (reliable frequency components) rather than the original resolution of the system. In other words, FRC may be greatly affected by the noise amplitude. When the noise amplitude is large, the frequency components become dominant within the OTF, which may affect FRC's inference of the original resolution of the system (resulting in a larger resolution). RLD requires a static OTF instead of the effective OTF of the system. Therefore, such an operation introduces the natural hypothesis that the images processed by RLD are under a sufficiently high SNR. (Example 8)

[0313] SMLM fusion based on the rFRC map By developing such an rFRC quality metric, different localization results may be fused according to the weights of the rFRC map to combine the advantages of each of the different models.

[0314]

Number

[0315] Here, L n is the result of the nth localization model, and G(σ) represents a Gaussian kernel with a standard deviation of σ. max(F 1~n ) is the maximum FRC value of the N localization results,

Number

[0316] Determination of the number of iterations of inverse convolution by rFRC The optimal number of iterations of Richardson-Lucy deconvolution (RLD) is generally determined by manual inspection by observing the reconstruction result and convergence. However, manual inspection is usually cumbersome and limits automatic image analysis. On the other hand, under low SNR conditions, RLD may not converge, and in such cases, it is necessary to stop the iteration at an early stage (before convergence) to avoid excessive sharpness of the structure and the generation of unpleasant artifacts. The reported rFRC is used to automatically determine the optimal number of iterations by estimating a locally reliable cut-off frequency. Specifically, the optimal iteration is the one with the smallest rFRC value in order to obtain a more reliable cut-off frequency equal to the rFRC value that is higher in resolution and smaller by the RLD process.

[0317] The images captured by TIRF and TIRF-SIM are used as data pairs shown in FIGS. 21G to 21J and FIGS. 30A to 30I. The TIRF-SIM image is used as the ground truth that is the target of RLD on the TIRF image. As can be seen from FIG. 21J, as the iteration progresses, the PSNR curve is expressed in a quadratic form, indicating that a good result is not necessarily obtained even with a large number of iterations. In fact, excessive iterations may induce artifacts (white arrows in FIGS. 21I and 30B to 30D) and reduce the PSNR value. In this embodiment, considering the optimal PSNR, it can be seen that "80" times is the optimal number of iterations. According to the observations for "80" iterations and "200" iterations in FIGS. 30B and 30C, the "80" iterations improve the contrast without inducing artifacts, while the "200" iterations generate artifacts like snow crystals indicated by the white arrows, which can be verified not to exist in the referenced TIRF-SIM image (FIG. 30D). Also, the rFRC value in FIG. 21J can determine the optimal choice of the number of iterations, and its curve is also the same as the PSNR curve calculated for the ground truth. Conversely, the resolution-scaled Pearson coefficient (RSP) metric could not represent such a beneficial distribution. (Example 10)

[0318] rFRC as a method for error mapping of data with ground truth rFRC was first proposed to map errors when there is no ground truth in actual biological imaging applications, and it can also be regarded as an excellent option for error mapping in the case of data with attached ground truth compared to SSIM to enable a more reasonable evaluation.

[0319] As can be seen from FIGS. 39A to 39F, the filaments were generated at different distances and convolved with a wide-field PSF (NA = 1.4). Then, gradually increasing noise (along the arrow in FIG. 39B) was involved in the image of FIG. 39B. The resulting image was obtained using Richardson-Lucy deconvolution as shown in FIG. 39C. Compared with the ground truth shown in FIG. 39A, the largest error appears in the white box of FIG. 39B, and it was found that the rFRC map can accurately detect such errors as can be seen from FIGS. 39D and 30E. Conversely, as shown in FIG. 39F, a spatial distance estimation approach such as SSIM induced high false negatives, making it difficult to separate or invisible true negatives. It should be noted that the rFRC maps formed by Reconstruction 1 and Reconstruction 2, or Reconstruction 1 and the ground truth, completely match, indicating that the PANEL can find errors without the need for the ground truth. FIG. 39A shows the ground truth. FIG. 39B shows the image with gradually increasing noise added. FIG. 39C shows the reconstructed image. FIG. 39D shows the rFRC map generated based on Reconstruction 1 and Reconstruction 2. FIG. 39E shows the rFRC map generated based on Reconstruction 1 and the ground truth. FIG. 39F shows the inverted SSIM. (Example 11)

[0320] Resolution as an additional metric In addition to detecting the exact degree of error in reconstruction, a quantitative metric of resolution is added to the rFRC map. To construct an accurate and practical quantification tool, three major contributions were realized: no false negatives due to background, a more suitable color map sJet for visualization, and pixel-level resolution mapping. Therefore, the rFRC map can be used not only for reconstruction results but also as an excellent option for evaluating the resolution of raw data captured by photodetectors, such as stimulated emission depletion (STED) microscopy. Due to the absence of background false negatives, the average resolution of STED (Figure 40A) is "92 nm" in the rFRC map of Figure 40C, which is clearly more reasonable than "146 nm" in the FRC map of Figure 40B. Also, the "sJet" color map can visualize the resolution distribution with higher contrast compared to the conventionally used "SQUIRREL-FRC" color map. Figure 40A shows SiR-tubulin-labeled microtubules observed under gSTED. Figure 40B shows an FRC map with a block size of 64 pixels color-coded with the SQUIRREL-FRC color map. Figure 40C shows an rFRC map color-coded with the sJet color map.

[0321] Based on the 3σ curve instead of the 1 / 7 threshold, rFRC is more stable and accurate for local FRC resolution mapping to determine an effective cut-off frequency. As shown in Figures 41A and 41B, the resolution of "64 nm" determined by the 1 / 7 threshold in the region indicated by the white arrow in the inset seems overconfident for such poorly resolved filaments, while the resolution of "160 nm" obtained by the 3σ curve was clearly milder and more reasonable. The so-called 1 / 7 threshold may be considered unstable in local resolution mapping, but it should be emphasized that both features of resolution mapping based on the 1 / 7 threshold and the 3σ curve are provided for further potential applications in the PANELJ FIJI / ImageJ plugin. The reason is that the same local resolution mapping is provided for a wider user community because the 1 / 7 threshold is common and widely used.

[0322] The fixed-value threshold method has been claimed to be based on a false statistical hypothesis, but the 1 / 7 threshold is widely applied in the super-resolution field. Interestingly, regardless of the criteria, the final resolutions obtained in experiments of SMLM and stimulated emission depletion microscopy (STED) are almost the same. The FRC curve was calculated between small image blocks to map the local SR error included in the reconstruction. As shown in FIGS. 40C and 40D, in small images, the 1 / 7 threshold becomes overconfident, that is, smaller than all correlation values of the FRC curve, and the cut-off frequency of the corresponding image cannot be determined (the cross in FIG. 40C). In contrast, in large images, the 1 / 7 threshold obtained similar results compared to the 3σ curve. This indicates that the 1 / 7 threshold is suitable for the determination of conventional global resolution but not for the determination of local cut-off frequency (local resolution). On the other hand, in the determination of the resolution described above, instead of avoiding conservative threshold selection such as the 1 / 7 threshold, a gentle threshold for error mapping is intended to reduce false positives. Therefore, considering the robustness and accuracy in the case of small image blocks, the 3σ curve is used as the threshold for calculating the FRC value of the PANEL.

[0323]

Table 1

[0324]

Table 2

[0325] Thus, although the basic concepts have been described, it will be apparent to those skilled in the art that after reading the detailed disclosure herein, the foregoing detailed disclosure is presented by way of example and is not intended to be limiting. Although not explicitly stated herein, various changes, improvements, and modifications will be possible and intended for those skilled in the art. These changes, improvements, and modifications are intended to be suggested by this disclosure and are within the spirit and scope of the exemplary embodiments of this disclosure.

[0326] Furthermore, specific terms are used to describe embodiments of the present disclosure. For example, the terms "one embodiment", "an embodiment", and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, it should be emphasized and understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily all refer to the same embodiment. Further, the particular features, structures, or characteristics may be suitably combined in one or more embodiments of the present disclosure.

[0327] Also, as will be understood by those skilled in the art, aspects of the present disclosure can be illustrated and described herein in any of several patentable types or situations, including any novel and useful process, machine, manufacturing process, or composition of matter, or any novel and useful improvement thereof. As such, aspects of the present disclosure may be implemented entirely in hardware or in software (including firmware, resident software, microcode, etc.), or in a combination of software and hardware, all of which may generally be referred to herein as a "unit", "module", or "system". Further, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0328] A computer-readable signal medium may include a propagated data signal embodying computer-readable program code therein, for example, as part of a baseband or a carrier wave. Such a propagated signal may take any of a variety of forms including, for example, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is any computer-readable medium that is not a computer-readable storage medium and that can be used by, or is associated with, a command execution system, apparatus, or device for the communication, propagation, or transmission of a program. The program code embodied on the computer-readable signal medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0329] The computer program code for performing operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as the "C" programming language, Visual Basic, Fortran 2103, Perl, COBOL 2102, PHP, ABAP, etc., dynamic programming languages such as Python, Ruby, Groovy, etc., or other programming languages. The program code may be executed entirely on a user computer, partially executed on a user computer as a stand-alone software package, partially executed on a user computer and partially executed on a remote computer, or executed entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user computer via any type of network including a local area network (LAN) or a wide area network (WAN), and this connection may be made to an external computer (e.g., via the Internet using an Internet service provider), or in a cloud computing environment, and may be provided as a service such as software as a service (SaaS).

[0330] Furthermore, the order of the described processing elements or sequences, or the use of numbers, characters, or other names, is not intended to limit the processes and methods described in the claims to any order other than that which may be specified in the claims. The above disclosure has been discussed by way of various specific examples that are presently considered to be various useful embodiments of the present disclosure, but such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments. Rather, it should be understood that the appended claims are intended to cover modifications and equivalent arrangements within the spirit and scope of the disclosed embodiments. For example, the implementation of the various components described above may be embodied in a hardware device, but may also be realized as an implementation only in software, such as an existing server or mobile device, etc.

[0331] Similarly, in the foregoing description of the embodiments of the present disclosure, it should be understood that various features may be grouped together in a single embodiment, drawing, or description thereof in order to simplify the disclosure and assist in the understanding of one or more of the various embodiments. However, the methods of the present disclosure should not be construed as reflecting an intention that the subject matter of the claims requires more features than are expressly recited in each claim. Rather, the embodiments of the present disclosure do not include all of the features of the embodiments described above.

[0332] In some embodiments, it is understood that numerical values representing amounts or characteristics used to describe particular embodiments of the present application may, in some cases, be modified by the terms "about," "approximate," or "substantially." For example, "about," "approximate," or "substantially" may indicate a variation of ±20% of the recited value, unless otherwise specified. Accordingly, in some embodiments, the numerical parameters set forth in the present specification and the appended claims are approximate values that may vary depending on the desired characteristics obtained by a particular embodiment. In some embodiments, numerical parameters should be construed by applying ordinary rounding techniques in light of the number of significant digits reported. Notwithstanding that the numerical ranges and parameters defining the broad scope of some embodiments of the present application are approximations, the numerical values set forth in the specific examples are reported as accurately as practicable.

[0333] Each of the patents, patent applications, published patent publications, and other materials, such as papers, books, specifications, publications, documents, articles, etc., referred to herein is hereby incorporated by reference in its entirety for all purposes, except to the extent that any of the associated prosecution documents, those that are inconsistent with or conflict with this document, or those that may have a limiting effect on the broadest scope of the claims that are or may become relevant to this document. By way of example, in the event of a conflict or contradiction between the descriptions, definitions, and / or uses of terms associated with any of the incorporated materials and the descriptions, definitions, and / or uses of terms associated with this document, the descriptions, definitions, and / or uses of terms in this document shall prevail.

[0334] Finally, it should be understood that the embodiments of the present application disclosed herein are illustrative of the principles of the embodiments of the present application. Other variations that may be utilized may be within the scope of the present application. Thus, by way of example and not limitation, alternative configurations of the embodiments of the present application can be utilized in accordance with the teachings of this document. Accordingly, the embodiments of the present application are not limited to exactly what is illustrated and described.

Description of Reference Numerals

[0335] 100 Image processing system 110 Image acquisition device 120 Network 130 Terminal 131 Mobile device 132 Tablet computer 133 Laptop computer 140 Processing device 150 Storage device 200 Arithmetic unit 210 Processor 220 Storage 230 I / O unit 240 Communication port 300 Mobile device 310 Communication unit 320 Display unit 330 Graphics processing unit 340 Central processing unit 350 I / O unit 360 Memory 361 Operating system 362 Application 370 Storage unit 402 Acquisition module 404 Expansion module 406 Rolling module 408 Map generation module 410 Filtering module 412 Display module 414 Image quality determination module 416 Decomposition module 418 Neural network processing module

Claims

1. A method for image processing, which is executed on at least one machine each having at least one processor and at least one storage device, comprising: obtaining a first image and a second image associated with the same object; determining a plurality of first blocks of the first image and a plurality of second blocks of the second image that correspond one-to-one; determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks; generating a first target map associated with the first image and the second image based on the plurality of first characteristic values; A method for image processing, including the above steps.

2. The step of determining a plurality of first blocks of the first image includes the step of determining a plurality of first blocks of the first image by performing a rolling operation on the first image. The method according to claim 1.

3. The rolling operation is performed by sliding a window on the first image and skipping a predetermined number of pixels on the first image. The method according to claim 2.

4. The step of determining a plurality of second blocks of the second image includes the step of determining a plurality of second blocks of the second image by performing a rolling operation on the second image. The method according to claim 1.

5. The rolling operation is performed by sliding a window on the second image and skipping a predetermined number of pixels on the second image. The method according to claim 4.

6. At least one of the first characteristic values is determined based on the correlation between one first block of the plurality of first blocks and its corresponding second block. The method according to claim 1.

7. The at least one first characteristic value is determined by a process including: determining a first intermediate image in the frequency domain based on the first block; determining a second intermediate image in the frequency domain based on the second block; determining a target frequency based on the first intermediate image and the second intermediate image; determining the at least one first characteristic value based on the target frequency. The method according to claim 6.

8. The step of determining a first intermediate image in the frequency domain based on the first block 8. The method of claim 7, further comprising determining the first intermediate image by performing a Fourier transform on the first block.

9. determining a second intermediate image in the frequency domain based on the second block, 8. The method of claim 7, further comprising determining the second intermediate image by performing a Fourier transform on the second block.

10. determining the at least one first characteristic value based on the target frequency, specifying the target frequency as the at least one first characteristic value; or determining the at least one first characteristic value based on the inverse of the target frequency; The method of claim 7, comprising:

11. determining a target frequency based on the first intermediate image and the second intermediate image, determining a first relationship between a plurality of correlation values and a plurality of frequencies based on the first intermediate image and the second intermediate image; determining a second relationship between the plurality of correlation values and the plurality of frequencies based on a predetermined function; determining the target frequency based on the first relationship and the second relationship, the target frequency corresponding to an intersection of a first curve representing the first relationship and a second curve representing the second relationship; The method of claim 7, comprising:

12. determining a first relationship between a plurality of correlation values and a plurality of frequencies based on the first intermediate image and the second intermediate image, determining a plurality of said correlation values, each corresponding to one of said plurality of frequencies; and each said correlation value is determining, from the first intermediate image, a first ring image based on the frequency, the first ring image including pixels within a ring having a diameter equal to the frequency; determining, from the second intermediate image, a second ring image based on the frequency, the second ring image including pixels within a ring having a diameter equal to the frequency; determining each of the correlation values based on the first ring image and the second ring image; The method of claim 11 , wherein the determination is made according to a process comprising:

13. performing a padding operation around the first image to expand the first image; performing a padding operation around the second image to extend the second image; The method of claim 1 further comprising:

14. The step of obtaining a first image and a second image associated with the same object includes: obtaining an initial image; decomposing the initial image into the first image, the second image, the third image, and the fourth image; The method according to claim 1, comprising:

15. determining a plurality of third blocks of the third image and a plurality of fourth blocks of the fourth image that correspond one-to-one; determining a plurality of second characteristic values based on the plurality of third blocks and the plurality of fourth blocks; generating a second target map associated with the third image and the fourth image based on the plurality of second characteristic values; generating a third target map based on the first target map and the second target map; The method according to claim 14, further comprising:

16. The step of generating a third target map based on the first target map and the second target map includes: averaging the first target map and the second target map to generate the third target map; resizing the third target map to the size of the initial image; The method according to claim 15, comprising:

17. performing a padding operation around the first image to expand the first image; performing a padding operation around the second image to expand the second image; performing a padding operation around the third image to expand the third image; performing a padding operation around the fourth image to expand the fourth image; The method according to claim 14, further comprising:

18. The step of determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks includes: for each of the first blocks or each of the second blocks, determining an average value of pixel values of pixels in a central region of each of the first blocks or each of the second blocks; comparing the average value with a threshold value; in response to a determination that the average value is greater than the threshold value, determining a first characteristic value based on a first intermediate image corresponding to each of the first blocks and a second intermediate image corresponding to each of the second blocks, or in response to a determination that the average value is less than the threshold value, determining the first characteristic value based on a predetermined value; The method according to claim 1, comprising:

19. The method according to claim 1, further comprising the step of filtering the first target map using an adaptive median filter.

20. The method according to claim 1, further comprising the step of determining a global image quality metric of the first image or the second image based on the first target map.

21. The method according to claim 1, further comprising the step of displaying the first target map using a displacement jet color map.

22. The step of determining a reference map associated with the first image, and The step of fusing the first target map and the reference map to determine a fourth target map, and The method according to claim 1, further comprising.

23. The step of determining a reference map associated with the first image includes The step of obtaining a reference image having a lower resolution than the first image, The step of determining a convolution kernel based on the first image and the reference image, The step of determining an intermediate image based on the first image and the convolution kernel, and The step of determining the reference map based on the difference between the intermediate image and the reference image, and The method according to claim 22, comprising.

24. The step of obtaining the first image and the second image associated with the same object includes The step of obtaining a first initial image and a second initial image associated with the same object, The step of generating the first image by processing the first initial image using a neural network, The step of generating the second image by processing the second initial image using the neural network, and The method according to claim 1, comprising.

25. The step of obtaining the first initial image and the second initial image associated with the same object includes The step of obtaining an initial image as the first initial image, The step of adding noise to the first initial image to generate the second initial image, and The method according to claim 24, comprising.

26. The step of obtaining the first initial image and the second initial image associated with the same object includes The step of obtaining an initial image, The step of adding first noise to the initial image to generate the first initial image, The step of adding second noise to the initial image to generate the second initial image, and The method according to claim 24, comprising.

27. 1. A system for image processing, comprising: at least one storage medium containing an instruction set; at least one processor in communication with the at least one storage medium, wherein upon executing the set of instructions, the at least one processor: acquiring a first image and a second image associated with the same object; determining a plurality of first blocks of the first image and a plurality of second blocks of the second image in one-to-one correspondence; determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks; generating a first target map associated with the first image and the second image based on the first plurality of characteristic values; 1. A system for image processing, wherein the system is instructed to perform operations including:

28. 1. A non-transitory computer-readable medium comprising at least one set of instructions for image processing, the at least one set of instructions, when executed by one or more processors of a computing device, to: acquiring a first image and a second image associated with the same object; determining a plurality of first blocks of the first image and a plurality of second blocks of the second image in one-to-one correspondence; determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks; generating a first target map associated with the first image and the second image based on the first plurality of characteristic values; A non-transitory computer-readable medium that causes the computing device to perform a method including:

Images