Systems and methods for image processing - Patents.com

Abstract

The present disclosure provides systems and methods for image processing, which may include acquiring a first image and a second image associated with a same object, determining a first number of blocks in the first image and a second number of blocks in the second image that correspond one-to-one, determining a first number of characteristic values ​​based on the first number of blocks and the second number of blocks, and / or generating a first target map associated with the first image and the second image based on the first number of characteristic values.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to the field of image processing, and more particularly to systems and methods for determining errors or artifacts that occur during image reconstruction. [Background technology]

[0002] Original images generated or collected by an image capture device (e.g., microscope, telescope, camera, webcam) are usually reconstructed using reconstruction techniques (e.g., physics-based reconstruction, learning-based reconstruction). However, the reconstruction process may cause image errors or artifacts, which may lead to misinterpretation of information in the image if the image quality is relatively low. Therefore, it is desirable to provide a system and method that can efficiently perform image processing to more accurately and effectively determine errors / artifacts. Summary of the Invention [Means for solving the problem]

[0003] In one aspect of the disclosure, a method for image processing is provided, which may include obtaining a first image and a second image associated with a same object, determining a first number of blocks in the first image and a second number of blocks in the second image that correspond one-to-one, determining a first number of characteristic values ​​based on the first number of blocks and the second number of blocks, and / or generating a first target map associated with the first image and the second image based on the first number of characteristic values.

[0004] In another aspect of the present disclosure, a system for image processing is provided. The system may include at least one storage device storing executable instructions and at least one processor in communication with the at least one storage device. Upon executing the set of instructions, the at least one processor may cause the system to perform operations including acquiring a first image and a second image associated with a same object, determining a first number of blocks in the first image and a second number of blocks in the second image that correspond one-to-one, determining a first number of characteristic values ​​based on the first number of blocks and the second number of blocks, and / or generating a first target map associated with the first image and the second image based on the first number of characteristic values.

[0005] In another aspect of the present disclosure, a non-transitory computer readable medium is provided, the non-transitory computer readable medium may include at least one set of instructions for image processing, the at least one set of instructions, when executed by one or more processors of a computing device, may cause the computing device to perform a method for image processing.

[0006] Other features are set forth in part in the following description, and in part will become apparent to one skilled in the art upon examination of the following and the accompanying drawings, or may be learned by the manufacture or operation of the embodiments. The features of the present disclosure may be realized and attained by practice or use of various aspects of the methodologies, apparatus, and combinations described in the detailed embodiments that follow.

[0007] The present disclosure will be described in more detail below with reference to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These embodiments are not limited, and the same numbers in these embodiments indicate the same configurations. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 is a schematic diagram illustrating an example application scenario of an image processing system according to some embodiments of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram illustrating an example computing device according to some embodiments of the present disclosure. [Diagram 3] 1 is a block diagram illustrating an example mobile device in which a terminal may be implemented, according to some embodiments of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating an exemplary processing apparatus according to some embodiments of the present disclosure. [Diagram 5] 1 is a flowchart illustrating an example process for determining a target map according to some embodiments of the present disclosure. [Figure 6A] 1 is a flowchart illustrating an exemplary process for determining a property value according to some embodiments of the present disclosure. [Figure 6B] 1 is a flowchart illustrating an example process for determining a property value according to some embodiments of the present disclosure. [Figure 6C] 1 is a flowchart illustrating an exemplary process for determining a property value according to some embodiments of the present disclosure. [Figure 6D] 1 is a flowchart illustrating an example process for determining a property value according to some embodiments of the present disclosure. [Figure 7] 1 is a flowchart illustrating an example process for determining a reference map according to some embodiments of the present disclosure. [Figure 8] 11 is a flowchart illustrating another example process for determining a target map according to some embodiments of the present disclosure. [Figure 9] 9A-9D illustrate an example process for determining a target map according to some embodiments of the present disclosure. [Figure 10] 1 illustrates an example process for determining a reference map according to some embodiments of the present disclosure. [Figure 11A] 1 illustrates an example process for fusing a first target map and a reference map according to some embodiments of the present disclosure. [Figure 11B] 1 illustrates an example process for fusing a first target map and a reference map according to some embodiments of the present disclosure. [Figure 11C] 1 illustrates an example process for fusing a first target map and a reference map according to some embodiments of the present disclosure. [Figure 12A] 1 illustrates an example process for determining a target map according to some embodiments of the present disclosure. [Figure 12B] 1 illustrates an example process for determining a target map according to some embodiments of the present disclosure. [Figure 12C] 1 illustrates an example process for determining a target map according to some embodiments of the present disclosure. [Figure 12] 12D-12G illustrate an example process for determining a target map according to some embodiments of the present disclosure. [Figure 12H] 1 illustrates an example process for determining a target map according to some embodiments of the present disclosure. [Figure 13] 1 illustrates another example process for determining a target map according to some embodiments of the present disclosure. [Figure 14] 14A-14B show example displacement jet color maps for error indication according to some embodiments of the present disclosure. [Figure 14C] 13 illustrates an example displacement jet color map for error indication according to some embodiments of the present disclosure. [Figure 14D] 13 illustrates an example displacement jet color map for error indication according to some embodiments of the present disclosure. [Figure 15] 1 illustrates an exemplary principle of a target map according to some embodiments of the present disclosure. [Figure 16] 1 illustrates an exemplary PANEL framework according to some embodiments of the present disclosure. [Figure 17A]1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17B] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17C] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17D] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17E] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17F] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17G] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17H] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17I] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17J]1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17K] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17L] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17M] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17N] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17O] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17P] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17Q] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17R] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17S]1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17T] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17U] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17V] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17W] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Fig. 17X] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 17Y] 1 shows exemplary results of simulations of single molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error position (PANEL), according to some embodiments of the present disclosure. [Figure 18A] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18B] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18C]1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18D] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18E] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18F] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18G] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18H] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18I] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18J] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18K] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 18L] 1 shows exemplary results of open source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 19A]13 shows exemplary results of full data from a 2D-SMLM experiment evaluated and fused with rFRC maps according to some embodiments of the present disclosure. [Figure 19B] 13 shows exemplary results of full data from a 2D-SMLM experiment evaluated and fused with rFRC maps according to some embodiments of the present disclosure. [Figure 19C] 13 shows exemplary results of full data from a 2D-SMLM experiment evaluated and fused with rFRC maps according to some embodiments of the present disclosure. [Figure 19D] 13 shows exemplary results of full data from a 2D-SMLM experiment evaluated and fused with rFRC maps according to some embodiments of the present disclosure. [Figure 19E] 13 shows exemplary results of full data from a 2D-SMLM experiment evaluated and fused with rFRC maps according to some embodiments of the present disclosure. [Figure 20A] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20B] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20C] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20D] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20E] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20F] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20G] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20H] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20I] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20J] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20K] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20L] 1 shows exemplary results of STORM fusion using rFRC maps according to some embodiments of the present disclosure. [Figure 20M] 1 shows an example of STORM fusion according to some embodiments of the present disclosure. [Figure 20N] 1 shows an example of STORM fusion according to some embodiments of the present disclosure. [Figure 21A] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21B] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21C] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21D] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21E] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21F] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21G]13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Fig. 21H] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21I] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 21J] 13A-13C show exemplary results of various physics-based imaging approaches aided by rFRC maps according to some embodiments of the present disclosure. [Figure 22A] 1 shows exemplary results of full data from a 3D-STORM experiment, according to some embodiments of the present disclosure. [Figure 22B] 1 shows exemplary results of full data from a 3D-STORM experiment, according to some embodiments of the present disclosure. [Figure 22C] 1 shows exemplary results of full data from a 3D-STORM experiment, according to some embodiments of the present disclosure. [Figure 23A] 13 shows exemplary results of another example of a 3D-STORM experiment evaluated by rFRC volumes, according to some embodiments of the present disclosure. [Figure 23B] 13 shows exemplary results of another example of a 3D-STORM experiment evaluated by rFRC volumes, according to some embodiments of the present disclosure. [Figure 23C] 13 shows exemplary results of another example of a 3D-STORM experiment evaluated by rFRC volumes, according to some embodiments of the present disclosure. [Figure 23D] 13 shows exemplary results of another example of a 3D-STORM experiment evaluated by rFRC volumes, according to some embodiments of the present disclosure. [Figure 23E] 13 shows exemplary results of another example of a 3D-STORM experiment evaluated by rFRC volumes, according to some embodiments of the present disclosure. [Figure 23F]13 shows exemplary results of another example of a 3D-STORM experiment evaluated by rFRC volumes, according to some embodiments of the present disclosure. [Figure 24A] 1 shows exemplary results of a PANEL evaluating coherent computational imaging and Fourier pticography microscopy (FPM) according to some embodiments of the present disclosure. [Figure 24B] 1 shows exemplary results of a PANEL evaluating coherent computational imaging and Fourier pticography microscopy (FPM) according to some embodiments of the present disclosure. [Figure 24C] 1 shows exemplary results of a PANEL evaluating coherent computational imaging and Fourier pticography microscopy (FPM) according to some embodiments of the present disclosure. [Figure 24D] 1 shows exemplary results of a PANEL evaluating coherent computational imaging and Fourier pticography microscopy (FPM) according to some embodiments of the present disclosure. [Figure 24E] 1 shows exemplary results of a PANEL evaluating coherent computational imaging and Fourier pticography microscopy (FPM) according to some embodiments of the present disclosure. [Figure 24F] 1 shows exemplary results of a PANEL evaluating coherent computational imaging and Fourier pticography microscopy (FPM) according to some embodiments of the present disclosure. [Figure 25A] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25B] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25C] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25D] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25E]13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25F] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25G] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Fig. 25H] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 25I] 13 illustrates example results of single-frame rFRC computation for 3D-SIM according to some embodiments of the present disclosure. [Figure 26] 13 illustrates example results of a full sparse sampling simulation according to some embodiments of the present disclosure. [Figure 27A] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27B] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27C] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27D] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27E] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27F] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27G]13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Fig. 27H] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27I] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27J] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27K] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 27L] 13 illustrates example results of a simulation of a learning-based application evaluated by PANEL, according to some embodiments of the present disclosure. [Figure 28A] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28B] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28C] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28D] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28E] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28F] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28G] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Fig. 28H] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28I] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28J] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28K] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28L] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28M] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28N] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28O] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 28P] 13 illustrates experimental results of a learning-based application evaluated by PANEL according to some embodiments of the present disclosure. [Figure 29A] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29B] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29C]1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29D] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29E] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29F] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29G] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Fig. 29H] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 29I] 1 shows exemplary results of full data from a TIRF-SIM experiment according to some embodiments of the present disclosure. [Figure 30A] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30B] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30C] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30D] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30E] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30F] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30G] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30H] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 30I] 1 shows exemplary results of a full TIRF deconvolution experiment according to some embodiments of the present disclosure. [Figure 31A] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Figure 31B] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Figure 31C] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Figure 31D] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Figure 31E] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Fig. 31F] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Figure 31G] 1 illustrates exemplary results of a full-pixel super-resolution simulation according to some embodiments of the present disclosure. [Figure 32A] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32B] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32C] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Fig. 32D] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32E] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Fig. 32F]1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Fig. 32G] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Fig. 32H] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Fig. 32I] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Fig. 32J] 1 illustrates example results of a single-frame rFRC computation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 33A] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33B] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33C] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33D] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33E] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33F] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33G] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Fig. 33H] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 33I] 1 shows exemplary results of a full ANNA-PALM experiment according to some embodiments of the present disclosure. [Figure 34A]1 shows exemplary results of a full CARE experiment according to some embodiments of the present disclosure. [Figure 34B] 1 shows exemplary results of a full CARE experiment according to some embodiments of the present disclosure. [Figure 34C] 1 shows exemplary results of a full CARE experiment according to some embodiments of the present disclosure. [Fig. 34D] 1 shows exemplary results of a full CARE experiment according to some embodiments of the present disclosure. [Figure 34E] 1 shows exemplary results of a full CARE experiment according to some embodiments of the present disclosure. [Fig. 34F] 1 shows exemplary results of a full CARE experiment according to some embodiments of the present disclosure. [Figure 35A] 1 shows exemplary results of a full Noise2Noise experiment according to some embodiments of the present disclosure. [Figure 35B] 1 shows exemplary results of a full Noise2Noise experiment according to some embodiments of the present disclosure. [Figure 35C] 1 shows exemplary results of a full Noise2Noise experiment according to some embodiments of the present disclosure. [Figure 35D] 1 shows exemplary results of a full Noise2Noise experiment according to some embodiments of the present disclosure. [Figure 35E] 1 shows exemplary results of a full Noise2Noise experiment according to some embodiments of the present disclosure. [Figure 36A] 1 illustrates example results of an adaptive low-pass filter for Richardson-Lucy deconvolution (RLD), according to some embodiments of the present disclosure. [Figure 36B] 1 illustrates example results of an adaptive low-pass filter for Richardson-Lucy deconvolution (RLD), according to some embodiments of the present disclosure. [Figure 36C] 1 illustrates example results of an adaptive low-pass filter for Richardson-Lucy deconvolution (RLD), according to some embodiments of the present disclosure. [Figure 36D]1 illustrates example results of an adaptive low-pass filter for Richardson-Lucy deconvolution (RLD), according to some embodiments of the present disclosure. [Figure 36E] 1 illustrates example results of an adaptive low-pass filter for Richardson-Lucy deconvolution (RLD), according to some embodiments of the present disclosure. [Figure 36F] 1 illustrates example results of an adaptive low-pass filter for Richardson-Lucy deconvolution (RLD), according to some embodiments of the present disclosure. [Figure 37A] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 37B] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 37C] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 37D] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 37E] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 37F] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 37G] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Fig. 37H] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Fig. 37I] 1 illustrates an exemplary SMLM result fusion of MLE and FALCON according to some embodiments of the present disclosure. [Figure 38A] 13 shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of the present disclosure. [Figure 38B]13 shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of the present disclosure. [Figure 38C] 13 shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of the present disclosure. [Figure 38D] 13 shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of the present disclosure. [Figure 38E] 13 shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of the present disclosure. [Figure 38F] 13 shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of the present disclosure. [Figure 39A] 13 shows exemplary results of simulations of different noise amplitudes evaluating the performance of SSIM and rFRC, according to some embodiments of the present disclosure. [Figure 39B] 13 shows exemplary results of simulations of different noise amplitudes evaluating the performance of SSIM and rFRC, according to some embodiments of the present disclosure. [Figure 39C] 13 shows exemplary results of simulations of different noise amplitudes evaluating the performance of SSIM and rFRC, according to some embodiments of the present disclosure. [Figure 39D] 13 shows exemplary results of simulations of different noise amplitudes evaluating the performance of SSIM and rFRC, according to some embodiments of the present disclosure. [Figure 39E] 13 shows exemplary results of simulations of different noise amplitudes evaluating the performance of SSIM and rFRC, according to some embodiments of the present disclosure. [Figure 39F] 13 shows exemplary results of simulations of different noise amplitudes evaluating the performance of SSIM and rFRC, according to some embodiments of the present disclosure. [Figure 40A] 1 shows example results illustrating background induced false negatives according to some embodiments of the present disclosure. [Figure 40B]1 shows example results illustrating background induced false negatives according to some embodiments of the present disclosure. [Figure 40C] 1 shows example results illustrating background induced false negatives according to some embodiments of the present disclosure. [Diagram 41] 41A-41D show example results of resolution mapping using a 3σ curve versus a 1 / 7 hard threshold, according to some embodiments of the present disclosure. [Diagram 42] 1 shows an overview of network architectures, learning parameter configurations, and data used for different applications according to some embodiments of the present disclosure. [Diagram 43] 1 illustrates an exemplary neural network backbone used for image restoration according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following detailed description, a number of specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without such details. In other instances, well-known methods, procedures, systems, components, and / or circuits, etc., have been described at a relatively high level and have not been described in detail in order to avoid unnecessarily obscuring aspects of the present disclosure. Various modifications of the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined in this disclosure may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the scope of the claims.

[0010] It should be noted that the terms used herein are for the purpose of describing particular exemplary embodiments and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly dictates otherwise. Furthermore, as used herein, it should be understood that the terms "comprise", "comprises" and / or "comprising", "include", "includes" and / or "including" specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be noted that the terms "object" and "subject" are used interchangeably with respect to the subject on which the imaging procedures of the present disclosure are performed.

[0011] It will be appreciated that the terms "system," "engine," "unit," "module," and / or "block" used herein are a way of distinguishing between different components, elements, parts, sections, or assemblies at different levels in ascending order, although the terms may be substituted with other terms that accomplish the same purpose.

[0012] The words "module", "unit" or "block" as used herein generally refer to a collection of software instructions or logic implemented in hardware or firmware. The modules, units or blocks described herein may be implemented as software and / or hardware and may be stored in any type of non-transitory computer-readable medium or another storage device. In some embodiments, software modules / units / blocks may be compiled and linked into an executable program. A software module may be called from other modules / units / blocks or from itself, and / or in response to a detected event or interrupt. A software module / unit / block executing on a computing device (e.g., processor 210 shown in FIG. 2) may be provided on a computer-readable medium such as a compact disc, digital video disc, flash drive, magnetic disk, or other tangible medium, or provided as a digital download (originally stored in a compressed or installable format file that must be installed, decompressed or decrypted before execution). Such software code may be stored in part or in whole in the storage device of the computing device on which it executes, for execution by the computing device. The software instructions may be embedded in firmware such as an EPROM. Furthermore, the hardware modules / units / blocks may be included in connected logic components such as gates and flip-flops, and / or in a programmable unit such as a programmable gate array or a processor. The modules / units / blocks or computing device functions described herein may be realized as software modules / units / blocks, but may also be expressed in hardware or firmware. The modules / units / blocks described herein generally refer to logical modules / units / blocks that may be combined with other modules / units / blocks and may be divided into sub-modules / sub-units / sub-blocks regardless of physical organization or storage. The description may apply to a system, an engine, or parts thereof.

[0013] When a unit, engine, module, or block is described as "on," "connected to," or "coupled to" another unit, engine, module, or block, it will be understood that it may be directly on the other unit, engine, module, or block, directly connected or coupled to the other unit, engine, module, or block, or in direct communication with the other unit, engine, module, or block, or there may be intervening units, engines, modules, or blocks, unless the context clearly indicates otherwise. As used herein, "and / or" includes any and all combinations of one or more items of the associated plurality of items.

[0014] It should be noted that, when describing operations performed on an image, the term "image" as used herein may be a data set (e.g., a matrix) that includes values ​​of pixels (pixel values) in the image. As used herein, for convenience of description, a representation of an object (e.g., a person, an organ, a cell, or a portion thereof) in an image may be referred to as an object. For example, for convenience of description, a representation of a cell or organelle (e.g., a mitochondria, an endoplasmic reticulum, a centrosome, a Golgi apparatus, etc.) in an image may be referred to as a cell or organelle. As used herein, for convenience of description, an operation on a representation of an object in an image may be referred to as an operation on an object. For example, for convenience of description, a segmentation of a portion of an image that includes a representation of a cell or organelle from an image may be referred to as a segmentation of the cell or organelle.

[0015] It should be understood that the term "resolution" as used herein is a measure of the sharpness of an image. The terms "super-resolution", "super-resolved" or "SR" as used herein refer to improved (or increased) resolution, which may be obtained, for example, by a process of combining a series of lower resolution images to produce a higher resolution image or sequence.

[0016] These and other features and characteristics of the present disclosure, its method of operation, function of the combination of associated elements and parts of construction, and economy of manufacture will become more apparent from a consideration of the following description in conjunction with the accompanying drawings, which form a part of this disclosure. It is to be expressly understood, however, that the drawings are for the purposes of illustration and description and are not intended to limit the scope of the present disclosure. It is to be understood that the drawings are not to scale.

[0017] The flowcharts used in this disclosure illustrate operations performed by systems according to some embodiments of the present disclosure. It should be clearly understood that the operations of the flowcharts may be performed out of order. Conversely, the operations may be performed in reverse order or simultaneously. Furthermore, one or more other operations may be added to the flowcharts. One or more operations may be deleted from the flowcharts.

[0018] With the development of technology, a series of super-resolution approaches may break through the diffraction limit (e.g., 200-300 nm) through the use of computational processes. Such computational operations may generally be performed on images generated based on several categories of techniques, including, for example: (i) molecular localization, such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM); (ii) molecular intensity fluctuations, such as super-resolution fluctuation imaging (SOFI) and super-resolution radial fluctuation (SRRF); (iii) high-frequency information mixing, such as structured illumination microscopy (SIM); and (iv) deconvolution, or other techniques that rely on high computational reconstruction to extend the spatial resolution of the image. In addition to the original physical configuration, multiple learning-based algorithms may be actively developed to reduce the photon / hardware budget in image generation, reconstruction, and / or processing. For example, artificial neural network accelerated PALM (ANNA-PALM) may be applied to reduce the total number of frames (or images, or image frames) without trading off the spatial resolution of PALM. As another example, conditional generative adversarial networks (cGANs) can be used to directly transform Total Internal Reflection Fluorescence (TIRF) microscopy images to match those obtained with TIRF-SIM. As a further example, the Content-Aware Image Restoration (CARE) network framework can make denoising and deconvolution applications more effective.

[0019] Such computational modalities are considered to be promising imaging techniques. In some embodiments, image errors or artifacts may occur during image reconstruction, and relatively low image quality may lead to misinterpretation of biological information. In some embodiments, the achieved image quality and resolution may be related to one or more comprehensive factors, including, for example, the photophysics of the fluorophores used for imaging, the chemical environment of the sample being imaged, the optical setup conditions, the analytical approach to generate the image (e.g., super-resolution images), network learning procedures, etc., or a combination thereof. For quantitative mapping and minimization of image defects, most of the existing quality assessments may rely on subjective comparison of reconstructed images against standard reference structures, benchmarking of reconstructed image data against other high-resolution imaging techniques such as electron microscopy, or corresponding specific design analysis algorithms. As a mere example, in the case of STORM / PLAM, given the input of precise imaging models and noise statistics, the confidence of individual localizations in a STORM / PLAM dataset without knowledge of the ground truth may be accessible based on a Wasserstein-guided flux computation process. In some embodiments, SIM checks, which require complex pipelines and specialized use, may be used to identify sources of errors and artifacts in the SIM system. In some embodiments, to capture the uncertainties of a learning-based approach (e.g., a Bayesian Neural Network (BNN) framework, learning distributions over weights, etc.), modifications may be required to the existing learning procedures of the learning-based approach. In some embodiments, such modifications may be complex to apply and computationally expensive compared to standard techniques (e.g., common learning-based approaches).

[0020] In some embodiments, a resolution-scale error map (RSM) (e.g., a reference map) may be used as an evaluation tool for the super-resolution image to generally and accurately evaluate local errors / artifacts. In some embodiments, a wide-field reference image may be used to generate the RSM. In some embodiments, the RSM may be generated based on one or more hypotheses, including, for example: (i) the wide-field reference image may have a relatively high SNR, (ii) the Gaussian kernel convolution process performed in generating the RSM map may be spatially invariant, and (iii) the illumination used in generating the raw data of the super-resolution image may be uniform. According to these hypotheses, in some embodiments, the RSM may be quite likely to induce false negatives in the estimated error map. On the other hand, when converting the super-resolution image to its low-resolution region, the detectable scale of the error of the RSM may be limited by Abbe diffraction. In some embodiments, a reference-free process (e.g., pixel-level analysis of error location (PANEL)) may be used to detect errors / artifacts at the super-resolution scale. In some embodiments, the PANEL technique may build on rolling Fourier ring correlation (rFRC) using one or more individual reconstructed super-resolution frames and / or may further work with a modified RSM. In some embodiments, one or more strategies to achieve single-frame rFRC computation may be involved in the PANEL framework for physics-based and / or learning-based applications.

[0021] In some embodiments, the RSM and rFRC maps may be integrated together in the PANEL framework, which may be developed as a multi-functional and model-free metric. Thus, the PANEL framework may have the ability to infer errors contained in multi-dimensional signals up to the super-resolution scale as well as mitigate possible false negative defects. The PANEL framework may be used in various scenarios, such as physical super-resolution imaging approaches (2D-STORM or 3D-STORM, TIRF-SIM or 3D-SIM, SRRF, Richardson-Lucy deconvolution, etc.), various learning-based image restoration techniques (ANNA-PALM, TIRF~TIRF-SIM, CARE, etc.), and / or non-super-resolution applications such as denoising tasks. In some embodiments, one or more minute quantitative error maps generated using the PANEL framework may be utilized to drive the design of multiple computational processes (e.g., STORM image fusion, adaptive low-pass filtering, automatic deconvolution iterative decision, etc.). In some embodiments, the PANEL framework can be used as a tool for robust and detailed evaluation of the image quality of physics-based or learning-based super-resolution imaging approaches, facilitating the optimization of super-resolution imaging and network learning procedures.

[0022] A wide range of super-resolution microscopy techniques may rely on corresponding computational operations that may shape reconstruction errors at one or more scales. Reconstruction errors in images may lead to misinterpretation of biological information. Quantitative mapping of such errors in super-resolution reconstruction may be limited to identify minute errors that result from the inevitable use of diffraction-limited images as references. According to some embodiments of the present disclosure, a rolling Fourier ring correlation (rFRC) operation may be used to trace pixels in the reconstructed image, providing a relatively precise model-free metric that allows pixel-level analysis of error location (PANEL). In some embodiments, a target image or map (e.g., an rFRC map) may be generated according to the pixel-level analysis of error location (PANEL). The target image or map may illustrate defects in the reconstructed image up to the super-resolution scale. In some embodiments, a reliability assessment of the PANEL approach may be performed on images reconstructed based on one or more computational modalities (e.g., physical super-resolution microscopy, deep learning super-resolution microscopy, etc.).

[0023] The present disclosure relates to a system and method for image processing. The system and method may acquire a first image and a second image associated with the same object. The system and method may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image. The plurality of first blocks and the plurality of second blocks may be in one-to-one correspondence. The system and method may determine a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks. The system and method may generate a target map (e.g., a first target map) associated with the first image and the second image based on the plurality of first characteristic values. The first target map may quantitatively map errors in the first image and / or the second image generated by an image capture device without using a reference (e.g., a ground truth) and / or any prior information of the image capture device. According to some embodiments of the present disclosure, the first target map may be used as a tool for quantitative assessment of image quality.

[0024] Furthermore, although the systems and methods disclosed in the present disclosure are primarily described with respect to generating an rFRC map reflecting the error of an image reconstructed by a physical super-resolution microscope and / or a deep learning super-resolution microscope, it should be understood that the description is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. The systems and methods of the present disclosure may be applied to any other type of system including an image acquisition device for image processing. For example, the systems and methods of the present disclosure may be applied to a microscope, a telescope, a camera (e.g., a security camera, a camera phone, a wedcam), an unmanned aerial vehicle, a medical imaging device, etc., or any combination thereof.

[0025] By way of example only, the methods disclosed in the present disclosure may be used to generate an rFRC map based on dual frames (e.g., sequential first and second images) or a single frame (e.g., an initial image) generated by physical super-resolution imaging and / or deep learning super-resolution reconstruction.

[0026] It should be understood that the application scenarios of the system and method disclosed herein are merely some exemplary embodiments provided for illustrative purposes, and are not intended to limit the scope of the present disclosure. Those skilled in the art can make multiple modifications and variations based on the teachings of the present disclosure.

[0027] 1 is a schematic diagram illustrating an example application scenario of an image processing system according to some embodiments of the present disclosure. As shown in FIG. 1, the image processing system 100 may include an image capture device 110, a network 120, one or more terminals 130, a processing device 140, and a storage device 150.

[0028] The components of image processing system 100 may be connected in one or more of a variety of ways. By way of example only, image capture device 110 may be connected to processing device 140 via network 120. As another example, image capture device 110 may be connected directly to processing device 140, as indicated by the dotted double-headed arrow connecting image capture device 110 and processing device 140. As yet another example, storage device 150 may be connected to processing device 140 directly or via network 120. As a further example, terminal 130 may be connected to processing device 140 directly (as indicated by the dotted double-headed arrow connecting terminal 130 and processing device 140) or via network 120.

[0029] The image processing system 100 may be configured to generate a first target map (e.g., an rFRC map) based on the first and second images using an rFRC process (e.g., one or more operations shown in FIG. 5). The first target map may illustrate or display errors and / or artifacts of the first and / or second images generated by the image capture device 110 and / or reflect image quality (e.g., the degree of errors). The image processing system 100 may be configured to determine a reference map associated with the first image. In some embodiments, the reference map may be used to identify errors not reported by the first target map. The image processing system 100 may be configured to determine a fourth target map by fusing the first target map and the reference map. The fourth target map may be configured to comprehensively illustrate or display errors and / or artifacts of the first and / or second images. In some embodiments, the errors and / or artifacts displayed by the target maps (e.g., the first target map, the fourth target map) may be related to or include errors generated in the reconstruction of the first image and / or the second image.

[0030] The image capture device 110 may be configured to capture one or more images (e.g., a first image, a second image, an initial image, etc.) associated with an object within its detection region. The object may include one or more biological or non-biological objects. In some embodiments, the image capture device 110 may be an optical imager, a radiation-based imager (e.g., a computed tomography imager), a nuclide-based imager (e.g., a positron emission tomography imager, a nuclear magnetic resonance imager), etc. Exemplary optical imagers may include a microscope 111 (e.g., a fluorescent microscope), a surveillance device 112 (e.g., a security camera), a mobile terminal device 113 (e.g., a camera phone), a scanning device 114 (e.g., a flatbed scanner, a drum scanner, etc.), a telescope, a webcam, etc., or any combination thereof. In some embodiments, the optical imager may include a capture device (e.g., a detector or a camera) that collects image data. For purposes of illustration, the present disclosure may use the microscope 111 as an example to describe exemplary functions of the image capture device 110. Exemplary microscopes may include structured illumination microscopes (SIM) (e.g., two-dimensional SIM (2D-SIM), three-dimensional SIM (3D-SIM), total internal reflection SIM (TIRF-SIM), spinning disk confocal-based SIM (SD-SIM), etc.), photoactivated localization microscopes (PALM), stimulated emission depletion microscopes (STED), stochastic optical reconstruction microscopes (STORM), etc.). SIM may include detectors such as EMCCD cameras, sCMOS cameras, etc. Objects detected by SIM may include one or more objects, such as biological structures, biological subjects, proteins, cells, microorganisms, etc., or any combination. Exemplary cells may include INS-1 cells, COS-7 cells, Hela cells, hepatic sinusoidal endothelial cells (LSECs), human umbilical vein endothelial cells (HUVECs), HEK293 cells, etc., or any combination thereof. In some embodiments, one or more objects may be fluorescent or fluorescently labeled. Fluorescent or fluorescently labeled objects may be excited to fluoresce for imaging.

[0031] Network 120 may include any suitable network capable of facilitating image processing system 100 to exchange information and / or data. In some embodiments, one or more of the components of image processing system 100 (e.g., image capture device 110, terminal 130, processing device 140, storage device 150, etc.) may communicate information and / or data with each other via network 120. For example, processing device 140 may obtain image data (e.g., first image, second image) from image capture device 110 via network 120. As another example, processing device 140 may obtain user instructions from terminal 130 via network 120. Network 120 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi network, etc.), a cellular network (e.g., a Long Term Evolution (LTE) network), an image relay network, a virtual private network ("VPN"), a satellite network, a telephone network, a router, a hub, a switch, a server computer, and / or a combination of one or more thereof. For example, network 120 may include a cable network, a wired network, a fiber network, a telecommunications network, a local area network, a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth® network, a ZixBee® network, a near field communication network (NFC), etc., or a combination thereof. In some embodiments, network 120 may include one or more network access points.For example, network 120 may include wired and / or wireless network access points, such as base stations and / or network switching points, through which one or more components of image processing system 100 may access network 120 for data and / or information exchange.

[0032] In some embodiments, a user can operate the image processing system 100 via a terminal 130. The terminal 130 may include a mobile device 131, a tablet computer 132, a laptop computer 133, or the like, or a combination thereof. In some embodiments, the mobile device 131 may include a smart home device, a wearable device, a mobile device, a virtual reality device, an augmented reality device, or the like. In some embodiments, the smart home device may include a smart lighting fixture, an intelligent electrical appliance controller, a smart surveillance device, a smart television, a smart video camera, an intercom, or the like, or a combination thereof. In some embodiments, the wearable device may include a bracelet, footwear, glasses, a helmet, a watch, clothing, a backpack, a smart accessory, or the like, or a combination thereof. In some embodiments, the mobile device may include a mobile phone, a personal digital assistant (PDA), a gaming device, a navigation device, a point of sale (POS) device, a laptop, a tablet computer, a desktop, or the like, or a combination thereof. In some embodiments, the virtual reality device and / or augmented reality device may include a virtual reality helmet, virtual reality glasses, virtual reality eyewear, augmented reality helmet, augmented reality glasses, augmented reality eyewear, etc., or combinations thereof. For example, the virtual reality device and / or augmented reality device may include Google Glass®, Oculus Rift®, Hololens®, GearVR®, etc. In some embodiments, terminal 130 may be part of processing unit 140.

[0033] The processing device 140 may process data and / or information obtained from the image capture device 110, the terminal 130, and / or the storage device 150. For example, the processing device 140 may process one or more images (e.g., a first image, a second image, etc.) generated by the image capture device 110 to generate an rFRC map (e.g., a first target map, a third target map, etc.). In some embodiments, the processing device 140 may be a server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 140 may be local or remote. For example, the processing device 140 may access information and / or data stored in the image capture device 110, the terminal 130, and / or the storage device 150 via the network 120. As another example, the processing device 140 may be directly connected to the image capture device 110, the terminal 130, and / or the storage device 150 to access the stored information and / or data. In some embodiments, the processing device 140 may be implemented on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an interconnected cloud, a multi-cloud, etc., or a combination thereof. In some embodiments, the processing device 140 may be implemented by a computing device 200 having one or more components, as shown in FIG.

[0034] The storage device 150 may store data, instructions, and / or any other information. In some embodiments, the storage device 150 may store data acquired from the terminal 130, the image capture device 110, and / or the processing device 140. In some embodiments, the storage device 150 may store data and / or instructions that the processing device 140 may execute or use to perform the exemplary methods described in this disclosure. In some embodiments, the storage device 150 may include mass storage devices, removable storage devices, volatile read-write memory, read-only memory (ROM), and the like. Exemplary mass storage devices may include magnetic disks, optical disks, solid-state drives, and the like. Exemplary removable storage devices may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tapes, and the like. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAM may include dynamic RAM (DRAM), double date rate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), etc. Exemplary ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disc ROM (CD-ROM), digital versatile disc ROM, etc. In some embodiments, the storage device 150 may run on a cloud platform. For example, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an interconnected cloud, a multi-cloud, etc., or a combination thereof.

[0035] In some embodiments, storage device 150 may be connected to network 120 and in communication with one or more other components of image processing system 100 (e.g., processing device 140, terminal 130, etc.). One or more components of image processing system 100 may access data or instructions stored in storage device 150 via network 120. In some embodiments, storage device 150 may be directly connected to and in communication with one or more other components of image processing system 100 (e.g., processing device 140, terminal 130, etc.). In some embodiments, storage device 150 may be part of processing device 140.

[0036] 2 is a schematic diagram illustrating an exemplary computing device according to some embodiments of the present disclosure. The computing device 200 may be used to implement any of the components of the image processing system 100 described herein. For example, the processing unit 140 and / or the terminal 130 may each be implemented on the computing device 200 via its hardware, software program, firmware, or a combination thereof. Although only one such computing device is shown, for convenience, the computer functionality for the image processing system 100 described herein may be implemented in a distributed manner on multiple similar platforms to distribute the processing load.

[0037] As shown in FIG. 2, the computing device 200 may include a processor 210, a storage 220, an input / output (I / O) unit 230, and a communication port 240.

[0038] The processor 210 may execute computer instructions (e.g., program code) to perform functions of the image processing system 100 (e.g., processing unit 140) in accordance with the techniques described herein. The computer instructions may include, for example, routines, programs, objects, components, data structures, procedures, modules, and functions that perform certain functions described herein. For example, the processor 210 may process image data obtained from any component of the image processing system 100. In some embodiments, the processor 210 may include one or more hardware processors such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or the like, or a combination thereof.

[0039] For purposes of illustration only, only one processor is described in computing device 200. However, it should be noted that computing device 200 in the present disclosure may also include multiple processors, such that operations and / or method operations performed by one processor as described in the present disclosure may also be performed jointly or individually by multiple processors. For example, in the present disclosure, if a processor of computing device 200 performs both operation A and operation B, it should be understood that operation A and operation B may also be performed jointly or individually by two or more different processors in computing device 200 (e.g., a first processor performs operation A and a second processor performs operation B, or the first processor and the second processor perform operation A and operation B jointly).

[0040] Storage 220 may store data / information obtained from any component of image processing system 100. In some embodiments, storage 220 may include mass storage devices, removable storage devices, volatile read / write memory, read only memory (ROM), etc., or any combination thereof. Exemplary mass storage devices may include magnetic disks, optical disks, solid state drives, etc. Removable storage devices may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tapes, etc. Volatile read / write memory may include random access memory (RAM). RAM may include dynamic RAM (DRAM), double date rate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), etc. ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disc ROM (CD-ROM), digital versatile disc ROM, etc.

[0041] In some embodiments, storage 220 may store one or more programs and / or instructions for performing the example methods described in this disclosure. For example, storage 220 may store a program for processing device 140 to process images generated by image capture device 110.

[0042] The I / O section 230 may input and / or output signals, data, information, etc. In some embodiments, the I / O section 230 may enable user interaction with the image processing system 100 (e.g., the processing unit 140). In some embodiments, the I / O section 230 may include input devices and output devices. Examples of input devices may include a keyboard, a mouse, a touch screen, a microphone, etc., or a combination thereof. Examples of output devices may include a display device, a speaker, a printer, a projector, etc., or a combination thereof. Examples of display devices may include a liquid crystal display (LCD), a light emitting diode (LED) based display, a thin display, a curved screen, a television set, a cathode ray tube (CRT), a touch screen, etc., or a combination thereof.

[0043] The communication port 240 may be connected to a network to facilitate data communication. The communication port 240 may establish a connection between the processing device 140 and the image capture device 110, the terminal 130, and / or the storage device 150. The connection may be a wired connection, a wireless connection, any other communication connection that may enable data transfer and / or data reception, and / or any combination of such connections. The wired connection may include, for example, an electrical cable, an optical cable, a telephone line, etc., or a combination thereof. The wireless connection may include a Bluetooth link, a Wi-Fi link, a WiMax link, a WLAN link, a ZikuBee link, a mobile network link (e.g., 3G, 4G, 5G), etc., or a combination thereof. In some embodiments, the communication port 240 may be and / or include a standardized communication port, such as RS232, RS485, etc. In some embodiments, the communication port 240 may be a specially designed communication port. For example, the communications port 240 may be designed according to the Digital Imaging and Communications in Medicine (DICOM) protocol.

[0044] FIG. 3 is a block diagram illustrating an example mobile device in which terminal 130 may be implemented, according to some embodiments of the disclosure.

[0045] As shown in FIG. 3 , the mobile device 300 may include a communication unit 310, a display unit 320, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, an I / O section 350, a memory 360, a storage unit 370, etc. In some embodiments, any other suitable components may also be included in the mobile device 300, including, but not limited to, a system bus or controller (not shown). In some embodiments, an operating system 361 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications (apps) 362 may be loaded from the storage unit 370 into the memory 360 for execution by the CPU 340. The applications 362 may include a browser or any other suitable mobile app for receiving and rendering information related to image capture, image processing, or other information from the image processing system 100 (e.g., the processing unit 140). User interaction with the information stream may be accomplished via I / O section 350 and provided to processing unit 140 and / or other components of image processing system 100 via network 120. In some embodiments, a user may input parameters to image processing system 100 via portable device 300.

[0046] A computer hardware platform may be used as a hardware platform for one or more elements (e.g., the processing unit 140 and / or other components of the image processing system 100 shown in FIG. 1) to implement the various modules, units and their functions described above. Since these hardware elements, operating systems and programming languages ​​are common, it is assumed that a person skilled in the art may be familiar with these technologies and may be able to provide the information required for imaging and evaluation according to the techniques described in this disclosure. The computer with the user interface may be used as a personal computer (PC), or other type of workstation or terminal device. After being appropriately programmed, the computer with the user interface may be used as a server. A person skilled in the art may also be considered familiar with such structure, program, or general operation of this type of computing device.

[0047] 4 is a schematic diagram illustrating an example processing device according to some embodiments of the present disclosure. As shown in FIG. 4, the processing device 140 may include an acquisition module 402, an expansion module 404, a rolling module 406, a map generation module 408, a filtering module 410, a display module 412, an image quality determination module 414, a decomposition module 416, and a neural network processing module 418.

[0048] The acquisition module 402 may be configured to acquire one or more images (eg, a first image and / or a second image, an initial image) associated with the same object.

[0049] The augmentation module 404 may be configured to augment one or more images (eg, the first image, the second image, the third image, and / or the fourth image).

[0050] The rolling module 406 may be configured to generate one or more image blocks (e.g., a plurality of first blocks of a first image, a plurality of second blocks of a second image, a plurality of third blocks of a third image, a plurality of fourth blocks of a fourth image, etc.).

[0051] The map generation module 408 may be configured to determine a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks, generate a first target map associated with the first image and the second image based on the plurality of first characteristic values, determine a reference map associated with the first image or the second image, and determine a fourth target map by fusing the first target map and the reference map. In some embodiments, the map generation module 408 may determine a plurality of second characteristic values ​​based on the plurality of third blocks and the plurality of fourth blocks, generate a second target map associated with the third image and the fourth image based on the plurality of second characteristic values, and generate the third target map based on the first target map and the second target map.

[0052] The filtering module 410 may be configured to filter one or more images (eg, the first target map, the second target map, the third target map, the fourth target map, etc.).

[0053] The display module 412 may be configured to display one or more images (e.g., the first target map, the second target map, the third target map, the fourth target map, a reference map, etc., or any other image generated by or used in image processing). In some embodiments, the display module 412 may display one or more images using a displacement jet color map.

[0054] The image quality determination module 414 may be configured to determine an image quality of one or more images. For example, the image quality determination module 414 may determine a global image quality metric for the first image and / or the second image based on the first target map.

[0055] The decomposition module 416 may be configured to decompose one or more images. For example, the decomposition module 416 may decompose an initial image into a first image, a second image, a third image, and a fourth image.

[0056] The neural network processing module 418 may be configured to generate one or more images by processing one or more initial images using a neural network.

[0057] A more detailed description of the modules of processing unit 140 may be found elsewhere in this disclosure (eg, FIGS. 5-8 and their descriptions).

[0058] It should be noted that the above description of the modules of the processing unit 140 is provided for illustrative purposes only and is not intended to limit the present disclosure. Those skilled in the art will appreciate that the modules may be combined in various ways and connected with other modules as subsystems under the teachings of the present disclosure without departing from the principles of the present disclosure. In some embodiments, one or more modules may be added or omitted in the processing unit 140. For example, the expansion module 404 and the rolling module 406 may be integrated into a single module.

[0059] FIG. 5 is a flow chart illustrating an exemplary process for determining a target map, according to some embodiments of the present disclosure. In some embodiments, the process 500 may be performed by the image processing system 100. For example, the process 500 may be configured as a set of instructions (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules illustrated in FIG. 4) may execute the set of instructions and cause the process 500 to be performed accordingly. The operations of the process illustrated below are intended to be illustrative. In some embodiments, the process 500 may be accomplished without one or more additional operations not described and / or one or more operations described. Furthermore, the order of operations of the process 500 illustrated in FIG. 5 and described below is not intended to be limiting.

[0060] In some embodiments, an imaging system (e.g., image processing system 100) may observe one or more objects using an optical device mathematically represented as a transfer function and collect corresponding signals using one or more sensors. In some embodiments, the signals may be sampled by the sensors and / or may be mixed with noise. In some embodiments, the artificial observations may be offset from real-world objects in a relatively high-dimensional space (e.g., as shown in the left panel of FIG. 15). Under such circumstances, recovering the hidden real signal from the measurement data may be considered an ill-posed problem, and the reconstructions from these observations may be offset from the real-world objects. If two separately measured signals are captured on the same object, these two signals may relate to the same object. It may be considered that the greater the distance between the reconstructions of these two signals (e.g., as shown in the right panel of FIG. 15), the more offset the reconstructions may show from the real object. If this distance (e.g., the distance between reconstruction 1 and reconstruction 2) is related to the reconstruction error (e.g., the deviation of the reconstruction from the real object), and following the intuition above, a key hypothesis holds: the distance between the two reconstructions is positively correlated with the magnitude of the reconstruction error.

[0061] To convincingly test this hypothesis, the derivation may be restricted to a specific general situation for fluorescence imaging with wide-field (WF) microscopy, in which case the corresponding images may be processed, for example, by a minimalist form of Wiener deconvolution.

[0062] The lemma may be stated as follows: Under ideal conditions, an object may be observed using an aberration-free optical system, and such observations may be sampled at an infinite sampling rate without noise. The object may be reconstructed using a minimalist form of Wiener deconvolution, as expressed in equation (1).

[0063]

number

[0064] where ω represents the spatial coordinate, and I, o, and h represent the point spread functions (PSFs) of the illumination, object, and microscope, respectively. F and F% are the operators of Fourier transform and its inverse Fourier transform. However, considering the joint effect of sampling and noise, a realistic imaging model may be expressed as follows:

[0065]

number

[0066] where s and n are the sampling model and noise model, respectively, and the image 広視野 represents the final image collected by the wide-field microscope. This imaging model may be considered closer to the situation in the real physical world and may differ from the Wiener model, as shown in equation (2).

[0067]

number

[0068] The minimalist form of Wiener deconvolution with an aberration-free PSF (unbiased estimate) is given by 広視野 may be used to process

[0069]

number

[0070] The image corresponding to the result of Eq. (4) 逆畳込 may be clearly detached from the real object.

[0071] The above lemma may suggest that the distance between the real object and the reconstruction result is mainly caused by the joint effect of sampling rate and mixed noise. According to this lemma, when imaging the same real object and controlling the variables to statistically capture independent image pairs, such distance between the reconstruction and the object may be highlighted by the difference between the reconstructions from the image pairs if an appropriate reconstruction model is used.

[0072] The explanation of the theorem may be explained as follows: If there are two wide-field images measured independently of the same sample (or object), these two images may show the same object. The larger the remaining distance between the two corresponding reconstruction results, the more the reconstruction results may appear to deviate from the real object (see FIG. 15).

[0073] This distance (e.g., Wiener result 1 vs. Wiener result 2) may be related to the error of the reconstruction (e.g., the difference between Wiener result 1 and Wiener result 2 vs. the real object). In some embodiments, the distance may be expressed as:

[0074]

number

[0075] Here, U represents the union operation, and || Wiener result 1, Wiener result 2 || p represents the distance between Wiener result 1 and Wiener result 2. Wiener result 1 and Wiener result 2 may each relate to the result of the reconstruction generated based on the Wiener model. n1, n2, s1, and s2 represent two separate noise models (n1 and n2) and sampling models (s1 and s2), respectively. In some embodiments, equation (5) may be modified by taking the Fourier transform of both sides.

[0076]

number

[0077] In some embodiments, the following two direct combinations of the distances of "Wiener result 1 and Wiener result 2" from the real object (i.e., using addition of Euclidean distances or multiplication of cross-correlation distances instead of union) may not be accurate compared to the union operation of equation (6). In some embodiments, the actual model of the union operation in the real world may be very complex, and therefore the combination of the two distances (reconstruction and real object) may be difficult to express explicitly. In some embodiments, the following two simple examples may be used to explain the relationship between the distances of the two individual reconstructions and the actual error of the reconstruction.

[0078] The first example concerns the Euclidean distance, which may be used to define the distance between two reconstructions. Assuming that the images are captured with an infinite sampling rate and are corrupted only by additive noise, denoted n, an addition operation may be utilized to combine the distances.

[0079]

number

[0080] where || Wiener result 1, Wiener result 2 || 2 represents the Euclidean distance between Wiener result 1 and Wiener result 2. n1 and n2 represent the individual additive noises in the two observations. In some embodiments, equation (7) may be simplified to obtain the following equation:

[0081]

number

[0082] The Fourier transform and PSF of the microscope may be eliminated to simplify equation (8) as follows:

[0083]

number

[0084] Equation (9) shows the close relationship between the Euclidean distances of Wiener result 1 and Wiener result 2, the Euclidean distances of Wiener result 1 and the real object, and the Euclidean distances of Wiener result 2 and the real object.

[0085] A second example concerns the cross-correlation distance: Alternatively, if cross-correlation is used as the distance, the addition operations in the Euclidean distance calculation need to be changed to multiplications.

[0086]

number

[0087] Where:

number

[0088]

number

[0089] where O, n1, and n2 denote I(ω)×o(ω), n1 / h(ω,0), and n2 / h(ω,0), respectively. In some embodiments, equation (11) may be further simplified by removing the Fourier transform.

[0090]

number

[0091] Similar to the Euclidean distance, equation (12) shows the close relationship between cross-correlation (e.g., the cross-correlation of Wiener result 1 with Wiener result 2) and multiplication of cross-correlations (e.g., the cross-correlation of Wiener result 1 with a real object, the cross-correlation of Wiener result 2 with a real object, etc.).

[0092] As discussed above, Euclidean distance and / or cross-correlation may be used to estimate the distance between the real object and the reconstruction. In some embodiments, one of Euclidean distance and / or cross-correlation may be selected to quantitatively map such distance in multiple dimensions (2D image or 3D volume). In some embodiments, a distance map may be generated to describe the distance. Conventional spatial processing algorithms such as root mean square error in RSM may generally be sensitive to intensity and micro-movement during measurement. In addition, many of the processing algorithms may be used to generate "absolute differences", which may misleadingly induce relatively high false negative results, overwhelming true negatives in the distance map.

[0093] Based on the above considerations, in some embodiments, an algorithm may be introduced to measure the Fourier Ring Correlation (FRC) or Spectral Signal to Noise Ratio (SSNR), which may describe the distance between two signals in the Fourier domain, i.e., the highest acceptable frequency components between the two signals. In some embodiments, the FRC metric may be utilized as an effective resolution criterion for super-resolution fluorescence and electron microscopy. In some embodiments, the FRC metric may be used to quantify the similarity (or distance) between two signals. The FRC metric may be insensitive to intensity changes and micro-movements, and may quantify the "relative error" or "saliency-based error" (e.g., the most reliable frequency components), and false negatives induced by quantifying the "absolute distance" may be significantly reduced. Also, it may be assumed that the aberrations of the system (i.e., the system used to generate the object's signal) do not change during such two independent observations. Errors due to aberrations (i.e., biased estimates) may be difficult to estimate using simple spatial or frequency algorithms. These errors may be visible because the FRC defines the most reliable frequency components. In some embodiments, the FRC may be used to quantify the distance between two signals. A more detailed description of the FRC may be found elsewhere in this disclosure (e.g., in operation 508 of FIG. 5 and in the operations and descriptions thereof of FIGS. 6A-6D).

[0094] In some embodiments, considering FRC as a global similarity estimate between two images, the FRC metric may be extended to a rolling FRC (rFRC) map to provide a local distance measure at the pixel level. In some embodiments, even without a ground truth, the errors of the multi-dimensional reconstructed signal may be quantitatively mapped. In some embodiments, if the same regions of both images are blank (but the ground truth has content), such information may be lost in the same regions of the image, leading to small FRC values. In other words, if two reconstructed signals are missing identical components (even though this situation is rare), rFRC may introduce false positives. Therefore, in some embodiments, to mitigate such imperfections of the rFRC map, a modified resolution-scaled error map (RSM) may be introduced as an additional error map.

[0095] In some embodiments, the RSM may be constructed based on one or more of three hypotheses.

[0096] (i) Because RSM needs to refer to the corresponding diffraction-limited wide-field image, the wide-field image needs to have a sufficiently high SNR, otherwise the noise contained in the wide-field image may induce a relatively high false negative result.

[0097] (ii) The transformation from the super-resolution (SR) scale to the low-resolution (LR) scale may be approximated as a spatially invariant global Gaussian kernel convolution (in the entire field of view). If the transformation from the super-resolution image to the low-resolution image is not limited to such a spatially invariant Gaussian model, it may generate a relatively high number of false negatives.

[0098] (iii) The illumination intensity and emitted fluorescence intensity must be uniform. The generated RSMs may be weighted by the illumination intensity and emitted fluorescence intensity of the corresponding labels.

[0099] According to the above hypotheses, the RSM itself may introduce false negatives (i.e., model errors) into the estimated error map. In the configuration of single molecule localization microscopy (SMLM), hypotheses (i) and (ii) may be met. For the first hypothesis, the wide-field image of the SMLM may be created by averaging thousands of blinking frames, which may remove noise and generate a reference wide-field image with a relatively high SNR. For the second hypothesis, if the imaging field of view is a relatively small field of view, it may be treated as being generally uniformly illuminated. Furthermore, since the error estimate of such an RSM is calculated at a low-resolution scale, the RSM may be capable of finding errors (usually large error components) at the low-resolution scale, such as misrepresentation or disappearance of structures. The errors (small error components) at the super-resolution scale estimated by the RSM may generally be false negatives. To reduce such potential false negatives, the normalized RSM may be segmented (e.g., by setting them to 0 if they are less than 0.5) to remove the corresponding small error components, leaving the large components to visualize the lost information of the reconstruction that may be ignored by the rFRC map. Overall, the rFRC map and the segmented normalized RSM may be integrated (e.g., in the green and red channels, respectively) to generate a complete PANEL map (see FIG. 16 ), which may accurately and comprehensively map and visualize the error of the reconstructed result. A more detailed description of the determination of the RSM may be found elsewhere in this disclosure (e.g., operation 518 of FIG. 5 , operations of FIG. 7 , and their descriptions). A more detailed description of the determination of the rFRC map may be found elsewhere in this disclosure (e.g., operations 502-516 of FIG. 5 , operations of FIG. 8 , and their descriptions). A more detailed description of the determination of the PANEL map may be found elsewhere in this disclosure (e.g., operation 520 of FIG. 5 , and their descriptions).

[0100] At 502, the processing device 140 (e.g., the acquisition module 402) may acquire a first image and a second image associated with the same object. In some embodiments, the processing device 140 may acquire the first image and / or the second image from one or more components of the image processing system 100. For example, the image acquisition device 110 may collect and / or generate the first image and / or the second image and store the image in the storage device 150 or any other storage associated with the image processing system 100. The processing device 140 may retrieve and / or acquire the image from the storage device 150 or any other storage. As another example, the processing device 140 may acquire the image directly from the image acquisition device 110. In some embodiments, the processing device 140 may acquire one or more initial images, process the initial images, and / or generate the first image and / or the second image. In this disclosure, "image," "image frame," and "frame" are used interchangeably. In some embodiments, the first image and the second image may be 2D images or 3D images. It should be noted that the description of the images (e.g., the first image, the second image) being 2D images is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure.

[0101] The objects may include non-biological objects or biological objects. Exemplary non-biological objects may include phantoms, engineered materials, etc., or any combination thereof. Exemplary biological objects may include biological structures, biological subjects, proteins, cells, microorganisms, etc., or any combination thereof. In some embodiments, the objects may be fluorescent or fluorescently labeled. Fluorescent or fluorescently labeled objects may be excited to fluoresce for imaging.

[0102] In some embodiments, the first and second images may be generated from physical imaging. In some embodiments, the first and second images may be two consecutive images (e.g., frame 1 and frame 2 shown in FIG. 9A) acquired from two image captures of the same object by the image capture device 110. In some embodiments, the first and second images may be acquired from a single image capture of the same object by the image capture device 110. For example, the first and second images may be acquired by decomposing an initial image captured by the image capture device 110. As shown in FIG. 12A, frame 1 represents an initial image, which may be decomposed into two or more images (e.g., a first image, a second image, a third image, and a fourth image, etc.). A more detailed description of generating a target map (e.g., a third target map) based on a single image may be found elsewhere in this disclosure (e.g., FIG. 8 and its description). In some embodiments, the first and / or second images may have a relatively high resolution (e.g., super-resolution).

[0103] In some embodiments, the first and second images may be obtained from two initial images based on a neural network. The processing unit 140 may obtain the first and second initial images associated with the same object from one or more components of the image processing system 100 (e.g., the image acquisition unit 110, the storage unit 150). In some embodiments, the first and / or second initial images may have a relatively low resolution, and the first and / or second images may have a relatively high resolution. In some embodiments, the processing unit 140 (e.g., the neural network processing module 418) may generate the first and second images by processing the first and second initial images, respectively, using a neural network (or an artificial neural network (ANN)). Exemplary neural networks may include a constructive neural network (CNN), a deep learning network, a support vector machine (SVM), a k-nearest neighbor (KNN), a backpropagation neural network (BPNN), a deep neural network (DNN), a Bayesian neural network (BNN), etc., or any combination thereof. In some embodiments, the processing unit 140 may generate the first and second images by processing the first and second initial images, respectively, based on a neural network model. For example, the first and / or second initial images may be input to the neural network model. The neural network model may output the first and / or second images. The first and second images may be two frame images used to generate a first target map (e.g., an rFRC map), as shown at 510. In some embodiments, the rFRC map may reflect errors (e.g., at a pixel level) generated during processing (e.g., reconstruction) using the neural network. As shown in FIG. 13, observer 1 and observer 2 represent the first and second initial images, respectively, and r1 and r2 represent the first and second images output by the neural network, respectively.

[0104] In some embodiments, the first image and / or the second image may be generated from a single initial image using a neural network. In some embodiments, the processing unit 140 (e.g., neural network processing module 418) may generate the first image and the second image by processing the single initial image using a neural network. The neural network reconstruction technique may be different from the physical imaging reconstruction technique. For example, adjacent pixels in an image output from a neural network (e.g., DNN) may be considered to be noise independent. Thus, the determination of an rFRC map based on a single frame used in physical imaging may not be directly applicable to such neural network reconstruction. Different single frame strategies may be used to determine the rFRC map of a neural network (e.g., DNN). In some embodiments, noise (e.g., Gaussian noise) may be added to the initial image. Exemplary noise may include Gaussian noise, impulsive noise, Rayleigh noise, gamma noise, exponential noise, uniform noise, etc., or any combination thereof. For example, a single initial image may be acquired as a first initial image, and a second initial image may be generated by adding noise (e.g., the noise magnitude is 0.5%, 1.5%, 2%, etc.) to the first initial image. Thus, two images (i.e., the first initial image and the second initial image) may be used as inputs of a neural network. Then, after reconstructing these two input images (i.e., the first initial image, the second initial image) respectively using the neural network, the two resulting output images may be acquired as the first image and the second image. The two output images may be further used to determine an rFRC map. In some embodiments, two noises may be added to the initial image to acquire two different images. For example, a first initial image (see the image represented by “observer+n1” in FIG. 13) may be generated by adding a first noise to the initial image, and a second initial image (see the image represented by “observer+n2” in FIG. 13) may be generated by adding a second noise to the initial image.In some embodiments, the type of the first noise may be the same as the type of the second noise, but the magnitude of the noise may be different. In some embodiments, the type of the first noise may be different from the type of the second noise, but the magnitude of the noise may be the same. In some embodiments, the type of the first noise and the type of the second noise may be different, and the magnitude of the first noise and the magnitude of the second noise may be different. Thus, two images (i.e., the first initial image and the second initial image) may be used as inputs of a neural network. Then, after reconstructing the two input images (i.e., the first initial image and the second initial image) respectively using a neural network, two output images may be obtained as the first image and the second image as a result. The two output images may be further used to determine an rFRC map. In some embodiments, the magnitude of the noise added to the initial image may be determined according to a specific situation, and this specification is not intended to limit the scope of the present disclosure.

[0105] At 504, the processing unit 140 (eg, the augmentation module 404) may augment the first image and / or the second image.

[0106] In some embodiments, the processing unit 140 may extend the first and / or second images by performing a padding operation around the first and / or second images, respectively. Performing a padding operation around the images means that data may be padded into one or more regions adjacent to the edges of the images. In some embodiments, the padded regions may have one or more predefined sizes. For example, the padded regions may have a width that is half the width of a block (e.g., the block described in 506). In some embodiments, the padded data may have one or more predefined values, e.g., 0, 1, 2, or any other value. In some embodiments, the padded data may be determined based on pixel values ​​of the image to be extended. In some embodiments, the data may be padded symmetrically around the image (see FIG. 9A).

[0107] At 506, the processing unit 140 (eg, the rolling module 406) may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image.

[0108] In some embodiments, the processing unit 140 may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image by performing a rolling operation on the first image and the second image, respectively. In some embodiments, the rolling operation may be performed by sliding a window over the images (e.g., the (expanded) first image and the (expanded) second image). In the rolling operation, the window may slide row-by-row and / or column-by-column over the images, and a designated center of the window may traverse one or more pixels (e.g., each pixel) of the images. In some embodiments, the window may have a predefined size (e.g., 32×32, 64×64, 128×128, etc.). The predefined size of the window may be determined by the image processing system 100 or may be preset by a user or operator via the terminal 130. In some embodiments, the size of the window may be related to the size of the first image and / or the second image. For example, the window may have a size smaller than the size of the first image and / or the second image. The window may have various shapes. For example, the window may be square, rectangular, circular, oval, etc. In some embodiments, the window used for the first image may be the same as the window used for the second image.

[0109] In some embodiments, a window may be slid pixel-by-pixel over the (extended) first image to generate a plurality of first blocks of the first image. As used herein, a block of an image may mean a portion of an image. When a window slides over an image (e.g., the (extended) first image), a portion of the image (e.g., the (extended) first image) may be enclosed by the window, and the portion of the image (e.g., the (extended) first image) enclosed by the window may be designated as a block of the image (e.g., the first block of the first image). Thus, a plurality of first blocks of the first image may be determined by sliding the window over the (extended) first image. Similarly, a plurality of second blocks of the second image may be determined by sliding the window over the (extended) second image. In some embodiments, the plurality of second blocks and the plurality of first blocks may have a one-to-one correspondence. In some embodiments, the number of first blocks (or second blocks) may be the same as the number of pixels of the first image (or second image) acquired at 502.

[0110] Although rFRC is an error map that allows evaluation at pixel level, the finest scale of detectable error can only reach the highest resolution of the corresponding images (e.g., first image and second image). Thus, if the images (e.g., first image, second image) satisfy the Nyquist-Shannon sampling theorem, pixel-by-pixel rolling operations may not be necessary. In some embodiments, the minimum error may be greater than 3×3 pixels. This allows 2-4 pixels to be skipped for each rolling operation to obtain a 4×-16× acceleration for the rFRC mapping operation. In some embodiments, the first blocks (or the second blocks) may be determined by sliding a window over the (extended) first image (or the (extended) second image) and skipping a predetermined number of pixels over the (extended) first image (or the (extended) second image). For example, when the window slides to a position of the (extended) first image (or the (extended) second image), a first block of the first image (or a second block of the second image) may be determined, and then the window may skip a predetermined number of pixels on the (extended) first image (or the (extended) second image) and slide to a next position of the (extended) first image (or the (extended) second image) to determine another first block of the first image (or another second block of the second image). Thereby, the number of first blocks (or second blocks) may be less than the number of pixels of the first image (or the (extended) second image) acquired in 502. The predetermined number of pixels may be determined by the image processing system 100 or may be preset by a user or operator via the terminal 130. In some embodiments, the predetermined number of pixels may be associated with the size of the window. For example, the predetermined number of pixels (e.g., 2, 3, or 4) may be close to the width of the window (e.g., 3). In some embodiments, the predetermined number of pixels may be random.

[0111] At 508, the processing unit 140 (eg, the map generation module 408) may determine a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks.

[0112] The first characteristic value may refer to a value associated with a pixel value of the first target map (i.e., rFRC map). For example, the first characteristic value corresponding to a pixel of the first target map may be associated with a correlation between a first block and a second block whose center pixel corresponds to the pixel of the first target map. In some embodiments, at least one of the first characteristic values ​​may be determined based on a correlation between one of the first blocks and a corresponding second block. The correlation between the first block and the corresponding second block may be represented by a correlation value (or referred to as an FRC value). In some embodiments, the correlation between the first block and the corresponding second block may be determined by performing a Fourier transform on the first block and the second block, respectively. For example, the processing unit 140 may determine a first intermediate image in the frequency domain based on the first block (e.g., by performing a Fourier transform on the first block). The processing unit 140 may determine a second intermediate image in the frequency domain based on the second block (e.g., by performing a Fourier transform on the second block). In some embodiments, the processing device 140 may determine a target frequency (also referred to as a cutoff frequency) based on the first intermediate image and the second intermediate image. The processing device 140 may determine a first characteristic value based on the target frequency. For example, the processing device 140 may determine the target frequency, the inverse of the target frequency, or the product of the inverse of the target frequency and the pixel size of the first image (and / or the second image) (e.g., the physical dimensions of the camera pixel divided by the magnification of the image (first image, second image, etc.)) as the first characteristic value, etc. More detailed descriptions regarding determining multiple first characteristic values ​​may be found elsewhere in this disclosure (e.g., FIGS. 6A-6D).

[0113] In some embodiments, background regions may exist in the first image and / or the second image. In some embodiments, the information of the background regions may remain unchanged or may change slightly in the first image and the second image. In some embodiments, the FRC calculation for the background regions may be avoided to reduce the computational burden. The FRC calculation may refer to the operation of determining an FRC value. In this case, the first characteristic value for the background region may be determined based on one or more predetermined values. In some embodiments, for the first block and the corresponding second block, the processing unit 140 may determine whether to perform the FRC calculation (or whether the first block and / or the corresponding second block are background regions) before determining the first characteristic value. A more detailed description of determining the first characteristic value may be found elsewhere in this disclosure (e.g., FIG. 6A).

[0114] At 510, the processing device 140 (eg, the map generation module 408) may generate a first target map associated with the first image and the second image based on the first plurality of characteristic values.

[0115] As used herein, a first target map (e.g., an rFRC map) may refer to an image indicative of reconstruction errors and / or artifacts of the first image and / or the second image acquired in 502. The first target map may include a plurality of pixels, and a value of each pixel may be determined based on the first characteristic value. The first target map may have the same size as the first image and / or the second image. In some embodiments, the pixel value determined based on the first characteristic value may be assigned to a pixel (of the target map) that corresponds to a central pixel of the first block or the second block (used to determine the first characteristic value). That is, the pixel of the first target map may correspond to a pixel of the first image and / or the second image.

[0116] In some embodiments, the first target map may reflect the image quality (e.g., the first image and / or the second image) at the pixel level, since there is a one-to-one correspondence between the pixels of the first target map and the pixels of the first image (and / or the second image). In some embodiments, if the pixel values ​​of the pixels of the first target map are determined as the inverse of the target frequency, i.e., meaning the resolution, the smaller the pixel value (i.e., the resolution), the higher the pixel accuracy of the pixels of the first image and / or the second image, and the larger the pixel value (i.e., the resolution), the lower the pixel accuracy of the pixels of the first image and / or the second image. A relatively low pixel accuracy here may indicate an error. In some embodiments, if the pixel values ​​of the pixels of the first target map are determined as the target frequency, i.e., meaning the spatial frequency, the smaller the pixel value (i.e., the spatial frequency), the lower the pixel accuracy of the pixels of the first image and / or the second image, and the larger the pixel value (i.e., the spatial frequency), the higher the pixel accuracy of the pixels of the first image and / or the second image. In some embodiments, when the pixel value of a pixel in the first target map is determined as the product of the inverse of the target frequency and the pixel size of the first image and / or the second image, a smaller pixel value represents a higher accuracy of the pixel in the first image and / or the second image, and a larger pixel value represents a lower accuracy of the pixel in the first image and / or the second image.

[0117] In some embodiments, the first target map may quantitatively present the resolution of each pixel of the first image and / or the second image, thereby indicating errors (and / or artifacts) present in the first image and / or the second image. Thus, using a target map (e.g., the first target map), the errors (and / or artifacts) may be more intuitively visualized. By way of example only, if the first image and / or the second image have a resolution of 100 nm and the resolution of a pixel is determined to be 10 nm, it may indicate that the pixel is accurate and there is no error in the reconstruction of the pixel, and if the first image and / or the second image have a resolution of 100 nm and the resolution of a pixel is determined to be 200 nm, it may indicate that the pixel is not accurate and there is an error in the reconstruction of the pixel.

[0118] In some embodiments, the first target map may be represented in the form of a grayscale image. In some embodiments, the grayscale value of each pixel of the first target map may be positively correlated with the first characteristic value, and the grayscale of the first target map may reflect the image quality (or error) of the first image and / or the second image.

[0119] In some embodiments, if a skip operation is performed at 506, the first target map may be resized (e.g., by bilinear interpolation) to the original image size of the first image and / or the second image for better visualization.

[0120] At 512, the processing unit 140 (e.g., the filtering module 410) may filter the first target map. In some embodiments, the processing unit 140 may filter the first target map using an adaptive median filter.

[0121] In some embodiments, according to the Nyquist sampling theorem, the minimum error of the first image and / or the second image that the motion of the present disclosure can detect may be greater than, for example, 3×3 pixels. Thus, the rolling motion in a small region (e.g., 3×3 pixels) may change smoothly without any isolated pixels (e.g., very high-value pixels or very low-value pixels compared to neighboring pixels). However, due to improper determination of the target frequency, the FRC calculation may become unstable and abnormal characteristic values ​​may be determined. In some embodiments, an adaptive median filter may be used to remove these improper characteristic values ​​in these pixels. In some embodiments, the adaptive median filter may replace each pixel with the median value of the neighboring pixel values ​​in the filtering window of the first target map. In some embodiments, a threshold value may be used in the adaptive median filter. For example, if the pixel value (e.g., gray value) of a pixel is greater than the product of the threshold value and the median value of the neighboring pixel values ​​in the filtering window, the pixel value may be replaced with the median value, otherwise the filtering window may move to the next pixel. By using the adaptive median filter, the isolated pixels may be filtered without blurring the rFRC map. In some embodiments, the threshold value (eg, 2) may be determined by image processing system 100 or may be preset by a user or operator via terminal 130.

[0122] At 514, the processing unit 140 (eg, image quality determination module 414) may determine a global image quality metric for the first image and / or the second image based on the first target map.

[0123] The rFRC map generated at 510 may be used to quantitatively visualize errors contained in the images (e.g., the first and second images). In addition to such local quality assessments, a global image quality metric may be determined for the first and / or second images. In some embodiments, the global image quality metric may have a dimension as a resolution. In some embodiments, the global image quality metric may be normalized without considering the dimension.

[0124] In some embodiments, a global image quality metric with respect to the dimension being the resolution may be expressed as:

[0125]

number

[0126] Here, ||FV||0 is the l0 norm representing the number of non-zeros in the rFRC map, and FV represents the rFRC map.

[0127] In some embodiments, the global image quality metric normalized without considering dimensionality may be expressed as:

[0128]

number

[0129] Here, ||FV||0 is the l0 norm representing the number of non-zeros in the rFRC map, and FV represents the rFRC map.

[0130] In addition, these two metrics shown in equations (13) and (14) may both be extended to three dimensions, and the two-dimensional coordinates (x, y) in equations (13) and (14) may be directly raised to three-dimensional coordinates (x, y, z).

[0131] At 516, the processing unit 140 (eg, the display module 412) may display the first target map using the displacement jet color map.

[0132] In some embodiments, the first target map may be displayed using one or more color maps. The color maps may include one or more existing common color maps, such as a jet color map. The jet color map may use colors from blue to red that index different error magnitudes to highlight all error regions. In some embodiments, such a jet color map may be in some way counter to human intuition or habits. The human visual system may intend to define black (dark colors) as small magnitudes and light (pale colors) as large magnitudes, which is the same as the logic of a shade color map. In some embodiments, human vision may be insensitive to pale or dark shade levels and more sensitive to different colors. Therefore, a color map (e.g., a displacement jet color map) may be used that includes color indexes to better match human intuition.

[0133] In some embodiments, the jet color map (see FIG. 14A) may be displaced to create a displaced jet (sJet) color map (see FIG. 14B) in which red components at relatively low magnitudes and green components at relatively high magnitudes are removed and blue components are expanded to the entire color index. In some embodiments, considering that human intuition may be more sensitive to green, green may be moved to a relatively large magnitude to highlight large errors. In some embodiments, humans may be more accustomed to light colors (e.g., white) as a relatively large magnitude and dark colors (e.g., black) as a relatively small magnitude, and accordingly, a black zone (0,0,0) and a white zone (1,1,1) may be involved in the displaced jet (sJet) color map. In some embodiments, with the expansion of the blue component, a white zone may be created to represent the maximum error. In some embodiments, the background of the rFRC map may mean positive (i.e., no error), and accordingly, a black zone (rather than the blue of the jet color map) may be used to display such a background, which is more suitable for human intuition. As shown in Figures 14C-14D, compared with the original Jet color map (see Figure 14C), the sJet color map (see Figure 14D) would obviously be more intuitive for the visualization of the error map.

[0134] At 518, the processing unit 140 (eg, the map generation module 408) may determine a reference map associated with the first image or the second image.

[0135] In some embodiments, the reference map may include a modified resolution-scaled error map (RSM). The reference map may reflect errors and / or artifacts in the first image and / or the second image. In some embodiments, the reference map may be in the form of a grayscale map.

[0136] In some embodiments, the reference map may be determined based on a difference between the intermediate image and a reference image (e.g., a relatively low resolution image). The intermediate image may be determined based on the first image (or the second image) and a convolution kernel. The convolution kernel may be determined based on the first image (or the second image) and the reference image. A more detailed description of the determination of the reference map may be found elsewhere in this disclosure (e.g., FIG. 7 and its description).

[0137] It should be noted that in some embodiments, if the first image (or the second image) is three-dimensional or in a non-Gaussian convolution relationship (e.g., between a low-resolution scale and a high-resolution scale), the reference map may not be determined, i.e., operation 518 may be omitted. Thus, for such first or second images, the RSM may not involve a fourth target map, and operation 520 may also be omitted.

[0138] At 520, the processing unit 140 (eg, the map generation module 408) may determine a fourth target map by fusing the first target map and the reference map.

[0139] In some embodiments, the first target map (e.g., rFRC map) may illustrate small and subtle errors in the reconstruction of the first image (and / or the second image). In some embodiments, the first target map may be impossible when structures are simultaneously missing in both the first and second images. By way of example only, a 2D SMLM dataset may be used to illustrate this possible false negative (see Figs. 17A-17C). As shown by the light arrows in Fig. 17A, a small portion of a filament may be artificially removed in the same region of the two reconstructions. As shown in Fig. 17B, the rFRC map may successfully detect most of the error components in the image, except for the artificially removed region. In some embodiments, such loss of structure may be detected by a reference map (e.g., RSM (see Fig. 17C)). In some embodiments, the reference map (e.g., RSM) may be introduced as an additional error map for the first target map (e.g., rFRC map).

[0140] The fourth target map may reflect the overall errors and / or artifacts of the first image (and / or the second image) indicated by the first target map and the reference map. The fourth target map may be referred to as a PANEL map.

[0141] In some embodiments, the processing unit 140 may process the first target map and / or the reference map in the form of a grayscale map. In some embodiments, the processing unit 140 may convert the first target map and the reference map into an RGB image. For example, the first target map may be displayed in green (as shown in the left panel of FIG. 11A ) and the reference map may be displayed in red (as shown in the right panel of FIG. 11A ). In some embodiments, the processing unit 140 may normalize the first target map and / or the reference map, for example, by variance normalization, mean normalization, etc. In some embodiments, the processing unit 140 may determine the fourth target map based on a weighted sum of the normalized first target map and the reference map.

[0142] In some embodiments, the processing unit 140 may perform a thresholding operation on the first target map and / or the reference map to filter background regions therein, thereby highlighting important information (e.g., critical errors in the reconstruction of the first image and / or the second image) (see FIG. 11B). In some embodiments, the threshold of the first target map may be the same as or different from the threshold of the reference map. In some embodiments, the threshold of the first target map and / or the reference map may range from 0 to 1.

[0143] In some embodiments, the threshold of the reference map may be determined by the image processing system 100 or may be preset by a user or operator via the terminal 130. For example, the threshold of the reference map may be 0.5. In some embodiments, the same missing information (e.g., misrepresentation or disappearance of a structure) in both the first image and the second image may have a relatively large magnitude and may be present in a relatively large area of ​​the reference map. In some embodiments, components of small magnitude included in the reference map may induce a relatively high false negative. In some embodiments, the reference map may be segmented based on the threshold by the following formula before involving the reference map in the fourth target map:

[0144]

number

[0145] where R(x,y) represents the normalized RSM value at coordinate (x,y) and R% represents the segmented RSM. In some embodiments, low false negatives (e.g., less than 0.5) may be filtered according to equation (15) to leave only relatively strong low-resolution errors and focus on true negatives detected by the reference map.

[0146] In some embodiments, the rFRC map may indicate the degree of error, and the smallest FRC value in the map may indicate no acceptable error. In some embodiments, the threshold of the first target map may be determined by Otsu's binarization method (also called Otsu for short) used to perform automatic image thresholding. Using Otsu's method, the threshold may be automatically determined by maximizing the inter-class variance. Image thresholding may be performed to filter the background of the rFRC map, thereby highlighting critical errors in the reconstruction of the first and second images. An exemplary threshold of the first target map may be 0.4. Those pixels whose pixel values ​​are smaller than the two thresholds may be filtered from the first target map and the reference map, respectively. For example, the pixel values ​​of such pixels may be designated as 0.

[0147] In some embodiments, the processing unit 140 may determine a fourth target map (e.g., a PANEL map) by fusing the first target map and the reference map to collectively indicate the reconstruction errors and / or artifacts of the first image and / or the second image (see FIG. 11C). In some embodiments, the fourth target map may be determined by fusing the third target map and the reference map determined in process 800.

[0148] It should be noted that the above description of process 500 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. Upon understanding the principles of operation, one of ordinary skill in the art would understand to combine any of the operations, add or remove any of the operations, or apply the principles of operation to other image processing processes without departing from the principles. In some embodiments, one or more operations in process 500 may be omitted. For example, operation 504, operation 512, operation 514, operation 516, operation 518, and / or operation 520 may be omitted. In some embodiments, two or more operations in process 500 may be combined into one operation. For example, operations 506-510 may be combined into one operation. As another example, operations 518-520 may be combined into one operation. As a further example, operations 506-510 and operations 518-520 may be combined into one operation.

[0149] In some embodiments, the first and second images may be 3D images. The procedure for processing the 3D images may be similar to the procedure for processing the 2D images. The rolling operation in 506 may be operable in three dimensions. For example, the processing unit 140 may perform the rolling operation in a manner that performs a 3D filter. The window used for the rolling operation in three dimensions may be cubic in shape, and therefore the first and second blocks may be cubic in shape. In some embodiments, the FRC operation may be extended to a 3D version called Fourier Shell Correlation (FSC). Thus, the first and second ring images (see FIG. 6D) generated from the first and second blocks may be spherical. In some embodiments, to avoid the relatively heavy computational burden, the three-dimensional rFRC operation may be generated directly from the 2D rFRC operation per plane.

[0150] FIG. 6A is a flow chart illustrating an example process for determining whether to perform an FRC operation on a first block or a second block, according to some embodiments of the present disclosure. In some embodiments, the process 600 may be performed by the image processing system 100. For example, the process 600 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules shown in FIG. 4) may execute the instruction set and may be configured to perform the process 600 accordingly. The illustrated process operations presented below are intended to be illustrative. In some embodiments, the process 600 may be accomplished with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of the process 600 as shown in FIG. 6A and described below is not intended to be limiting. In some embodiments, operation 508 shown in FIG. 5 may be performed in accordance with one or more operations of process 600.

[0151] As explained above, FRC may measure the global similarity between two images (e.g., a first image and a second image), and the FRC value may be extended in the form of a rolling FRC (rFRC) map to provide a local distance measure at the pixel level. In some embodiments, the form of FRC computation may be considered as a filter where the image is processed block by block, and each block is assigned a corresponding FRC value. In some embodiments, as shown at 504, the images (e.g., the first image and / or the second image) may be symmetrically padded with half the size of the block to facilitate FRC computation on the image boundaries. In some embodiments, a background threshold may be set for a central pixel of a block (e.g., 1×1 pixels to 3×3 pixels) to determine whether to determine an FRC value for this block, thereby avoiding FRC computation on background regions. In some embodiments, if the average of the central pixels of the block is greater than the background threshold, the FRC computation may be performed and an FRC value may be assigned to the central pixel of the block. In some embodiments, if the average is less than the background threshold, a zero value may be set to the central pixel of the block. This procedure block may be performed block by block until the entire image has been processed.

[0152] At 601, processing unit 140 (e.g., map generation module 408) may determine an average value of pixel values ​​of pixels in a central region of each first block and / or an average value of pixel values ​​of pixels in a central region of each second block. In some embodiments, the central region may be located in or near the center of each first block or each second block. The central region may be any shape, e.g., square, circular, etc. In some embodiments, the size of the central region may be smaller than or equal to the size of each first block or each second block.

[0153] The average pixel value of a pixel may be the sum of the pixel values ​​divided by the total number of pixels in the central region. In some embodiments, the processing unit 140 may determine an average pixel value of the pixels in the central region of each first block and the pixels in the central region of each second block. In some embodiments, the processing unit 140 may determine a plurality of average values ​​corresponding to a plurality of first blocks and / or a plurality of average values ​​corresponding to a plurality of second blocks. In some embodiments, the processing unit 140 may determine a plurality of average values ​​corresponding to a plurality of first blocks and a plurality of second blocks.

[0154] At 602, the processing unit 140 (eg, the map generation module 408) may compare the average value to a threshold value (eg, a background threshold value).

[0155] The first image and / or the second image may have background regions. In some embodiments, the background regions may have noise (e.g., background noise, readout noise). In some embodiments, such background regions may result in relatively large FRC values, which may introduce false negatives during generation of the rFRC map, thereby preventing highlighting true negatives (e.g., errors generated in image reconstruction). Therefore, a background threshold may be set to avoid such FRC calculations for these background regions. In this disclosure, true negative refers to a true error determined by the processing unit 140, false negative refers to a false error determined by the processing unit 140, and false positive refers to an error determined by the processing unit 140 to be not an error.

[0156] In some embodiments, the threshold may be determined by the image processing system 100 or may be preset by a user or operator via the terminal 130. In some embodiments, a fixed threshold may be used for the entire first image and / or the entire second image (i.e., for multiple first blocks of the first image and / or multiple second blocks of the second image). In some embodiments, an adaptive threshold may be used. In some embodiments, an adaptive threshold may be determined depending on the (each) first block and / or the (each) second block. In some embodiments, one or more thresholds may be adaptively determined for small local regions of the first image (or first block) or the second image (or second block). That is, different thresholds may be used for different small local regions. In some embodiments, an iterative wavelet transform may be performed to determine the (adaptive) threshold. Merely by way of example, a local background intensity distribution of the corresponding images (e.g., the first image, the second image, the first block, and the second block, etc.) may be estimated. Specifically, the background may be iteratively estimated from the lowest frequency wavelet band of the corresponding image, and at each iteration, one or more (e.g., all) values ​​of the corresponding image that are above the current estimated background level may be clipped.

[0157] At 603, the processing unit 140 (e.g., map generation module 408) may determine whether the average value is greater than a threshold value. In some embodiments, in response to determining that the average value is greater than or equal to the threshold value, process 600 may proceed to operation 605. In some embodiments, in response to determining that the average value is not greater than or equal to the threshold value, process 600 may proceed to operation 604.

[0158] In some embodiments, the processing unit 140 may compare the average value for each first block, the average value for each second block, and / or the average value for each first block and each second block to a threshold value. In some embodiments, in response to determining that the average value for each first block and the average value for each second block are both greater than or equal to the threshold value, the process may proceed to operation 605; otherwise, the process 600 may proceed to 604. In some embodiments, in response to determining that the average value for each first block and each second block are greater than or equal to the threshold value, the process may proceed to operation 605; otherwise, the process 600 may proceed to 604.

[0159] At 604, the processing unit 140 (e.g., the map generation module 408) may determine the characteristic values ​​(e.g., the first characteristic values ​​corresponding to each first block and / or each second block) based on a predetermined value. For example, the processing unit 140 may assign a predetermined value to the characteristic values. The predetermined value may be, for example, 0. Thus, if the average value of the first block and / or the second block is not greater than the threshold, the characteristic values ​​corresponding to the first block and the second block may be 0, thereby avoiding unnecessary FRC calculations for the background region.

[0160] At 605, the processing unit 140 (eg, the map generation module 408) may determine a characteristic value based on each of the first blocks and each of the second blocks.

[0161] If the average value is greater than the threshold, the processing unit 140 may determine the characteristic value by performing an FRC operation based on each of the first blocks and each of the second blocks. In some embodiments, the processing unit 140 may determine the characteristic value based on a first intermediate image and a second intermediate image obtained by performing a Fourier transform on each of the first blocks and each of the second blocks. A more detailed description of determining the characteristic value by performing an FRC operation may be found elsewhere in this disclosure (e.g., FIG. 6B and its description).

[0162] The processing unit 140 may perform the process 600 for all the first blocks and / or the second blocks to determine the plurality of characteristic values.

[0163] FIG. 6B is a flow chart illustrating an exemplary process for determining characteristic values, according to some embodiments of the present disclosure. In some embodiments, the process 610 may be performed by the image processing system 100. For example, the process 610 may be configured as a set of instructions (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules illustrated in FIG. 4) may execute the set of instructions and may be configured to perform the process 610 accordingly. The illustrated process operations presented below are intended to be illustrative. In some embodiments, the process 610 may be accomplished with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of the process 610 as illustrated in FIG. 6B and described below is not intended to be limiting. In some embodiments, operation 605 shown in FIG. 6A may be performed in accordance with one or more operations of process 610.

[0164] At 611, the processing unit 140 (eg, the map generation module 408) may determine a first intermediate image and a second intermediate image in the frequency domain based on the first block and the second block, respectively.

[0165] In some embodiments, the processing unit 140 may determine a first intermediate image in the frequency domain by performing a Fourier transform on each first block. In some embodiments, the processing unit 140 may determine a second intermediate image in the frequency domain by performing a Fourier transform on each second block. As used herein, the first intermediate image may be considered as a first transformed image (in the frequency domain) from the first block, and the second intermediate image may be considered as a second transformed image (in the frequency domain) from the second block. The Fourier transform may be used to transform information of an image or block (e.g., the first block and the second block) from the spatial domain to the frequency domain. In some embodiments, the Fourier transform may be a discrete Fourier transform (DFT). In some embodiments, each first block may correspond to a first intermediate image, and each second block may correspond to a second intermediate image. Thus, multiple first intermediate images corresponding to multiple first blocks may be determined, and multiple second intermediate images corresponding to multiple second blocks may be determined.

[0166] At 612, the processing unit 140 (e.g., the map generation module 408) may determine a target frequency based on the first intermediate image and the second intermediate image. In some embodiments, each first intermediate image and the corresponding second intermediate image may be used to determine a target frequency. Thus, multiple target frequencies may be determined based on multiple first intermediate images and multiple second intermediate images.

[0167] In some embodiments, a target frequency may be determined using a criterion to quantitatively determine the characteristic value. For example, when the FRC curve falls below a threshold at a certain point, the frequency corresponding to that point may be defined as the target frequency. The FRC threshold may indicate the spatial frequency location of meaningful information above the random noise level. In some embodiments, a fixed threshold or sigma factor curve (see curve b in FIG. 9C) may be used to determine the threshold. For example, a fixed value of 1 / 7 of the FRC curve may be used as a hard threshold.

[0168] In some embodiments, the target frequency may be determined based on a graph (see FIG. 9C) with spatial frequency on the horizontal axis and correlation value on the vertical axis. The target frequency may correspond to an intersection of a first curve (i.e., curve a in FIG. 9C) and a second curve (i.e., curve b in FIG. 9C). The first curve may indicate a first relationship between the correlation values ​​and the frequencies based on the first intermediate image and the second intermediate image. The second curve may indicate a second relationship between the correlation values ​​and the frequencies based on a predetermined function. A more detailed description of the determination of the target frequency may be found elsewhere in this disclosure (e.g., FIG. 6C and FIG. 6D and their description).

[0169] In 613, the processing unit 140 (e.g., the map generation module 408) may determine a characteristic value based on the target frequencies. In some embodiments, the processing unit 140 may designate each target frequency as a characteristic value. In some embodiments, the processing unit 140 may determine a characteristic value based on the inverse of the target frequency. For example, the processing unit 140 may designate the inverse of the target frequency as the characteristic value. In some embodiments, the processing unit 140 may determine a characteristic value based on the inverse of the target frequency and a pixel size of the first image and / or the second image acquired in 502. For example, the processing unit 140 may designate the product of the inverse of the target frequency and the pixel size as the characteristic value. As shown in FIG. 9D, FV represents a characteristic value (e.g., an rFRC value) determined based on the target frequency determined based on the first block and the second block.

[0170] FIG. 6C is a flow chart illustrating an exemplary process for determining a target frequency, according to some embodiments of the present disclosure. In some embodiments, the process 620 may be performed by the image processing system 100. For example, the process 620 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules illustrated in FIG. 4) may execute the instruction set and may be configured to perform the process 620 accordingly. The illustrated process operations presented below are intended to be illustrative. In some embodiments, the process 620 may be accomplished with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of the process 620 as illustrated in FIG. 6C and described below is not intended to be limiting. In some embodiments, operation 612 shown in FIG. 6B may be performed in accordance with one or more operations of process 620.

[0171] At 621, the processing unit 140 (eg, the map generation module 408) may determine a first relationship between the plurality of correlation values ​​and the plurality of frequencies based on the first intermediate image and the second intermediate image.

[0172] In some embodiments, the processing unit 140 may determine a plurality of correlation values ​​between each first intermediate image and a corresponding second intermediate image. In some embodiments, the correlation values ​​may be determined based on a first ring image associated with the first intermediate image and a corresponding second ring image associated with the second intermediate image. Thus, the plurality of correlation values ​​may be determined based on a plurality of first ring images associated with the first intermediate image and a plurality of second ring images associated with the second intermediate image. A more detailed description of determining correlation values ​​may be found elsewhere in this disclosure (e.g., FIG. 6D and its description).

[0173] In some embodiments, each of the correlation values ​​may correspond to one frequency of the plurality of frequencies. The correlation values ​​determined based on the first ring image and the corresponding second ring image may correspond to the frequency of the first ring image and / or the second ring image. In some embodiments, a first curve (also referred to as an FRC curve) may show a first relationship between the plurality of correlation values ​​and the plurality of frequencies. For example, the first curve may be shown as curve a in the graph shown in FIG. 9C.

[0174] At 622, the processing unit 140 (eg, the map generation module 408) may determine a second relationship between the plurality of correlation values ​​and the plurality of frequencies based on a predetermined function.

[0175] In some embodiments, the second relationship may be represented by a second curve. In some embodiments, the second curve may be a threshold curve. In some embodiments, the second relationship may be determined based on a predetermined function described as equation (16). The threshold curve may also be referred to as a sigma factor curve or a 3σ curve, as shown in curve b of FIG. 9C.

[0176]

number

[0177] Here, N i is the radius q i represents the number of pixels of the first ring image and / or the second ring image, and σ 因子 may be 3. The extra factor of "2" may be needed because the FRC summation operation involves all Hermitian pairs in the Fourier domain. If two images (e.g., first block, second block) contain no signal but entirely noise, the second curve is i =1 / √N iThus, the corresponding 3σ curve means that three times the standard deviation of the expected random noise fluctuation may be selected as the threshold for determining the target frequency. In some embodiments, the second curve represented by Equation (16) may be more suitable for PENEL to identify the category (noise or not) of components in an image (e.g., the first image, the second image). Note that σ in Equation (16) 因子 may be other values, e.g., 5, 7, 9, etc., and is not intended to be limiting.

[0178] In some embodiments, the second curve may be a straight line since the threshold for determining the target frequency may be a fixed value. The threshold may be determined, for example, as 1 / 7 of the FRC curve. In some embodiments, the second curve may indicate frequencies of meaningful information above the random noise level. For example, information corresponding to frequencies smaller than the second curve may be less certain, whereas information corresponding to frequencies larger than the second curve may be more certain.

[0179] In some embodiments, the 3σ curve may be suitable for small images due to its robustness and accuracy, and may be adapted to determine the local resolution (corresponding to the cutoff frequency). A fixed threshold value may be suitable for large images, and may be adapted to determine the traditional global resolution. A more detailed explanation of the comparison between the two second curves may be found elsewhere in this disclosure (e.g., Example 11).

[0180] At 623, the processing unit 140 (e.g., map generation module 408) may determine a target frequency based on the first relationship and the second relationship. In some embodiments, the target frequency may correspond to an intersection point between a first curve illustrating the first relationship and a second curve illustrating the second relationship. The target frequency may be a frequency at the intersection point. Multiple target frequencies may be determined for multiple first intermediate images and corresponding second intermediate images.

[0181] FIG. 6D is a flow chart illustrating an exemplary process for determining a correlation value, according to some embodiments of the present disclosure. In some embodiments, the process 630 may be performed by the image processing system 100. For example, the process 630 may be configured as a set of instructions (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules illustrated in FIG. 4) may execute the set of instructions and may be configured to perform the process 630 accordingly. The illustrated process operations presented below are intended to be illustrative. In some embodiments, the process 630 may be accomplished with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of the process 630 as illustrated in FIG. 6D and described below is not intended to be limiting. In some embodiments, operation 621 shown in FIG. 6C may be performed in accordance with one or more operations of process 630.

[0182] At 631, processing unit 140 (eg, map generation module 408) may determine a first ring image from the first intermediate image based on one of the multiple frequencies.

[0183] The first ring image refers to a portion of the first intermediate image (in the frequency domain) that is ring-shaped with a diameter equal to the frequency. The first ring image may include pixels in a ring (see light gray circle in FIG. 9B) with a diameter equal to the frequency. The frequency may range from 0 to half the reciprocal of the pixel size of the first image or the second image. The first ring image may correspond to a frequency, and accordingly multiple first ring images may be determined based on multiple frequencies.

[0184] At 632, the processing unit 140 (e.g., the map generation module 408) may determine a second ring image from the second intermediate image based on the frequency. The second ring image may include pixels within a ring (see light gray circle in FIG. 9B) having a diameter equal to the frequency. The second ring image may be determined in a similar manner to the first ring image. The second ring image may correspond to a frequency, and accordingly, multiple second ring images may be determined based on multiple frequencies. The multiple second ring images may have a one-to-one correspondence with the multiple first ring images. The second ring image and the corresponding first ring image may correspond to the same frequency.

[0185] At 633, the processing unit 140 (e.g., the map generation module 408) may determine a correlation value based on the first ring image and the second ring image. In some embodiments, the processing unit 140 may determine multiple correlation values ​​based on multiple first ring images and multiple second ring images.

[0186] In some embodiments, the processing unit 140 may determine the correlation value through a Fourier Ring Correlation (FRC) (or so-called Spectral Signal to Noise Ratio) operation. The FRC operation may measure the statistical correlation between two images (e.g., a first block, a second block) over a series of concentric rings (i.e., ring images) in the Fourier domain. In some embodiments, FRC may be proposed as a way to measure image resolution in super-resolution microscopy. The FRC operation may be performed to determine the correlation value based on frequency. The correlation value (or FRC value) is the correlation coefficient (or FRC value) for a corresponding spatial frequency q i It may be a function with variables, and may be determined based on equation (17).

[0187]

number

[0188] where F1 and F2 represent the first and second intermediate images after a Fourier transform (e.g., DFT) is performed;

number

[0189] Additionally, a Hamming window may be used in the FRC operation to suppress edge effects and other spurious correlations caused by the DFT operation. An exemplary Hamming window may be defined as follows:

[0190]

number

[0191] Here, N represents the number of elements in the mask (eg, the image size of the first intermediate image or the second intermediate image), and the coefficients α and β may be set to 0.5.

[0192] In some embodiments, in the FRC calculation or in the determination of the first and / or second ring images in 631-632, it is necessary to discretize the spatial frequencies of the first and second intermediate images. In some embodiments, it is necessary to define a discretization of the spatial frequencies of the FRC curve in order to determine the corresponding discrete values ​​of the spatial frequencies. In some embodiments, the maximum frequency f max is the pixel size of the first or second image (p s ), i.e., half the reciprocal of f max =1 / (2p s ) In some embodiments, a non-square image (i.e., a ring image) may be padded (e.g., zero padded) to a square image to calculate the FRC curve. In some embodiments, the FRC curve may include N / 2 points, and the discretization step (Δf) may be obtained for the spatial frequency by equation (19).

[0193]

number

[0194] Here, N represents the number of elements in the mask (e.g., the image size of the first intermediate image or the second intermediate image). max represents the maximum frequency, and p s represents the pixel size.

[0195] In some embodiments, an averaging filter with a half-width of the averaging window (eg, equal to 3 frequency bins) may be applied to smooth the FRC curve.

[0196] It should be noted that the above description of process 600, process 610, process 620, and process 630 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. Upon understanding the principles of operation, a person skilled in the art would understand to arbitrarily combine any operations, add or remove any operations, or apply the principles of operation to other image processing processes without departing from the principles. In some embodiments, one or more operations in process 600 may be omitted. For example, operations 601 to 604 may be omitted. That is, processing unit 140 may perform FRC operations on each first block and / or each second block without filtering the background region. In some embodiments, process 610 may be integrated into process 600. In some embodiments, process 620 may be integrated into process 610. In some embodiments, one or more operations may be integrated into one operation. For example, operations 631 and 632 may be integrated into one operation.

[0197] An exemplary FRC computation process may be illustrated in Figures 9A-9D. Figures 9A-9D illustrate an exemplary process for determining a target map according to some embodiments of the present disclosure. Figures 9A-9D are provided for illustrative purposes only and are not intended to limit the scope of the present disclosure. As illustrated in Figure 9A, a first image (frame 1) and a second image (frame 2) are 4x4 in size. Frame 1 (and frame 2) are expanded by symmetrically padding around the frame using some pixel values ​​of frame 1 (and frame 2). For example, pixel values ​​(denoted by number 2) are symmetrically padded around the left boundary of frame 1, pixel values ​​(denoted by number 6) are symmetrically padded around the left boundary of frame 1 and the top boundary of frame 1, and pixel values ​​(denoted by number 5) are symmetrically padded around the top boundary of frame 1. Similarly, other portions of frame 1 (and frame 2) may be symmetrically padded. Explaining the rolling operation with respect to frame 1 in Figure 9A, a window (3x3) is represented by a solid box. The image of size 3×3 surrounded by the window represents the first block. Initially, the window is located at an initial position of the expanded first image, and the first block has a center represented by “c”. Then, the window may slide to a next position of the expanded first image, and a next first block is obtained, and the center (represented by “c”) of the next first block may coincide with a pixel (represented by number 2) of the first image. The window may slide over the entire expanded first image to generate a plurality of first blocks. The upper solid black box of FIG. 9B represents a first intermediate image corresponding to the upper first block of FIG. 9A, and the lower solid black box of FIG. 9B represents a second intermediate image corresponding to the upper second block of FIG. 9B. As shown in FIG. 9B, a circle with a radius indicated by a dotted arrow represents a ring (or ring image). A plurality of first ring images associated with the first intermediate image and a plurality of second ring images associated with the second intermediate image may be determined. A plurality of correlation values ​​(also called FRC curves, as shown by curve a in FIG. 9C) may be determined based on the first ring image and the second ring image, and an intersection point between each FRC curve and the 3σ curve (as shown by curve b in FIG. 9C) may be determined.A characteristic value may be determined based on the target frequency corresponding to each intersection point, and an rFRC map may be generated based on the plurality of characteristic values ​​(see FIG. 9D).

[0198] FIG. 7 is a flow chart illustrating an exemplary process for determining a reference map, according to some embodiments of the present disclosure. In some embodiments, the process 700 may be performed by the image processing system 100. For example, the process 700 may be configured as a set of instructions (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules illustrated in FIG. 4) may execute the set of instructions and may be configured to perform the process 700 accordingly. The illustrated process operations presented below are intended to be illustrative. In some embodiments, the process 700 may be accomplished with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of the process 700 as illustrated in FIG. 7 and described below is not intended to be limiting. In some embodiments, operation 518 shown in FIG. 5 may be performed in accordance with one or more operations of process 700.

[0199] At 702, processing unit 140 (eg, map generation module 408) may obtain a reference image having a lower resolution than the first image.

[0200] In some embodiments, processing device 140 may obtain the reference image from one or more components of image processing system 100. For example, image capture device 110 may collect and / or generate the reference image and store it in storage device 150. Processing device 140 may retrieve and / or obtain the reference image from storage device 150. As another example, processing device 140 may obtain the reference image directly from image capture device 110.

[0201] In some embodiments, the reference image may be generated with the same measurements of the same object as the first image. In some embodiments, the reference image may be acquired by capturing the object using the image acquisition device 110. In some embodiments, the reference image and the first image may be reconstructed based on the same measurement signal, but using different reconstruction algorithms. The reference image may have a lower resolution than the first image. The reference image herein may be considered a low-resolution image, and the first image may be considered a high-resolution (or super-resolution) image. For example, the first image may be a super-resolution (SR) image, and the reference image may be a low-resolution (LR) image corresponding to the first image. As shown in FIG. 10, the SR image represents the first image, and the LR image represents the reference image.

[0202] In some embodiments, the reference image may be used to reconstruct the first image. In some embodiments, some error (e.g., information loss) may be introduced in the reconstruction. In some embodiments, the reference image may be used as a ground truth for determining the error.

[0203] At 704, processing unit 140 (eg, map generation module 408) may determine a convolution kernel based on the first image and the reference image.

[0204] In some embodiments, the convolution kernel may include a resolution scaling function (RSF), a point spread function (PSF), etc. In some embodiments, the convolution kernel may be an RSF. In some embodiments, the RSF may be used to convert a high-resolution image to a low-resolution scale. In some embodiments, the RSF may be defined by a 2D Gaussian function with a parameter σ. Since the RSF may typically be anisotropic in the x and y directions, σ may be divided into two components, namely σ x and σ y Thus, the RSF function may be expressed as:

[0205]

number

[0206] To discretely calculate the RSF, a 2D Gaussian function may be integrated over a finite number of pixels, so that the Gaussian error function (erf) may be given as:

[0207]

number

[0208] In some embodiments, a discrete equation for the RSF may be derived as follows:

[0209]

number

[0210] At 706, the processing unit 140 (eg, the map generation module 408) may determine an intermediate image based on the first image and the convolution kernel.

[0211] The intermediate image may be considered as an image whose resolution is scaled back from the first image. HL ) may be determined by transforming the first image with a convolution kernel. For example, the intermediate image may be determined based on a convolution of the first image with the convolution kernel.

[0212] In some embodiments, four parameters (e.g., μ, θ, σ x , σ y ) may be optimized. μ and θ may be parameters for rescaling the image intensity, and σ x and σ ymay be parameters for the parameterization of the RSF. Image intensity rescaling may refer to linearly rescaling the intensity of the first image.

[0213] In some embodiments, the first image I is processed to globally equalize the intensity between the reference image (e.g., the LR image) and the first image (e.g., the high-resolution image) and maximize the similarity between them. H The intensity of needs to be linearly rescaled by equation (23).

[0214]

number

[0215] Here, I HS represents the first image after being linearly rescaled, μ represents the intensity parameter, θ represents the parameter associated with the background region, and I H represents the first image. μ and θ in equation (23) represent the reference image I L and the first image I after rescaling convolved with the RSF. HS may be selected to maximize the similarity between

[0216]

number

[0217] Here, I L represents the reference image, and I HS represents the first image after being linearly rescaled, μ represents an intensity parameter, θ represents a parameter associated with the background region,

number

[0218] In some embodiments, μ and θ for image intensity rescaling, and σ for RSF parameterization x and σy To estimate x , σ y ) may be jointly optimized to minimize the following function:

[0219]

number

[0220] In some embodiments, since it may be difficult to calculate the gradient of equation (25), a derivative-free optimizer may be used to search for the four optimal parameters. Unlike particle swarm optimization (PSO), a pattern search method (PSM) may be used to optimize equation (25). PSO may search a relatively large space of candidate solutions, and may not be the choice required to optimize such four parameters. Compared to the unstable and slow metaheuristic optimization approach PSO, PSM may be a stable and computationally efficient way to directly search for solutions, and is commonly used for small-scale parameter optimization. In some embodiments, PSM may be more suitable for determining the RSM.

[0221] In some embodiments, μ and θ for image intensity rescaling, and σ for RSF parameterization x and σ y After optimizing, the high-resolution image (i.e., the first image) I H is estimated by convolving the RSF at its lower resolution scale (i.e., intermediate image) I HL In some embodiments, the intermediate image I HL may be determined according to equation (26).

[0222]

number

[0223] In some embodiments, the global quality of the intermediate images may be estimated. In some embodiments, the original low-resolution image (i.e., the reference image) I L Resolution scaling back image (i.e., intermediate image) I HL Commonly used metrics to evaluate the global quality of may be utilized in this operation, such as the root mean square error (referred to as RSE for resolution scaled error) and the Pearson correlation coefficient (referred to as RSP for resolution scaled Pearson coefficient). RSE and RSP may be expressed as follows:

[0224]

number

[0225] At 708, processing unit 140 (eg, map generation module 408) may determine a reference map based on the difference between the intermediate image and the reference image.

[0226] In some embodiments, the reference map may include a resolution-scaled error map (RSM). In some embodiments, the reference map may be determined based on the difference (e.g., absolute difference) between the intermediate image and the reference image (see FIG. 10). Thus, the reference map may indicate at least a portion of the errors and / or artifacts in the reconstruction process of the first image. To visualize the pixel-wise absolute difference, I HL and I L The reference map RSM between may be calculated by the following formula:

[0227]

number

[0228] It should be noted that the above description of process 700 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. After understanding the principles of operation, one skilled in the art would understand to combine any of the operations, add or remove any of the operations, or apply the principles of operation to other image processing processes without departing from the principles. In some embodiments, one or more of the operations may be combined into one operation. For example, operations 704-708 may be combined into one operation. In some embodiments, the first image in process 700 may be replaced with a second image.

[0229] FIG. 8 is a flow chart illustrating another exemplary process for determining a target map, according to some embodiments of the present disclosure. In some embodiments, the process 800 may be performed by the image processing system 100. For example, the process 800 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., the memory device 150, the storage 220, and / or the storage unit 370). In some embodiments, the processing device 140 (e.g., the processor 210 of the computing device 200, the CPU 340 of the portable device 300, and / or one or more modules illustrated in FIG. 4) may be configured to execute the instruction set and perform the process 800 accordingly. The process operations illustrated below are intended to be illustrative. In some embodiments, the process 800 may be accomplished without one or more additional operations not described and / or one or more operations described. Furthermore, the order of operations of the process 800 illustrated in FIG. 8 and described below is not intended to be limiting. In some embodiments, the process 800 may be a process for generating an rFRC map (e.g., a second target map) based on a single initial image (e.g., a single frame).

[0230] At 802, the processing unit 140 (e.g., acquisition module 402) may acquire an initial image. As shown in FIG. 12A, frame 1 represents an initial image of size 8×8. In some embodiments, the initial image may be generated by physical imaging. In some embodiments, the initial image may be a single frame image captured by the image acquisition device 110. The initial image may be generated based on a single measurement of the object. The object may be a biological object or a non-biological object as described elsewhere in this disclosure.

[0231] In some embodiments, processing device 140 may obtain the initial image from one or more components of image processing system 100. For example, image capture device 110 may collect and / or generate the initial image and store it in storage device 150. Processing device 140 may retrieve and / or obtain the initial image from storage device 150. As another example, processing device 140 may obtain the initial image directly from image capture device 110.

[0232] At 804, the processing unit 140 (e.g., decomposition module 416) may decompose the initial image into a first image, a second image, a third image, and a fourth image. In some embodiments, the first image and the second image at 502 may be acquired according to 804. In some embodiments, the FRC operation may require statistically independent image pairs that share the same detailed content but different noise realizations. In some physical imaging applications, such as SMLM, SRRF, SOFI, these modalities may suitably generate statistically independent images by splitting an input image sequence (e.g., the initial image) into two subsets and reconstructing them independently.

[0233] In some embodiments, the processing unit 140 may decompose the initial image using wavelet filters. Exemplary wavelet filters may include a Daubechies wavelet filter, a biorthogonal wavelet filter, a Morlet wavelet filter, a Gaussian wavelet filter, a Marr wavelet filter, a Mayer wavelet filter, a Shannon wavelet filter, a Battle-Lemarie wavelet filter, etc., or any combination thereof.

[0234] In some embodiments, since each pixel of the initial image may be sampled independently from the physical image, the processing unit 140 may divide the initial image into four sub-images (e.g., a first image, a second image, a third image, and a fourth image). In some embodiments, the pixels at row / column indexes of the initial image of (even, even), (odd, odd), (even, odd), and (odd, even) may be extracted to form the four sub-images, respectively. As shown in FIG. 12B, the pixels numbered 1 are at (odd, odd) row / column indexes, the pixels numbered 2 are at (odd, even) row / column indexes, the pixels numbered 3 are at (even, odd) row / column indexes, and the pixels numbered 4 are at (even, even) row / column indexes of frame 1 shown in FIG. 12A. As shown in FIG. 12C, the pixels numbered 1, 2, 3, and 4 may be extracted to form the four sub-images numbered 1, 2, 3, and 4, respectively. As shown in FIG. 12C, the four dotted boxes may represent a first image (e.g., the dotted box numbered 1), a second image (e.g., the dotted box numbered 4), a third image (e.g., the dotted box numbered 3), and a fourth image (e.g., the dotted box numbered 2), each of which is 4×4 in size. In some embodiments, the four sub-images may be of the same size. The four sub-images may be used to create two image pairs with identical detail content but different noise realizations. In some embodiments, the two image pairs may be randomly created from the four sub-images. For example, the first image and the second image may be randomly combined to form an image pair, and the other two sub-images may be combined to form another image pair. In some embodiments, the two image pairs may be formed according to the location of the sub-image pixels in the initial image.For example, as shown in FIG. 12C, a sub-image numbered 1 having pixels at (odd, odd) row / column indices and a sub-image numbered 4 having pixels at (even, even) row / column indices may form an image pair, while a sub-image numbered 2 having pixels at (odd, even) row / column indices and a sub-image numbered 3 having pixels at (even, odd) row / column indices may form another image pair.

[0235] In 806, the processing unit 140 (e.g., the rolling module 406) may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image (see FIG. 12D, which is similar to FIG. 9A). Operation 806 may be similar to operation 506, and the relevant description thereof will not be repeated here. In some embodiments, the first blocks and the second blocks may have a one-to-one correspondence.

[0236] In 808, the processing device 140 (e.g., the map generating module 408) may determine a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks (see FIGS. 12E-12G, which are similar to FIGS. 9B-9D). Operation 808 may be similar to operation 508, and the relevant description thereof will not be repeated here.

[0237] In some embodiments, the first characteristic value may be modified or revised. In some embodiments, during the FRC computation of a single frame image (i.e., the initial image), a single pixel displacement in both the x and y directions may be generated in the image pair. This spatial displacement is then calculated during the FRC computation. -i2πsr / N may result in frequency phase modulation, where s=√x0 2 +y0 2 represents the total length of the spatial displacement and r represents the radius in the FRC calculation. Additionally or alternatively, a calibration procedure may be used to correct the determined characteristic values ​​using the following formula:

[0238]

number

[0239] where r tf represents the corrected characteristic value, and r sf represents the target frequency corresponding to the image pair (i.e., the first image and the second image), and the four parameters a, b, c, and d are experimentally fitted (e.g., a=0.9599, b=0.9798, c=13.9044, d=0.5515). In some embodiments, the characteristic value may be corrected to calibrate such bias caused by spatial displacement, making the characteristic value more accurate.

[0240] At 810, the processing device 140 (e.g., the map generation module 408) may generate a first target map associated with the first image and the second image based on the first plurality of characteristic values. Operation 810 may be similar to operation 510, and the relevant description thereof will not be repeated here.

[0241] In 812, the processing device 140 (e.g., the rolling module 406) may determine a plurality of third blocks of the third image and a plurality of fourth blocks of the fourth image. In some embodiments, similar to the dilation of the first image and / or the second image, the processing device 140 (e.g., the dilation module 404) may dilate the third image and / or the fourth image before the third and fourth blocks are generated. The processing device 140 may determine the plurality of third blocks of the third image and the plurality of fourth blocks of the fourth image in a manner similar to the determination of the first and second blocks, and the relevant description thereof will not be repeated here. The plurality of third blocks and the plurality of fourth blocks may have a one-to-one correspondence.

[0242] At 814, the processing device 140 (e.g., the map generating module 408) may determine a plurality of second characteristic values ​​based on the plurality of third blocks and the plurality of fourth blocks. Operation 814 may be performed similarly to 808, and the relevant description thereof will not be repeated here.

[0243] At 816, the processing device 140 (e.g., the map generation module 408) may generate a second target map associated with the third image and the fourth image based on the second plurality of characteristic values. The processing device 140 may determine the second target map associated with the third image and the fourth image in a manner similar to the determination of the first target map at 810 or 510, and the relevant description thereof will not be repeated here.

[0244] At 818, the processing unit 140 (eg, the map generation module 408) may generate a third target map based on the first target map and the second target map.

[0245] The third target map may be considered as an rFRC map generated from the initial image. In some embodiments, the processing unit 140 may generate the third target map based on a weighted sum of the first target map and the second target map. In some embodiments, the processing unit 140 may generate the third target map by directly combining the first target map and the second target map. In some embodiments, the processing unit 140 may average the first target map and the second target map to obtain the third target map.

[0246] In some embodiments, through image decomposition at 804, the lateral dimensions of the four sub-images may be the same or half the size of the original image (i.e., the initial image at 802). In some embodiments, the third target map may be resized to the size of the original image (i.e., the size of the initial image) (see FIG. 12H). Alternatively, the first target map and the second target map may be resized to the size of the original image, and the third target map may be generated based on the resized first target map and the resized second target map. In some embodiments, the resizing may be performed using bilinear interpolation.

[0247] It should be noted that the above description of process 800 is provided for illustrative purposes only and is not intended to limit the scope of the present disclosure. After understanding the principles of operation, one skilled in the art would understand to combine any of the operations, add or remove any of the operations, or apply the principles of operation to other image processing processes without departing from the principles. In some embodiments, one or more of the operations may be combined into one operation. For example, operations 806-814 may be combined into one operation. As another example, operations 816 and 818 may be combined into one operation. As a further example, the processing device (e.g., filtering module 410) may filter the first target map, the second target map, and / or the third target map. In some embodiments, one or more of the operations may be added. For example, an operation of filtering the first target map and an operation of filtering the second target map may be added. As another example, an operation of filtering the third target map may be added. As a further example, an operation of displaying a third target map using the displacement jet color map and / or an operation of determining a global image quality metric of the initial image based on the third target map may be added.

[0248] The present disclosure is further illustrated by the following examples, which should not be construed as limiting the scope of the disclosure.

[0249] Working Example method

[0250] STORM imaging The microscope setup is described below. After washing with phosphate-buffered saline (PBS), 100 × 10 -3 M cysteamine, 0.8 mg mL -1 of glucose oxidase, and 40 μg mL -1Samples were mounted on glass slides using a standard STORM imaging buffer consisting of 100x catalase. Data were then collected by 3D-STORM performed in a self-defined setup using an oil immersion objective (100x / 1.45NA, CFI Plan Apochromat λ, Nikon) with a modified commercial inverted fluorescence microscope (Eclipse Ti-E, Nikon). 405 nm and 647 nm lasers were introduced into the cell sample from the back focal plane of the objective, displaced towards the end of the objective, and irradiated approximately 1 μm within the glass-water interface. A powerful 647 nm (approximately 2 kW cm) laser was used. -2 ) excitation laser photoswitched most of the labeled dye molecules to a dark state while simultaneously exciting the fluorescence from the remaining sparsely distributed light-emitting dye molecules for single molecule localization. -2 A 647 nm laser was used in conjunction with a 500 nm laser to reactivate the fluorophores to their luminescent state, so that at any given moment only a small, optically resolvable amount of fluorophores were luminescent. A cylindrical lens was placed in the imaging path to introduce astigmatism and encode the depth (z) position into the ellipticity of the single molecule images. An EMCCD (iXon Ultra 897, Andor) camera recorded images at a frame rate of 110 for a frame size of 256 × 256 pixels, and typically about 50000 frames were recorded in each experiment. Additionally, the cylindrical lens was removed from the optical layout to form a 2D-STORM image.

[0251] STORM reconstruction is described below. The open source software package Thunder-STORM and customized 3D-STORM software were used for STORM image reconstruction. Images labeled "ME-MLE" and "SE-MLE" were reconstructed by Thunder-STORM with maximum likelihood estimation (integrated PSF method) and multi-emitter fittable ("ME-MLE") or multi-emitter fittable ("SE-MLE"). Images labeled "SE-Gaussian" were reconstructed using customized 3D-STORM software by fitting the extreme values ​​with an (elliptical) Gaussian function. Drift correction was performed after localization and images were rendered using a normalized Gaussian function (σ is 2 pixels).

[0252] Cell culture, fixation, and immunofluorescence are described below. COS-7 cells were cultured in a humidified CO2 incubator with 5% CO2 at 37 °C according to standard tissue culture protocols in DMEM supplemented with 10% fetal bovine serum (FBS). Cells were then seeded onto 12 mm glass coverslips in 24-well plates at approximately 2 × 104 cells per well and cultured for 12 h. For STORM of actin filaments, a previously established fixation protocol was used: samples were first fixed in cytoskeleton buffer (CB, 10 × 10 -3 MES, pH: 6.1, 150 x 10 -3 M NaCl, 5 x 10 -3 M EGTA, 5×10 -3 M glucose, 5 x 10 -3 The cells were fixed and extracted for 1 min using 0.3% (v / v) glutaraldehyde and 0.25% (v / v) Triton® X-100 in 100 mM MgCl2, then post-fixed for 15 min using 2% (v / v) glutaraldehyde in CB and reduced with freshly prepared 0.1% sodium borohydride solution in PBS. Alexa Fluor 647-conjugated phalloidin was added at approximately 0.4 × 10 -6M concentration for 1 h. Samples were briefly washed 2-3 times with PBS and then immediately mounted for imaging. For imaging of other targets, samples were fixed with 3 w / v% paraformaldehyde and 0.1 w / v% glutaraldehyde in PBS for 20 min. After reduction with freshly prepared 0.1% sodium borohydride solution in PBS for 5 min, samples were permeabilized and blocked with blocking buffer (3 w / v% BSA, 0.5 v / v% Triton X-100 in PBS) for 20 min. Cells were then incubated with primary antibodies (as described above) in blocking buffer for 1 h. After washing three times with washing buffer (0.2 w / v% BSA and 0.1 v / v% Triton X-100 in PBS), cells were incubated with secondary antibodies for 1 h at room temperature. Samples were then washed three times with washing buffer and then mounted for imaging.

[0253] SIM imaging TIRF-SIM is described below. A commercial inverted fluorescence microscope (IX83, Olympus) equipped with a TIRF objective (100x / 1.7NA, Apo N, HI Oil, Olympus) and a multiband dichroic mirror (DM, ZT405 / 488 / 561 / 640 Phase R, Chroma) was used to construct a SIM system as described above. In short, laser light with wavelengths of 488 nm (Sapphire 488LP-200) and 561 nm (Sapphire 561LP-200, Coherent) and an acousto-optical tunable filter (AOTF, AA Opto-Electronic, France) were used to combine, switch, and adjust the illumination power. A collimating lens (focal length: 10 mm, Lightpath) was used to couple the lasers into a polarization-maintaining single-mode fiber (QPMJ-3AF3S, Oz Optics). The output laser was then collimated by an objective lens (CFI Plan Apochromat Lambda 2x NA 0.10, Nikon) and diffracted by a pure phase grating composed of a polarizing beam splitter (PBS), a half-wave plate, and an SLM (3DM-SXGA, ForthDD). The diffracted beam was then focused by another achromatic lens (AC508-250, Thorlabs) onto the intermediate pupil plane, where a carefully designed stop mask was placed to block the zeroth order and other stray light and allow only the ±1st order light pairs to pass. A homemade polarization rotator was placed after the stop mask to maximally modulate the illumination pattern while eliminating the switching time between different excitation polarizations. The light then passed through another lens (AC254-125, Thorlabs) and a tube lens (ITL200, Thorlabs) to be focused onto the back focal plane of the objective lens and interfered on the image plane after passing through the objective lens. The emission fluorescence collected by the same objective lens passed through a dichroic mirror (DM), an emission filter, and another tube lens. Finally, the emission fluorescence was split by an image splitter (W-VIEW GEMINI, Hamamatsu Japan) and then captured by an sCMOS (Flash 4.0 V3, Hamamatsu Japan) camera.

[0254] Hessian SIM is described below. We applied the Hessian denoising algorithm without the t-continuity constraint to the result of the Wiener SIM reconstruction to obtain the Hessian SIM image shown in Figure 21E.

[0255] 3D-SIM is described below. The 3D-SIM data sets in Figures 25A-I were acquired using a Nikon 3D-SIM microscope equipped with a TIRF objective (100x / 1.49NA, CFI Apochromat, Oil, Nikon).

[0256] Cell maintenance and preparation are described below. Human umbilical vein endothelial cells were isolated and cultured in M199 medium (Thermo Fisher Scientific, 31100035) supplemented with fibroblast growth factors, heparin, and 20% fetal bovine serum (FBS), or in endothelial cell medium (ECM) (ScienCell, 1001) supplemented with endothelial cell growth supplement (ECGS) and 10% FBS. To express LifeAct-EGFP, cells were infected with a retroviral system. Transfected cells were cultured for 24 hours, detached using trypsin-EDTA, and transfected with poly- L -Lysine-coated coverslips (H-LAF10L glass, reflectance: 1.788, thickness: 0.15 mm, customized) and cultured for another 20–28 h in an incubator with 5% CO2 at 37 °C before the experiment.

[0257] Isolate LSECs and plate them on coverslips coated with 100 μg / ml collagen and 10% FBS, 1% L The cells were cultured in high glucose DMEM supplemented with β-glutamine, 50 U / ml penicillin, and 50 μg / ml streptomycin for 6 h in an incubator at 37°C with 5% CO2 before imaging. Live cells were incubated with DiI (100 μg / ml, Biotium, 60010) for 15 min at 37°C, and fixed cells were fixed with 4% formaldehyde for 15 min at room temperature before labeling with DiI.

[0258] STED imaging Image acquisition by STED was performed using a gated STED (gSTED) microscope (Leica TCS SP8 STED 3X, Leica Microsystems, Germany) equipped with a wide-field objective (100x / 1.40 NA, HCX PL APO, Oil, Leica). The excitation wavelength was 647 nm and the extinction wavelength was 775 nm. All images were acquired using LAS AF software (Leica). To label microtubules in live cells shown in Figures 40A-C, COS-7 cells were incubated with SiR-Tubulin (cytoskeleton, CY-SC002) for 20 min without washing before imaging.

[0259] PANEL Framework In principle, the imaging process can be described as a measurement system observing an object with a transfer function and a corresponding signal collected by a sensor. It is generally recognized that such artificial observations will always be offset from the real-world object in high-dimensional space (left panel of Fig. 15) because the signal is actually sampled by the sensor subject to various kinds of noise. The errors / artifacts included in the reconstruction are mainly caused by such joint effects due to the sampling rate and mixed noise when using an appropriate reconstruction model. In other words, the reconstruction with such unavoidably offset observations may even be offset from the real-world object in the target / super-resolution domain (right panel of Fig. 15). If the variables are controlled to image the same object and statistically capture independent image pairs, the distance between the ground truth and its reconstruction can be highlighted by the difference between such individual reconstructions (Fig. 9A-D). To quantify such distance between a recorded image and its ground truth, conventional evaluation methods in the spatial domain, such as the relative error and absolute error used in RSM, are overly sensitive to the intensity and minute movements during the measurement. These quantization algorithms, which can be thought of as "absolute differences", can induce a high number of false negatives that overwhelm the true negatives in the distance map, potentially causing biologists to be severely misled about the presence of existing errors / artifacts.

[0260] Based on the above analysis, a criterion-free approach to measure such unmeasurable distances between two signals in the Fourier domain, namely Fourier Ring Correlation (FRC), is introduced. FRC describes the highest allowable frequency component of the distance between two signals. Traditionally, the FRC metric has been widely used as an effective resolution criterion for super-resolution fluorescence microscopy and electron microscopy. It can also be applied to quantify the similarity or distance between two images. Since FRC is calculated as a "relative error" or "saliency-based error" with respect to an image, it has the inherent advantage of being insensitive to intensity changes and minute movements. Moreover, since it estimates the most reliable frequency components, FRC is a more quantitative and easy-to-understand metric that only highlights the saliency error. As a result, FRC is an excellent choice for quantifying the distance between two signals, while greatly reducing the potential false negative problem. Notably, we extend the conventional FRC framework in the form of rolling FRC (rFRC) maps (see Fig. 9A-D) with the aim of considering FRC as a global similarity estimate between two images and providing a more accurate local distance measure down to the pixel level, enabling a quantitative assessment of image quality at the super-resolution scale. The rolling FRC operation may be like a moving filter on the image plane, where a sliding window is used for each block of the image, and each block is assigned a corresponding FRC value. First, the input image is symmetrically padded with half the size of the block (see Fig. 9A) to ensure that the FRC operation is performed at the border of the image. Second, a suitable background threshold may be used for the central pixel (1x1-3x3) to decide whether to calculate the FRC value of this block or not, thus avoiding the FRC operation on the background region. Third, if the average value of the central pixel is greater than the threshold, the FRC operation is performed and the central pixel of each block is assigned an FRC value. Otherwise, the central pixel may be set to zero to avoid unnecessary operations on the background region. This procedure may then be performed block by block until the entire image has been completely scanned.

[0261] In addition, to visualize the error map, a corresponding metric, rFRC value, and color map (displacement jet, or sJet), which are more suitable for human intuition, are developed. The rFRC map allows to quantitatively map the errors of multidimensional reconstructed signals without prior information of the ground truth and imaging system. However, it should be noted that if the two reconstructed signals are missing identical components (even though such a situation is rare in practical experiments), the rFRC method may fail and report false positives. If the same region in both images is blank (but the ground truth has content), such information may be lost in the same region, leading to erroneous small FRC values. Therefore, to remove such possible false positives, a modified resolution-scaled error map (see RSM in Figure 10) is introduced as an additional error map and integrated into the PANEL framework. The full RSM may introduce high false negatives at small magnitudes due to the three aforementioned hypotheses. To reduce such false negatives in RSM, RSM may be segmented with a 0.5 hard threshold to highlight common large-scale artifacts such as misrepresentation or disappearance of structures before integrating them into the final PANEL map (Figure 11A-C). On the other hand, rFRC maps show the degree of error, so that the smallest FRC value in the map does not necessarily represent an error. To address this situation, we introduce a segmentation method called Otsu, which automatically determines the threshold by maximizing the inter-class variance, and then performs image thresholding to filter the background of the original rFRC map, thereby highlighting crucial errors in the reconstruction (Figure 16).

[0262] Open Source Datasets

[0263] We illustrated the broad applicability of PANEL using freely available simulation / experimental datasets as well as custom-curated datasets.

[0264] The 2D-SMLM simulation data sets are described below. As shown in FIG. 17A, the "Bundle Tube High Density" (361 frames) and "Bundle Tube Long Sequence" (12000 frames) data sets from the "Localization Microscopy Challenge Data Set" on the EPFL website were used as high-density and low-density 2D-SMLM simulation data sets. The NA of the optical system was 1.4 (oil immersion objective lens), and the fluorescence wavelength was 723 nm.

[0265] The 3D-SMLM simulation dataset is described below. As shown in Figure 17C, the "MT1.N1.LD" (19996 frames, 3D astigmatism PSF) dataset from the "Localization Microscopy Challenge Dataset" on the EPFL website was used as the low-density 3D-SMLM simulation dataset. The NA of the optical system was 1.49 (oil immersion objective lens), and the wavelength of the fluorescence was 660 nm. All images were 64 × 64 pixels (pixel size is 100 nm). Then, every 20 frames were averaged from this low-density 3D-SMLM simulation dataset into one frame to generate the corresponding high-density 3D-SMLM simulation dataset (leading to 998 frames).

[0266] The 2D-SMLM experimental dataset is described below. The "Localization Microscopy Challenge Dataset" also contains experimental data, 500 high-density images of tubulin were obtained from the EPFL website (Figures 18A-H). The NA of the optical system was 1.3 (oil immersion objective) and the wavelength of the fluorescence was 690 nm. Images were recorded with a camera with a frame size of 64 × 64 pixels (pixel size is 100 nm) and a frame rate of 25.

[0267] Live-cell SRRF datasets are described below. GFP-tagged microtubules in live HeLa cells were imaged in TIRF mode using a TIRF objective (100x / 1.46NA, Plan Apochromat, Oil, Zeiss) and further illuminated with a 488 nm laser at 1.6x magnification (total of 200 frames), and super-resolution SRRF results were reconstructed using the open source ImageJ plugin (Figures S18I-L).

[0268] Simulation of gridding of samples imaged by SMLM A regular grid with pixel size of 10 nm (Figure 17M) is created and the density of randomly activated molecules is set to gradually increase from the center to the sides. Then, the acquired image sequence is convolved with a Gaussian kernel with full width at half maximum (FWHM) of 280 nm and downsampled 10 times (pixel size: 100 nm). As a result, Poisson noise and 20% Gaussian noise are included in the image sequence. Finally, the image sequence was reconstructed by Thunder-STORM with maximum likelihood estimation (integrated PSF method) and multi-emitter fitting capability.

[0269] Simulation for FPM The United States Air Force (USAF) resolution target was used as the ground truth sample for FPM (Figure 24A). Both the intensity and phase of the imaged sample were set as a USAF target with 240 × 240 pixels (pixel size: 406.3 nm). It was illuminated from different angles by a 7 × 7 LED matrix with an emission wavelength of 532 nm and a distance to the sample of 90 mm. The sample was illuminated by each LED unit, filtered by an objective lens (4x / 0.1NA), and sampled by a camera (the acquired image size was 60 × 60, pixel size was 1.625 μm). After the LEDs illuminated the sample one by one, 49 low-resolution images were finally acquired. The image illuminated by the central LED was used as the initial image. Then, with each iteration of FPM, the amplitude and phase of the corresponding aperture were updated in sequence. After 10 rounds of iteration, the final high-resolution complex amplitude image (240 × 240) was acquired, which had a size enlarged by 4 times compared to the corresponding low-resolution image.

[0270] Data Generation Process for Learning-Based Applications Sparse sampling is explained below. A deep neural network (DNN) was trained on the sparsely sampled geometric structures and their corresponding intact geometric structures as ground truth. Four simple and common geometric structures were selected for simulation: triangle, circle, rectangle, and square. The spatial size of one structure and the number of structures in one input image are shown in Table 1. After obtaining the structures, the images were randomly sampled with a sampling rate of 8%. A rectangular structure was selected, and 5000 images were used as the training dataset. For each geometric structure, 200 images were generated as the test dataset.

[0271] Pixel super-resolution is described below. The DNN was trained with RAW images as ground truth and corresponding downsampled RAW images as input. As a training dataset, we selected Drosophila melanogaster expressing the membrane marker Ecad::GFP imaged with a commercial spinning disk microscope (camera exposure / laser power: 240 ms / 20%). First, to obtain the surface-projected 2D ground truth, we used PreMosa. Second, we discarded images containing strong background. Third, we estimated the background by repeated wavelet transform and then subtracted it to generate a background-free image as the ground truth. Finally, we used images downsampled four times with 4 × 4 pixels as 1 pixel as input. Before inputting the DNN, the images were upsampled four times with bilinear interpolation.

[0272] The deconvolution is described below. The DNN was trained using the synthetic sub-diffraction images as ground truth and the corresponding diffraction-limited wide-field images as input. Specifically, a random walk process was used to create synthetic tubulin structures to simulate two-dimensional trajectories with randomly changing orientation. Meanwhile, the maximum curvature was set as a constrained value with respect to the known physical stiffness of tubulin. These images were used as ground truth with a pixel size of 65 nm and processed to create input images. These images were convolved with a Gaussian kernel (FWHM: 300 nm) to obtain blurred images. To simulate a realistic fluorescent background, the blurred images were convolved with a larger Gaussian kernel (FWHM: 1.2 μm) and added to the blurred images. Then, Poisson noise and 2% Gaussian noise were added to the images to create the final input images for DNN training. A similar procedure was used to generate another 781 images as a test dataset.

[0273] Noise2Noise is described below. Noise2Noise is an unsupervised learning procedure that denoises noisy images without a clear image. During training, the DNN only sees noisy image pairs (two images with independent noise that share the same detailed content), i.e., one as the input target and the other as the output target. The Fluorescence Microscopy Denoising (FMD) dataset is used for this Noise2Noise task. Fixed zebrafish embryos [2-day-old EGFP-labeled Tg(sox10:megfp) zebrafish] were selected as the dataset imaged with a commercially available Nikon A1R-MP confocal laser scanning microscope at very low excitation power. There are five noise levels in this image configuration. The RAW image has the highest noise level, and the other noise level images were generated by averaging RAW images with multiple frames (2, 4, 8, 16) using the circular averaging method. To test extreme conditions, the RAW image with the highest noise level was selected as the input for the training set (every second RAW image). 200 noise-to-noise data pairs were randomly selected for each field of view (total 20) with 50 different noise realizations. Meanwhile, the raw images with size 512 × 512 were cropped into 4 non-overlapping patches with size 256 × 256. Finally, 20 × 200 × 4 = 16000 images were obtained as the training dataset. The ground truth reference for evaluating the prediction accuracy of Noise2Noise was generated by averaging 50 noisy raw images.

[0274] Network architecture and training procedure The network architecture is described below. This network architecture includes a reduction path and an augmentation path, and is a so-called U-shaped architecture (Unet). In the reduction path, the input layer is followed by a series of down-convolution blocks, which consist of a 4×4 kernel convolution with a stride step of 2, a batch normalization (BN), and a leaky regularized linear unit (LeakyReLU) function. The convolution layer is at the bottom of this U-shaped structure, connecting the down-convolution blocks and the up-convolution blocks. The augmentation pathway combines the features and spatial information from the reduction path by a series of up-convolution blocks (upsampling 2D operations with a stride step of 1+BN+ReLU+4×4 kernel convolution) and concatenation with the high-resolution features. The last layer is another convolution layer that maps the 32-channel image to a 1-channel image. Two kinds of U-shaped network architectures (Unet) were used for different tasks. 1 and Unet 2 ) was used (see Figure 43). 1 and Unet 2 The difference with Unet 1 has 7 down convolutional blocks and 7 up convolutional blocks, while Unet 2 has four down convolution blocks and four up convolution blocks.

[0275] The training procedures are described below. All networks were trained using stochastic gradient descent with adaptive moment estimation (Adam). The detailed input patch size of training images, number of epochs, batch size, number of training images, learning rate, network architecture, number of parameters, and loss function for each task are shown in Figure 42 and Table 2. Figure 42 provides an overview of the network architecture, training parameter configuration, and data used for different applications. All training procedures were performed on a local workstation equipped with an NVIDIA Titan Xp GPU card. The associated training framework was implemented in Tensorflow framework version 1.8.0 and Python version 3.6.

[0276] Use of open source deep learning models ANNA-PALM is described below. ANNA-PALM reconstructs super-resolution images from rapidly acquired sparse localization data. ANNA-PALM was trained using densely sampled PALM images (long sequences) as ground truth and corresponding sparsely sampled PALM images (short sequences) as input. ANNA-PALM is a novel neural network based on Unet 1 cGAN when using as a generator 1 The performance of ANNA-PALM was tested using 500 dense images of tubulin from the EPFL website. Fluorophores in frames 1–25 and 26–50 were localized using the ME-MLE estimator to construct two sparse super-resolution inputs, after which ANNA-PALM predicted the corresponding densely sampled images.

[0277] CARE is explained below. The CARE framework is 3 It is a computational approach that can extend the spatial resolution of a microscope using a 3D architecture. We fill the CARE open source pre-trained model with the first and second raw images from the open source SRRF dataset to generate two corresponding super-resolution images.

[0278] Cross-modality super-resolution is described below. The cross-modality imaging capability of DNNs is preliminarily demonstrated by mapping TIRF to the TIRF-SIM modality (TIRF2SIM) using a cGAN approach. cGAN on TIRF2SIM 2 is based on Unet with a generator residual convolution block (Res-Unet). It was trained and tested using AP2-eGFP tagged clathrin from gene-edited SUM159 cells. Results were reproduced straightforwardly using the provided ImageJ plugin and example data.

[0279] Image Rendering and Processing A custom developed color map, the displacement jet (sJet) color map, was used to visualize the rFRC maps. The color map SQUIRREL-Error was used to present the RSM in Figures 17P, 28F, bottom right of Figures 17A-17B, 18G, and 33H. Jet projections were used to show depth in Figures 23D and 25A. All data processing was accomplished using MATLAB® and ImageJ. All figures were created using MATLAB®, ImageJ, Microsoft Visio, and OriginPro.

[0280] Data availability All data that support the findings of this study are available from the corresponding author on request.

[0281] Code Availability The latest versions of the custom-written MATLAB® library (with user manual and example data), custom-designed color maps, and corresponding ImageJ plugin used in this disclosure are available as software. An updated version of the PANEL in MATLAB® library can be found at https: / / github.com / WeisongZhao / PANELM. An updated version of the ImageJ plugin and its source code can be found at https: / / github.com / WeisongZhao / PANELJ. Example 1

[0282] Verification of PANEL by SMLM simulation To evaluate the ability of PANEL to identify minute defects in super-resolution images, we used 2D single molecule localization microscopy (SMLM) simulation datasets from the EPFL Challenge. The 2D-SMLM simulation datasets (e.g., high-density (HD) 2D-SMLM simulation dataset and low-density (LD) 2D-SMLM simulation dataset) included HD-emitting fluorophores and LD-emitting fluorophores, respectively (see Figures 17A-17F). Figures 17A-17F show the simulations of 2D-SMLM using high-density (HD)-emitting fluorophores and low-density (LD)-emitting fluorophores. As shown in Figures 17A-17D, the integrated ground truth structures are represented in the dark gray channels labeled HD-GT or LD-GT, and the maximum likelihood estimation (MLE) reconstructions are represented in the light gray channels labeled MLE. Figures 17B-17E show the rFRC maps of the MLE, respectively. Figures 17C-17F show the complete PANEL with MLE. The RSM was represented as the channel indicated by the white arrow, and the rFRC map was represented as the other channel. The white and light grey arrows indicated the errors found by the RSM or rFRC map, respectively. In this quantitative mapping, one set of images was divided into two statistically independent subsets to obtain two independent super-resolution images. After using maximum likelihood estimation (MLE) reconstruction, two corresponding super-resolution images of the HD 2D-SMLM simulation dataset were obtained. Then, some parts of the filaments were artificially removed to create known structural artifacts as indicated by the white arrows in Figures 17A-F to visualize potential false positives induced by considering only the rFRC map. As indicated by the light grey arrows, it was found that the rFRC map is able to detect all subtle errors (rFRC value: 0.66) but is unable to detect missing structures in both super-resolution frames. Therefore, as mentioned above, to verify any errors in the reconstruction, the RSM map and the rFRC map were integrated and represented as the channel indicated by the white arrow and the other channels, respectively (Figures 17A to 17F), to create the final full-PANEL map.On the other hand, compared with the ground truth, the MLE results of the LD 2D-SMLM simulation dataset were found to contain almost no error (rFRC value: 0.16), and the corresponding rFRC map also realistically described such appearance during the computation process (Figures 17A-F).

[0283] Comparison of the respective LD and HD 2D-SMLM simulation data sets revealed that the performance of the SMLM results is related to the density of the emitting fluorophores, which may be highly dependent on the induced illumination intensity. To further quantitatively evaluate the effect of the emission density on the MLE reconstruction in a single frame image, a regular grid was created where the center is illuminated with high intensity and the illumination intensity decreases towards the edges, showing that the blinking in the center is more separated in time than the edges (Figure 17G). Figure 17G shows the MLE results of the 2D-SMLM simulation with inhomogeneous illumination (high intensity in the center and decreasing illumination towards the edges). Figure 17H ​​shows the rFRC map generated based on Figure 17G. As shown in Figure 17H, it is clearly shown that the rFRC map accurately describes the reconstruction performance induced by the blinking density transition. In contrast, the estimated RSM cannot properly present the error (even though it is opposite to the reference) due to the RSM assumptions not being met (Figures 17M-P). Figure 17M shows the simulated ground truth. Figure 17N shows the widefield image of Figure 17M with high illumination in the center and decreasing illumination towards the edges. Figure 17O shows the reconstructed MLE image of Figure 17G after convolving it with the estimated RSF. Figure 17P shows the RSM of the reconstructed MLE image of Figure 17G.

[0284] In the final step, we extended the rFRC map directly to a 3D version by removing the RSM from the PANEL for 3D super-resolution imaging (Figures 17I-17L) and applying plane-wise operations as a constraint in 3D model evaluation. Figure 17I shows an image showing the integrated ground truth structure (red channel labeled "LD-GT") and the MLE reconstruction (green channel labeled "3D-MLE"). Figure 17J shows the rFRC map of the low-density 3D-MLE, where the rFRC value was 2.2. Figure 17K shows an image showing the integrated ground truth structure (red channel labeled "HD-GT") and the MLE reconstruction (green channel labeled "3D-MLE"). Figure 17L shows the rFRC map of the high-density "3D-MLE", where the rFRC value was 4.5. The 3D simulation datasets used for the validation of the PANEL are taken from the EPFL SMLM challenge, including the LD and HD cases (per 20 frames averaged from the LD dataset). Similar to the 2D case, the rFRC results show that the global performance of the 3D-MLE reconstruction is strongly influenced by the density of the emitting fluorophores, all in good agreement with the actual physical experience (see rFRC values ​​of 2.2 vs. 4.5 for 3D, LD and HD in Figures 17I-L, 17Q and 17X, 17Y). Figure 17Q shows the dark-colored plots of the ground truth and 3D-MLE reconstructions of the low-density (LD) and high-density (HD) datasets, respectively, from left to right. Figure 17X shows the horizontal cross-sections (z position 0 nm) of the 3D-MLE reconstructions (LD and HD) and a representative frame (frame 19) of the LD dataset, respectively, from left to right. Figure 17Y shows the rFRC maps of the horizontal cross-sections of Figure 17X and a representative frame (frame 99) of the HD dataset, respectively, from left to right.

[0285] We also found that while the rFRC map can detect all tiny and subtle errors, this method is not possible when structures are simultaneously missing in both adjacent super-resolution frames. For clarity, we create an example using the 2D SMLM challenge dataset to illustrate this possible false negative (see Figures 17A-C and 17R-T). Figure 17R shows the rFRC map after Otsu threshold filtering for Figure 17B. Figure 17S shows the RSM. Figure 17T shows the RSM after 0.5 threshold filtering. As shown in Figure 17B, the rFRC map successfully detects most error components in the entire image, except for the artificially removed regions. However, the disappearance of such structures can be successfully detected by the RSM (see Figure 17R). Therefore, to address this possible false positive, the RSM was introduced as an error map to append to the rFRC map. It should also be noted that such missing identical information in the two reconstructions may contain large magnitudes and be present in large regions of the RSM. Furthermore, due to the three hypotheses of RSM, the small magnitude components included in RSM may introduce high false negatives. Therefore, RSM was partitioned using 0.5 threshold filtering before involving RSM in PANEL. After this operation, low false negatives are accurately filtered and only strong low-resolution error components are left to focus on the true negatives detected by RSM.

[0286] On the other hand, in practice, the rFRC map indicates the degree of error, so the smallest FRC value in the map may not indicate the acceptable error. Similarly, a segmentation method called Otsu was introduced to automatically determine the threshold by maximizing the inter-class variance, and to threshold the image to filter the background of the rFRC map and highlight the critical errors of the reconstruction (Figures 17U-17W). After that, the segmented rFRC map was integrated as a light gray channel and the RSM was segmented as a dark gray channel to create a complete PANEL to visualize the integrated error map of the corresponding reconstruction (Figures 17A-17C, 17R-17T, and 17U-17W). It should be noted that if the dataset is in a three-dimensional or non-Gaussian convolution relationship (between low-resolution and high-resolution scales), the corresponding RSM cannot be estimated. As a result, for such datasets, the RSM may not be involved in the PANEL. Figure 17U shows the rFRC map displayed with a displacement jet color map for the same SRRF dataset as Figures 18I-18L. 17V and 17W show the full rFRC map and the rFRC map after Otsu thresholding to highlight critical errors in the SRRF reconstruction shown in the green channel of the PANEL. Example 2

[0287] Minimizing analysis errors with PANEL After synthesizing ground truth images including the grid structure and basic simulations, an experimental 2D-SMLM dataset from the EPFL challenge was evaluated using the proposed PANEL (Figures 18A-H). Figure 18A shows an image showing the MLE localization results of 500 high-density images of tubulin from the EPFL website. Figure 18B shows the rFRC map of the "MLE". Figure 18C shows the complete PANEL of the "MLE" where the rFRC value was 1.2. Figure 18D shows an image showing the PANEL after Otsu threshold segmentation. Figure 18E shows the corresponding "wide-field" image. Figure 18F shows the "MLE" image deconvolved to the original low-resolution scale. Figure 18G shows an image showing the RSM of the "MLE". Figure 18H shows the FRC map of the "MLE" where the FRC value was 584 nm. Based on the corresponding rFRC maps obtained, it can be seen that large FRC values ​​tend to appear in the filament intersection regions, which is consistent with the SRRF results revealed in Figures 18I-18L. Figure 18I shows a diffraction-limited TIRF image. Figure 18J shows an image showing the SRRF reconstruction results of 100 fluctuating images (GFP-tagged microtubules in live HeLa cells, see Methods). Figure 18K shows the rFRC map of "SRRF", in which the rFRC value was 2.25. Figure 18L shows an image showing the PANEL after Otsu threshold segmentation. The main possible reason lies in the effect of emission density in the 2D-SMLM simulation, where the intersection regions contain a relatively large emitter density, leading to poor performance of localization. It is an experimentally recognized fact that in the actual experimental process, emission density (related to the complexity of the structure) changes significantly within the field of view, but there is no practical quantification method to identify the minute errors on that super-resolution scale. So far, most existing algorithms have to accept such a performance trade-off by being designed to consider only uniform density instead of HD or LD.

[0288] The high-resolution error mapping feature allows to integrate advantages from existing HD or LD focusing algorithms, respectively, thereby minimizing the errors included in all selected methods. To verify this statement, a 2D-STORM dataset of immunolabeled α-tubulin in fixed COS-7 cells was analyzed using two different algorithms, namely multi-emitter MLE (MEM-LE) and single-emitter Gaussian fitting (SE-Gaussian) (see methods in Fig. 20A and Fig. 19A-E). Fig. 19A shows an image showing the reconstruction of the multi-emitter MLE (ME-MLE). Fig. 19B shows an image showing the single-emitter Gaussian fitting (SE-MLE). Fig. 19C shows an image showing the fusion result of ME-MLE and SE-MLE with rFRC map. Fig. 19D shows the corresponding rFRC map of Fig. 19A-C. Fig. 19E shows an enlarged view of the white box in Fig. 19A. FIG. 20A shows images showing fused STORM results (COS-7 cells, α-tubulin labeled with Alexa Fluor 647) from reconstructions with multi-emitter MLE (ME-MLE) and single-emitter Gaussian fitting (SE-MLE). FIG. 20B shows the rFRC map of FIG. 20A. FIG. 20C-FIG. 20E show magnified images of the area enclosed in box 2001 in FIG. 20B, in which FIG. 20C shows an image showing the rFRC map results, FIG. 20D shows an image showing the fused STORM results, and FIG. 20E shows an image showing the RSM results. 20F-20H show images showing a close-up of box 2002 in FIG. 20A, and 20I-20K show images showing a close-up of box 2003 in FIG. 20B, of which FIG. 20F shows an image showing the results of ME-MLE, FIG. 20G shows an image showing the results of SE-MLE, FIG. 20H shows an image showing the results of fused STORM, FIG. 20I shows the rFRC map of FIG. 20F where the rFRC value was 1.01, FIG. 20J shows the rFRC map of FIG. 20G where the rFRC value was 4.51, and FIG. 20K shows the rFRC map of FIG. 20H where the rFRC value was 0.87. FIG. 20L shows a close-up of the dashed circle in FIG. 20F-20H.SE-Gaussian is more suitable for the reconstruction of simple structures (LD emitters), whereas ME-MLE is more suitable for complex structures (HD emitters) as shown in Figures 20C-20K. By utilizing the rFRC error map at the super-resolution scale, the spatial details of the local accuracy of each algorithm can be mapped, which can be translated into fusion weights (Figures 20B-20E), after which the lowest error features of each reconstruction may be used to generate a new composite image with minimal defects. As expected, as shown in the two selected regions of interest (Figures 20F-20K), it can be seen that no individual algorithm (ME-MLE or SE-Gaussian alone) can achieve stable performance for all structures in the entire field of view. Combining the information generated by ME-MLE and SE-Gaussian using the proposed rFRC map allows us to take advantage of the strengths of both algorithms and achieve excellent performance in the entire field of view (Figures 20C-20K, 20M). Moreover, such a fusion approach was further applied to a 2D-STORM dataset of heavy chain clathrin-coated pits (CCPs) of COS-7 cells, where the average resolution was significantly improved (Figure 20N). Figure 20M shows ME-MLE rFRC maps, fusion dominance maps, and TIRF images from top left to bottom left, and SE-Gaussian rFRC maps, fusion results, and fusion results (Fused) from top right to bottom right. It was found that the ME-MLE method achieved superior performance in regions with strong background, whereas the SE-Gaussian method obtained better reconstruction quality in regions with weak background. Figure 20N shows the zoomed results of ME-MLE, SE-Gaussian, and fusion single CCPs from top left to bottom left, and the corresponding rFRC maps are shown on the right. The average resolution is described in the top left of the rFRC map. As highlighted in Figure 20M, in addition to the stable fusion performance across the entire field of view, the rFRC map also lends itself to the fusion of small structures such as a single ring-shaped CCP, achieving a higher average resolution. Example 3

[0289] Diverse physics-based imaging applications supported by rFRC maps The designed rFRC was identified and manipulated into a typical 3D-STORM reconstruction (Figure 21A), and the resulting rFRC map (Figure 21B) and its PANEL map (Figure 21C) showed that the large errors occurred mainly between the entangled filament voxels (Figures 22A-C). Figure 21A shows an image showing the 3D-MLE reconstruction (COS-7 cells, α-tubulin labeled with Alexa Fluor 647). Figure 21B shows the 3D rFRC map of Figure 21A. Figure 21C shows an image showing the PANEL after Otsu thresholding in Figure 21B. Figure 21D shows an image showing the curve of rFRC values ​​along the axial position. Figure 21E shows representative images of live human umbilical vein endothelial cells (HUVECs) labeled with LifeAct-EGFP under Wiener SIM (top), Hessian SIM (middle), and TIRF (bottom) imaging. FIG. 21F shows the rFRC map of the Hessian SIM, and the rFRC, RSP, and RSE values ​​of the Wiener SIM and Hessian SIM were rFRC=1.24, RSP=0.98, and RSE=0.27. FIG. 21G shows images showing representative results of fixed liver sinusoidal endothelial cells (LSECs) labeled with DiI under RL deconvolution (top) and TIRF (bottom) imaging. FIG. 21H shows the rFRC map of the RL deconvolution result. FIG. 21I shows an enlarged view of the box in FIG. 21G, including the original TIRF image (top left), images showing the RL deconvolution results of 80 iterations and 200 iterations (top right and bottom left), and an image showing the TIRF-SIM result (bottom right). FIG. 21J shows images showing the curves of PSNR (vs. TIRF-SIM), RSP (vs. TIRF), and rFRC values ​​along the iterations. FIG. 22A shows an image showing a maximum intensity projection (MIP) image of the 3D-MLE reconstruction. FIG. 22B shows an image showing a MIP image of the rFRC volume of FIG. 22A. FIG. 22C shows an image showing the corresponding magnified horizontal section of the 3D-MLE (left) and rFRC volume (right) from the white box of FIG. 22A. Also, in the experimental setup, it was found that the most accurate plane of reconstruction depth for microtubules (FIG. 21D) and actin filaments (FIGS. 23A-F) was located at the focal position.FIG. 23A shows an image showing a maximum intensity projection (MIP) image of TIRF (COS-7 cells labeled with Alexa Fluor 647-phalloidin). FIG. 23B shows an image showing a dark color-coded view of the 3D-MLE reconstruction. FIG. 23C shows an image showing a horizontal section of the 3D-MLE reconstruction at a z position of −50 nm. FIG. 23D shows the corresponding rFRC map of FIG. 23C. FIG. 23E shows an image showing a horizontal section of the 3D-MLE reconstruction at a z position of +300 nm. FIG. 23F shows the corresponding rFRC map of FIG. 23E.

[0290] The Hessian denoising algorithm (Hessian SIM) can be used on the conventional Wiener SIM to separate random discontinuity artifacts from real structures (see Fig. 21E and Figs. 29A-I). Fig. 29A shows an image of the Wiener SIM. Fig. 29B shows the corresponding rFRC map. Fig. 29C shows the corresponding RSM. Fig. 29D shows an image of the Hessian SIM. Fig. 29E shows the corresponding rFRC map. Fig. 29F shows the corresponding RSM. Fig. 29G shows TIRF images of the Wiener SIM and Hessian SIM. Fig. 29H shows a TIRF image convolved with a large Gaussian kernel and encoded with the inverted sJet color map. Fig. 29I shows a merged image of the PANEL (light grey channel) and the Wiener SIM (dark grey channel). However, due to the need for conversion to TIRF mode, the RSM may miss such artifacts, as can be seen from Fig. 21E, where the RSE value remains at 0.27. Conversely, when utilizing the rFRC map, an improvement of the Hessian SIM over the conventional Wiener SIM can be detected, as can be seen from FIG. 21F, where the corresponding rFRC value improves significantly from 1.36 to 1.24.

[0291] Richardson-Lucy (RL) deconvolution has been actively researched for many reasons, including its potential to improve the resolution and contrast of raw images. However, conventional RL algorithms can also generate artifacts when performing excessive iterations, greatly limiting their application. A common use of RL requires tedious visual inspection to determine the optimal number of iterations. Here, to confirm the readout of rFRC values ​​that guide such a decision for the number of RL iterations, its associated rFRC value is calculated for each iteration case (right panel of Figure 21H and Figure 21J) after RL is applied to process TIRF images (Figure 21G, Figure 30A-Figure 30I). Figure 30A shows the TIRF results. Figure 30B and Figure 30C show the RL deconvolution results for 80 and 200 iterations, respectively. Figure 30D shows the corresponding TIRF-SIM images. Figure 30E and Figure 30F show the rFRC maps of Figures 30B and 30C. FIG. 30G shows a TIRF image convolved with a large Gaussian kernel and colored with the inverted sJet color map. The inverted illumination intensity map (FIG. 30G) is proportional to the rFRC map (FIG. 30E), indicating that the degree of error in the RLD results is highly correlated with the SNR. FIG. 30H and FIG. 30I show the panels of FIG. 30B and FIG. 30C. Interestingly, when compared with the curve distribution of the peak signal-to-noise ratio (PSNR in the left panel of FIG. 21J) where the corresponding TIRF-SIM image is utilized as the ground truth, it was found that the curve of the rFRC value shows a similar distribution (quadratic form, with a minimum value appearing at 80 iterations). In contrast, the curve of the resolution-scaled Pearson coefficient (RSP) obviously could not represent such an informative distribution (middle panel of FIG. 21J). As shown in Figure 21I, RL deconvolution with 200 iterations produced snowflake-like artifacts, as pointed out by the white arrows, which were verified to be absent by the referenced TIRF-SIM images. Overall comparison demonstrated that RL with 80 iterations optimally enhanced image contrast while minimizing noise amplification artifacts.

[0292] Not only typical fluorescence modalities but also coherent computational imaging techniques, e.g., Fourier-Ptychographic Microscopy (FPM), can be effectively evaluated by using PANEL to develop and fully achieve their practical potential (see Methods in Figures 24A-F). Figure 24A shows an image illustrating the simulated ground truth. Figure 24B shows the wide-field image of Figure 24A. Figure 24C shows an image illustrating the corresponding FPM reconstruction. Figure 24D shows the rFRC map of FPM. Figure 24E shows an image illustrating the RSM of FPM. Figure 24F shows an integrated image of PANEL (light gray channel) and FPM (dark gray channel). The proposed PANEL framework was also extended to a single-frame version (see Figure 25), aiming to access the image quality of modalities when constrained to form statistically independent image subsets such as 3D-SIM. Figure 25A shows an image showing the color-coded volume of HUVECs transfected with LifeAct-EGFP and imaged with a Nikon 3D-SIM microscope. Figures 25B-C show 3D-rendered rFRC maps displayed in sJet color map (Figure 25B) and after Otsu thresholding in the green channel (Figure 25C). Figures 25D-F show a 3D-SIM image at an axial position of 500 nm (Figure 25D), the corresponding rFRC map (Figure 25E), and the PANEL result (Figure 25F). Figures 25G-I show a 3D-SIM image at an axial position of 700 nm (Figure 25G), the corresponding rFRC map (Figure 25H), and the PANEL result (Figure 25I). As indicated by the white arrows (FIG. 25I), we found that large errors can be present in regions containing thick actin stress fibers (or strong stress fibers) that scatter and distort the structured incident light. Example 4

[0293] Validating PANEL through Simulation of a Learning-Based Approach Deep learning algorithms can learn effective representations that map high-dimensional data to arrays of outputs. However, these mappings often lack rational explanations and are therefore blindly assumed to be accurate. Characterizing the uncertainties of the corresponding network predictions and representations is therefore crucial for further profiling, especially in safety-critical applications. In general, to obtain the uncertainties for such learning-based approaches, the traditional Bayesian neural network framework requires significant modifications to the existing learning procedure by learning a distribution over the weights. However, a potential drawback of BNNs is their rather complex application and high computational cost compared to native learning-based techniques. To alleviate this conflict, we intend to port the concept of PANEL to learning-based models, allowing them to identify subtle and irregular errors generated by black-box deep neural networks.

[0294] First, four types of simple structures are created and sampled into sparse shapes (see Fig. 26). Fig. 26 shows images illustrating a full sparse sampling simulation, where a rectangle (top) is used as the training data set, and (from top to bottom) other geometric shapes (square, circle, triangle) are used as the test data set. "Input" means a representative sparsely sampled input of the corresponding geometric shape, "Prediction 1" means the network prediction of "Input", "Ground Truth" means before sparse sampling, "GT+Prediction" means the integrated image of the ground truth of the green channel and the predicted result of the red channel, "Prediction 1+Prediction 2" means the integrated image of the two predicted results, and "Integrated" means the integrated image of the predicted result of the PANEL of the green channel and the white channel. The corresponding input images of "Prediction 1" and "Prediction 2" are sampled independently. To investigate the generalization effect, the obtained structures that are sparsely sampled (taken as model input) or not sparsely sampled (taken as ground truth) are used as data pairs to verify the applicability of PANEL in learning-based approaches. Rectangular shapes are utilized as the training data set, while squares, triangles, and circles are used as the test data set. As expected, squares can be considered as a subset of rectangles, and thus the network trained on rectangular shapes showed good results on the square data set. Interestingly, as can be seen from FIG. 27A, when the network was faced with data not presented in the training set (out-of-distribution data), such as triangle and circle shapes, the returned results were still similar to rectangular-like shapes. FIG. 27A shows an image illustrating representative integrated images (prediction 1: cyan channel, prediction 2: magenta channel) of rectangular, square, triangular, and circle structures, as shown from left to right. FIG. 27B shows an image illustrating the average IoU values ​​by predictions versus ground truth. FIG. 27C shows an image illustrating the average IoU values ​​between two predictions. FIG. 27D shows an image illustrating the average rFRC values ​​between two predictions. FIG. 27E shows an image showing a representative result after pixel super-resolution ("PSR") (top left) and its counterpart with an undersampled input (bottom right).FIG. 27F shows an image showing the ground truth ("GT") reference of FIG. 27E. FIG. 27G shows an rFRC map of the PSR results. FIG. 27H shows an integrated image of the PANEL (green channel) and the PSR results (gray channel). FIG. 27I shows an image showing a representative result of the deconvolution network. FIG. 27J shows an image showing the corresponding ground truth ("GT") of FIG. 27I. FIG. 27K shows an image showing a zoomed-in view from the open boxes of FIG. 27I and FIG. 27J. FIG. 27L shows an integrated image of the PANEL (green channel) and the deconvolution results (gray channel).

[0295] To verify the ability of PANEL to detect such structural artifacts (model errors) introduced by deep learning frameworks, the same structure was sampled twice independently to generate two predictions (denoted as Prediction 1 and Prediction 2 in Fig. 27A). To directly evaluate the difference between the two predictions, the metric intersection is chosen instead of the set operation (IoU, also known as Jaccard index). As can be seen from Fig. 27B and Fig. 27C, the average IoU values ​​between Prediction 1 and Prediction 2 showed the same distribution as that between the ground truth and Prediction 1. Moreover, Fig. 27D clearly shows that the pattern of the averaged rFRC values ​​is remarkably similar to the one of the IoU values, verifying that the proposed concept and rFRC metric have the ability to quantitatively analyze learning-based approaches.

[0296] Next, to examine PANEL in more detail, we also perform another simulation test, namely pixel super-resolution (PSR in Fig. 27E and Fig. 31A-G), to evaluate its usability for learning-based applications. Fig. 31A shows the result after pixel super-resolution (PSR) (top left) and the undersampled input (bottom right). Fig. 31B shows a close-up of the undersampled input in the white box of Fig. 31A. Fig. 31C shows the network prediction of Fig. 31B. Fig. 31D shows the corresponding ground truth ("GT") of Fig. 31B. Fig. 31E shows the rFRC map of Fig. 31C. Fig. 31F shows the estimated RSM of Fig. 31C. Fig. 31G shows the merged image of PANEL (light gray channel) and PSR result (dark gray channel). In this case, the corresponding deep neural network was trained by a data pair when the RAW image was used as the ground truth and the corresponding downsampled one was considered as the input image. Comparison with the ground truth (Figure 27F), where the artifact regions of interest are indicated by white arrows, fully verifies that the rFRC map (Figure 27G) can accurately detect errors in the network predictions (Figure 27H) and provide a quantitative assessment at the pixel level.

[0297] For convenient applications, a special single-frame version of the rFRC computation strategy is also involved in learning-based applications. Another learning-based task, namely deconvolution, is used to evaluate such strategies (Fig. 27I, Fig. 32A-J). Fig. 32A shows the input image. Fig. 32B shows the DNN result. Fig. 32C shows the ground truth (GT) of Fig. 32A. Fig. 32D shows a zoomed-in view from the white box of Fig. 32A-C. Fig. 32E shows the RSM of Fig. 32A. Fig. 32F shows an integrated image of the PANEL (white) and the DNN result (gray). Fig. 32G shows the inverted SSIM map of Fig. 32B and Fig. 32C. Fig. 32H shows the deconvolution of Fig. 32B. Fig. 32I and Fig. 32J show the rFRC map of the DNN result from the input image with 1% and 1.5% additive Gaussian noise. TN is true negative, FN is false negative, and FP is false positive. Intuitively, in contrast to the ground truth (Figures 27J-K), the single-frame strategy also showed superior quantitative performance for accessing image quality, as indicated by the white arrow (Figure 27L). Example 5

[0298] Reliability of PANEL in various learning-based imaging methods After being analyzed and examined, the usability of PANEL is then applied to evaluate the performance of three typical open source learning-based models, including artificial neural network accelerated PALM17 (ANNA-PALM in FIG. 28A), content-aware image restoration (CARE in FIG. 28E), and TIRF conversion to TIRF-SIM (TIRF2SIM in FIG. 28I). FIG. 28A shows an image showing ANNA-PALM output (MLE result with 25 frames of tubulin as input). FIG. 28B shows an image showing MLE reconstruction with full 500 frames. FIG. 28C shows the rFRC map of FIG. 28A. FIG. 28D shows a merged image of PANEL (green channel) and ANNA-PALM result (gray channel). FIG. 28E shows an image showing CARE output (raw TIRF image as input) of GFP-tagged microtubules in live HeLa cells. FIG. 28F shows an image showing CARE deconvolved to the original low-resolution scale (top) and its RSM (bottom). FIG. 28G shows the corresponding TIRF image. FIG. 28H shows a merged image of PANEL (green channel) and CARE results (gray channel). FIG. 28I shows an image showing TIRF2SIM results of CCP (gene-edited SUM159 cells expressing AP2-eGFP). FIG. 28J shows a TIRF-SIM image. FIG. 28K shows an image showing TIRF input. FIG. 28L shows a merged image of PANEL (green channel) and TIRF2SIM results (gray channel). FIG. 28M shows an image showing Noise2Noise results of an EGFP-labeled Tg(sox10:megfp) zebrafish at 2 days post-fertilization. FIG. 28N shows a ground truth reference image, generated by averaging 50 noise images with identical content. FIG. 28O shows an image showing a representative noise input. FIG. 28P shows a merged image of PANEL (green channel) and Noise2Noise results (gray channel).

[0299] For ANNA-PALM, a sparse MLE reconstruction (containing 25 frames) is used as the input image for the network (Figure 28A, Figures 33A-I), and rFRC is utilized in combination with RSM for complementarity. Figures 33A-C show the MLE reconstruction with 25 frames (Figure 33A) and the MLE reconstruction with the full 500 frames (Figure 33B), and the corresponding ANNA-PALM output (MLE result with 25 frames as input) (Figure 33C). Figures 33D-I show the integrated result of MLE (gray) and ANNA-PALM (white) (Figure 33D), the PANEL result (Figure 33E), the TIRF image (Figure 33F), the deconvolved ANNA-PALM (Figure 33G), the RSM (Figure 33H), and the rFRC map (Figure 33I). Compared with the ground truth dense MLE reconstruction results (including the full 500 frames in Fig. 28B), the rFRC map (Fig. 28C and white channel in Fig. 28D) successfully partitions the small errors, as indicated by the cyan arrows. Meanwhile, the RSM (white channel in Fig. 28D) finds large missing structures, as indicated by the white arrows. Based on these practical results, it follows that the integral PANEL map can effectively detect all types of errors at different scales included in the ANNA-PALM method.

[0300] In the deconvolution application of CARE20, we consider that the output (Fig. 28E) and input (Fig. 28G) of the network do not satisfy the Gaussian convolution assumption, so in this case the RSM cannot accurately convert the output to its lower resolution scale (Fig. 28F, Fig. 34A-F), and possible false negatives are indicated by purple circles. Fig. 34A shows the input of a TIRF image. Fig. 34B shows the CARE prediction of Fig. 34A. Fig. 34C shows the rFRC result colored in full color. Fig. 34D shows an integrated image of the PANEL (light gray channel) and the CARE result (white channel). Fig. 34E shows the CARE prediction deconvolved with the estimated RSF. Fig. 34F shows the result of the RSM of Fig. 34B. RSF is the resolution scaling function. RSM is the resolution scale error map. Conversely, obvious non-biological structures occurring near the field of view boundary, as indicated by the cyan circles, are effectively detected by the rFRC map (green channel in Figure 28H). By applying the single-frame rFRC computation strategy, we can see that the TIRF2SIM19 application (Figures 28I-L) can also be evaluated by PANEL. As shown in the insets of Figures 28I-K, two adjacent CCPs in TIRF-SIM (indicated by two green circles in Figure 28J) are erroneously reconstructed by the network into a single large CCP (indicated by one white circle in Figure 28I), and the rFRC map (white channel in Figure 28L) approach accurately partitions such refined errors.

[0301] The performance of PANEL is further tested to access the performance of non-super-resolution learning-based applications, e.g., denoising tasks. Noise2Noise38 is a widely used unsupervised learning-based approach that has the advantage of denoising noisy images without clear images. In such unsupervised learning models, the network only sees noisy image pairs during training, and two images with independent noise share identical details, in other words, one image is used as the input target and the other image is used as the output target. In this part, 59 fluorescence microscopy denoising (FMD) datasets are involved in training the network of Noise2Noise (see methods in Figures 28M-P and 35A-E), and RSM is removed from the PANEL framework in this evaluation example since RSM does not meet the assumptions. From the reconstructions and evaluations in Figures 28M-P, it is clear that Noise2Noise works effectively except in the highlighted areas in the insets of Figures 28M-P (suspicious areas are indicated by white arrows), indicating that the rFRC map (white channel in Figure 28P) can accurately detect such types of errors. Figure 35A shows the result after Noise2Noise (N2N) (top left) and noise input (bottom right, noise1). Figure 35B shows the Noise2Noise result from the white box in Figure 35A. Figure 35C shows the reference image averaged over 50 noise images with identical content. Figure 35D shows the merged image of PANEL (light grey channel) and Noise2Noise result (dark grey channel). Figure 35E shows the rFRC map of Figure 35B.

[0302] As explained above, a comprehensive, reliable and universal quantitative mapping of errors at the super-resolution scale, independent of any reference information, may be crucial as computational and learning-based super-resolution techniques emerge in biomedical imaging science. Tracing the quantitative error map up to the super-resolution scale allows the assessment of reconstruction quality at pixel level, which allows further design of computational processes such as STORM image fusion, adaptive low-pass filters, and automatic iterative decision making. Moreover, the proposed rFRC map may also be a good choice to access quality metrics against the ground truth and estimate local resolution maps of images.

[0303] In principle, there are two major categories of reconstruction errors in imaging by computational microscopy: model errors and data errors. The main source of model errors is the distance that exists between an artificially created estimated model and its real model in the physical world, and in the case of learning-based microscopy, such distance may come from ignoring the network for out-of-distribution data. The main source of data errors is due to the joint effect of the noise conditions and sampling capabilities of the hardware equipment, such as the sensors and cameras, of the microscope system. Based on the theory of corresponding models, model-related errors may be found and reduced in physical imaging systems by careful system calibration, and in learning-based applications, they can be explained by specifically designed strategies and sufficient training data. On the other hand, data errors may be essentially model-free, unavoidable in artificial system calibration methods, and difficult to explain with larger training data sets. Taken together, the above suggests that in biological analysis, such estimates of data errors may be more decisive. In some embodiments, with the aim of developing an absolute model-free quantitative method, PANEL may be based only on one hypothesis, namely that the model for reconstruction is unbiased. This means that the proposed PANEL method can accurately detect data errors present in various methods, but it should also be noted that it is limited in estimating model errors.

[0304] In particular, considering learning-based applications, the model error can be simply estimated by the mismatch of an ensemble of models trained independently and repeatedly on the same data set through multiple random initialization and optimization processes. Moreover, due to the pure data-driven approach of learning representations of training data, model error and data error in such learning-based applications are not necessarily mutually exclusive. As shown in Figure 27A, the PANEL framework shows the possibility of detecting both model error and data error. Model error from predictions of out-of-distribution test samples (rectangles: training data, triangles / circles: test data) can be effectively detected by the PANEL concept and rFRC metric (Figures 27B-D). Alternatively, data uncertainty and model uncertainty can be estimated by applying the rFRC map to the twin predictions from the two inputs (from two data samplings) and the two models (from two network trainings), respectively.

[0305] As a model-independent and criterion-free metric, the PANEL method can partition the local reconstruction quality of images from various modalities without requiring extra prior knowledge. PANEL may place more emphasis on detecting data errors than model errors. However, when considering data-driven approaches, data errors may generally be mixed with some of the model errors, leading PANEL to evaluate the model errors to some extent. After fully considering the capabilities and boundaries of the corresponding approaches, PANEL may provide a universal and reliable local reconstruction quality assessment framework, which may not only benefit image-based biological profiling but also promote further progress in the rapidly developing field of computational microscopy imaging.

[0306] In the following Examples 6 to 11, new findings and extended applications of PANEL will be described. Example 6

[0307] Errors in SMLM, SIM, and deconvolution Errors in SMLM: We found that the accuracy of SMLM is highly related to the molecular activation density in each frame. In the case of a small field of view, which can be considered as a uniform illumination (uniform activation density), errors are more likely to occur in complex structures, e.g., common assemblies of filaments (Figures 18A-L). In the case of a large field of view, since the molecular activation density is related to the illumination intensity (the higher the intensity, the lower the activation density), errors are more likely to occur in areas of low intensity illumination (Figures 17G, 17H, 17P).

[0308] Errors in SIM and deconvolution: SIM and deconvolution cannot detect high frequency errors (smaller than the resolution), such as snowflake-like artifacts. We found that the magnitude of errors in SIM and deconvolution is related to the emission intensity of the fluorescent signal (Figures 29A-I, 30A-I), and the quality of the reconstruction results of SIM and deconvolution is related to the SNR of the image, which is reasonable to assume that the SNR is somehow proportional to the magnitude of the signal. Example 7

[0309] Adaptive low-pass filter based on rFRC FRC determines a reliable cutoff frequency (COF) of an image, and indicates that noise and errors are concentrated in frequency components larger than the cutoff frequency. Since rFRC calculates local cutoff frequencies in different regions of an image, here we apply local cutoff frequencies to adaptively apply low-pass filters to different block box regions throughout the image.

[0310]

number

[0311] Here, I x,y represents a subset image of the input image with a center pixel at spatial location (x, y), and the OTF (F x,y ) is the cutoff frequency F x,yis an optical transfer function (OTF) with a cutoff frequency of 0 outside and 1 within the cutoff frequency. When considering Richardson-Lucy deconvolution (RLD), the image quality of the reconstruction result is highly related to the corresponding local SNR, so such reconstruction results usually have a spatially varying cutoff frequency. Figure 36A shows a simulated wide-field image. Figure 36B shows an image after performing RLD with 500 iterations on the simulated wide-field (WF) image of Figure 36A. Figure 36C shows the workflow of an adaptive filter on an image, where the block size of the filter is set as 64 × 64 pixels and the overlap between adjacent blocks is set as 4 pixels. Figure 36D shows an image after performing a global estimated cutoff frequency filter on the image of Figure 36C. Figure 36E shows an image after performing an adaptive local filter on the image of Figure 36A. Figure 36F shows a ground truth (GT) image of Figure 36A. The ground truth image is convolved with the PSF (FWHM = 240 nm) and involves Poisson noise and 10% Gaussian noise. The global FRC filter does not achieve optimal results (see Figure 36D, SSIM = 0.32, PSNR = 19.30), and in comparison, the designed adaptive rFRC filter is a superior choice for filtering the image after Richardson-Lucy deconvolution (see Figure 36E, SSIM = 0.42, PSNR = 21.90).

[0312] The global FRC value was used to estimate the OTF of the entire image, and such estimated OTF was applied to blind Richardson-Lucy deconvolution. However, FRC can estimate the effective OTF (reliable frequency content) instead of the native resolution of the system. In other words, FRC may be significantly affected by noise amplitude. Large noise amplitude may lead to dominant frequency content even in the OTF, affecting FRC's estimation of the native resolution of the system (larger resolution). RLD requires a static OTF instead of the effective OTF of the system. Such behavior therefore introduces a natural assumption that the image processed by RLD is under a sufficiently high SNR. Example 8

[0313] SMLM fusion based on rFRC maps By developing such an rFRC quality metric, different localization results may be fused according to the weights of the rFRC map to combine the respective advantages of different models.

[0314]

number

[0315] Here, L n is the result of the n-th localization model, and G(σ) represents the Gaussian kernel with standard variance of σ. max(F 1~n ) is the maximum FRC value of N localization results,

number

[0316] Determining the number of deconvolution iterations using rFRC The optimal number of iterations for Richardson-Lucy deconvolution (RLD) is generally determined by artificial testing by observing the reconstruction results and convergence. However, artificial testing is usually tedious and limits automatic image analysis. On the other hand, under low SNR conditions, RLD may not reach convergence, and in such cases it is necessary to stop the iterations early (before convergence) to avoid oversharpening of structures and the appearance of unpleasant artifacts. The reported rFRC is used to estimate the local reliable cutoff frequency to automatically determine the optimal number of iterations. Specifically, the optimal iteration is the one with the smallest rFRC value, since the intention is to obtain a more reliable cutoff frequency, which is equal to the rFRC value that is smaller due to the higher resolution and RLD processing.

[0317] The images captured by TIRF and TIRF-SIM are used as data pairs shown in Figs. 21G-21J and 30A-30I. The TIRF-SIM image is used as the ground truth, which is the subject of RLD on the TIRF image. As can be seen from Fig. 21J, as the iterations proceed, the PSNR curve is expressed in a quadratic form, indicating that more iterations do not necessarily produce better results. In fact, excessive iterations may induce artifacts (white arrows in Figs. 21I and 30B-30D) and reduce the PSNR value. In this example, it can be seen that "80" iterations is the optimal number of iterations considering the optimal PSNR. According to the observations on "80" iterations and "200" iterations in Figs. 30B and 30C, "80" iterations improve the contrast without inducing artifacts, whereas "200" iterations generate snowflake-like artifacts, indicated by white arrows, which can be verified not to exist by the referenced TIRF-SIM image (Fig. 30D). Also, the rFRC value in Fig. 21J can determine the optimal choice of the number of iterations, and its curve is also identical to that of the PSNR calculated for the ground truth. Conversely, the Resolution Scaled Pearson Coefficient (RSP) metric could not represent such a useful distribution. Example 10

[0318] rFRC as an error mapping method for data with ground truth rFRC is the first proposed method for mapping errors in the absence of ground truth in real bioimaging applications, and can be considered as a superior option for error mapping in the case of ground truth-accompanying data compared to SSIM to allow a more reasonable evaluation.

[0319] As can be seen from Figures 39A-39F, filaments were generated by different distances and convolved with a wide-field PSF (NA=1.4). Then, gradually increasing noise (along the arrows in Figure 39B) was involved in the image in Figure 39B. The resulting image was obtained using Richardson-Lucy deconvolution, as displayed in Figure 39C. Compared with the ground truth shown in Figure 39A, the largest errors appear in the open boxes in Figure 39B, and the rFRC map was found to accurately detect such errors, as can be seen from Figures 39D and 30E. Conversely, as shown in Figure 39F, spatial distance estimation approaches such as SSIM induced high false negatives and made true negatives difficult to segment or invisible. It should be noted that the rFRC maps formed by reconstruction 1 and reconstruction 2, or reconstruction 1 and the ground truth, were perfectly consistent, indicating that PANEL can find errors without the need for a ground truth. Figure 39A shows the ground truth. Figure 39B shows the image with gradually increasing noise added. Fig. 39C shows the reconstructed image. Fig. 39D shows an rFRC map generated based on reconstruction 1 and reconstruction 2. Fig. 39E shows an rFRC map generated based on reconstruction 1 and the ground truth. Fig. 39F shows the inverted SSIM. Example 11

[0320] Resolution as an additional metric In addition to detecting the exact degree of errors in the reconstruction, the rFRC map also adds a quantitative metric: resolution. To build an accurate and practical quantification tool, we realized three main contributions: the absence of false negatives due to background, the color map sJet that is more suitable for visualization, and pixel-wise resolution mapping. To that end, the rFRC map can be used as an excellent choice to evaluate the resolution of not only the reconstruction results but also the raw data captured by photoelectric detectors, e.g., stimulated emission depletion (STED) microscopy. It is clear that due to the absence of false negatives due to background, the average resolution of STED (Figure 40A) is “92 nm” for the rFRC map in Figure 40C, which is more reasonable than “146 nm” for the FRC map in Figure 40B. Moreover, the “sJet” color map can visualize the resolution distribution with higher contrast compared to the conventionally used “SQUIRREL-FRC” color map. Figure 40A shows SiR-tubulin-labeled microtubules observed under gSTED. Figure 40B shows the FRC map for a block size of 64 pixels colored with the SQUIRREL-FRC color map, and Figure 40C shows the rFRC map colored with the sJet color map.

[0321] Based on the 3σ curve rather than the 1 / 7 threshold, rFRC is more stable and accurate for local FRC resolution mapping to determine the effective cutoff frequency. As shown in Figures 41A and 41B, the resolution of "64 nm" determined by the 1 / 7 threshold in the area indicated by the white arrow in the inset seems overconfident for such poorly resolved filaments, and the resolution of "160 nm" obtained by the 3σ curve was apparently moderate and reasonable. Although the so-called 1 / 7 threshold may be considered unstable in local resolution mapping, it should be emphasized that in the PANELJ FIJI / ImageJ plugin, the characteristics of both the 1 / 7 threshold and the resolution mapping based on the 3σ curve are provided for further potential applications. The reason is that the 1 / 7 threshold was common and widely used, so the same local resolution mapping is provided for a wider user community.

[0322] Although the fixed threshold method was claimed to be based on an incorrect statistical hypothesis, the 1 / 7 threshold has been widely applied in the field of super-resolution. Interestingly, whatever the criteria, the final resolution obtained in SMLM and stimulated emission depletion microscopy (STED) experiments is almost the same. The FRC curves mapped the local SR errors calculated among small image blocks and included in the reconstruction. As shown in Figures 40C and 40D, for small images, the 1 / 7 threshold is overconfident, i.e., smaller than all correlation values ​​of the FRC curve, and the cutoff frequency of the corresponding image cannot be determined (cross in Figure 40C). In contrast, for large images, the 1 / 7 threshold obtained similar results compared to the 3σ curve. This indicates that the 1 / 7 threshold is suitable for the conventional global resolution determination, but not for the local cutoff frequency determination (local resolution). On the other hand, in the above-mentioned resolution determination, instead of avoiding conservative threshold selection such as the 1 / 7 threshold, a gentle threshold for error mapping is intended to reduce false positives. Therefore, the 3σ curve is used as the threshold for calculating the FRC value of the PANEL, considering its robustness and accuracy in the case of small image blocks.

[0323] [Table 1]

[0324] [Table 2]

[0325] Thus, having described the basic concepts, it will be apparent to those skilled in the art after reading the detailed disclosure herein that the detailed disclosure is intended to be presented by way of example only and is not limiting. Although not expressly stated herein, various changes, improvements and modifications are possible and contemplated by those skilled in the art. These changes, improvements and modifications are intended to be suggested by the present disclosure and are within the spirit and scope of the exemplary embodiments of the present disclosure.

[0326] Furthermore, certain terms are used to describe embodiments of the present disclosure. For example, the terms "one embodiment," "one embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present disclosure. Thus, it is emphasized and understood that two or more references to "one embodiment" or "an embodiment" or "alternative embodiments" in various parts of this specification do not necessarily all refer to the same embodiment. Furthermore, certain features, structures, or characteristics may be suitably combined in one or more embodiments of the present disclosure.

[0327] Also, as will be appreciated by those skilled in the art, aspects of the present disclosure may be illustrated and described herein in any of a number of patentable varieties or contexts, including any new and useful process, machine, manufacturing step, or composition of matter, or any new and useful improvement thereof. Thus, aspects of the present disclosure may be implemented entirely in hardware or software (including firmware, resident software, microcode, etc.), or in combinations of software and hardware, all of which may be referred to generally herein as "units," "modules," or "systems." Moreover, aspects of the present disclosure may be in the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied therein.

[0328] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein, for example, as part of a baseband or carrier wave. Such a propagated signal may take any of a variety of forms, including electromagnetic, optical, etc., or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium, but may be any computer-readable medium capable of communicating, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code embodied in a computer-readable signal medium may be transmitted using any suitable medium, including wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0329] Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, the "C" programming language, traditional procedural programming languages ​​such as Visual Basic, Fortran 2103, Perl, COBOL 2102, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby, Groovy, or other programming languages. The program code may be executed entirely on the user computer, partially on the user computer as a standalone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), which may be to an external computer (e.g., via the Internet using an Internet Service Provider) or in a cloud computing environment, or may be provided as a service, such as Software as a Service (SaaS).

[0330] Furthermore, the order of the described processing elements or sequences, or the use of numbers, letters, or other designations, are not intended to limit the processes and methods described in the claims to any other order than may be set forth in the claims. While the above disclosure has been discussed in terms of various specific examples that are presently believed to be various useful embodiments of the present disclosure, it should be understood that such details are for illustrative purposes only, and that the appended claims are not limited to the disclosed embodiments, but rather are intended to encompass modifications and equivalent arrangements within the spirit and scope of the disclosed embodiments. For example, implementations of the various components described above may be embodied in hardware devices, but may also be realized as software-only implementations, such as existing servers or mobile devices.

[0331] Similarly, in the foregoing description of the embodiments of the present disclosure, it should be understood that various features may be grouped together in a single embodiment, figure, or description thereof in order to simplify the disclosure to aid in understanding one or more of the various embodiments. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, an embodiment of the present disclosure does not include all features of the embodiments described above.

[0332] In some embodiments, numerical values ​​expressing quantities or properties used to describe certain embodiments of the present application are understood to be modified in some cases by the terms "about", "approximate", or "substantially". For example, "about", "approximate", or "substantially" may indicate a variation of ±20% of the stated value, unless otherwise specified. Thus, in some embodiments, the numerical parameters set forth in the present specification and appended claims are approximations that may vary depending on the desired properties to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of some embodiments of the present application are approximations, the numerical values ​​set forth in the specific examples are reported as precisely as practicable.

[0333] Each of the patents, patent applications, published patent applications, and other materials, such as articles, books, specifications, publications, literature, articles, etc., referenced herein are incorporated herein by this reference in their entirety for all purposes, except for any prosecution history documents related thereto, any of which are inconsistent with or contradictory to this document, or any of which may have a limiting effect on the broadest scope of any claims now or later related to this document. By way of example, in the event of a conflict or inconsistency between a description, definition, and / or use of a term associated with any of the incorporated materials and a description, definition, and / or use of a term associated with this document, the description, definition, and / or use of the term in this document shall control.

[0334] Finally, it should be understood that the embodiments of the present application disclosed herein are illustrative of the principles of the embodiments of the present application. Other variations that may be utilized may be within the scope of the present application. Thus, by way of example, but not of limitation, alternative configurations of the embodiments of the present application may be utilized in accordance with the teachings herein. Thus, the embodiments of the present application are not limited to those precisely as shown and described. [Explanation of symbols]

[0335] 100 Image Processing System 110 Image acquisition device 120 Network 130 terminals 131 Mobile Devices 132 Tablet Computer 133 Laptop Computer 140 Processing equipment 150 Storage device 200 Computing equipment 210 Processor 220 Storage 230 I / O section 240 communication port 300 Mobile Devices 310 Communication Unit 320 Display Unit 330 Graphics Processing Unit 340 Central Processing Unit 350 I / O section 360 Memory 361 Operating Systems 362 Applications 370 Storage Unit 402 Acquisition Module 404 Extension Module 406 Rolling Module 408 Map Generation Module 410 Filtering Module 412 Display module 414 Image Quality Decision Module 416 Decomposition Module 418 Neural Network Processing Module

Claims

1. 1. A method for image processing executed on at least one machine, each machine having at least one processor and at least one storage device, comprising: acquiring a first image and a second image associated with the same object; determining a number of first blocks of the first image and a number of second blocks of the second image in one-to-one correspondence; determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks; generating a first target map associated with the first image and the second image based on the plurality of first characteristic values, the first target map indicative of reconstruction errors or artifacts of the first image and the second image, each of the first characteristic values ​​corresponding to a pixel of the first target map; 23. A method for image processing comprising:

2. The step of determining a plurality of first blocks of the first image comprises: The method of claim 1 , comprising determining a plurality of first blocks of the first image by performing a rolling operation on the first image.

3. The method of claim 2 , wherein the rolling operation is performed by sliding a window over the first image, skipping a predetermined number of pixels over the first image.

4. The method according to any one of claims 1 to 3, wherein at least one of the first characteristic values ​​is determined based on a correlation between a first block of the plurality of first blocks and its corresponding second block.

5. The at least one first characteristic value is determining a first intermediate image in the frequency domain based on the first block; determining a second intermediate image in the frequency domain based on the second block; determining a target frequency based on the first intermediate image and the second intermediate image; determining the at least one first characteristic value based on the target frequency; The method of claim 4 , wherein the determination is made by a process comprising:

6. determining a target frequency based on the first intermediate image and the second intermediate image, determining a first relationship between a plurality of correlation values ​​and a plurality of frequencies based on the first intermediate image and the second intermediate image; determining a second relationship between the plurality of correlation values ​​and the plurality of frequencies based on a predetermined function; determining the target frequency based on the first relationship and the second relationship, the target frequency corresponding to an intersection of a first curve representing the first relationship and a second curve representing the second relationship; The method of claim 5 , comprising:

7. The step of determining a first relationship between a plurality of correlation values ​​and a plurality of frequencies based on the first intermediate image and the second intermediate image includes: determining a plurality of correlation values, each of the correlation values ​​corresponding to one of the plurality of frequencies, each of the correlation values ​​being: determining from the first intermediate image, based on the frequency, a first ring image that includes pixels within a ring having a diameter equal to the frequency; determining, from the second intermediate image, a second ring image based on the frequency, the second ring image including pixels within a ring having a diameter equal to the frequency; determining each of the correlation values ​​based on the first ring image and the second ring image; The method of claim 6 , wherein the determination is made according to a process comprising:

8. The step of acquiring a first image and a second image associated with the same object comprises: acquiring an initial image; decomposing the initial image into the first image, the second image, a third image, and a fourth image; The method according to any one of claims 1 to 7, comprising:

9. determining a number of third blocks in the third image and a number of fourth blocks in the fourth image in one-to-one correspondence; determining a plurality of second characteristic values ​​based on the plurality of third blocks and the plurality of fourth blocks; generating a second target map associated with the third image and the fourth image based on the second plurality of characteristic values; generating a third target map based on the first target map and the second target map; The method of claim 8 further comprising:

10. The step of generating a third target map based on the first target map and the second target map includes: averaging the first target map and the second target map to generate the third target map; resizing the third target map to a size of the initial image; 10. The method of claim 9, comprising:

11. The step of determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks includes: For each of the first blocks or each of the second blocks, determining an average value of pixel values ​​of pixels in a central region of each of the first blocks or each of the second blocks; comparing the average value with a threshold value; determining a first characteristic value based on a first intermediate image corresponding to each of the first blocks and a second intermediate image corresponding to each of the second blocks in response to a determination that the average value is greater than the threshold, or determining the first characteristic value based on a predetermined value in response to a determination that the average value is less than the threshold; The method according to any one of claims 1 to 10, comprising:

12. determining a reference map associated with the first image; fusing the first target map and the reference map to determine a fourth target map; The method of any one of claims 1 to 11, further comprising:

13. The step of determining a reference map associated with the first image comprises: acquiring a reference image having a lower resolution than the first image; determining a convolution kernel based on the first image and the reference image; determining an intermediate image based on the first image and the convolution kernel; determining the reference map based on a difference between the intermediate image and the reference image; The method of claim 12, comprising:

14. 1. A system for image processing, comprising: an acquisition module for acquiring a first image and a second image associated with the same object; a rolling module for determining a plurality of first blocks of the first image and a plurality of second blocks of the second image in one-to-one correspondence; determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks; a map generation module that generates a first target map associated with the first image and the second image based on the plurality of first characteristic values, the first target map indicative of reconstruction errors or artifacts of the first image and the second image, each of the first characteristic values ​​corresponding to a pixel of the first target map; A system for image processing comprising:

15. 1. A non-transitory computer-readable medium comprising at least one set of instructions for image processing, the at least one set of instructions, when executed by one or more processors of a computing device, acquiring a first image and a second image associated with the same object; determining a number of first blocks of the first image and a number of second blocks of the second image in one-to-one correspondence; determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks; generating a first target map associated with the first image and the second image based on the plurality of first characteristic values, the first target map indicative of reconstruction errors or artifacts of the first image and the second image, each of the first characteristic values ​​corresponding to a pixel of the first target map; A non-transitory computer-readable medium for causing the computing device to perform a method including:

Images