System and method for image processing

Abstract

To provide systems and methods for image processing.SOLUTION: A system may: obtain a first image and a second image associated with a same object; determine a plurality of first blocks of the first image and a plurality of second blocks of the second image, the plurality of second blocks and the plurality of first blocks being in one-to-one correspondence; determine a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks; and / or generate a first target map associated with the first image and the second image based on the plurality of first characteristic values.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and more particularly to a system and method for determining errors or artifacts that occur during image reconstruction.

Background Art

[0002] Original images generated or collected by an image acquisition device (e.g., microscope, telescope, camera, webcam) are typically reconstructed using reconstruction techniques (e.g., physics-based reconstruction, learning-based reconstruction). However, the reconstruction process may introduce image errors or artifacts, which can cause misinterpretation of information in the image, especially when the image quality is relatively low. Therefore, there is a need to provide a system and method that can efficiently perform image processing to more accurately and effectively determine errors / artifacts.

Summary of the Invention

Means for Solving the Problems

[0003] In one aspect of the present disclosure, a method for image processing is provided. The method may include steps of acquiring a first image and a second image associated with the same object, determining a plurality of first blocks of the first image and a plurality of second blocks of the second image that correspond one-to-one, determining a plurality of first characteristic values based on the plurality of first blocks and the plurality of second blocks, and / or generating a first target map associated with the first image and the second image based on the plurality of first characteristic values.

[0004] In another aspect of the present disclosure, a system for image processing is provided. The system may include at least one storage device for storing executable instructions, and at least one processor capable of communicating with the at least one storage device. When executing the instruction set, the at least one processor may cause the system to perform operations including the steps of: acquiring first and second images associated with the same object; determining a plurality of first blocks in the first image and a plurality of second blocks in the second image that correspond one-to-one; determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks; and / or generating a first target map associated with the first and second images based on the plurality of first characteristic values.

[0005] In another aspect of the present disclosure, a non-temporary computer-readable medium is provided. The non-temporary computer-readable medium may include at least one instruction set for image processing, which, when executed by one or more processors of an arithmetic unit, causes the arithmetic unit to perform a method for image processing.

[0006] In addition, some of the other features described below may be apparent to those skilled in the art through the following discussion and accompanying drawings, or may be understood through the manufacture or operation of the embodiments. The features of this disclosure may be realized and achieved through the practice or use of various forms of methodologies, apparatus, and combinations described in the detailed embodiments below.

[0007] The present disclosure will be described in further detail below with reference to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These embodiments are not limiting, and the same numbers in these embodiments indicate the same configuration. [Brief explanation of the drawing]

[0008] [Figure 1]This is a schematic diagram illustrating exemplary application scenarios of an image processing system according to some embodiments of the present disclosure. [Figure 2] This is a schematic diagram showing an exemplary computing device according to some embodiments of the present disclosure. [Figure 3] A block diagram showing an exemplary portable device in which a terminal may be implemented, according to some embodiments of this disclosure. [Figure 4] This is a schematic diagram showing an exemplary apparatus according to some embodiments of the present disclosure. [Figure 5] This is a flowchart illustrating an exemplary process for determining a target map according to some embodiments of this disclosure. [Figure 6A] This flowchart shows an exemplary process for determining characteristic values ​​according to some embodiments of the present disclosure. [Figure 6B] This flowchart shows an exemplary process for determining characteristic values ​​according to some embodiments of the present disclosure. [Figure 6C] This flowchart shows an exemplary process for determining characteristic values ​​according to some embodiments of the present disclosure. [Figure 6D] This flowchart shows an exemplary process for determining characteristic values ​​according to some embodiments of the present disclosure. [Figure 7] This flowchart shows an exemplary process for determining a reference map according to some embodiments of the present disclosure. [Figure 8] This flowchart shows another exemplary process for determining a target map according to some embodiments of the present disclosure. [Figure 9] Figures 9A to 9D illustrate exemplary processes for determining a target map according to some embodiments of this disclosure. [Figure 10] An exemplary process for determining a reference map, according to some embodiments of this disclosure, is shown. [Figure 11A] This disclosure describes an exemplary process for merging a first target map and a reference map according to some embodiments of this disclosure. [Figure 11B] This disclosure describes an exemplary process for merging a first target map and a reference map according to some embodiments of this disclosure. [Figure 11C] This disclosure describes an exemplary process for merging a first target map and a reference map according to some embodiments of this disclosure. [Figure 12A] This disclosure describes an exemplary process for determining a target map according to some embodiments of this disclosure. [Figure 12B] This disclosure describes an exemplary process for determining a target map according to some embodiments of this disclosure. [Figure 12C] This disclosure describes an exemplary process for determining a target map according to some embodiments of this disclosure. [Figure 12] Figures 12D to 12G illustrate an exemplary process for determining a target map according to some embodiments of this disclosure. [Figure 12H] This disclosure describes an exemplary process for determining a target map according to some embodiments of this disclosure. [Figure 13] Another exemplary process for determining a target map, according to some embodiments of this disclosure, is shown. [Figure 14] Figures 14A-14B show exemplary displacement jet color maps for error indication according to some embodiments of the present disclosure. [Figure 14C] An exemplary displacement jet color map for error indication, according to some embodiments of this disclosure, is shown. [Figure 14D] An exemplary displacement jet color map for error indication, according to some embodiments of this disclosure, is shown. [Figure 15] The following are illustrative principles of target maps according to some embodiments of this disclosure. [Figure 16] The following are exemplary panel frameworks according to some embodiments of this disclosure. [Figure 17A]The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17B] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17C] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17D] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17E] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17F] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17G] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17H] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17I] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17J]The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17K] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17L] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17M] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17N] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17O] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17P] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17Q] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17R] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17S]The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17T] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17U] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17V] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17W] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17X] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 17Y] The following are illustrative results of simulations of single-molecule localization microscopy (SMLM) evaluated by pixel-level analysis of error positions (PANEL) according to some embodiments of this disclosure. [Figure 18A] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18B] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18C]The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18D] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18E] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18F] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18G] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18H] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18I] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18J] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18K] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 18L] The following are illustrative results of open-source 2D-SMLM and SRRF experimental datasets evaluated by PANEL, according to some embodiments of this disclosure. [Figure 19A]The following are exemplary results of all data from 2D-SMLM experiments evaluated and fused by rFRC maps according to some embodiments of this disclosure. [Figure 19B] The following are exemplary results of all data from 2D-SMLM experiments evaluated and fused by rFRC maps according to some embodiments of this disclosure. [Figure 19C] The following are exemplary results of all data from 2D-SMLM experiments evaluated and fused by rFRC maps according to some embodiments of this disclosure. [Figure 19D] The following are exemplary results of all data from 2D-SMLM experiments evaluated and fused by rFRC maps according to some embodiments of this disclosure. [Figure 19E] The following are exemplary results of all data from 2D-SMLM experiments evaluated and fused by rFRC maps according to some embodiments of this disclosure. [Figure 20A] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20B] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20C] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20D] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20E] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20F] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20G] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20H] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20I] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20J] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20K] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20L] The following are exemplary results of STORM fusion using rFRC maps according to some embodiments of this disclosure. [Figure 20M] Examples of STORM fusion according to some embodiments of this disclosure are shown. [Figure 20N] Examples of STORM fusion according to some embodiments of this disclosure are shown. [Figure 21A] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21B] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21C] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21D] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21E] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21F] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21G]The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21H] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21I] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 21J] The following are exemplary results of various physical-based imaging approaches supported by rFRC maps, according to some embodiments of this disclosure. [Figure 22A] The following are exemplary results of all data from a 3D-STORM experiment according to some embodiments of this disclosure. [Figure 22B] The following are exemplary results of all data from a 3D-STORM experiment according to some embodiments of this disclosure. [Figure 22C] The following are exemplary results of all data from a 3D-STORM experiment according to some embodiments of this disclosure. [Figure 23A] The following are illustrative results from another example of a 3D-STORM experiment evaluated by rFRC volume according to some embodiments of this disclosure. [Figure 23B] The following are illustrative results from another example of a 3D-STORM experiment evaluated by rFRC volume according to some embodiments of this disclosure. [Figure 23C] The following are illustrative results from another example of a 3D-STORM experiment evaluated by rFRC volume according to some embodiments of this disclosure. [Figure 23D] The following are illustrative results from another example of a 3D-STORM experiment evaluated by rFRC volume according to some embodiments of this disclosure. [Figure 23E] The following are illustrative results from another example of a 3D-STORM experiment evaluated by rFRC volume according to some embodiments of this disclosure. [Figure 23F]The following are illustrative results from another example of a 3D-STORM experiment evaluated by rFRC volume according to some embodiments of this disclosure. [Figure 24A] The following are exemplary panel results for evaluating coherent computational imaging and Fourier ptichography microscopy (FPM) according to some embodiments of this disclosure. [Figure 24B] The following are exemplary panel results for evaluating coherent computational imaging and Fourier ptichography microscopy (FPM) according to some embodiments of this disclosure. [Figure 24C] The following are exemplary panel results for evaluating coherent computational imaging and Fourier ptichography microscopy (FPM) according to some embodiments of this disclosure. [Figure 24D] The following are exemplary panel results for evaluating coherent computational imaging and Fourier ptichography microscopy (FPM) according to some embodiments of this disclosure. [Figure 24E] The following are exemplary panel results for evaluating coherent computational imaging and Fourier ptichography microscopy (FPM) according to some embodiments of this disclosure. [Figure 24F] The following are exemplary panel results for evaluating coherent computational imaging and Fourier ptichography microscopy (FPM) according to some embodiments of this disclosure. [Figure 25A] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25B] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25C] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25D] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25E]The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25F] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25G] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25H] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 25I] The following are exemplary results of a single-frame rFRC calculation for 3D-SIM according to some embodiments of this disclosure. [Figure 26] The following are exemplary results of a fully sparse sampling simulation according to some embodiments of this disclosure. [Figure 27A] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27B] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27C] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27D] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27E] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27F] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27G]The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27H] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27I] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27J] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27K] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 27L] The following are illustrative results of simulations of learning-based applications evaluated by PANEL, according to some embodiments of this disclosure. [Figure 28A] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28B] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28C] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28D] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28E] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28F] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28G] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28H] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28I] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28J] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28K] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28L] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28M] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28N] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28O] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 28P] The experimental results of a learning-based application evaluated by PANEL, according to some embodiments of this disclosure, are shown. [Figure 29A] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29B] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29C]The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29D] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29E] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29F] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29G] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29H] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 29I] The following are exemplary results of all data from a TIRF-SIM experiment according to some embodiments of this disclosure. [Figure 30A] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30B] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30C] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30D] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30E] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30F] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30G] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30H] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 30I] The following are exemplary results from a full TIRF deconvolution experiment according to some embodiments of this disclosure. [Figure 31A] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 31B] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 31C] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 31D] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 31E] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 31F] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 31G] The following are exemplary results of a full-pixel super-resolution simulation according to some embodiments of this disclosure. [Figure 32A] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32B] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32C] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32D] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32E] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32F]The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32G] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32H] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32I] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 32J] The following are exemplary results of a single-frame rFRC operation for learning-based deconvolution according to some embodiments of the present disclosure. [Figure 33A] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33B] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33C] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33D] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33E] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33F] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33G] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33H] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 33I] The following are exemplary results from a complete ANNA-PALM experiment according to some embodiments of this disclosure. [Figure 34A]The following are exemplary results from a complete CARE experiment according to some embodiments of this disclosure. [Figure 34B] The following are exemplary results from a complete CARE experiment according to some embodiments of this disclosure. [Figure 34C] The following are exemplary results from a complete CARE experiment according to some embodiments of this disclosure. [Figure 34D] The following are exemplary results from a complete CARE experiment according to some embodiments of this disclosure. [Figure 34E] The following are exemplary results from a complete CARE experiment according to some embodiments of this disclosure. [Figure 34F] The following are exemplary results from a complete CARE experiment according to some embodiments of this disclosure. [Figure 35A] The following are exemplary results from a complete Noise2Noise experiment according to some embodiments of this disclosure. [Figure 35B] The following are exemplary results from a complete Noise2Noise experiment according to some embodiments of this disclosure. [Figure 35C] The following are exemplary results from a complete Noise2Noise experiment according to some embodiments of this disclosure. [Figure 35D] The following are exemplary results from a complete Noise2Noise experiment according to some embodiments of this disclosure. [Figure 35E] The following are exemplary results from a complete Noise2Noise experiment according to some embodiments of this disclosure. [Figure 36A] The following are exemplary results of adaptive low-pass filters for Richardson-Lucy deconvolution (RLD) according to some embodiments of this disclosure. [Figure 36B] The following are exemplary results of adaptive low-pass filters for Richardson-Lucy deconvolution (RLD) according to some embodiments of this disclosure. [Figure 36C] The following are exemplary results of adaptive low-pass filters for Richardson-Lucy deconvolution (RLD) according to some embodiments of this disclosure. [Figure 36D]The following are exemplary results of adaptive low-pass filters for Richardson-Lucy deconvolution (RLD) according to some embodiments of this disclosure. [Figure 36E] The following are exemplary results of adaptive low-pass filters for Richardson-Lucy deconvolution (RLD) according to some embodiments of this disclosure. [Figure 36F] The following are exemplary results of adaptive low-pass filters for Richardson-Lucy deconvolution (RLD) according to some embodiments of this disclosure. [Figure 37A] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37B] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37C] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37D] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37E] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37F] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37G] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37H] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 37I] The following are exemplary SMLM result fusions of MLE and FALCON according to some embodiments of this disclosure. [Figure 38A] This disclosure shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of this disclosure. [Figure 38B]This disclosure shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of this disclosure. [Figure 38C] This disclosure shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of this disclosure. [Figure 38D] This disclosure shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of this disclosure. [Figure 38E] This disclosure shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of this disclosure. [Figure 38F] This disclosure shows a comparison between exemplary fusion results using RSM or rFRC maps according to some embodiments of this disclosure. [Figure 39A] The following are illustrative results of simulations evaluating the performance of SSIM and rFRC at different noise amplitudes according to some embodiments of this disclosure. [Figure 39B] The following are illustrative results of simulations evaluating the performance of SSIM and rFRC at different noise amplitudes according to some embodiments of this disclosure. [Figure 39C] The following are illustrative results of simulations evaluating the performance of SSIM and rFRC at different noise amplitudes according to some embodiments of this disclosure. [Figure 39D] The following are illustrative results of simulations evaluating the performance of SSIM and rFRC at different noise amplitudes according to some embodiments of this disclosure. [Figure 39E] The following are illustrative results of simulations evaluating the performance of SSIM and rFRC at different noise amplitudes according to some embodiments of this disclosure. [Figure 39F] The following are illustrative results of simulations evaluating the performance of SSIM and rFRC at different noise amplitudes according to some embodiments of this disclosure. [Figure 40A] The following are exemplary results demonstrating background-induced false negatives according to some embodiments of this disclosure. [Figure 40B]The following are exemplary results demonstrating background-induced false negatives according to some embodiments of this disclosure. [Figure 40C] The following are exemplary results demonstrating background-induced false negatives according to some embodiments of this disclosure. [Figure 41] Figures 41A to 41D show exemplary results of resolution mapping using a 3σ curve versus a 1 / 7 hard threshold according to some embodiments of the present disclosure. [Figure 42] This disclosure outlines network architectures, learning parameter configurations, and data used for different applications, according to several embodiments of this disclosure. [Figure 43] An exemplary neural network backbone used for image restoration, according to some embodiments of this disclosure, is shown. [Modes for carrying out the invention]

[0009] In the following detailed description, several specific details are provided to provide a complete understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that the disclosure can be implemented without such details. In other examples, well-known methods, procedures, systems, components, and / or circuits, etc., are described at a relatively high level and not in detail, in order to avoid unnecessarily obscuring aspects of the disclosure. Various modifications of the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined in the disclosure can be applied to other embodiments and uses without departing from the spirit and scope of the disclosure. Therefore, the disclosure should be limited to the broadest scope consistent with the claims, and not limited to the embodiments shown.

[0010] The terms used herein are for illustrative purposes only and are not intended to limit any particular exemplary embodiment. As used herein, the singular forms "a," "an," and "the" are intended to include the plural form unless otherwise specified in the context. Furthermore, as used herein, the terms "comprise," "comprises," and / or "comprising," "include," "includes," and / or "including" identify the presence of a described feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "object" and "subject" are used synonymously with respect to the object on which the imaging procedure of this disclosure is performed.

[0011] It should be understood that the terms “system,” “engine,” “unit,” “module,” and / or “block” as used herein are one way of distinguishing, in ascending order, different components, elements, parts, sections, or assemblies at different levels. However, terms may be replaced by other expressions that achieve the same purpose.

[0012] As used herein, the terms “module,” “unit,” or “block” generally refer to a set of logic or software instructions implemented in hardware or firmware. Modules, units, or blocks described herein may be implemented as software and / or hardware and may be stored in any type of non-temporary computer-readable medium or other storage device. In some embodiments, software modules / units / blocks may be compiled and linked into an executable program. Software modules may be called from other modules / units / blocks or from themselves, and / or in response to detected events or interrupts. Software modules / units / blocks executed on an arithmetic unit (e.g., the processor 210 shown in Figure 2) may be provided on a computer-readable medium such as a compact disk, digital video disk, flash drive, magnetic disk, or other tangible medium, or as a digital download (originally stored in a compressed or installable file format requiring installation, decompression, or decryption before execution). For execution by the arithmetic unit, some or all of such software code may be stored in the storage device of the executing arithmetic unit. Software instructions may be embedded in firmware such as an EPROM. Furthermore, hardware modules / units / blocks may include connected logic components such as gates and flip-flops, and / or may include programmable units such as programmable gate arrays or processors. Modules / units / blocks or arithmetic unit functions described herein may be implemented as software modules / units / blocks, but may also be represented in hardware or firmware. Modules / units / blocks described herein generally refer to logical modules / units / blocks that may be combined with other modules / units / blocks and may be divided into submodules / subunits / subblocks, regardless of their physical configuration or storage. This description may apply to systems, engines, or parts thereof.

[0013] Where it is stated that a unit, engine, module, or block is “on top of,” “connected to,” or “combined with” another unit, engine, module, or block, unless otherwise explicitly stated in the context, it will be understood that it may be directly on top of another unit, engine, module, or block, directly connected to or combined with another unit, engine, module, or block, directly communicate with another unit, engine, module, or block, or intervening unit, engine, module, or block may be present. As used herein, “and / or” includes any and all combinations of one or more items from the multiple items associated.

[0014] When describing actions performed on an image, the term "image" as used herein may refer to a dataset (e.g., a matrix) containing the values ​​of pixels (pixel values) within the image. For convenience of explanation, as used herein, a representation of an object in an image (e.g., a person, organ, cell, or part thereof) may be referred to as an object. For example, for convenience of explanation, a representation of a cell or organelle in an image (e.g., mitochondria, endoplasmic reticulum, centrosome, Golgi apparatus, etc.) may be referred to as a cell or organelle. For convenience of explanation, as used herein, an action performed on a representation of an object in an image may be referred to as an action on an object. For example, for convenience of explanation, segmentation of a part of an image containing a representation of a cell or organelle from an image may be referred to as cell or organelle segmentation.

[0015] As used herein, the term "resolution" refers to a measure of the sharpness of an image. As used herein, the terms "super-resolution," "super-resolution," or "SR" refer to improved (or increased) resolution, which can be achieved, for example, by a process that combines a series of low-resolution images to produce a higher-resolution image or sequence.

[0016] These and other features and characteristics, methods of operation, functions of the structural elements and component combinations, and manufacturing economics of this disclosure will become more apparent when considering the following description with reference to the accompanying drawings which constitute part of this disclosure. However, it should be clearly understood that the drawings are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure. It should also be understood that the drawings are not to scale.

[0017] The flowcharts used in this disclosure illustrate the actions performed by a system according to some embodiments of this disclosure. It should be clearly understood that the actions in the flowchart may be performed in no particular order. Conversely, the actions may be performed in reverse order or simultaneously. Furthermore, one or more other actions may be added to the flowchart. One or more actions may be removed from the flowchart.

[0018] With technological advancements, a range of super-resolution approaches may break the diffraction limit (e.g., 200-300 nm) through the use of computational processes. Such computational operations may generally be performed on images generated based on several categories of techniques, including, for example: (i) molecular localization such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM); (ii) molecular intensity fluctuations such as super-resolution fluctuation imaging (SOFI) and super-resolution radial fluctuation (SRRF); (iii) high-frequency information mixing such as structured illumination microscopy (SIM); and (iv) deconvolution, or other techniques that rely on high-computational reconstruction to extend the spatial resolution of images. In addition to the original physical configuration, multiple learning-based algorithms may be actively developed to reduce the photon / hardware budget in image generation, reconstruction, and / or processing. For example, artificial neural network-accelerated PALM (ANNA-PALM) may be applied to reduce the total number of frames (or images, or image frames) without trading off the spatial resolution of PALM. As another example, a conditional generative adversarial network (cGAN) can be used to directly transform total internal reflection fluorescence (TIRF) microscope images to match results obtained with TIRF-SIM. Further examples include content-aware image restoration (CARE) network frameworks, which can perform denoising and deconvolution applications more effectively.

[0019] Such computational modalities are considered promising imaging techniques. In some embodiments, image errors or artifacts may occur during image reconstruction, and relatively low image quality may lead to misinterpretation of biological information. In some embodiments, the image quality and resolution achieved may be related to one or more combined factors, including, for example, the photophysics of the phosphor used for imaging, the chemical environment of the sample being imaged, optical setup conditions, analytical approaches for generating images (e.g., super-resolution images), network learning procedures, etc., or a combination thereof. For quantitative mapping and minimization of image defects, most existing quality assessments may rely on subjective comparison of reconstructed images to standard reference structures, benchmarking of reconstructed image data against other high-resolution imaging techniques such as electron microscopy, or corresponding specific design and analysis algorithms. As just one example, in the case of STORM / PLAM, given precise imaging models and noise statistics as inputs, the confidence of individual localizations within a STORM / PLAM dataset may be accessible based on a Wasserstein-induced flux computation process without knowledge of ground truth. In some embodiments, SIM checks requiring complex pipelines and specialized use may be used to identify the causes of errors and artifacts in the SIM system. In some embodiments, modifications to the existing training procedures of a learning-based approach (e.g., a Bayesian neural network (BNN) framework, learning a distribution for weights, etc.) may be necessary to capture the uncertainty of the learning-based approach. In some embodiments, such modifications may be more complex to apply and computationally expensive compared to standard techniques (e.g., general learning-based approaches).

[0020] In some embodiments, a resolution-scale error map (RSM) (e.g., a reference map) may be used as an evaluation tool for super-resolution images to generally and accurately assess local errors / artifacts. In some embodiments, a wide-field reference image may be used to generate the RSM. In some embodiments, the RSM may be generated based on one or more hypotheses, including, for example: (i) the wide-field reference image may have a relatively high SNR; (ii) the Gaussian kernel convolution performed in generating the RSM map may be spatially invariant; and (iii) the illumination used when generating the raw data of the super-resolution image may be uniform. According to these hypotheses, in some embodiments, the RSM is quite likely to induce false negatives in the estimated error map. On the other hand, when the super-resolution image is converted to its lower-resolution region, the detectable scale of errors in the RSM may be limited by Abbe diffraction. In some embodiments, a reference empty process (e.g., pixel-level analysis of error locations (PANEL)) may be used to detect errors / artifacts at the super-resolution scale. In some embodiments, the PANEL technique may be built upon rolling Fourier correlation (rFRC) using one or more separate reconstructed super-resolution frames and / or further collaborate with a modified RSM. In some embodiments, one or more strategies for achieving single-frame rFRC computation may be involved in the PANEL framework for physics-based and / or learning-based applications.

[0021] In some embodiments, the RSM and rFRC maps are integrated into the PANEL framework, and the PANEL framework may be developed as a multifunctional, model-free metric. Thus, the PANEL framework may have the ability to infer errors contained in multidimensional signals up to super-resolution scales and to mitigate possible false negative defects. The PANEL framework can be used in a variety of scenarios, such as physical super-resolution imaging approaches (2D-STORM or 3D-STORM, TIRF-SIM or 3D-SIM, SRRF, Richardson-Lucy deconvolution, etc.), various learning-based image reconstruction techniques (ANNA-PALM, TIRF-TIRF-SIM, CARE, etc.), and / or non-super-resolution applications such as denoising tasks. In some embodiments, one or more minute quantitative error maps generated using the PANEL framework can be used to design multiple computational processes (e.g., STORM image fusion, adaptive low-pass filtering, automated deconvolution iterative decision, etc.). In some embodiments, the PANEL framework can be used as a tool for robust and detailed evaluation of the image quality of physically based or learning-based super-resolution imaging approaches, and can facilitate the optimization of super-resolution imaging and network learning procedures.

[0022] A wide range of super-resolution microscopy techniques may rely on corresponding computational operations that can form reconstruction errors at one or more scales. Image reconstruction errors can lead to misinterpretations of biological information. Quantitative mapping of such errors in super-resolution reconstruction may be limited to identifying minute errors resulting from the inevitable use of diffraction-limited images as a reference. According to some embodiments of this disclosure, a rolling Fouriering correlation (rFRC) operation may be used to trace pixels in the reconstructed image and can provide a relatively precise model-free metric that enables pixel-level analysis of error locations (PANEL). In some embodiments, a target image or map (e.g., an rFRC map) can be generated according to the pixel-level analysis of error locations (PANEL). The target image or map can illustrate defects in the reconstructed image down to the super-resolution scale. In some embodiments, the reliability evaluation of the PANEL approach may be performed on images reconstructed based on one or more computational modalities (e.g., physical super-resolution microscopy, deep learning super-resolution microscopy, etc.).

[0023] This disclosure relates to a system and method for image processing. The system and method may acquire a first image and a second image associated with the same object. The system and method may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image. The plurality of first blocks and the plurality of second blocks may correspond one-to-one. The system and method may determine a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks. The system and method may generate a target map (e.g., a first target map) associated with the first image and the second image based on the plurality of first characteristic values. The first target map may quantitatively map errors in the first image and / or second image generated by the image acquisition device without using a reference (e.g., ground truth) and / or any prior information of the image acquisition device. According to some embodiments of this disclosure, the first target map may be used as a tool for quantitative evaluation of image quality.

[0024] Furthermore, while the systems and methods disclosed herein primarily relate to the generation of rFRC maps that reflect errors in images reconstructed by physical super-resolution microscopes and / or deep learning super-resolution microscopes, it should be understood that such descriptions are provided solely for illustrative purposes and are not intended to limit the scope of this disclosure. The systems and methods disclosed herein may be applied to any other type of system, including image acquisition devices for image processing. For example, the systems and methods disclosed herein may be applied to microscopes, telescopes, cameras (e.g., surveillance cameras, camera phones, webcams), unmanned aerial vehicles, medical imaging devices, etc., or any combination thereof.

[0025] As merely an example, the methods disclosed herein may be used to generate an rFRC map based on dual frames (e.g., sequential first and second images) or a single frame (e.g., initial image) generated by physical super-resolution imaging and / or deep learning super-resolution reconstruction.

[0026] It should be understood that the application scenarios of the systems and methods disclosed herein are merely illustrative embodiments provided for illustrative purposes and are not intended to limit the scope of this disclosure. Those skilled in the art can make numerous modifications and variations based on the teachings of this disclosure.

[0027] Figure 1 is a schematic diagram illustrating exemplary application scenarios of an image processing system according to some embodiments of the present disclosure. As shown in Figure 1, the image processing system 100 may include an image acquisition device 110, a network 120, one or more terminals 130, a processing device 140, and a storage device 150.

[0028] The components of the image processing system 100 may be connected in one or more ways. As just one example, the image acquisition device 110 may be connected to the processing device 140 via the network 120. As another example, the image acquisition device 110 may be connected directly to the processing device 140, as indicated by the dotted double-headed arrow connecting the image acquisition device 110 and the processing device 140. As yet another example, the storage device 150 may be connected to the processing device 140 directly or via the network 120. As yet another example, the terminal 130 may be connected to the processing device 140 directly or via the network 120 (as indicated by the dotted double-headed arrow connecting the terminal 130 and the processing device 140).

[0029] The image processing system 100 may be configured to generate a first target map (e.g., an rFRC map) based on the first and second images using an rFRC process (e.g., one or more operations shown in Figure 5). The first target map may illustrate or display errors and / or artifacts of the first and / or second images generated by the image acquisition device 110, and / or reflect the image quality (e.g., the degree of error). The image processing system 100 may be configured to determine a reference map associated with the first image. In some embodiments, the reference map may be used to identify errors not reported by the first target map. The image processing system 100 may be configured to determine a fourth target map by fusing the first target map and the reference map. The fourth target map may be configured to comprehensively illustrate or display errors and / or artifacts of the first and / or second images. In some embodiments, errors and / or artifacts displayed by target maps (e.g., first target map, fourth target map) may relate to errors generated in the reconstruction of the first and / or second image, or include errors generated in the reconstruction of the first and / or second image.

[0030] The image acquisition device 110 may be configured to acquire one or more images (e.g., a first image, a second image, an initial image, etc.) associated with an object within its detection area. The object may include one or more biological or non-biological objects. In some embodiments, the image acquisition device 110 may be an optical imaging device, a radiation-based imaging device (e.g., a computed tomography device), a nuclide-based imaging device (e.g., a positron emission tomography device, a nuclear magnetic resonance imaging device), etc. Exemplary optical imaging devices may include a microscope 111 (e.g., a fluorescence microscope), a surveillance device 112 (e.g., a security camera), a mobile terminal device 113 (e.g., a camera phone), a scanning device 114 (e.g., a flatbed scanner, a drum scanner, etc.), a telescope, a webcam, etc., or any combination thereof. In some embodiments, the optical imaging device may include an acquisition device (e.g., a detector or a camera) for collecting image data. For illustrative purposes, the present disclosure may use a microscope 111 as an example to illustrate the exemplary functions of the image acquisition device 110. Exemplary microscopes may include structured illumination microscopes (SIMs) (e.g., two-dimensional SIMs (2D-SIMs), three-dimensional SIMs (3D-SIMs), total internal reflection SIMs (TIRF-SIMs), spinning disk confocal-based SIMs (SD-SIMs), etc.), photoactivated localization microscopes (PALMs), stimulated emission suppression microscopes (STEDs), stochastic optical reconstruction microscopes (Storms), etc.). SIMs may include detectors such as EMCCD cameras and sCMOS cameras. Objects detected by SIMs may include one or more objects, or any combination thereof, such as biological structures, biological issues, proteins, cells, and microorganisms. Exemplary cells may include INS-1 cells, COS-7 cells, HeLa cells, hepatic sinusoidal endothelial cells (LSECs), human umbilical vein endothelial cells (HUVECs), HEK293 cells, etc., or any combination thereof. In some embodiments, one or more objects may be fluorescent or fluorescently labeled. Fluorescent or fluorescently labeled objects may be excited to emit fluorescence for imaging.

[0031] The network 120 may include any suitable network that facilitates the exchange of information and / or data by the image processing system 100. In some embodiments, one or more components of the image processing system 100 (e.g., image acquisition device 110, terminal 130, processing unit 140, storage device 150, etc.) may communicate information and / or data with each other via the network 120. For example, the processing unit 140 may acquire image data (e.g., first image, second image) from the image acquisition device 110 via the network 120. In another example, the processing unit 140 may acquire user commands from the terminal 130 via the network 120. Network 120 may include public networks (e.g., the Internet), private networks (e.g., local area networks (LANs), wide area networks (WANs), etc.), wired networks (e.g., Ethernet), wireless networks (e.g., 802.11 networks, Wi-Fi networks, etc.), cellular networks (e.g., Long-Term Evolution (LTE) networks), image relay networks, virtual private networks ("VPNs"), satellite networks, telephone networks, routers, hubs, switches, server computers, and / or combinations of one or more of these. For example, Network 120 may include cable networks, wired networks, fiber optic networks, telecommunications networks, local area networks, wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth® networks, Zikbi® networks, near-field communication networks (NFC), etc., or combinations thereof. In some embodiments, Network 120 may include one or more network access points.For example, the network 120 may include wired network access points such as base stations and / or network switching points and / or wireless network access points, and one or more components of the image processing system 100 may access the network 120 for data and / or information exchange via these access points.

[0032] In some embodiments, a user can operate the image processing system 100 via a terminal 130. The terminal 130 may include a portable device 131, a tablet computer 132, a laptop computer 133, etc., or a combination thereof. In some embodiments, the portable device 131 may include a smart home device, a wearable device, a portable device, a virtual reality device, an augmented reality device, etc. In some embodiments, the smart home device may include a smart lighting fixture, a control device for intelligent electrical appliances, a smart surveillance device, a smart TV, a smart video camera, an intercom, etc., or a combination thereof. In some embodiments, the wearable device may include a bracelet, footwear, glasses, a helmet, a watch, clothing, a backpack, a smart accessory, etc., or a combination thereof. In some embodiments, the portable device may include a mobile phone, a personal digital assistant (PDA), a gaming device, a navigation device, a point-of-sale (POS) device, a laptop, a tablet computer, a desktop, etc., or a combination thereof. In some embodiments, the virtual reality device and / or augmented reality device may include a virtual reality helmet, virtual reality glasses, virtual reality eyewear, augmented reality helmet, augmented reality glasses, augmented reality eyewear, etc., or a combination thereof. For example, the virtual reality device and / or augmented reality device may include Google Glass®, Oculus Rift®, HoloLens®, GearVR®, etc. In some embodiments, the terminal 130 may be part of the processing unit 140.

[0033] The processing unit 140 may process data and / or information acquired from the image acquisition device 110, the terminal 130, and / or the storage device 150. For example, the processing unit 140 may process one or more images (e.g., a first image, a second image, etc.) generated by the image acquisition device 110 to generate rFRC maps (e.g., a first target map, a third target map, etc.). In some embodiments, the processing unit 140 may be a server or a group of servers. The server group may be centralized or distributed. In some embodiments, the processing unit 140 may be local or remote. For example, the processing unit 140 may access information and / or data stored in the image acquisition device 110, the terminal 130, and / or the storage device 150 via the network 120. As another example, the processing unit 140 may be directly connected to the image acquisition device 110, the terminal 130, and / or the storage device 150 to access the stored information and / or data. In some embodiments, the processing unit 140 may be implemented on a cloud platform. For example, the cloud platform may include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, interconnected clouds, multi-clouds, etc., or combinations thereof. In some embodiments, the processing unit 140 may be implemented by a computing unit 200 having one or more components, as shown in Figure 2.

[0034] The storage device 150 may store data, instructions, and / or any other information. In some embodiments, the storage device 150 may store data acquired from the terminal 130, the image acquisition device 110, and / or the processing device 140. In some embodiments, the storage device 150 may store data and / or instructions that the processing device 140 may execute or use to perform the exemplary methods described herein. In some embodiments, the storage device 150 may include mass storage, removable storage, volatile read / write memory, read-only memory (ROM), etc. Exemplary mass storage may include magnetic disks, optical disks, solid-state drives, etc. Exemplary removable storage may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tapes, etc. Exemplary volatile read / write memory may include random access memory (RAM). Exemplary RAM may include dynamic RAM (DRAM), double-datarate synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero-capacitor RAM (Z-RAM), etc. Exemplary ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), and digital versatile disk ROM, etc. In some embodiments, the storage device 150 may run on a cloud platform. For example, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, interconnected cloud, multi-cloud, etc., or a combination thereof.

[0035] In some embodiments, the storage device 150 may be connected to a network 120 and communicate with one or more other components of the image processing system 100 (e.g., a processing unit 140, a terminal 130, etc.). One or more components of the image processing system 100 may access data or instructions stored in the storage device 150 via the network 120. In some embodiments, the storage device 150 may be directly connected to one or more other components of the image processing system 100 (e.g., a processing unit 140, a terminal 130, etc.) and communicate with them. In some embodiments, the storage device 150 may be part of the processing unit 140.

[0036] Figure 2 is a schematic diagram showing an exemplary computing device according to some embodiments of the present disclosure. The computing device 200 may be used to implement any component of the image processing system 100 described herein. For example, the processing unit 140 and / or terminal 130 may be implemented on the computing device 200, respectively, via their hardware, software programs, firmware, or a combination thereof. Although only one such computing device is shown, for convenience, the computer functions relating to the image processing system 100 described herein may be implemented in a distributed manner on multiple similar platforms to distribute the processing load.

[0037] As shown in Figure 2, the arithmetic unit 200 may include a processor 210, storage 220, input / output (I / O) unit 230, and communication port 240.

[0038] The processor 210 may execute computer instructions (e.g., program code) and perform functions of the image processing system 100 (e.g., processing unit 140) in accordance with the techniques described herein. Computer instructions may include, for example, routines, programs, objects, components, data structures, procedures, modules, and functions that perform specific functions described herein. For example, the processor 210 may process image data acquired from any component of the image processing system 100. In some embodiments, the processor 210 may include one or more hardware processors such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physics processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or a combination thereof.

[0039] For illustrative purposes only, only one processor is described in the arithmetic unit 200. However, it should be noted that the arithmetic unit 200 in this disclosure may include multiple processors, and thereafter, operations and / or method operations performed by one processor as described in this disclosure may also be performed jointly or individually by multiple processors. For example, if in this disclosure a processor in the arithmetic unit 200 performs both operations A and B, it should be understood that operations A and B may also be performed jointly or individually by two or more different processors in the arithmetic unit 200 (for example, a first processor performs operation A and a second processor performs operation B, or the first and second processors jointly perform operations A and B).

[0040] The storage 220 may store data / information obtained from any component of the image processing system 100. In some embodiments, the storage 220 may include mass storage devices, removable storage devices, volatile read / write memory, read-only memory (ROM), etc., or any combination thereof. Exemplary mass storage devices may include magnetic disks, optical disks, solid-state drives, etc. Removable storage devices may include flash drives, floppy disks, optical disks, memory cards, zip disks, magnetic tapes, etc. Volatile read / write memory may include random access memory (RAM). RAM may include dynamic RAM (DRAM), double-digital-rate-synchronous dynamic RAM (DDR SDRAM), static RAM (SRAM), thyristor RAM (T-RAM), and zero-capacitor RAM (Z-RAM), etc. ROM may include mask ROM (MROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), and digital multipurpose disk ROM, etc.

[0041] In some embodiments, the storage 220 may store one or more programs and / or instructions for performing exemplary methods described herein. For example, the storage 220 may store a program for the processing unit 140 to process images generated by the image acquisition unit 110.

[0042] The I / O unit 230 may input and / or output signals, data, information, etc. In some embodiments, the I / O unit 230 may enable user interaction with the image processing system 100 (e.g., the processing unit 140). In some embodiments, the I / O unit 230 may include input and output devices. Examples of input devices include keyboards, mice, touchscreens, microphones, etc., or combinations thereof. Examples of output devices include display devices, speakers, printers, projectors, etc., or combinations thereof. Examples of display devices include liquid crystal displays (LCDs), light-emitting diode (LED) based displays, thin-screen displays, curved screens, televisions, cathode ray tubes (CRTs), touchscreens, etc., or combinations thereof.

[0043] The communication port 240 may be connected to a network to facilitate data communication. The communication port 240 may establish a connection between the processing unit 140 and the image acquisition device 110, terminal 130, and / or storage device 150. The connection may be a wired connection, a wireless connection, any other communication connection that can enable data transfer and / or data reception, and / or any combination thereof. Wired connections may include, for example, electrical cables, optical cables, telephone lines, etc., or a combination thereof. Wireless connections may include Bluetooth® links, Wi-Fi® links, WiMAX® links, WLAN links, Zikbee® links, mobile network links (e.g., 3G, 4G, 5G), etc., or a combination thereof. In some embodiments, the communication port 240 may be a standardized communication port such as RS232 or RS485, and / or may include one. In some embodiments, the communication port 240 may be a specially designed communication port. For example, communication port 240 may be designed according to the Digital Imaging and Communication in Medical (DICOM) protocol.

[0044] Figure 3 is a block diagram showing an exemplary portable device in which terminal 130 may be implemented, according to some embodiments of the present disclosure.

[0045] As shown in Figure 3, the mobile device 300 may include a communication unit 310, a display unit 320, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, an I / O unit 350, memory 360, a storage unit 370, and the like. In some embodiments, the mobile device 300 may also include, but is not limited to, any other suitable components including a system bus or controller (not shown). In some embodiments, an operating system 361 (e.g., IOS®, Android®, Windows® Phone, etc.) and one or more applications (apps) 362 may be loaded from the storage unit 370 into memory 360 so that they are executed by the CPU 340. The applications 362 may include a browser or any other suitable mobile app for receiving and rendering information about imaging, image processing, or other information from an image processing system 100 (e.g., a processing unit 140). User interaction with the information stream is realized via the I / O unit 350 and may be provided to the processing unit 140 and / or other components of the image processing system 100 via the network 120. In some embodiments, the user may input parameters to the image processing system 100 via a portable device 300.

[0046] To implement the various modules, units, and their functions described above, a computer hardware platform may be used as the hardware platform for one or more elements (for example, the processing unit 140 and / or other components of the image processing system 100 shown in Figure 1). Since these hardware elements, operating systems, and programming languages ​​are common, it is assumed that those skilled in the art may be familiar with these technologies and may be able to provide the information required for imaging and evaluation in accordance with the technologies described herein. The computer with a user interface may be used as a personal computer (PC) or as another type of workstation or terminal device. After being appropriately programmed, the computer with a user interface may be used as a server. Those skilled in the art may also be assumed to be familiar with such structures, programs, or general operations of this type of computing device.

[0047] Figure 4 is a schematic diagram showing an exemplary processing apparatus according to some embodiments of the present disclosure. As shown in Figure 4, the processing apparatus 140 may include an acquisition module 402, an expansion module 404, a rolling module 406, a map generation module 408, a filtering module 410, a display module 412, an image quality determination module 414, a decomposition module 416, and a neural network processing module 418.

[0048] The acquisition module 402 may be configured to acquire one or more images (for example, a first image and / or a second image associated with the same object, an initial image).

[0049] The expansion module 404 may be configured to expand one or more images (for example, a first image, a second image, a third image, and / or a fourth image).

[0050] The rolling module 406 may be configured to generate one or more image blocks (for example, multiple first blocks of the first image, multiple second blocks of the second image, multiple third blocks of the third image, multiple fourth blocks of the fourth image, etc.).

[0051] The map generation module 408 may be configured to determine a plurality of first characteristic values ​​based on a plurality of first blocks and a plurality of second blocks, generate a first target map associated with a first image and a second image based on the plurality of first characteristic values, determine a reference map associated with the first image or the second image, and determine a fourth target map by fusing the first target map and the reference map. In some embodiments, the map generation module 408 may determine a plurality of second characteristic values ​​based on a plurality of third blocks and a plurality of fourth blocks, generate a second target map associated with a third image and a fourth image based on the plurality of second characteristic values, and generate a third target map based on the first target map and the second target map.

[0052] The filtering module 410 may be configured to filter one or more images (for example, a first target map, a second target map, a third target map, a fourth target map, etc.).

[0053] The display module 412 may be configured to display one or more images (e.g., a first target map, a second target map, a third target map, a fourth target map, a reference map, etc., or any other images generated by or used in image processing). In some embodiments, the display module 412 may display one or more images using a displacement jet color map.

[0054] The image quality determination module 414 may be configured to determine the image quality of one or more images. For example, the image quality determination module 414 may determine global image quality metrics for the first and / or second images based on a first target map.

[0055] The decomposition module 416 may be configured to decompose one or more images. For example, the decomposition module 416 may decompose the initial image into a first image, a second image, a third image, and a fourth image.

[0056] The neural network processing module 418 may be configured to generate one or more images by processing one or more initial images using a neural network.

[0057] A more detailed description of the modules of the processing unit 140 may be found elsewhere in this disclosure (for example, in Figures 5 to 8 and their descriptions).

[0058] The above description of the modules of the processing unit 140 is provided for illustrative purposes only and is not intended to limit the disclosure. Those skilled in the art will know that the modules may be combined in various ways, without departing from the principles of the disclosure, under the teachings of the disclosure, and may be connected to other modules as subsystems. In some embodiments, one or more modules may be added to or omitted in the processing unit 140. For example, the expansion module 404 and the rolling module 406 may be integrated into a single module.

[0059] Figure 5 is a flowchart illustrating an exemplary process for determining a target map according to some embodiments of the present disclosure. In some embodiments, process 500 may be performed by an image processing system 100. For example, process 500 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules illustrated in Figure 4) may execute the instruction set and, accordingly, execute process 500. The operation of the process shown below is intended to be illustrative. In some embodiments, process 500 may be achieved with one or more additional operations not described, and / or without one or more operations described. Furthermore, the order of operations of process 500 shown in Figure 5 and described below is not intended to be limiting.

[0060] In some embodiments, an imaging system (e.g., an image processing system 100) may observe one or more objects using optical devices mathematically expressed as transfer functions, and collect corresponding signals using one or more sensors. In some embodiments, the signals may be sampled by the sensors and / or mixed with noise. In some embodiments, the artificial observations may deviate from the real-world objects in a relatively high-dimensional space (e.g., as shown in the left panel of Figure 15). Under such circumstances, reconstructing hidden real signals from the measurement data may be considered a poorly set-up problem, and the reconstructions from these observations may deviate from the real-world objects. If two separately measured signals are captured on the same object, these two signals may be related to the same object. It may be conceivable that the greater the distance between the reconstructions of these two signals (e.g., as shown in the right panel of Figure 15), the greater the deviation the reconstruction may show from the real object. This distance (for example, the distance between reconstruction 1 and reconstruction 2) is related to the reconstruction error (for example, the deviation of the reconstruction from the real object), and if we follow the intuition above, we can make the important hypothesis that the distance between the two reconstructions is positively correlated with the magnitude of the reconstruction error.

[0061] To convincingly test this hypothesis, the derivation may be limited to specific general circumstances for fluorescence imaging with a wide-field (WF) microscope, in which case the corresponding image may be processed, for example, by a minimalist form of Wiener deconvolution.

[0062] The lemma may also be described as follows: Under ideal conditions, an object may be observed using an aberration-free optical system, and such observations may be sampled at an infinite sampling rate without noise. The object may also be reconstructed using a minimalist form of Wiener deconvolution, as expressed by equation (1).

[0063]

number

[0064] Here, ω represents the spatial coordinate, and I, o, and h represent the point spread functions (PSF) of the illumination, object, and microscope, respectively. F and F% are the operators of the Fourier transform and its inverse Fourier transform. However, considering the combined effects of sampling and noise, a realistic imaging model may be expressed as follows:

[0065]

number

[0066] Here, s and n represent the sampling model and the noise model, respectively, and the image 広視野 This represents the final image collected by the wide-field microscope. This imaging model may be considered to be closer to the state in the actual physical world, as shown in equation (2), and may differ from that of the Wiener model.

[0067]

number

[0068] The minimalist form of Wiener deconvolution with aberration-free PSF (unbiased estimation) is given by the following image: 広視野 It may be used to process [something].

[0069]

number

[0070] The image is the corresponding result of equation (4). 逆畳込 It may be clearly separated from the actual object.

[0071] The lemma described above may suggest that the distance between the real object and the resulting reconstruction is primarily caused by the combined effect of the sampling rate and mixed noise. According to this lemma, if the same real object is imaged and variables are controlled to statistically capture independent image pairs, then, with a suitable reconstruction model, such a distance between the reconstruction and the object may be amplified by the difference between the reconstructions from the image pairs.

[0072] The theorem may also be explained as follows: Given two wide-field images taken separately of the same sample (or object), these two images may represent the same object. The greater the distance remaining between the two corresponding reconstruction results, the more the reconstruction may appear to deviate from the actual object (see Figure 15).

[0073] This distance (e.g., Wiener result 1 vs. Wiener result 2) may also relate to the reconstruction error (e.g., the difference between Wiener result 1 and Wiener result 2 vs. the real object). In some embodiments, the distance may be expressed as follows:

[0074]

number

[0075] Here, U represents the union operation, ||Wiener result 1, Wiener result 2|| p represents the distance between Wiener result 1 and Wiener result 2. Wiener result 1 and Wiener result 2 may each relate to the results of reconstructions generated based on the Wiener model. n1, n2, s1, and s2 represent two separate noise models (n1 and n2) and sampling models (s1 and s2), respectively. In some embodiments, equation (5) may be modified by taking the Fourier transform of both sides.

[0076]

number

[0077] In some embodiments, the following two direct combinations of distances from the real object to “Wiener result 1 and Wiener result 2” (i.e., combinations using addition of Euclidean distances or multiplication of cross-correlation distances instead of union) may not be exact compared to the union operation in equation (6). In some embodiments, the real model of the union operation in the real world may be very complex, and therefore, the combination of two distances (reconstruction and real object) may be difficult to express explicitly. In some embodiments, the following two concise examples may be used to illustrate the relationship between the distances of two individual reconstructions and the real errors of the reconstructions.

[0078] The first example concerns the Euclidean distance. The Euclidean distance may be used to define the distance between two reconstructions. Assuming that the image is acquired at an infinite sampling rate and is corrupted only by additive noise denoted as n, an addition operation may be used to combine the distances.

[0079]

number

[0080] Here, ||Wiener result 1, Wiener result 2||2 represents the Euclidean distance between Wiener result 1 and Wiener result 2. n1 and n2 represent the separate additive noise in the two observations. In some embodiments, equation (7) may be simplified to obtain the following equation.

[0081]

number

[0082] The Fourier transform and PSF of the microscope may be omitted to simplify equation (8) as follows:

[0083]

number

[0084] Equation (9) shows the close relationship between the Euclidean distances of Wiener result 1 and Wiener result 2, the Euclidean distance between Wiener result 1 and the real object, and the Euclidean distance between Wiener result 2 and the real object.

[0085] The second example concerns cross-correlation distance. Alternatively, when using cross-correlation as a distance, the addition operation in Euclidean distance calculation needs to be changed to multiplication.

[0086]

number

[0087] Here,

number

[0088]

number

[0089] Here, O, n1, and n2 represent I(ω)×o(ω), n1 / h(ω,0), and n2 / h(ω,0), respectively. In some embodiments, equation (11) may be further simplified by removing the Fourier transform.

[0090]

number

[0091] Similar to the Euclidean distance, equation (12) shows a close relationship between cross-correlations (e.g., the cross-correlation between Wiener result 1 and Wiener result 2) and the multiplication of cross-correlations (e.g., the cross-correlation between Wiener result 1 and the real object, the cross-correlation between Wiener result 2 and the real object, etc.).

[0092] As described above, Euclidean distance and / or cross-correlation may be used to estimate the distance between the real object and the reconstruction. In some embodiments, one of Euclidean distance and / or cross-correlation may be selected to quantitatively map such distances in multiple dimensions (2D image or 3D volume). In some embodiments, a distance map may be generated to explain the distances. Conventional spatial processing algorithms, such as root mean square error in RSM, may generally be sensitive to intensity and small displacements during measurement. In addition, many processing algorithms may be used to generate "absolute differences," which may lead to misleadingly high false negative results and overwhelm true negatives in the distance map.

[0093] Based on the above considerations, in some embodiments, the algorithm may be introduced to measure the distance between two signals in the Fourier domain, i.e., the Fouriering correlation (FRC) or spectral signal-to-noise ratio (SSNR), which may describe the highest acceptable frequency component between the two signals. In some embodiments, the FRC metric may be used as an effective resolution criterion for super-resolution fluorescence microscopy and electron microscopy. In some embodiments, the FRC metric may be used to quantify the similarity (or distance) between two signals. The FRC metric may be insensitive to intensity changes and minute shifts, may quantify “relative errors” or “striance-based errors” (e.g., the most reliable frequency component), and may significantly reduce false negatives induced by quantifying “absolute distance.” It may also be assumed that the aberrations of the system (i.e., the system used to generate the signal of the object) do not change during two such independent observations. Errors due to aberrations (i.e., biased estimates) may be difficult to estimate using simple spatial or frequency algorithms. These errors may be visible because the FRC defines the most reliable frequency component. In some embodiments, FRC may be used to quantify the distance between two signals. A more detailed description of FRC can be found elsewhere in this disclosure (e.g., operation 508 in Figure 5, operation in Figures 6A–6D, and their descriptions).

[0094] In some embodiments, when considering the FRC as a global similarity estimation between two images, the FRC metric may be extended to a rolling FRC (rFRC) map to provide a pixel-level local distance measurement. In some embodiments, even without ground truth, the error of the multi-dimensional reconstruction signal may be quantitatively mapped. In some embodiments, if the same region of both images is blank (however, the ground truth has content), such information may be lost in the same region of the images and may lead to a small FRC value. In other words, if two reconstruction signals lose the same component (even if this situation is rare), the rFRC may introduce false positives. Therefore, in some embodiments, to mitigate such incompleteness of the rFRC map, a modified resolution scale error map (RSM) may be introduced as an additional error map.

[0095] In some embodiments, the RSM may be constructed based on one or more of three hypotheses.

[0096] (i) Since the RSM needs to refer to the corresponding diffraction-limited wide-field image, the wide-field image needs to have a sufficiently high SNR. Otherwise, the noise contained in the wide-field image may induce a relatively high false negative result.

[0097] (ii) The conversion from the super-resolution (SR) scale to the low-resolution (LR) scale may be approximated as a spatially invariant global Gaussian kernel convolution (across the entire field of view). If the conversion from the super-resolution image to the low-resolution image is not limited to such a spatially invariant Gaussian model, it may generate relatively high false negatives.

[0098] (iii) The irradiation intensity and the emitted fluorescence intensity need to be uniform. The generated RSM may be weighted by the irradiation intensity and the emitted fluorescence intensity of the corresponding label.

[0099] [[ID=十九]] According to the above hypothesis, the RSM itself may introduce false negatives (i.e., model errors) into the estimated error map. In a single-molecule localization microscopy (SMLM) configuration, hypotheses (i) and (ii) may be satisfied. In the first hypothesis, the SMLM wide-field image may be created by averaging thousands of flickering frames, which may remove noise and produce a reference wide-field image with a relatively high SNR. In the second hypothesis, if the imaging field is relatively small, it may usually be treated as being uniformly lit. Furthermore, since the error estimation of such an RSM is calculated on a low-resolution scale, the RSM may be able to detect errors on a low-resolution scale (usually large error components), such as misrepresentation or disappearance of structures. Errors on a super-resolution scale estimated by the RSM (small error components) may generally be false negatives. To reduce such potential false negatives, the normalized RSM may be segmented (e.g., set to 0 if less than 0.5) to remove the corresponding small error components and leave the large components to visualize the lost information of reconstructions that may be ignored by the rFRC map. Overall, the rFRC map and the segmented normalized RSM (for example, in the green and red channels, respectively) may be integrated to generate a complete PANEL map (see Figure 16), thereby accurately and comprehensively mapping and visualizing the errors of the reconstructed results. A more detailed explanation of the RSM determination may be found elsewhere in this disclosure (e.g., operation 518 in Figure 5, operation in Figure 7, and their descriptions). A more detailed explanation of the rFRC map determination may be found elsewhere in this disclosure (e.g., operations 502-516 in Figure 5, operation in Figure 8, and their descriptions). A more detailed explanation of the PANEL map determination may be found elsewhere in this disclosure (e.g., operation 520 in Figure 5 and its description).

[0100] In 502, the processing unit 140 (e.g., acquisition module 402) may acquire a first image and a second image associated with the same object. In some embodiments, the processing unit 140 may acquire the first image and / or the second image from one or more components of the image processing system 100. For example, the image acquisition device 110 may collect and / or generate the first image and / or the second image and store the images in the storage device 150 or any other storage associated with the image processing system 100. The processing unit 140 may retrieve and / or acquire the image from the storage device 150 or any other storage. As another example, the processing unit 140 may acquire the image directly from the image acquisition device 110. In some embodiments, the processing unit 140 may acquire one or more initial images, process the initial images, and / or generate the first image and / or the second image. In this disclosure, “image,” “image frame,” and “frame” are used synonymously. In some embodiments, the first image and the second image may be 2D images or 3D images. The explanation regarding the case where the images (e.g., Image 1, Image 2) are 2D images is provided solely for illustrative purposes and is not intended to limit the scope of this disclosure.

[0101] The object may include non-biological objects or biological objects. Exemplary non-biological objects may include phantoms, industrial materials, etc., or any combination thereof. Exemplary biological objects may include biological structures, biological issues, proteins, cells, microorganisms, etc., or any combination thereof. In some embodiments, the object may be fluorescent or fluorescently labeled. Fluorescent or fluorescently labeled objects may be excited to emit fluorescence for imaging.

[0102] In some embodiments, the first and second images may be generated from physical imaging. In some embodiments, the first and second images may be two consecutive images (e.g., frame 1 and frame 2 shown in Figure 9A) obtained from two image acquisitions of the same object by the image acquisition device 110. In some embodiments, the first and second images may be obtained from a single image acquisition of the same object by the image acquisition device 110. For example, the first and second images may be obtained by decomposing an initial image acquired by the image acquisition device 110. As shown in Figure 12A, frame 1 represents the initial image and may be decomposed into two or more images (e.g., first image, second image, third image, and fourth image, etc.). A more detailed explanation of generating a target map (e.g., a third target map) based on a single image may be found elsewhere in this disclosure (e.g., Figure 8 and its description). In some embodiments, the first and / or second images may have a relatively high resolution (e.g., super-resolution).

[0103] In some embodiments, the first and second images may be obtained from two initial images based on a neural network. The processing unit 140 may obtain the first and second initial images associated with the same object from one or more components of the image processing system 100 (e.g., an image acquisition device 110, a storage device 150). In some embodiments, the first and / or second initial images may have relatively low resolution, or the first and / or second images may have relatively high resolution. In some embodiments, the processing unit 140 (e.g., a neural network processing module 418) may generate the first and second images by processing the first and second initial images, respectively, using a neural network (or artificial neural network (ANN)). Exemplary neural networks may include constitutive neural networks (CNNs), deep learning networks, support vector machines (SVMs), k-nearest neighbors (KNNs), backpropagation neural networks (BPNNs), deep neural networks (DNNs), Bayesian neural networks (BNNs), etc., or any combination thereof. In some embodiments, the processing unit 140 may generate the first and second images by processing the first and second initial images, respectively, based on a neural network model. For example, the first and / or second initial images may be input to a neural network model. The neural network model may output the first and / or second images. The first and second images may be two frame images used to generate a first target map (e.g., an rFRC map), as shown in 510. In some embodiments, the rFRC map may reflect errors (e.g., at the pixel level) generated during processing using the neural network (e.g., reconstruction). As shown in Figure 13, observer 1 and observer 2 represent the first and second initial images, respectively, and r1 and r2 represent the first and second images output by the neural network, respectively.

[0104] In some embodiments, the first and / or second image may be generated from a single initial image using a neural network. In some embodiments, the processing unit 140 (e.g., a neural network processing module 418) may generate the first and second images by processing a single initial image using a neural network. The neural network reconstruction technique may differ from the physical imaging reconstruction technique. For example, adjacent pixels in an image output from a neural network (e.g., a DNN) may be considered noise-independent. Therefore, the determination of an rFRC map based on a single frame used in physical imaging may not be directly applicable to such neural network reconstruction. Different single-frame strategies may be used to determine the rFRC map of a neural network (e.g., a DNN). In some embodiments, noise (e.g., Gaussian noise) may be added to the initial image. Exemplary noises may include Gaussian noise, impulsive noise, Rayleigh noise, gamma noise, exponential noise, uniform noise, etc., or any combination thereof. For example, a single initial image may be acquired as the first initial image, and the second initial image may be generated by adding noise (e.g., noise levels of 0.5%, 1.5%, 2%, etc.) to the first initial image. Thus, the two images (i.e., the first and second initial images) may be used as input to a neural network. After reconstructing these two input images (i.e., the first and second initial images) using the neural network, the two resulting output images may be acquired as the first and second images. The two output images may be further used to determine the rFRC map. In some embodiments, two types of noise may be added to the initial image to acquire two different images. For example, the first initial image (see the image represented by "Observer + n1" in Figure 13) may be generated by adding a first type of noise to the initial image, and the second initial image (see the image represented by "Observer + n2" in Figure 13) may be generated by adding a second type of noise to the initial image.In some embodiments, the type of first noise may be the same as the type of second noise, but the magnitudes of the noises may be different. In some embodiments, the type of first noise may be different from the type of second noise, but the magnitudes of the noises may be the same. In some embodiments, the types of first and second noise may be different, and the magnitudes of the first and second noises may be different. Thus, two images (i.e., a first initial image and a second initial image) may be used as input to a neural network. After reconstructing these two input images (i.e., a first initial image and a second initial image) using the neural network, the two resulting output images may be obtained as the first image and the second image. The two output images may be further used to determine the rFRC map. In some embodiments, the magnitude of the noise added to the initial images may be determined on a case-by-case basis, and this specification is not intended to limit the scope of the disclosure.

[0105] In 504, the processing unit 140 (for example, the expansion module 404) may expand the first image and / or the second image.

[0106] In some embodiments, the processing unit 140 may expand the first and / or second images by performing a padding operation around the first and / or second images, respectively. Performing a padding operation around the image means that data may be padded into one or more regions adjacent to the edges of the image. In some embodiments, the padded regions may have one or more predetermined sizes. For example, the padded regions may have a width that is half the width of a block (e.g., the block described in 506). In some embodiments, the padded data may have one or more predetermined values, e.g., 0, 1, 2, or any other value. In some embodiments, the padded data may be determined based on the pixel values ​​of the image being expanded. In some embodiments, the data may be padded symmetrically around the image (see Figure 9A).

[0107] In step 506, the processing unit 140 (for example, the rolling module 406) may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image.

[0108] In some embodiments, the processing unit 140 may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image by performing a rolling operation on the first image and the second image, respectively. In some embodiments, the rolling operation may be performed by sliding a window over the image (e.g., the (expanded) first image and the (expanded) second image). In the rolling operation, the window may slide row by row and / or column by column over the image, and a designated center of the window may traverse one or more pixels (e.g., each pixel) of the image. In some embodiments, the window may have a predetermined size (e.g., 32×32, 64×64, 128×128, etc.). The predetermined size of the window may be determined by the image processing system 100 or may be preset by a user or operator via the terminal 130. In some embodiments, the size of the window may be associated with the size of the first image and / or the second image. For example, the window may have a size smaller than the size of the first image and / or the second image. The window may have a variety of shapes. For example, the window may be square, rectangular, circular, elliptical, or the like. In some embodiments, the window used for the first image may be the same as the window used for the second image.

[0109] In some embodiments, a window may be slid pixel by pixel on the (expanded) first image to generate a plurality of first blocks of the first image. As used herein, a block of an image may mean a portion of an image. When the window is slid on the image (e.g., the (expanded) first image), a portion of the image (e.g., the (expanded) first image) may be enclosed by the window, and the portion of the image (e.g., the (expanded) first image) enclosed by the window may be designated as a block of the image (e.g., the first block of the first image). Thus, a plurality of first blocks of the first image may be determined by sliding the window on the (expanded) first image. Similarly, a plurality of second blocks of the second image may be determined by sliding the window on the (expanded) second image. In some embodiments, a plurality of second blocks and a plurality of first blocks may correspond one to one. In some embodiments, the number of first blocks (or second blocks) may be the same as the number of pixels in the first image (or second image) acquired in 502.

[0110] While rFRC is an error map that enables evaluation at the pixel level, the finest scale of detectable error can only reach the highest resolution of the corresponding images (e.g., Image 1 and Image 2). Therefore, if the images (e.g., Image 1 and Image 2) satisfy the Nyquist-Shannon sampling theorem, pixel-by-pixel rolling may not be necessary. In some embodiments, the minimum error may be greater than ~3 × 3 pixels. This allows 2 to 4 pixels to be skipped for each rolling operation in order to obtain a 4x to 16x acceleration for the rFRC mapping operation. In some embodiments, a plurality of first blocks (or a plurality of second blocks) may be determined by sliding a window on the (extended) first image (or (extended) second image) and skipping a predetermined number of pixels on the (extended) first image (or (extended) second image). For example, when the window slides to the position of the (expanded) first image (or (expanded) second image), the first block of the first image (or the second block of the second image) may be determined. The window may then skip a predetermined number of pixels on the (expanded) first image (or (expanded) second image) and slide to the next position of the (expanded) first image (or (expanded) second image) to determine another first block of the first image (or another second block of the second image). Thus, the number of first blocks (or second blocks) may be less than the number of pixels in the first image (or second image) acquired in 502. The predetermined number of pixels may be determined by the image processing system 100 or may be preset by the user or operator via the terminal 130. In some embodiments, the predetermined number of pixels may be associated with the size of the window. For example, the predetermined number of pixels (e.g., 2, 3, or 4) may be close to the width of the window (e.g., 3). In some embodiments, the predetermined number of pixels may be random.

[0111] In 508, the processing unit 140 (for example, the map generation module 408) may determine a plurality of first characteristic values ​​based on a plurality of first blocks and a plurality of second blocks.

[0112] The first characteristic value may refer to a value associated with a pixel value of a first target map (i.e., an rFRC map). For example, the first characteristic value corresponding to a pixel of the first target map may be associated with the correlation between a first block and a second block, where the central pixel corresponds to a pixel of the first target map. In some embodiments, at least one of the first characteristic values ​​may be determined based on the correlation between one of a plurality of first blocks and the corresponding second block. The correlation between the first block and the corresponding second block may be represented by a correlation value (or referred to as an FRC value). In some embodiments, the correlation between the first block and the corresponding second block may be determined by performing a Fourier transform on the first block and the second block, respectively. For example, the processing unit 140 may determine a first intermediate image in the frequency domain based on the first block (for example, by performing a Fourier transform on the first block). The processing unit 140 may determine a second intermediate image in the frequency domain based on the second block (for example, by performing a Fourier transform on the second block). In some embodiments, the processing unit 140 may determine a target frequency (also called a cutoff frequency) based on a first intermediate image and a second intermediate image. The processing unit 140 may determine a first characteristic value based on the target frequency. For example, the processing unit 140 may determine the target frequency, the reciprocal of the target frequency, or the product of the reciprocal of the target frequency and the pixel size of the first image (and / or second image) (e.g., the physical dimensions of the camera pixels divided by the magnification of the images (first image, second image, etc.)) as the first characteristic value. A more detailed explanation of determining multiple first characteristic values ​​may be found elsewhere in this disclosure (e.g., Figures 6A to 6D).

[0113] In some embodiments, background regions may be present in the first and / or second images. In some embodiments, the information of the background regions may remain unchanged or change slightly in the first and second images. In some embodiments, FRC calculation for the background regions may be avoided to reduce the computational burden. FRC calculation may refer to the operation of determining the FRC value. In this case, the first characteristic value for the background region may be determined based on one or more predetermined values. In some embodiments, for the first block and the corresponding second block, the processing unit 140 may decide whether or not to perform FRC calculation (or whether the first block and / or the corresponding second block are background regions) before determining the first characteristic value. A more detailed explanation of determining the first characteristic value may be found elsewhere in this disclosure (e.g., Figure 6A).

[0114] In 510, the processing unit 140 (for example, the map generation module 408) may generate a first target map associated with the first image and the second image based on a plurality of first characteristic values.

[0115] As used herein, the first target map (e.g., rFRC map) may refer to an image showing the reconstruction errors and / or artifacts of the first and / or second images acquired by 502. The first target map may include multiple pixels, the value of which may be determined based on a first characteristic value. The first target map may have the same size as the first and / or second images. In some embodiments, the pixel values ​​determined based on the first characteristic value may be assigned to pixels (of the target map) corresponding to the central pixels of the first or second block (used to determine the first characteristic value). That is, pixels of the first target map may correspond to pixels of the first and / or second images.

[0116] In some embodiments, the pixels of the first target map correspond one-to-one with the pixels of the first image (and / or second image), so the first target map may reflect the image quality of the image (e.g., the first image and / or second image) at the pixel level. In some embodiments, if the pixel value of the pixels of the first target map is determined as the reciprocal of the target frequency, i.e., the resolution, then a smaller pixel value (i.e., resolution) results in higher pixel precision in the first and / or second image, and a larger pixel value (i.e., resolution) results in lower pixel precision in the first and / or second image. Relatively low pixel precision here may indicate errors. In some embodiments, if the pixel value of the pixels of the first target map is determined as the target frequency, i.e., the spatial frequency, then a smaller pixel value (i.e., spatial frequency) results in lower pixel precision in the first and / or second image, and a larger pixel value (i.e., spatial frequency) results in higher pixel precision in the first and / or second image. In some embodiments, when the pixel value of a pixel in the first target map is determined as the product of the reciprocal of the target frequency and the pixel size of the first and / or second image, a smaller pixel value results in higher pixel precision in the first and / or second image, while a larger pixel value results in lower pixel precision in the first and / or second image.

[0117] In some embodiments, the first target map may quantitatively present the resolution of each pixel in the first and / or second image, thereby indicating errors (and / or artifacts) present in the first and / or second image. Thus, using a target map (e.g., a first target map), errors (and / or artifacts) may be visualized more intuitively. As a mere example, if the first and / or second image has a resolution of 100 nm and the resolution of a pixel is determined to be 10 nm, it can be shown that the pixel is accurate and there are no errors in the pixel reconstruction. If the first and / or second image has a resolution of 100 nm and the resolution of a pixel is determined to be 200 nm, it can be shown that the pixel is inaccurate and there are errors in the pixel reconstruction.

[0118] In some embodiments, the first target map may be represented in the form of a grayscale image. In some embodiments, the grayscale value of each pixel of the first target map may have a positive correlation with the first characteristic value, and the degree of grayscale of the first target map may reflect the image quality (or error) of the first image and / or the second image.

[0119] In some embodiments, when the skip operation is executed at 506, the first target map may be resized (e.g., by bilinear interpolation) to the original image size of the first image and / or the second image for better visualization.

[0120] At 512, the processing device 140 (e.g., the filtering module 410) may filter the first target map. In some embodiments, the processing device 14 is0 may filter the first target map using an adaptive median filter.

[0121] In some embodiments, according to the Nyquist sampling theorem, the minimum error in the first and / or second image in which the behavior of the present disclosure can be detected may be greater than, for example, 3x3 pixels. Therefore, rolling behavior in a small region (e.g., 3x3 pixels) may change smoothly without any isolated pixels (e.g., pixels with extremely high or extremely low values ​​compared to neighboring pixels). However, due to inappropriate determination of the target frequency, the FRC calculation may become unstable and abnormal characteristic values ​​may be determined. In some embodiments, an adaptive median filter may be used to remove these inappropriate characteristic values ​​in these pixels. In some embodiments, the adaptive median filter may replace each pixel with the median of the adjacent pixel values ​​in the filtering window of the first target map. In some embodiments, a threshold may be used in the adaptive median filter. For example, if the pixel value of a pixel (e.g., grayscale value) is greater than the product of the threshold and the median of the adjacent pixel values ​​in the filtering window, the pixel value may be replaced with the median; otherwise, the filtering window may move to the next pixel. By using an adaptive median filter, isolated pixels may be filtered without blurring the rFRC map. In some embodiments, the threshold (e.g., 2) may be determined by the image processing system 100 or may be preset by a user or operator via the terminal 130.

[0122] In 514, the processing unit 140 (for example, the image quality determination module 414) may determine the global image quality metrics for the first image and / or the second image based on the first target map.

[0123] The rFRC map generated in 510 may be used to quantitatively visualize the errors contained in the images (e.g., the first and second images). In addition to such local quality assessments, a global image quality metric for the first and / or second images may be determined. In some embodiments, the global image quality metric may have a dimension as resolution. In some embodiments, the global image quality metric may be normalized without considering the dimension.

[0124] In some embodiments, a global image quality metric with resolution as the dimension may be expressed as follows:

[0125]

number

[0126] Here, ||FV||0 is the l0 norm representing the number of non-zero elements in the rFRC map, and FV represents the rFRC map.

[0127] In some embodiments, a global image quality metric normalized without regard to dimensions may be expressed as follows:

[0128]

number

[0129] Here, ||FV||0 is the l0 norm representing the number of non-zero elements in the rFRC map, and FV represents the rFRC map.

[0130] Furthermore, these two metrics shown in equations (13) and (14) may both be extended to three dimensions, and the two-dimensional coordinates (x,y) in equations (13) and (14) may be directly converted to three-dimensional coordinates (x,y,z).

[0131] In 516, the processing unit 140 (for example, the display module 412) may display the first target map using a displacement jet color map.

[0132] In some embodiments, the first target map may be displayed using one or more color maps. The color maps may include one or more existing common color maps, such as jet color maps. A jet color map may use colors from blue to red that index different magnitudes of error to highlight all error areas. In some embodiments, such a jet color map may be contrary in some way to human intuition or habit. The human visual system may be intended to define black (dark colors) as small grades and light (pale colors) as large grades, which is identical to the logic of a grayscale color map. In some embodiments, human vision may be less sensitive to light or dark levels of grayscale and more sensitive to different colors. Therefore, a color map that includes a color index to better suit human intuition (e.g., a displacement jet color map) may be used.

[0133] In some embodiments, the jet colormap (see Figure 14A) may be displaced to create a displacement jet (sJet) colormap (see Figure 14B) in which the red component at relatively low grades and the green component at relatively high grades are removed, and the blue component is extended to the entire color index. In some embodiments, considering that human intuition may be more sensitive to green, green may be moved as a relatively high grade to emphasize large errors. In some embodiments, humans may be accustomed to assigning lighter colors (e.g., white) as relatively high grades and darker colors (e.g., black) as relatively low grades, and accordingly, black zones (0,0,0) and white zones (1,1,1) may be involved in the displacement jet (sJet) colormap. In some embodiments, with the extension of the blue component, the white zone may be created to represent the maximum error. In some embodiments, the background of the rFRC map may mean positive (i.e., no error), and accordingly, black zones (rather than blue in the jet colormap) may be used to display such a background, which is more appropriate to human intuition. As shown in Figures 14C to 14D, the sJet colormap (see Figure 14D) is clearly more intuitive for visualizing the error map compared to the original jet colormap (see Figure 14C).

[0134] In step 518, the processing unit 140 (for example, the map generation module 408) may determine a reference map associated with the first image or the second image.

[0135] In some embodiments, the reference map may include a modified resolution-scaled error map (RSM). The reference map may reflect errors and / or artifacts in the first and / or second image. In some embodiments, the reference map may be in the form of a grayscale map.

[0136] In some embodiments, the reference map may be determined based on the difference between an intermediate image and a reference image (e.g., a relatively low-resolution image). The intermediate image may be determined based on a first image (or second image) and a convolution kernel. The convolution kernel may be determined based on a first image (or second image) and a reference image. A more detailed explanation of how the reference map is determined may be found elsewhere in this disclosure (e.g., Figure 7 and its description).

[0137] In some embodiments, if the first image (or second image) is three-dimensional or under a non-Gaussian convolution relationship (e.g., between a low-resolution scale and a high-resolution scale), the reference map may not be determined. That is, operation 518 may be omitted. As a result, for such a first or second image, the RSM may not be involved in the fourth target map, and operation 520 may also be omitted.

[0138] In step 520, the processing unit 140 (for example, the map generation module 408) may determine the fourth target map by fusing the first target map and the reference map.

[0139] In some embodiments, a first target map (e.g., an rFRC map) may illustrate minute and subtle errors in the reconstruction of the first image (and / or second image). In some embodiments, the first target map may be impossible if the structure is simultaneously missing in both the first and second images. As just one example, a 2D SMLM dataset may be used to account for this possible false negative (see Figures 17A–17C). As indicated by the bright arrow in Figure 17A, a small portion of the filament in the same region of the two reconstructions may be artificially removed. As shown in Figure 17B, the rFRC map may successfully detect most of the error components in the image, except for the artificially removed region. In some embodiments, such loss of structure may be detected by a reference map (e.g., an RSM (see Figure 17C)). In some embodiments, the reference map (e.g., an RSM) may be introduced as an additional error map for the first target map (e.g., an rFRC map).

[0140] The fourth target map may reflect the overall error and / or artifacts of the first image (and / or second image) as shown by the first target map and the reference map. The fourth target map may also be called a PANEL map.

[0141] In some embodiments, the processing unit 140 may process the first target map and / or reference map in the form of grayscale maps. In some embodiments, the processing unit 140 may convert the first target map and reference map into RGB images. For example, the first target map may be displayed in green (as shown in the left panel of Figure 11A), and the reference map may be displayed in red (as shown in the right panel of Figure 11A). In some embodiments, the processing unit 140 may normalize the first target map and / or reference map by, for example, variance normalization, mean normalization, etc. In some embodiments, the processing unit 140 may determine the fourth target map based on a weighted sum of the normalized first target map and reference map.

[0142] In some embodiments, the processing unit 140 may perform a thresholding operation on the first target map and / or reference map to filter out background areas therein, thereby highlighting important information (e.g., critical errors in the reconstruction of the first and / or second images) (see Figure 11B). In some embodiments, the threshold of the first target map may be the same as or different from the threshold of the reference map. In some embodiments, the thresholds of the first target map and / or reference map may be in the range of 0 to 1.

[0143] In some embodiments, the reference map threshold may be determined by the image processing system 100 or preset by a user or operator via terminal 130. For example, the reference map threshold may be 0.5. In some embodiments, identical lost information in both the first and second images (e.g., misrepresentation or disappearance of structure) may have a relatively high magnitude and reside in a relatively large area of ​​the reference map. In some embodiments, low-magnitude components included in the reference map may induce a relatively high false negative. In some embodiments, the reference map may be segmented based on a threshold by the following formula before involving the reference map in the fourth target map.

[0144]

number

[0145] Here, R(x,y) represents the normalized RSM value at coordinate (x,y), and R% represents the segmented RSM. In some embodiments, low false negatives (e.g., less than 0.5) may be filtered according to equation (15) to focus on true negatives detected by the reference map, leaving only relatively strong, low-resolution errors.

[0146] In some embodiments, the rFRC map may indicate the degree of error, and the smallest FRC value in the map may not indicate the tolerance. In some embodiments, the threshold of the first target map may be determined by Otsu's binarization method (also known as Otsu for short), which is used to perform automatic image thresholding. Using Otsu's method, the threshold may be determined automatically by maximizing the interclass variance. Image thresholding may be performed to filter the background of the rFRC map, thereby highlighting the decisive errors in the reconstruction of the first and second images. An exemplary threshold of the first target map may be 0.4. Pixels whose pixel values ​​are less than two thresholds may be filtered from the first target map and the reference map, respectively. For example, the pixel value of such pixels may be specified as 0.

[0147] In some embodiments, the processing apparatus 140 may determine a fourth target map (e.g., a panel map) by fusing a first target map and a reference map to comprehensively represent the reconstruction errors and / or artifacts of the first and / or second images (see Figure 11C). In some embodiments, the fourth target map may be determined by fusing a third target map and a reference map determined in process 800.

[0148] The above description of process 500 is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Those skilled in the art, having understood the operating principle, will understand, without departing from the principle, that any combination of operations, any addition or deletion of operations, or application of the operating principle to other image processing processes may be made. In some embodiments, one or more operations in process 500 may be omitted. For example, operations 504, 512, 514, 516, 518, and / or 520 may be omitted. In some embodiments, two or more operations of process 500 may be combined into a single operation. For example, operations 506-510 may be combined into a single operation. As another example, operations 518-520 may be combined into a single operation. As yet another example, operations 506-510 and operations 518-520 may be combined into a single operation.

[0149] In some embodiments, the first and second images may be 3D images. The procedure for processing 3D images may be the same as the procedure for processing 2D images. The rolling operation in 506 may be operable in three dimensions. For example, the processing unit 140 may perform the rolling operation in a manner such as performing a 3D filter. Since the window used for the three-dimensional rolling operation may be cubic, the first and second blocks may be cubic. In some embodiments, the FRC operation may be extended to a 3D version called Fourier Shell Correlation (FSC). In this case, the first and second ring images (see Figure 6D) generated from the first and second blocks may be spherical. In some embodiments, to avoid a relatively heavy computational burden, the three-dimensional rFRC operation may be generated directly by the 2D rFRC operation for each plane.

[0150] Figure 6A is a flowchart illustrating an exemplary process for determining whether to perform an FRC calculation on a first or second block, according to some embodiments of the present disclosure. In some embodiments, process 600 may be performed by an image processing system 100. For example, process 600 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules shown in Figure 4) may execute an instruction set and be configured to execute process 600 accordingly. The operation of the illustrated process presented below is intended to be illustrative. In some embodiments, process 600 may be achieved with one or more additional operations not described, and / or without one or more operations described. Furthermore, the order of operations of process 600, as shown in Figure 6A and described below, is not intended to be limiting. In some embodiments, the operation 508 shown in Figure 5 may be performed in accordance with one or more operations of process 600.

[0151] As described above, FRC may measure global similarity between two images (e.g., Image 1 and Image 2), and the FRC values ​​may be extended to the form of a rolling FRC (rFRC) map to provide local distance measurements at the pixel level. In some embodiments, the form of the FRC calculation may be considered as a filter in which the image is processed block by block, and each block is assigned a corresponding FRC value. In some embodiments, as shown in 504, the images (e.g., Image 1 and / or Image 2) may be symmetrically padded by half the size of the block to facilitate the FRC calculation on the boundaries of the image. In some embodiments, a background threshold may be set for the central pixels of a block (e.g., 1x1 to 3x3 pixels) to determine whether or not to determine an FRC value for this block, thereby avoiding the FRC calculation on the background region. In some embodiments, if the average of the central pixels of a block is greater than the background threshold, the FRC calculation may be performed and an FRC value may be assigned to the central pixels of the block. In some embodiments, if the average is less than the background threshold, a zero value may be set for the central pixels of the block. This procedure block may be executed block by block until the entire image has been processed.

[0152] In 601, the processing unit 140 (for example, the map generation module 408) may determine the average pixel value of pixels in the central region of each first block, and / or the average pixel value of pixels in the central region of each second block. In some embodiments, the central region may be located within or near the center of each first block or each second block. The central region may be any shape, such as a square or a circle. In some embodiments, the size of the central region may be smaller than or equal to the size of each first block or each second block.

[0153] The average value of a pixel may be the sum of the pixel values ​​divided by the total number of pixels in the central region. In some embodiments, the processing unit 140 may determine the average value of the pixels in the central region of each first block and the pixels in the central region of each second block. In some embodiments, the processing unit 140 may determine multiple average values ​​corresponding to a plurality of first blocks and / or a plurality of average values ​​corresponding to a plurality of second blocks. In some embodiments, the processing unit 140 may determine multiple average values ​​corresponding to a plurality of first blocks and a plurality of second blocks.

[0154] In 602, the processing unit 140 (for example, the map generation module 408) may compare the average value with a threshold (for example, a background threshold).

[0155] The first and / or second image may have background regions. In some embodiments, the background regions may have noise (e.g., background noise, readout noise). In some embodiments, such background regions may result in relatively large FRC values, which may introduce false negatives during the generation of the rFRC map, thereby hindering the emphasis on true negatives (e.g., errors generated in image reconstruction). Therefore, background thresholds may be set to avoid such FRC calculations on these background regions. In this disclosure, true negative means a true error determined by the processing unit 140, false negative means a false error determined by the processing unit 140, and false positive means that the processing unit 140 determines that an error is not an error.

[0156] In some embodiments, the threshold may be determined by the image processing system 100 or preset by a user or operator via terminal 130. In some embodiments, a fixed threshold may be used for the entire first image and / or the entire second image (i.e., a plurality of first blocks of the first image and / or a plurality of second blocks of the second image). In some embodiments, an adaptive threshold may be used. In some embodiments, the adaptive threshold may be determined according to (each) first block and / or (each) second block. In some embodiments, one or more thresholds may be adaptively determined for small local regions of the first image (or first block) or the second image (or second block). That is, different thresholds may be used for different small local regions. In some embodiments, an iterative wavelet transform may be performed to determine the (adaptive) threshold. As just one example, the local background intensity distribution of the corresponding images (e.g., the first image, the second image, the first block, and the second block, etc.) may be estimated. Specifically, the background may be iteratively estimated from the lowest frequency wavelet band of the corresponding image, and in each iteration, one or more (e.g., all) values ​​of the corresponding image that are above the current estimated background level may be clipped.

[0157] In step 603, the processing unit 140 (for example, the map generation module 408) may determine whether the average value is greater than or equal to a threshold. In some embodiments, in response to determining that the average value is greater than or equal to a threshold, process 600 may proceed to operation 605. In some embodiments, in response to determining that the average value is not greater than or equal to a threshold, process 600 may proceed to operation 604.

[0158] In some embodiments, the processing unit 140 may compare the average value for each first block, the average value for each second block, and / or the average values ​​for each first block and each second block with a threshold. In some embodiments, in response to determining that both the average value for each first block and the average value for each second block are greater than or equal to the threshold, the process may proceed to operation 605; otherwise, process 600 may proceed to 604. In some embodiments, in response to determining that the average values ​​for each first block and each second block are greater than or equal to the threshold, the process may proceed to operation 605; otherwise, process 600 may proceed to 604.

[0159] In 604, the processing unit 140 (for example, the map generation module 408) may determine characteristic values ​​(for example, first characteristic values ​​corresponding to each first block and / or each second block) based on a predetermined value. For example, the processing unit 140 may assign a predetermined value to the characteristic value. The predetermined value may be, for example, 0. Therefore, if the average value of the first block and / or the second block is not greater than a threshold, the characteristic values ​​corresponding to the first block and the second block may be 0, thereby avoiding unnecessary FRC calculations for the background region.

[0160] In 605, the processing unit 140 (for example, the map generation module 408) may determine characteristic values ​​based on each first block and each second block.

[0161] If the average value is greater than a threshold, the processing unit 140 may determine the characteristic value by performing an FRC calculation on each first block and each second block. In some embodiments, the processing unit 140 may determine the characteristic value based on a first intermediate image and a second intermediate image obtained by performing a Fourier transform on each first block and each second block. A more detailed explanation of determining the characteristic value by performing an FRC calculation may be found elsewhere in this disclosure (e.g., Figure 6B and its description).

[0162] The processing unit 140 may execute process 600 on all first and / or second blocks in order to determine multiple characteristic values.

[0163] Figure 6B is a flowchart illustrating an exemplary process for determining characteristic values ​​according to some embodiments of the present disclosure. In some embodiments, process 610 may be performed by an image processing system 100. For example, process 610 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules shown in Figure 4) may execute the instruction set and be configured to execute process 610 accordingly. The operation of the illustrated process presented below is intended to be illustrative. In some embodiments, process 610 may be achieved with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of process 610 as shown in Figure 6B and described below is not intended to be limiting. In some embodiments, the operation 605 shown in Figure 6A may be performed in accordance with one or more operations of process 610.

[0164] In 611, the processing unit 140 (for example, the map generation module 408) may determine a first intermediate image and a second intermediate image in the frequency domain, respectively, based on the first block and the second block.

[0165] In some embodiments, the processing unit 140 may determine a first intermediate image in the frequency domain by performing a Fourier transform on each first block. In some embodiments, the processing unit 140 may determine a second intermediate image in the frequency domain by performing a Fourier transform on each second block. As used herein, the first intermediate image may be considered a first transformed image (in the frequency domain) from the first block, and the second intermediate image may be considered a second transformed image (in the frequency domain) from the second block. The Fourier transform may be used to transform information in an image or block (e.g., the first and second blocks) from the spatial domain to the frequency domain. In some embodiments, the Fourier transform may be a Discrete Fourier Transform (DFT). In some embodiments, each first block may correspond to a first intermediate image, and each second block may correspond to a second intermediate image. This may result in the determination of multiple first intermediate images corresponding to multiple first blocks, and multiple second intermediate images corresponding to multiple second blocks.

[0166] In 612, the processing unit 140 (for example, the map generation module 408) may determine the target frequency based on the first intermediate image and the second intermediate image. In some embodiments, each first intermediate image and the corresponding second intermediate image may be used to determine the target frequency. This may result in multiple target frequencies being determined based on multiple first intermediate images and multiple second intermediate images.

[0167] In some embodiments, a criterion may be used to determine the target frequency in order to quantitatively determine the characteristic value. For example, when the FRC curve falls below a threshold at a certain point, the frequency corresponding to that point may be defined as the target frequency. The FRC threshold may indicate the spatial frequency position of meaningful information above the random noise level. In some embodiments, a fixed threshold or sigma factor curve (see curve b in Figure 9C) may be used to determine the threshold. For example, a fixed value of 1 / 7 of the FRC curve may be used as a hard threshold.

[0168] In some embodiments, the target frequency may be determined based on a graph (see Figure 9C) with spatial frequency on the horizontal axis and correlation value on the vertical axis. The target frequency may correspond to the intersection of a first curve (i.e., curve a in Figure 9C) and a second curve (i.e., curve b in Figure 9C). The first curve may represent a first relationship between multiple correlation values ​​and multiple frequencies based on a first and second intermediate image. The second curve may represent a second relationship between multiple correlation values ​​and multiple frequencies based on a predetermined function. A more detailed explanation of how to determine the target frequency may be found elsewhere in this disclosure (e.g., Figures 6C and 6D and their descriptions).

[0169] In 613, the processing unit 140 (e.g., the map generation module 408) may determine characteristic values ​​based on target frequencies. In some embodiments, the processing unit 140 may specify each target frequency as a characteristic value. In some embodiments, the processing unit 140 may determine characteristic values ​​based on the reciprocal of the target frequency. For example, the processing unit 140 may specify the reciprocal of the target frequency as a characteristic value. In some embodiments, the processing unit 140 may determine characteristic values ​​based on the reciprocal of the target frequency and the pixel size of the first and / or second image acquired in 502. For example, the processing unit 140 may specify the product of the reciprocal of the target frequency and the pixel size as a characteristic value. As shown in Figure 9D, FV represents a characteristic value (e.g., rFRC value) determined based on the target frequencies determined based on the first and second blocks.

[0170] Figure 6C is a flowchart illustrating an exemplary process for determining a target frequency according to some embodiments of the present disclosure. In some embodiments, process 620 may be performed by an image processing system 100. For example, process 620 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules shown in Figure 4) may execute an instruction set and be configured to execute process 620 accordingly. The operation of the illustrated process presented below is intended to be illustrative. In some embodiments, process 620 may be achieved with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of process 620 as shown in Figure 6C and described below is not intended to be limiting. In some embodiments, the operation 612 shown in Figure 6B may be performed in accordance with one or more operations of process 620.

[0171] In 621, the processing unit 140 (for example, the map generation module 408) may determine a first relationship between a plurality of correlation values ​​and a plurality of frequencies based on the first intermediate image and the second intermediate image.

[0172] In some embodiments, the processing unit 140 may determine a plurality of correlation values ​​between each first intermediate image and a corresponding second intermediate image. In some embodiments, the correlation values ​​may be determined based on a first ring image associated with the first intermediate image and a corresponding second ring image associated with the second intermediate image. Thus, the plurality of correlation values ​​may be determined based on a plurality of first ring images associated with the first intermediate image and a plurality of second ring images associated with the second intermediate image. A more detailed explanation of how to determine the correlation values ​​may be found elsewhere in this disclosure (e.g., Figure 6D and its description).

[0173] In some embodiments, each of the correlation values may correspond to one of a plurality of frequencies. The correlation value determined based on the first ring image and the corresponding second ring image may correspond to the frequency of the first ring image and / or the second ring image. In some embodiments, the first curve (also referred to as the FRC curve) may indicate a first relationship between a plurality of correlation values and a plurality of frequencies. For example, the first curve may be represented by curve a in the graph shown in FIG. 9C.

[0174] At 622, the processing device 140 (e.g., the map generation module 408) may determine a second relationship between a plurality of correlation values and a plurality of frequencies based on a predetermined function.

[0175] In some embodiments, the second relationship may be indicated by a second curve. In some embodiments, the second curve may be a threshold curve. In some embodiments, the second relationship may be determined based on a predetermined function described as Equation (16). The threshold curve may also be referred to as a sigma factor curve or a 3σ curve, as shown by curve b in FIG. 9C.

[0176]

Number

[0177] Here, N i represents the number of pixels of the first ring image and / or the second ring image having a radius q i and σ 因子 may be 3. Since the calculation of the FRC sum includes all Hermite pairs in the Fourier domain, an extra factor of "2" may be required. If the two images (e.g., the first block, the second block) completely contain noise rather than signals, the second curve is FRC i = 1 / √N iThis is expressed as follows. Therefore, the corresponding 3σ curve means that a standard deviation of three times the expected random noise variation may be selected as the threshold for determining the target frequency. In some embodiments, the second curve represented by equation (16) may be more suitable for PENEL to identify the category (noise or not) of components in the images (e.g., first image, second image). Note that in equation (16) σ 因子 This could be any other number, such as 5, 7, or 9, and is not intended to be limiting.

[0178] In some embodiments, the threshold for determining the target frequency may be a fixed value, so the second curve may be a straight line. The threshold may be determined, for example, as 1 / 7 of the FRC curve. In some embodiments, the second curve may represent frequencies of meaningful information above the random noise level. For example, information corresponding to frequencies below the second curve may not be certain, while information corresponding to frequencies above the second curve may be certain.

[0179] In some embodiments, the 3σ curve may be suitable for small images in terms of its robustness and accuracy and may be adapted to the determination of local resolution (corresponding to the cutoff frequency). A fixed threshold may be suitable for large images and may be adapted to the determination of conventional global resolution. A more detailed explanation of the comparison between the two second curves may be found elsewhere in this disclosure (e.g., Example 11).

[0180] In 623, the processing unit 140 (for example, the map generation module 408) may determine the target frequency based on the first and second relationships. In some embodiments, the target frequency may correspond to the intersection of a first curve representing the first relationship and a second curve representing the second relationship. The target frequency may also be the frequency at the intersection. Multiple target frequencies may be determined for a plurality of first intermediate images and corresponding second intermediate images.

[0181] Figure 6D is a flowchart illustrating an exemplary process for determining a correlation value according to some embodiments of the present disclosure. In some embodiments, process 630 may be performed by an image processing system 100. For example, process 630 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules shown in Figure 4) may execute an instruction set and be configured to execute process 630 accordingly. The operation of the illustrated process presented below is intended to be illustrative. In some embodiments, process 630 may be achieved with one or more additional operations not described and / or without one or more operations described. Furthermore, the order of operations of process 630 as shown in Figure 6D and described below is not intended to be limiting. In some embodiments, the operation 621 shown in Figure 6C may be performed in accordance with one or more operations of process 630.

[0182] In 631, the processing unit 140 (for example, the map generation module 408) may determine a first ring image from the first intermediate image based on one of a plurality of frequencies.

[0183] The first ring image refers to a portion of the first intermediate image (in the frequency domain) that is ring-shaped with a diameter equal to the frequency. The first ring image may include pixels within a ring with a diameter equal to the frequency (see the light gray circle in Figure 9B). The frequency may range from 0 to half the reciprocal of the pixel size of the first or second image. The first ring image may correspond to a frequency, and accordingly, multiple first ring images may be determined based on multiple frequencies.

[0184] In 632, the processing unit 140 (for example, the map generation module 408) may determine a second ring image from the second intermediate image based on frequency. The second ring image may include pixels within a ring having a diameter equal to the frequency (see the light gray circle in Figure 9B). The second ring image may be determined in the same manner as the first ring image. The second ring image may correspond to a frequency, and accordingly, multiple second ring images may be determined based on multiple frequencies. Multiple second ring images and multiple first ring images may correspond one-to-one. A second ring image and its corresponding first ring image may correspond to the same frequency.

[0185] In 633, the processing unit 140 (for example, the map generation module 408) may determine correlation values ​​based on the first ring image and the second ring image. In some embodiments, the processing unit 140 may determine multiple correlation values ​​based on a plurality of first ring images and a plurality of second ring images.

[0186] In some embodiments, the processing unit 140 may determine the correlation value through a Fourier Ring correlation (FRC) (or so-called spectral signal-to-noise ratio) calculation. The FRC calculation may measure the statistical correlation between two images (e.g., a first block and a second block) across a series of concentric rings (i.e., ring images) in the Fourier domain. In some embodiments, FRC may be proposed as a method for measuring image resolution in super-resolution microscopy. The FRC calculation may be performed to determine the correlation value based on frequency. The correlation value (or FRC value) is the corresponding spatial frequency q i It may also be a function with variable , and may be determined based on equation (17).

[0187]

number

[0188] Here, F1 and F2 represent the first and second intermediate images after a Fourier transform (e.g., DFT) has been performed.

number

[0189] Furthermore, a Hamming window may be used in the FRC calculation to suppress edge effects and other spurious correlations that arise from the DFT calculation. An example Hamming window may be defined as follows:

[0190]

number

[0191] Here, N represents the number of elements in the mask (for example, the image size of the first or second intermediate image). The coefficients α and β may be set to 0.5.

[0192] In some embodiments, the spatial frequencies of the first and second intermediate images need to be discretized in the FRC calculation or in the determination of the first and / or second ring images in 631-632. In some embodiments, the discretization of the spatial frequencies of the FRC curve needs to be defined in order to determine the discrete values ​​of the corresponding spatial frequencies. In some embodiments, the maximum frequency f max The pixel size (p) of the first or second image is... s Half of the reciprocal of ) that is, f max = 1 / (2p) s ) may also be. In some embodiments, a non-square image (i.e., a ring image) may be padded (e.g., zero-padding) to a square image in order to compute the FRC curve. In some embodiments, the FRC curve may include N / 2 points, and the discretization step (Δf) may be obtained for the spatial frequency by equation (19).

[0193]

number

[0194] Here, N represents the number of elements in the mask (for example, the image size of the first or second intermediate image). max represents the maximum frequency, and p s This represents the pixel size.

[0195] In some embodiments, an average filter with a half-width of the average window (e.g., equal to three frequency bins) may be applied to smooth the FRC curve.

[0196] The above descriptions of processes 600, 610, 620, and 630 are provided for illustrative purposes only and are not intended to limit the scope of this disclosure. Those skilled in the art, having understood the operating principles, will understand, without departing from the principles, that any combination of operations, any addition or deletion of operations, or application of the operating principles to other image processing processes may be performed. In some embodiments, one or more operations in process 600 may be omitted. For example, operations 601 to 604 may be omitted. That is, the processing unit 140 may perform FRC calculations on each first block and / or each second block without filtering the background region. In some embodiments, process 610 may be integrated into process 600. In some embodiments, process 620 may be integrated into process 610. In some embodiments, one or more operations may be integrated into a single operation. For example, operations 631 and 632 may be integrated into a single operation.

[0197] Exemplary FRC calculation processes may be shown in Figures 9A to 9D. Figures 9A to 9D illustrate an exemplary process for determining a target map according to some embodiments of the present disclosure. Figures 9A to 9D are provided for illustrative purposes only and are not intended to limit the scope of the present disclosure. As shown in Figure 9A, the first image (frame 1) and the second image (frame 2) are 4x4 in size. Frame 1 (and frame 2) are expanded by symmetrically padding around the frame using some of the pixel values ​​of frame 1 (and frame 2). For example, the pixel value (indicated by number 2) is symmetrically padded around the left boundary of frame 1, the pixel value (indicated by number 6) is symmetrically padded around the left boundary and the top boundary of frame 1, and the pixel value (indicated by number 5) is symmetrically padded around the top boundary of frame 1. Similarly, other parts of frame 1 (and frame 2) may be symmetrically padded. To illustrate the rolling operation in relation to frame 1 in Figure 9A, the window (3x3) is represented by a solid line box. A 3x3 image enclosed in a window represents the first block. Initially, the window is positioned at the initial location of the expanded first image, and the first block has a center represented by "c". Next, the window slides to the next position of the expanded first image, and the next first block is obtained, and the center of the next first block (represented by "c") may coincide with a pixel (represented by number 2) of the first image. The window may slide over the entire expanded first image to generate multiple first blocks. The upper black solid box in Figure 9B represents the first intermediate image corresponding to the upper first block in Figure 9A, and the lower black solid box in Figure 9B represents the second intermediate image corresponding to the upper second block in Figure 9B. As shown in Figure 9B, circles with radii indicated by dotted arrows represent rings (or ring images). Multiple first ring images associated with the first intermediate image and multiple second ring images associated with the second intermediate image may be determined. Multiple correlation values ​​(also called FRC curves, as shown by curve a in Figure 9C) may be determined based on the first and second ring images. The intersection points of each FRC curve and the 3σ curve (shown by curve b in Figure 9C) may also be determined.The characteristic values ​​may be determined based on the target frequency corresponding to each intersection. The rFRC map may be generated based on multiple characteristic values ​​(see Figure 9D).

[0198] Figure 7 is a flowchart illustrating an exemplary process for determining a reference map according to some embodiments of the present disclosure. In some embodiments, process 700 may be performed by an image processing system 100. For example, process 700 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules shown in Figure 4) may execute the instruction set and be configured to execute process 700 accordingly. The operation of the illustrated process presented below is intended to be illustrative. In some embodiments, process 700 may be achieved with and / or without one or more additional operations not described. Furthermore, the order of operations of process 700 as shown in Figure 7 and described below is not intended to be limiting. In some embodiments, the operation 518 shown in Figure 5 may be performed in accordance with one or more operations of process 700.

[0199] In step 702, the processing unit 140 (for example, the map generation module 408) may acquire a reference image with a lower resolution than the first image.

[0200] In some embodiments, the processing unit 140 may acquire a reference image from one or more components of the image processing system 100. For example, the image acquisition device 110 may collect and / or generate a reference image and store it in the storage device 150. The processing unit 140 may search for and / or acquire the reference image from the storage device 150. In another example, the processing unit 140 may directly acquire the reference image from the image acquisition device 110.

[0201] In some embodiments, the reference image may be generated from the same measurement of the same object as the first image. In some embodiments, the reference image may be acquired by capturing the object using an image acquisition device 110. In some embodiments, the reference image and the first image may be reconstructed based on the same measurement signal, but using different reconstruction algorithms. The reference image may have a lower resolution than the first image. The reference image herein may be considered a low-resolution image, and the first image may be considered a high-resolution (or super-resolution) image. For example, the first image may be a super-resolution (SR) image, and the reference image may be a low-resolution (LR) image corresponding to the first image. As shown in Figure 10, the SR image represents the first image, and the LR image represents the reference image.

[0202] In some embodiments, a reference image may be used to reconstruct the first image. In some embodiments, some errors (e.g., loss of information) may be introduced in the reconstruction. In some embodiments, the reference image may be used as a ground truth for determining the errors.

[0203] In 704, the processing unit 140 (for example, the map generation module 408) may determine a convolution kernel based on the first image and the reference image.

[0204] In some embodiments, the convolution kernel may include a resolution scaling function (RSF), a point spreading function (PSF), etc. In some embodiments, the convolution kernel may be an RSF. In some embodiments, the RSF may be used to convert a high-resolution image to a low-resolution scale. In some embodiments, the RSF may be defined by a 2D Gaussian function with parameter σ. Since the RSF may typically be anisotropic in the x and y directions, σ has two elements, i.e., σ x and σ y It may be set as a vector containing . In this case, the function of RSF may be expressed as follows:

[0205]

number

[0206] To compute the RSF discretely, the 2D Gaussian function may be integrated over a finite number of pixels, so the Gaussian error function (erf) may be given as follows:

[0207]

number

[0208] In some embodiments, the discrete mathematical formula for RSF may be obtained as follows.

[0209]

number

[0210] In 706, the processing unit 140 (for example, the map generation module 408) may determine an intermediate image based on the first image and the convolution kernel.

[0211] The intermediate image may be considered an image whose resolution has been scaled back from the first image. (For example, the intermediate image I described in equation (26)) HL The intermediate image may be determined by transforming the first image using a convolution kernel. For example, the intermediate image may be determined based on the convolution of the first image with the convolution kernel.

[0212] In some embodiments, four parameters (e.g., μ, θ, σ) are used. x , σ y ) may be optimized. μ and θ may be parameters for rescaling image intensity, σ x and σ yThis may be a parameter for parameterizing the RSF. Image intensity rescaling may refer to linearly rescaling the intensity of the first image.

[0213] In some embodiments, to globally equalize the intensity between a reference image (e.g., an LR image) and a first image (e.g., a high-resolution image) and maximize the similarity between them, the first image I H The intensity of needs to be linearly rescaled by equation (23).

[0214]

number

[0215] Here, I HS Here, represents the first image after linear rescaling, μ represents the intensity parameter, θ represents the parameter associated with the background region, and I H This represents the first image. μ and θ in equation (23) are given by reference image I L And the first image I after rescaling convolved with RSF. HS It may be selected to maximize the similarity between them.

[0216]

number

[0217] Here, I L This represents a reference image, I HS Here, μ represents the first image after linear rescaling, μ represents the intensity parameter, and θ represents the parameter associated with the background region.

number

[0218] In some embodiments, μ and θ are used for rescaling image intensity, and σ is used for RSF parameterization. x and σy To estimate this, we need these four variables (i.e., μ, θ, σ) x , σ y ) may be jointly optimized to minimize the following function:

[0219]

number

[0220] In some embodiments, since calculating the gradient of equation (25) may be difficult, a derivative-free optimizer may be used to search for the four optimal parameters. Unlike particle swarm optimization (PSO), pattern search (PSM) may be used to optimize equation (25). PSO may search a relatively large space of candidate solutions, which may not be the choice required to optimize such four parameters. Compared to the unstable and slow metaheuristic optimization approach PSO, PSM may be a stable and computationally efficient method that directly searches for solutions and is commonly used for small-scale parameter optimization. In some embodiments, PSM may be better suited to determining the RSM.

[0221] In some embodiments, μ and θ are used for rescaling image intensity, and σ is used for RSF parameterization. x and σ y After optimization, the high-resolution image (i.e., the first image) I H This is achieved by convolving the estimated RSF, which is then used to obtain the low-resolution scale (i.e., intermediate image) I HL It may be converted to an intermediate image I HL This may be determined by formula (26).

[0222]

number

[0223] In some embodiments, the global quality of the intermediate image may be estimated. In some embodiments, the original low-resolution image (i.e., reference image) I L Resolution scaling back image (i.e., intermediate image) I HL To evaluate the global quality of the system, commonly used metrics (e.g., root mean square error (referred to as RSE for resolution scaling error) and Pearson correlation coefficient (referred to as RSP for resolution scaling Pearson coefficient)) may be used in this operation. RSE and RSP may be expressed as follows:

[0224]

number

[0225] In 708, the processing unit 140 (for example, the map generation module 408) may determine the reference map based on the difference between the intermediate image and the reference image.

[0226] In some embodiments, the reference map may include a resolution scale error map (RSM). In some embodiments, the reference map may be determined based on the difference (e.g., absolute difference) between the intermediate image and the reference image (see Figure 10). Thus, the reference map may display at least some of the errors and / or artifacts in the reconstruction process of the first image. To visualize the absolute difference on a pixel-by-pixel basis, HL and I L The reference map RSM between and may be calculated by the following formula.

[0227]

number

[0228] The above description of process 700 is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Those skilled in the art, having understood the operating principle, will understand, without departing from the principle, that any combination of operations, any addition or deletion of operations, or application of the operating principle to other image processing processes may be performed. In some embodiments, one or more operations may be combined into a single operation. For example, operations 704 to 708 may be combined into a single operation. In some embodiments, the first image in process 700 may be replaced with a second image.

[0229] Figure 8 is a flowchart illustrating another exemplary process for determining a target map according to some embodiments of the present disclosure. In some embodiments, process 800 may be performed by an image processing system 100. For example, process 800 may be configured as an instruction set (e.g., an application) stored in a memory device (e.g., memory device 150, storage 220, and / or memory unit 370). In some embodiments, a processing unit 140 (e.g., a processor 210 of an arithmetic unit 200, a CPU 340 of a portable device 300, and / or one or more modules illustrated in Figure 4) may be configured to execute the instruction set and, accordingly, execute process 800. The operation of the process shown below is intended to be illustrative. In some embodiments, process 800 may be achieved with one or more additional operations not described, and / or without one or more operations described. Furthermore, the order of operations of process 800 shown in Figure 8 and described below is not intended to be limiting. In some embodiments, process 800 may be a process for generating an rFRC map (e.g., a second target map) based on a single initial image (e.g., a single frame).

[0230] In 802, the processing unit 140 (e.g., acquisition module 402) may acquire an initial image. As shown in Figure 12A, frame 1 represents an initial image of size 8 × 8. In some embodiments, the initial image may be generated by physical imaging. In some embodiments, the initial image may be a single-frame image captured by the image acquisition device 110. The initial image may be generated based on a single measurement of an object. The object may be a biological or non-biological object as described elsewhere in this disclosure.

[0231] In some embodiments, the processing unit 140 may acquire an initial image from one or more components of the image processing system 100. For example, the image acquisition device 110 may collect and / or generate an initial image and store it in the storage device 150. The processing unit 140 may retrieve and / or acquire the initial image from the storage device 150. In another example, the processing unit 140 may directly acquire the initial image from the image acquisition device 110.

[0232] In 804, the processing unit 140 (e.g., decomposition module 416) may decompose the initial image into a first image, a second image, a third image, and a fourth image. In some embodiments, the first and second images in 502 may be acquired according to 804. In some embodiments, the FRC calculation may require statistically independent image pairs that share the same detail but different noise realizations. In some physical imaging applications such as SMLM, SRRF, and SOFI, these modalities may suitably generate statistically independent images by dividing the input image sequence (e.g., initial image) into two subsets and reconstructing them independently.

[0233] In some embodiments, the processing unit 140 may use wavelet filters to decompose the initial image. Exemplary wavelet filters may include Dobsie wavelet filters, biorthogonal wavelet filters, Morlay wavelet filters, Gaussian wavelet filters, Marr wavelet filters, Meyer wavelet filters, Shannon wavelet filters, Battle-Lemarie wavelet filters, etc., or any combination thereof.

[0234] In some embodiments, each pixel of the initial image may be sampled independently of the physical imaging, so the processing unit 140 may divide the initial image into four sub-images (e.g., the first image, the second image, the third image, and the fourth image). In some embodiments, pixels at the (even, even), (odd, odd), (even, odd), and (odd, even) row / column indices of the initial image may be extracted to form four sub-images, respectively. As shown in Figure 12B, the pixel numbered 1 is at the (odd, odd) row / column index of frame 1 shown in Figure 12A, the pixel numbered 2 is at the (odd, even) row / column index, the pixel numbered 3 is at the (even, odd) row / column index, and the pixel numbered 4 is at the (even, even) row / column index. As shown in Figure 12C, the pixels numbered 1, 2, 3, and 4 may be extracted to form four sub-images numbered 1, 2, 3, and 4, respectively. As shown in Figure 12C, the four dotted boxes may represent a first image (e.g., the dotted box numbered 1), a second image (e.g., the dotted box numbered 4), a third image (e.g., the dotted box numbered 3), and a fourth image (e.g., the dotted box numbered 2), each having a size of 4x4. In some embodiments, the four sub-images may be the same size. The four sub-images may be used to create two image pairs that have the same detail but different noise realizations. In some embodiments, the two image pairs may be created randomly from the four sub-images. For example, the first and second images may be randomly combined to form an image pair, and the other two sub-images may be combined to form another image pair. In some embodiments, the two image pairs may be formed according to the positions of the sub-image pixels in the initial image.For example, as shown in Figure 12C, a sub-image numbered 1 having a pixel at an odd, odd row / column index and a sub-image numbered 4 having a pixel at an even, even row / column index may form an image pair, while a sub-image numbered 2 having a pixel at an odd, even row / column index and a sub-image numbered 3 having a pixel at an even, odd row / column index may form a different image pair.

[0235] In 806, the processing unit 140 (for example, the rolling module 406) may determine a plurality of first blocks of the first image and a plurality of second blocks of the second image (see Figure 12D, similar to Figure 9A). Operation 806 may be the same as operation 506, and its related description will not be repeated here. In some embodiments, the first blocks and the second blocks may correspond one-to-one.

[0236] In operation 808, the processing unit 140 (for example, the map generation module 408) may determine a plurality of first characteristic values ​​based on a plurality of first blocks and a plurality of second blocks (see Figures 12E to 12G, similar to Figures 9B to 9D). Operation 808 may be the same as operation 508, and its related description will not be repeated here.

[0237] In some embodiments, the first characteristic value may be modified or revised. In some embodiments, during the FRC calculation of a single frame image (i.e., the initial image), single pixel displacements in both the x and y directions may be generated in the image pair. This spatial displacement is generated during the FRC calculation. -i2πsr / N Frequency-phase modulation may occur, where s = √x0 2 +y0 2 r represents the total length of the spatial displacement, and r represents the radius in the FRC calculation. Additionally or alternatively, calibration procedures may be used to correct the determined characteristic values ​​using the following formula:

[0238]

number

[0239] Here, r tf This represents the corrected characteristic value, and r sf represents the target frequency corresponding to the image pair (i.e., the first and second images), and the four parameters a, b, c, and d are experimentally fitted (e.g., a=0.9599, b=0.9798, c=13.9044, d=0.5515). In some embodiments, the characteristic values ​​may be corrected to compensate for such biases caused by spatial displacement, making the characteristic values ​​more accurate.

[0240] In operation 810, the processing unit 140 (for example, the map generation module 408) may generate a first target map associated with the first and second images based on a plurality of first characteristic values. Operation 810 may be similar to operation 510, and its related description will not be repeated here.

[0241] In 812, the processing unit 140 (e.g., the rolling module 406) may determine a plurality of third blocks in the third image and a plurality of fourth blocks in the fourth image. In some embodiments, the processing unit 140 (e.g., the expansion module 404) may expand the third image and / or the fourth image before the third and fourth blocks are generated, similar to the expansion of the first and / or second image. The processing unit 140 may determine a plurality of third blocks in the third image and a plurality of fourth blocks in the fourth image in a similar manner to the determination of the first and second blocks, and the relevant explanation will not be repeated here. The plurality of third blocks and the plurality of fourth blocks may correspond one-to-one.

[0242] In 814, the processing unit 140 (for example, the map generation module 408) may determine a plurality of second characteristic values ​​based on a plurality of third blocks and a plurality of fourth blocks. Operation 814 may be performed in the same manner as 808, and the relevant description will not be repeated here.

[0243] In 816, the processing unit 140 (for example, the map generation module 408) may generate a second target map associated with the third and fourth images based on a plurality of second characteristic values. The processing unit 140 may determine the second target map associated with the third and fourth images in a manner similar to that used for determining the first target map in 810 or 510, and the relevant explanation will not be repeated here.

[0244] In 818, the processing unit 140 (for example, the map generation module 408) may generate a third target map based on the first target map and the second target map.

[0245] The third target map may be considered as an rFRC map generated from the initial image. In some embodiments, the processing unit 140 may generate the third target map based on a weighted sum of the first target map and the second target map. In some embodiments, the processing unit 140 may generate the third target map by directly combining the first target map and the second target map. In some embodiments, the processing unit 140 may average the first target map and the second target map to obtain the third target map.

[0246] In some embodiments, through image decomposition in 804, the lateral dimensions of the four sub-images may be identical and may be half the size of the original image (i.e., the initial image in 802). In some embodiments, the third target map may be resized to the size of the original image (i.e., the size of the initial image) (see Figure 12H). Alternatively, the first and second target maps may be resized to the size of the original image, and the third target map may be generated based on the resized first and second target maps. In some embodiments, resizing may be performed using bilinear interpolation.

[0247] The above description of process 800 is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Those skilled in the art, having understood the operating principle, will understand, without departing from the principle, that any combination of operations, any addition or deletion of operations, or application of the operating principle to other image processing processes may be performed. In some embodiments, one or more operations may be combined into a single operation. For example, operations 806 to 814 may be combined into a single operation. As another example, operations 816 and 818 may be combined into a single operation. As yet another example, the processing unit (e.g., filtering module 410) may filter a first target map, a second target map, and / or a third target map. In some embodiments, one or more operations may be added. For example, an operation to filter the first target map and an operation to filter the second target map may be added. As another example, an operation to filter the third target map may be added. As a further example, the operation of displaying a third target map using a displacement jet color map and / or the operation of determining the global image quality metric of the initial image based on the third target map may be added.

[0248] This disclosure is further described by the following embodiments, which should not be construed as limiting the scope of this disclosure.

[0249] Examples method

[0250] STORMTION The microscope setup is described below. After washing with phosphate-buffered saline (PBS), 5 w / v% glucose is added in Tris-HCl (pH: 7.5), and the sample is measured in 100 × 10⁴ -3 Cysteamine M, 0.8 mg / mL -1 glucose oxidase, and 40 μg / mL -1Samples were placed on glass slides using standard STORM imaging buffer consisting of catalase. Data were then acquired using a modified commercially available inverted fluorescence microscope (Eclipse Ti-E, Nikon) with a self-defined 3D-STORM setup using an oil immersion objective lens (100x / 1.45NA, CFI plan apochromat λ, Nikon). 405nm and 647nm lasers were introduced into the cell sample from the back focal plane of the objective lens, displaced towards the edge of the objective lens, and irradiated approximately 1 μm within the glass-water interface. A powerful 647nm laser (approximately 2 kW·cm²) was used. -2 The excitation laser optically switched most of the labeled dye molecules into the dark state, while simultaneously exciting the fluorescence from the remaining sparsely distributed luminescent dye molecules to achieve single-molecule localization. A weak 405 nm (typical range: 0-1 W·cm) was used. -2 By using a laser in combination with a 647nm laser to reactivate the phosphors into an emissive state, only the small, optically resolvable phosphors were emissive at any given moment. A cylindrical lens was introduced into the imaging path to introduce astigmatism, and the depth (z) position was encoded into the ellipticity of the single-molecule image. The EMCCD (iXon Ultra 897, Andor) camera recorded images at a frame rate of 110 frames per second for a 256x256 pixel frame size, typically recording approximately 50,000 frames per experiment. Furthermore, the cylindrical lens in the optical layout was removed to form 2D-STORM imaging.

[0251] STORM reconstruction is described below. The open-source software package Thunder-STORM and customized 3D-STORM software were used for STORM image reconstruction. Images labeled "ME-MLE" and "SE-MLE" were reconstructed using maximum likelihood estimation (integrated PSF method) and Thunder-STORM, which has multi-emitter fitting capability ("ME-MLE") or non-multi-emitter fitting capability ("SE-MLE"). Images labeled "SE-Gaussian" were reconstructed using customized 3D-STORM software by fitting the extrema with an (elliptic) Gaussian function. After localization, drift correction was performed and the images were rendered using a normalized Gaussian function (σ = 2 pixels).

[0252] Cell culture, fixation, and immunofluorescence are described below. COS-7 cells were cultured in DMEM supplemented with 10% fetal bovine serum (FBS) in a humidified CO2 incubator at 37°C with 5% CO2, following a standard tissue culture protocol. Next, the cells were seeded onto 12 mm glass coverslips in 24-well plates at a rate of approximately 2 × 10⁴ cells per well and cultured for 12 hours. For actin filament storming, a previously established fixation protocol was used: the samples were first fixed in cytoskeletal buffer (CB, 10 × 10⁴). -3 M's MES, pH: 6.1, 150 x 10 -3 M NaCl, 5 × 10 -3 M EGTA, 5x10 -3 M glucose, 5 × 10 -3 In MgCl2, the samples were fixed and extracted for 1 minute using 0.3 v / v% glutaraldehyde and 0.25 v / v% Triton® X-100, then fixed in CB with 2 (v / v)% glutaraldehyde for 15 minutes, and reduced with freshly prepared 0.1% sodium borohydride solution in PBS. Approximately 0.4 × 10⁶ Alexa Fluor 647-conjugated phalloidin were extracted. -6The sample was coated with concentration M for 1 hour. After briefly washing the sample 2-3 times with PBS, it was immediately placed for imaging. For imaging of other targets, the sample was fixed with 3 w / v% paraformaldehyde and 0.1 w / v% glutaraldehyde in PBS for 20 minutes. After reduction with freshly prepared 0.1% sodium borohydride solution in PBS for 5 minutes, the sample was permeabilized and blocked with blocking buffer (3 w / v% BSA and 0.5 v / v% Triton X-100 in PBS) for 20 minutes. The cells were then incubated with the primary antibody (described above) in blocking buffer for 1 hour. After washing 3 times with washing buffer (0.2 w / v% BSA and 0.1 v / v% Triton X-100 in PBS), the cells were incubated with secondary antibody at room temperature for 1 hour. The samples were then washed 3 times with washing buffer and placed for imaging.

[0253] SIMICAL The TIRF-SIM is described below. The SIM system was constructed as described above using a commercially available inverted fluorescence microscope (IX83, Olympus) equipped with a TIRF objective lens (100x / 1.7NA, Apo N, HI Oil, Olympus) and a multiband dichroic mirror (DM, ZT405 / 488 / 561 / 640 phase R, Chroma). Specifically, laser beams with wavelengths of 488nm (Sapphire 488LP-200) and 561nm (Sapphire 561LP-200, Coherent) and an acousto-optic tunable filter (AOTF, AA Opto-Electronic, France) were used to combine, switch, and adjust the irradiation power of the lasers. The lasers were coupled to a polarization-maintaining single-mode fiber (QPMJ-3AF3S, Oz Optics) using a collimating lens (focal length: 10mm, Lightpath). The output laser was then collimated by an objective lens (CFI Plan Apochromatic Lambda 2x NA 0.10, Nikon) and diffracted by a pure phase grating consisting of a polarizing beam splitter (PBS), a half-wave plate, and an SLM (3DM-SXGA, ForthDD). The diffracted beam was then focused onto the intermediate pupil plane by another achromatic lens (AC508-250, Thorabo), where a stop mask, meticulously designed to block zeroth-order light and other stray light and allow only ±1st-order light pairs to pass through, was placed. A custom-made polarization rotator was placed after the stop mask to maximize modulation of the irradiation pattern while eliminating switching time between different excitation polarizations. The light then passed through another lens (AC254-125, Thorabo) and a tube lens (ITL200, Thorabo) to focus onto the back focal plane of the objective lens and interfered with the image plane after passing through the objective lens. The fluorescence emitted, focused by the same objective lens, passed through a dichroic mirror (DM), a light-emitting filter, and another tube lens. Finally, the fluorescence was split by an image splitter (W-VIEW GEMINI, manufactured by Hamamatsu Japan Co., Ltd.) and then captured by an sCMOS camera (Flash 4.0 V3, manufactured by Hamamatsu Japan Co., Ltd.).

[0254] The Hessian SIM is described below. A Hessian denoising algorithm without t-continuity constraints was applied to the Wiener SIM reconstruction results to obtain the Hessian SIM image shown in Figure 21E.

[0255] 3D-SIM is explained below. The 3D-SIM datasets shown in Figures 25A to 25I were acquired using a Nikon 3D-SIM microscope equipped with a TIRF objective lens (100x / 1.49NA, CFI apochromatic, oil, manufactured by Nikon).

[0256] Cell maintenance and preparation are described below. Human umbilical vein endothelial cells were isolated and cultured in M199 medium (Thermo Fisher Scientific, 31100035) supplemented with fibroblast growth factor, heparin, and 20% fetal bovine serum (FBS), or in endothelial cell medium (ECM) (ScienCell, 1001) supplemented with endothelial cell growth supplement (ECGS) and 10% FBS. Cells were infected with a retroviral system to express LifeAct-EGFP. Transfected cells were cultured for 24 hours, detached using trypsin-EDTA, and poly- L - Seeds were seeded on lysine-coated coverslips (H-LAF10L glass, reflectivity: 1.788, thickness: 0.15 mm, customized) and incubated for an additional 20-28 hours in an incubator at 37°C with 5% CO2 before the experiment.

[0257] LSEC was isolated and plated onto coverslips coated with 100 μg / ml collagen, and then mixed with 10% FBS and 1% L Cells were cultured in high-glucose DMEM supplemented with glutamine, 50 U / ml penicillin, and 50 μg / ml streptomycin in an incubator at 37°C with 5% CO2 for 6 hours prior to imaging. Live cells were incubated with DiI (100 μg / ml, Biotium, 60010) at 37°C for 15 minutes, and fixed cells were fixed with 4% formaldehyde at room temperature for 15 minutes before labeling with DiI.

[0258] STED IMAGE Image acquisition by STED was performed using a gated STED (gSTED) microscope (Leica TCS SP8 STED 3X, Leica Microsystems, Germany) equipped with a wide-field objective lens (100x / 1.40 NA, HCX PL APO, oil, Leica). The excitation wavelength was 647 nm and the attenuation wavelength was 775 nm. All images were acquired using LAS AF software (Leica). To label microtubules in living cells, as shown in Figures 40A-40C, COS-7 cells were incubated with SiR-Tubulin (cytoskeleton, CY-SC002) for 20 minutes without washing before imaging.

[0259] PANEL Framework In principle, the imaging process can be described as a measurement system observing an object using a transfer function, with the corresponding signal being collected by a sensor. Since the signal is actually sampled by a sensor subject to various types of noise, it is generally recognized that such artificial observations will always deviate from the real-world object in high-dimensional space (left panel of Figure 15). Errors / artifacts included in the reconstruction, when a suitable reconstruction model is used, are primarily caused by such joint effects of sampling rate and mixed noise. In other words, reconstructions with such inevitably distorted observations can also deviate from the real-world object in the target / super-resolution region (right panel of Figure 15). When the same object is imaged and variables are controlled to statistically capture independent image pairs, the distance between the ground truth and its reconstruction can be amplified by the difference between these individual reconstructions (Figures 9A-9D). Conventional spatial evaluation methods, such as relative and absolute errors used in RSM, for quantifying such a distance between a recorded image and its ground truth are overly sensitive to intensity and minute movements during measurement. These quantization algorithms can be viewed as "absolute differences," potentially inducing a high number of false negatives that outweigh true negatives in distance maps, which could significantly mislead biologists in their perception of existing errors / artifacts.

[0260] Based on the above analysis, a reference empty approach, namely Fouriering correlation (FRC), was introduced to measure such an immeasurable distance between two signals in the Fourier domain. FRC describes the most acceptable frequency component of the distance between two signals. Traditionally, the FRC metric has been widely used as an effective resolution metric in super-resolution fluorescence microscopy and electron microscopy. It can also be applied to quantify the similarity or distance between two images. Because FRC is calculated as a "relative error" or "sampling-based error" for an image, it has the inherent advantage of being insensitive to changes in intensity and minute movements. Furthermore, to estimate the most reliable frequency component, FRC is a more quantitative and understandable metric that emphasizes only the sampling error. As a result, FRC is an excellent choice for quantifying the distance between two signals while significantly reducing the potential problem of false negatives. Notably, the conventional FRC framework is extended to the form of a rolling FRC (rFRC) map (see Figures 9A-9D) with the aim of considering FRC as a global similarity estimation between two images and providing more accurate local distance measurements down to the pixel level, enabling quantitative evaluation of image quality at super-resolution scales. The rolling FRC calculation may be like a moving filter on the image plane, using a sliding window for each block of the image, to which a corresponding FRC value is assigned. Firstly, the input image is symmetrically padded by half the size of the blocks to ensure that the FRC calculation is performed at the image boundaries (see Figure 9A). Secondly, an appropriate background threshold may be used for central pixels (1×1 to 3×3) to determine whether or not to calculate an FRC value for that block, thereby avoiding FRC calculations on the background region. Thirdly, if the average value of the central pixels is greater than the threshold, the FRC calculation is performed and an FRC value is assigned to the central pixels of each block. Otherwise, the central pixels may be set to zero to avoid unnecessary calculations on the background region. This procedure may then be performed block by block until the entire image has been completely scanned.

[0261] Furthermore, to visualize the error map, corresponding metrics, rFRC values, and color maps (displacement jets, or sJet) that are more in line with human intuition have been developed. The rFRC map allows for the quantitative mapping of errors in multidimensional reconstructed signals without prior information on the ground truth and imaging system. However, it should be noted that the rFRC method may fail and report false positives if the two reconstructed signals lose identical components (although such situations are rare in practical experiments). If the same region in both images is blank (but the ground truth has content), such information may be lost in the same region, leading to an incorrectly small FRC value. Therefore, to eliminate such possible false positives, a modified resolution-scale error map (see RSM in Figure 10) is introduced as an additional error map and integrated into the PANEL framework. A drawback of the full RSM is that, due to the three hypotheses mentioned above, it may introduce high false negatives at low magnitudes. To reduce such false negatives in the RSM, the RSM may be segmented with a hard threshold of 0.5 and common large artifacts such as misrepresentation or disappearance of structures may be highlighted before integration into the final PANEL map (Figures 11A-11C). On the other hand, the rFRC map indicates the degree of error, and consequently, the smallest FRC value in the map does not necessarily represent the error. To address this situation, a segmentation method called Otsu is introduced, which automatically determines the threshold by maximizing the interclass variance, and highlights the decisive error of the reconstruction by performing image thresholding that filters the background of the original rFRC map (Figure 16).

[0262] Open source datasets

[0263] In addition to custom-collected datasets, we demonstrated the broad applicability of PANEL using freely available simulation / experimental datasets.

[0264] The 2D-SMLM simulation datasets are described below. As shown in Figure 17A, the "High-Density Bundle Tubes" (361 frames) dataset and the "Long Bundle Tubes" (12,000 frames) dataset from the "Localized Microscopy Challenge Datasets" on the EPFL website were used as high-density and low-density 2D-SMLM simulation datasets. The optical system's NA was 1.4 (oil immersion objective lens), and the fluorescence wavelength was 723 nm.

[0265] The 3D-SMLM simulation dataset is described below. As shown in Figure 17C, the "MT1.N1.LD" dataset (19996 frames, 3D astigmatism PSF) from the "Localization Microscopy Challenge Datasets" on the EPFL website was used as the low-density 3D-SMLM simulation dataset. The optical system's NA was 1.49 (oil immersion objective lens), and the fluorescence wavelength was 660 nm. All images were 64 × 64 pixels (pixel size 100 nm). Next, 20 frames from this low-density 3D-SMLM simulation dataset were averaged to 1 frame to generate a corresponding high-density 3D-SMLM simulation dataset (sequentially 998 frames).

[0266] The 2D-SMLM experimental dataset is described below. The "Localization Microscopy Challenge Dataset" also includes experimental data, and 500 high-density images of tubulin were obtained from the EPFL website (Figures 18A to 18H). The optical system had a numerical aperture (NA) of 1.3 (oil immersion objective lens), and the fluorescence wavelength was 690 nm. Images were recorded with a 64x64 pixel frame size (pixel size of 100 nm) using a camera at a frame rate of 25 frames per second.

[0267] The live cell SRRF dataset is described below. GFP-tagged microtubules in live HeLa cells were imaged in TIRF mode using a TIRF objective lens (100x / 1.46NA, plan apochromatic, oil, Zeiss), and then irradiated with a 488nm laser at 1.6x magnification (200 frames in total). The super-resolution SRRF results were reconstructed using an open-source ImageJ plugin (Figures 18I-18L).

[0268] Simulation of Gridding Samples Imaged by SMLM A regular grid with a pixel size of 10 nm (Figure 17M) was created, and the density of randomly activated molecules was set to gradually increase from the center to the sides. Next, the acquired image sequence was convolved with a Gaussian kernel with a full width at half maximum (FWHM) of 280 nm and downsampled 10 times (pixel size: 100 nm). As a result, the image sequence contained Poisson noise and 20% Gaussian noise. Finally, the image sequence was reconstructed using maximum likelihood estimation (integrated PSF method) and Thunder-STORM, which has multi-emitter fitting capability.

[0269] Simulation in the case of FPM A US Air Force (USAF) resolution target was used as the ground truth sample for FPM (Figure 24A). Both the intensity and phase of the imaged sample were set as a 240×240 pixel (pixel size: 406.3 nm) USAF target. The sample was illuminated from different angles using a 7×7 LED matrix with an emission wavelength of 532 nm and a distance of 90 mm from the sample. The sample was illuminated by each LED unit, filtered with an objective lens (4x / 0.1 NA), and sampled with a camera (acquired image size: 60×60, pixel size: 1.625 μm). After each LED illuminated the sample, a total of 49 low-resolution images were acquired. The image illuminated by the central LED was used as the initial image. Next, with each iteration of the FPM, the amplitude and phase of the corresponding aperture were updated sequentially. After 10 rounds of iterations, the final high-resolution complex amplitude image (240×240) was acquired, which was four times larger in size than the corresponding low-resolution image.

[0270] Data generation process for learning-based applications Sparse sampling is explained below. The deep neural network (DNN) was trained using sparsely sampled geometric structures and their corresponding intact geometric structures as ground truths. Four simple and common geometric structures were selected for simulation: triangle, circle, rectangle, and square. Table 1 shows the spatial size of one structure and the number of structures in a single input image. After obtaining the structures, images were randomly sampled at a sampling rate of 8%. The rectangle structure was selected, and 5000 images were used as the training dataset. For each geometric structure, 200 images were generated as the test dataset.

[0271] The superresolution of pixels is explained below. The DNN was trained using a RAW image as the ground truth and the corresponding downsampled RAW image as input. As the training dataset, Drosophila melanogaster expressing the membrane marker Ecad::GFP was selected using a commercially available spinning disk microscope (camera exposure / laser power: 240ms / 20%). First, PreMosa was used to obtain the surface-projected 2D ground truth. Second, images containing a strong background were discarded. Third, the background was estimated by repeatedly applying wavelet transforms, and then subtracted to generate a background-free image as the ground truth. Finally, images downsampled four times with 4x4 pixels as one pixel were used as input. Before inputting to the DNN, the images were upsampled four times using bilinear interpolation.

[0272] The deconvolution process is described below. The DNN was trained using a synthetic subdiffraction image as the ground truth and a corresponding diffraction-limited wide-field image as input. Specifically, a synthetic tubulin structure was created using a random walk process to simulate two-dimensional trajectories with randomly changing orientations. Meanwhile, the maximum curvature was set as a restricted value with respect to the known physical stiffness of tubulin. These images were processed to create input images using a ground truth with a pixel size of 65 nm. These images were convolved with a Gaussian kernel (FWHM: 300 nm) to obtain blurred images. To simulate a realistic fluorescence background, the blurred images were convolved with a larger Gaussian kernel (FWHM: 1.2 μm) and added to the blurred images. Then, Poisson noise and 2% Gaussian noise were added to the images to create the final input images for DNN training. A further 781 images were generated as a test dataset using a similar procedure.

[0273] Noise2Noise is described below. Noise2Noise is an unsupervised learning procedure that denoises noisy images without sharp images. During training, the DNN only considers noisy image pairs (two images with independent noise that share the same detail), i.e., one as the input target and the other as the output target. The fluorescence microscopy denoising (FMD) dataset was used for this Noise2Noise task. Fixed zebrafish embryos [EGFP-labeled Tg(sox10:megfp) zebrafish 2 days post-fertilization] were selected as the dataset, imaged at very low excitation power with a commercially available Nikon A1R-MP re-confocal laser scanning microscope. This image composition has five noise levels. The RAW image has the highest noise level, and images with other noise levels were generated by averaging RAW images with multiple frames (2, 4, 8, 16) using the circumference averaging method. To test extreme conditions, the RAW image with the highest noise level was selected as the input for the training set (2 RAW images every 2). For each of the 20 fields of view (50 different noise realizations), 200 noise-to-noise data pairs were randomly selected. Meanwhile, the 512x512 RAW images were cropped into four non-overlapping 256x256 patches. Ultimately, 20x200x4 = 16,000 images were obtained as the training dataset. A ground truth reference to evaluate Noise2Noise's prediction accuracy was generated by averaging the 50 noisy RAW images.

[0274] Network Architecture and Learning Procedures The network architecture is described below. This network architecture includes a shrinking path and an expanding path, and is a so-called U-shaped architecture (UNet). In the shrinking path, following the input layer, there is a series of down-convolutional blocks consisting of a 4x4 kernel convolution with a stride step of 2, batch normalization (BN), and a leaky normalized linear unit (LeakyReLU) function. The convolutional layers are located at the bottom of this U-shaped structure, connecting the down-convolutional blocks and the up-convolutional blocks. The expanding path combines the features and spatial information from the shrinking path by concatenating a series of up-convolutional blocks (upsampling 2D operation + 4x4 kernel convolution with a stride step of 1 + BN + ReLU) and high-resolution features. The final layer is another convolutional layer that maps a 32-channel image to a 1-channel image. Two types of U-shaped network architectures (UNet) are used for different tasks. 1 and Unet 2 ) was used (see Figure 43). Unet 1 and Unet 2 The difference is Unet 1 While it has 7 down convolution blocks and 7 up convolution blocks, Unet 2 It has four down-convolution blocks and four up-convolution blocks.

[0275] The training procedure is described below. All networks were trained using stochastic gradient descent with adaptive moment estimation (Adam). Detailed input patch size, number of epochs, batch size, number of training images, learning rate, network architecture, number of parameters, and loss function for each task are shown in Figure 42 and Table 2. Figure 42 shows an overview of the network architectures, training parameter configurations, and data used for different applications. All training procedures were performed on a local workstation equipped with an NVIDIA Titan Xp GPU card. The relevant training framework was implemented using TensorFlow framework version 1.8.0 and Python version 3.6.

[0276] Use of open-source deep learning models ANNA-PALM is described below. ANNA-PALM reconstructs super-resolution images from rapidly acquired sparsely localized data. ANNA-PALM was trained using densely sampled PALM images (long sequences) as the ground truth and corresponding sparsely sampled PALM images (short sequences) as input. ANNA-PALM is based on Unet 1 cGAN when using as a generator 1 This is based on the following: The performance of ANNA-PALM was tested using 500 high-density images of tubulin from the EPFL website. After the phosphors in frames 1-25 and frames 26-50 were localized using the ME-MLE estimator to construct two sparse super-resolution inputs, ANNA-PALM predicted the corresponding high-density sampled images.

[0277] CARE will be explained below. The CARE framework is Unet 3 This is a computational approach that can extend the spatial resolution of a microscope using an architecture. The open-source pre-trained CARE model was populated with the first and second RAW images from the open-source SRRF dataset, generating two corresponding super-resolution images.

[0278] Cross-modality super-resolution is explained below. The cross-modality imaging capability of the DNN is demonstrated in advance by mapping TIFR to the TIRF-SIM modality (TIRF2SIM) using a cGAN approach. cGAN in TIRF2SIM 2 This model is based on Unet, which uses residual convolutional blocks (Res-Unet) as its generator. It was trained and tested using AP2-eGFP tagged clathrin in gene-edited SUM159 cells. The results were directly reproduced using the provided ImageJ plugin and sample data.

[0279] Image rendering and processing Custom-developed color maps, including the displacement jet (sJet) color map, were used to visualize the rFRC map. The SQUIRREL-Error color map was used to present the RSM in Figures 17P, 28F, the lower right corner of Figures 17A-17B, Figure 18G, and 33H. Jet projections were used to show depth in Figures 23D and 25A. All data processing was achieved using MATLAB® and ImageJ. All figures were created using MATLAB®, ImageJ, Microsoft Visio, and OriginPro.

[0280] Data availability All data supporting the results of this study are available upon request from the corresponding authors.

[0281] Code availability The latest versions of the custom-written MATLAB® library (with user manual and sample data), custom-designed colormaps, and corresponding ImageJ plugins used in this disclosure are available as software. Updated versions of the PANEL in MATLAB® library can be found at https: / / github.com / WeisongZhao / PANELM. Updated versions of the ImageJ plugin and its source code can be found at https: / / github.com / WeisongZhao / PANELJ. (Example 1)

[0282] PANEL verification using SMLM simulation To evaluate the PANEL's ability to identify minute defects in super-resolution images, 2D single-molecule localization microscopy (SMLM) simulation datasets from the EPFL Challenge were used. The 2D-SMLM simulation datasets (e.g., high-density (HD) 2D-SMLM simulation dataset and low-density (LD) 2D-SMLM simulation dataset) included HD-emitting phosphors and LD-emitting phosphors, respectively (see Figures 17A–17F). Figures 17A–17F show 2D-SMLM simulations using high-density (HD) and low-density (LD) emitting phosphors. As shown in Figures 17A–17D, the integrated ground truth structure is represented by dark gray channels labeled HD-GT or LD-GT, and the maximum likelihood estimation (MLE) reconstruction is represented by a light gray channel labeled MLE. Figures 17B–17E show the rFRC maps of the MLEs, respectively. Figures 17C–17F show the complete PANEL for the MLEs. RSM was represented as a channel indicated by the white arrow, and the rFRC map was represented as the other channel. The white and light gray arrows indicate errors found by the RSM or rFRC map, respectively. In this quantitative mapping, one set of images was split into two statistically independent subsets to obtain two independent super-resolution images. After using maximum likelihood estimation (MLE) reconstruction, the two corresponding super-resolution images of the HD 2D-SMLM simulation dataset were obtained. Next, known structural artifacts, as indicated by the white arrows in Figures 17A–17F, were artificially removed to visualize potential false positives induced by considering only the rFRC map. As indicated by the light gray arrow, the rFRC map was found to be able to detect all subtle errors (rFRC value: 0.66), but was unable to detect the missing structures in both super-resolution frames. Therefore, as mentioned above, in order to verify all errors in the reconstruction, the RSM map and rFRC map were merged and represented as channels indicated by white arrows and other channels, respectively (Figures 17A to 17F), to create the final full panel map.On the other hand, compared to the ground truth, the MLE results of the LD 2D-SMLM simulation dataset were found to contain almost no error (rFRC value: 0.16), and the corresponding rFRC maps also realistically described this appearance during the computation process (Figures 17A to 17F).

[0283] A comparison of 2D-SMLM simulation datasets for each LD and HD revealed that the performance of the SMLM results is related to the density of the luminescent phosphor, which may be highly dependent on the induced irradiation intensity. To further quantitatively evaluate the effect of emission density on MLE reconstruction within a single-frame image, a regular grid was created with high intensity illumination in the center and decreasing intensity towards the edges, demonstrating that the flashing in the center is temporally separated from the edges (Figure 17G). Figure 17G shows the MLE results of a 2D-SMLM simulation using heterogeneous illumination (high intensity in the center, decreasing towards the edges). Figure 17H ​​shows the rFRC map generated based on Figure 17G. As shown in Figure 17H, the rFRC map clearly demonstrates that it accurately describes the reconstruction performance induced by the flashing density transition. In contrast, the estimated RSM cannot adequately present the error (even contrary to the reference) because the RSM hypothesis is not met (Figures 17M-17P). Figure 17M shows the simulated ground truth. Figure 17N shows the wide-field image of Figure 17M, where the central part is illuminated with high brightness and the illumination decreases towards the edges. Figure 17O shows the image after convolving the reconstructed MLE image of Figure 17G with the estimated RSF. Figure 17P shows the RSM of the reconstructed MLE image of Figure 17G.

[0284] In the final step, the RSM was removed from the PANEL for 3D super-resolution imaging (Figures 17I-17L), and the rFRC map was directly extended to the 3D version by applying plane-by-plane calculations as a constraint in 3D model evaluation. Figure 17I shows an image showing the integrated ground truth structure (red channel labeled "LD-GT") and MLE reconstruction (green channel labeled "3D-MLE"). Figure 17J shows the rFRC map for low-density 3D-MLE with an rFRC value of 2.2. Figure 17K shows an image showing the integrated ground truth structure (red channel labeled "HD-GT") and MLE reconstruction (green channel labeled "3D-MLE"). Figure 17L shows the rFRC map for high-density "3D-MLE" with an rFRC value of 4.5. The 3D simulation datasets used to validate the PANEL, including for LD and HD (averaged per 20 frames from the LD dataset), were obtained from the EPFL SMLM challenge. Similar to the 2D case, the rFRC results show that the global performance of 3D-MLE reconstruction is greatly influenced by the density of the luminescent phosphor, which is all in good agreement with actual physical experience (see Figures 17I-17L, 17Q, and 17X, 17Y for 3D, LD, and HD with rFRC values ​​of 2.2 vs. 4.5). Figure 17Q shows the global ground truth for the low-density (LD) and high-density (HD) datasets, and the darkly colored 3D-MLE reconstructions, from left to right. Figure 17X shows the horizontal cross-sections (z position 0nm) of the 3D-MLE reconstructions (LD and HD) and a representative frame (frame 19) of the LD dataset, from left to right. Figure 17Y shows the rFRC map of the horizontal cross-section of Figure 17X and a representative frame (frame 99) of the HD dataset, from left to right.

[0285] Furthermore, while rFRC mapping can detect all minute and subtle errors, it was found that this method is not possible when a structure is simultaneously missing in both adjacent super-resolution frames. To clarify, an example is created using the 2D SMLM challenge dataset to illustrate this possible false negative (see Figures 17A–17C and 17R–17T). Figure 17R shows the rFRC map after Otsu threshold filtering for Figure 17B. Figure 17S shows the RSM. Figure 17T shows the RSM after 0.5 threshold filtering. As shown in Figure 17B, the rFRC map successfully detects most of the error components across the entire image, except for the artificially removed regions. However, such disappearances of structures can be successfully detected by the RSM (see Figure 17R). Therefore, to address this possible false positive, the RSM was introduced as an error map to be appended to the rFRC map. It should also be noted that such lost identical information in the two reconstructions may include higher magnitudes and reside in large areas of the RSM. Furthermore, three hypotheses regarding RSM suggest that low-grade components included in the RSM may introduce high false negatives. Therefore, the RSM was segmented using 0.5 threshold filtering before involving the RSM in the PANEL. After this operation, low false negatives are precisely filtered out, leaving only strongly low-resolution error components to focus on true negatives detected by the RSM.

[0286] On the other hand, in practice, since the rFRC map indicates the degree of error, the smallest FRC value in the map does not necessarily indicate the acceptable error. Similarly, a segmentation method called Otsu was introduced, which automatically determines the threshold by maximizing the interclass variance, performs image thresholding to filter the background of the rFRC map, and emphasizes the decisive error of the reconstruction (Figures 17U-17W). Subsequently, the segmented rFRC maps were integrated as light gray channels, and the RSM was segmented as a dark gray channel to create a complete panel and visualize the integrated error map of the corresponding reconstruction (Figures 17A-17C, 17R-17T, and 17U-17W). It should be noted that when the dataset is three-dimensional or in a non-Gaussian convolution relationship (between low-resolution and high-resolution scales), it is not possible to estimate the corresponding RSM. As a result, in such datasets, the RSM does not need to be involved in the panel. Figure 17U shows the rFRC map displayed in a displacement jet color map for the same SRRF dataset as Figures 18I-18L. Figures 17V and 17W show the full rFRC map and the rFRC map after Otsu thresholding to highlight the decisive error in SRRF reconstruction, indicated by the green channel in PANEL. (Example 2)

[0287] Minimizing analysis errors using PANEL After synthesizing ground truth images including grid structure and basic simulations, the experimental 2D-SMLM dataset from the EPFL Challenge was evaluated using the proposed PANEL (Figures 18A-18H). Figure 18A shows an image illustrating the MLE localization results of 500 high-density images of tubulin from the EPFL website. Figure 18B shows the rFRC map of "MLE". Figure 18C shows the complete PANEL of "MLE" with an rFRC value of 1.2. Figure 18D shows an image illustrating the PANEL after Otsu threshold segmentation. Figure 18E shows the corresponding "wide field" image. Figure 18F shows the "MLE" image deconvolved to the original low-resolution scale. Figure 18G shows an image illustrating the RSM of "MLE". Figure 18H shows the FRC map of "MLE" with an FRC value of 584 nm. Based on the acquired corresponding rFRC maps, it was found that large FRC values ​​tend to appear in the filament crossing regions, which is consistent with the SRRF results revealed in Figures 18I to 18L. Figure 18I shows the diffraction-limited TIRF image. Figure 18J shows the SRRF reconstruction results of 100 fluctuation images (GFP-tagged microtubules in live HeLa cells, see Methods). Figure 18K shows the rFRC map of "SRRF" with an rFRC value of 2.25. Figure 18L shows the PANEL after Otsu threshold segmentation. The main possible reason is the influence of emission density in 2D-SMLM simulations, where the crossing regions contain relatively large emitter densities, leading to a decrease in localization performance. In actual experimental processes, it is experimentally observed that emission density (related to structural complexity) changes significantly within the field of view, but there is no practical quantitative method to identify the minute errors in the super-resolution scale. Up to this point, most existing algorithms have had to accept such performance trade-offs by designing themselves to consider only uniform density instead of HD or LD.

[0288] The high-resolution error mapping capability allows for the integration of advantages from existing HD or LD focusing algorithms, thereby minimizing the errors inherent in all selected methods. To validate this statement, a 2D-STORM dataset of immunolabeled α-tubulin in fixed COS-7 cells was analyzed using two different algorithms: multi-emitter MLE (MEM-LE) and single-emitter Gaussian fitting (SE-Gaussian) (see methods in Figures 20A and 19A-19E). Figure 19A shows an image illustrating the reconstruction using multi-emitter MLE (ME-MLE). Figure 19B shows an image illustrating single-emitter Gaussian fitting (SE-MLE). Figure 19C shows an image illustrating the fusion results of ME-MLE and SE-MLE using rFRC maps. Figure 19D shows the corresponding rFRC maps from Figures 19A-19C. Figure 19E shows a magnified view of the white box in Figure 19A. Figure 20A shows images of fusion STORM results (COS-7 cells, α-tubulin labeled with Alexa Fluor 647) from reconstitution by multi-emitter MLE (ME-MLE) and single-emitter Gaussian fitting (SE-MLE). Figure 20B shows the rFRC map of Figure 20A. Figures 20C to 20E show enlarged images of the area enclosed by box 2001 in Figure 20B, of which Figure 20C shows the rFRC map results, Figure 20D shows the fusion STORM results, and Figure 20E shows the RSM results. Figures 20F to 20H show enlarged views of box 2002 in Figure 20A, and Figures 20I to 20K show enlarged views of box 2003 in Figure 20B. Of these, Figure 20F shows the ME-MLE result, Figure 20G shows the SE-MLE result, and Figure 20H shows the fusion STORM result. Figure 20I shows the rFRC map of Figure 20F where the rFRC value was 1.01, Figure 20J shows the rFRC map of Figure 20G where the rFRC value was 4.51, and Figure 20K shows the rFRC map of Figure 20H where the rFRC value was 0.87. Figure 20L shows enlarged views of the dashed circles in Figures 20F to 20H.SE-Gaussian is better suited for the reconstruction of simple structures (LD emitters), while ME-MLE is better suited for complex structures (HD emitters), as shown in Figures 20C–20K. By utilizing rFRC error maps at the super-resolution scale, spatial details regarding the local accuracy of each algorithm can be mapped and converted into fused weights (Figures 20B–20E), and the lowest error features of each reconstruction may then be used to generate a new composite image with minimal defects. As expected, as shown in the two selected regions of interest (Figures 20F–20K), the individual algorithms (ME-MLE or SE-Gaussian only) cannot achieve stable performance for all structures across the entire field of view. By combining the information generated by ME-MLE and SE-Gaussian using the proposed rFRC maps, the strengths of both algorithms can be leveraged to achieve superior performance across the entire field of view (Figures 20C–20K, 20M). Furthermore, when this fusion approach was applied to a 2D-STORM dataset of heavy-chain clathrin-coated pits (CCPs) in COS-7 cells, the average resolution was significantly improved (Figure 20N). Figure 20M shows the ME-MLE rFRC map, fusion superiority map, and TIRF image from top left to bottom left, and the SE-Gaussian rFRC map, fusion result, and fused result from top right to bottom right. The ME-MLE method achieved superior performance in areas with strong backgrounds, while the SE-Gaussian method obtained better reconstruction quality in areas with weak backgrounds. Figure 20N shows enlarged results of a single CCP for ME-MLE, SE-Gaussian, and fusion from top left to bottom left, with the corresponding rFRC map on the right. The average resolution is described in the upper left of the rFRC map. As highlighted in Figure 20M, in addition to stable fusion performance across the entire field of view, rFRC mapping also helps in the fusion of microstructures such as single ring-shaped CCPs, resulting in higher average resolution. (Example 3)

[0289] Diverse physical-based imaging applications supported by rFRC maps The designed rFRCs were identified and manipulated into a typical 3D-STORM reconstruction (Figure 21A). The resulting rFRC map (Figure 21B) and its panel map (Figure 21C) showed that large errors occurred mainly among entangled filament voxels (Figures 22A-22C). Figure 21A shows an image of the 3D-MLE reconstruction (COS-7 cells, α-tubulin labeled with Alexa Fluor 647). Figure 21B shows the 3D rFRC map of Figure 21A. Figure 21C shows an image of the panel after Otsu thresholding of Figure 21B. Figure 21D shows an image of the curve of rFRC values ​​along the axial position. Figure 21E shows representative images of live human umbilical vein endothelial cells (HUVECs) labeled with LifeAct-EGFP under Wiener SIM (top), Hessian SIM (middle), and TIRF (bottom) imaging. Figure 21F shows the rFRC map of the Hessian SIM, with rFRC=1.24, RSP=0.98, and RSE=0.27 for the Wiener SIM and Hessian SIM, respectively. Figure 21G shows representative results for DiI-labeled fixed hepatic sinusoidal endothelial cells (LSEC) under RL inverse convolution (top) and TIRF (bottom) imaging. Figure 21H shows the rFRC map of the RL inverse convolution results. Figure 21I shows a magnified view of the box in Figure 21G, including the original TIRF image (top left), images showing RL inverse convolution results for 80 and 200 repeats (top right and bottom left), and an image showing the TIRF-SIM result (bottom right). Figure 21J shows curves of PSNR (vs. TIRF-SIM), RSP (vs. TIRF), and rFRC values ​​along the repeats. Figure 22A shows the maximum projection (MIP) image of the 3D-MLE reconstruction. Figure 22B shows the MIP image of the rFRC volume from Figure 22A. Figure 22C shows the corresponding magnified horizontal sections of the 3D-MLE (left) and rFRC volume (right) from the white box in Figure 22A. Furthermore, in the experimental configuration, it was found that the most accurate plane at the reconstruction depth of microtubules (Figure 21D) and actin filaments (Figures 23A-23F) is located at the focal point.Figure 23A shows the maximum projection (MIP) image of TIRF (COS-7 cells labeled with Alexa Fluor 647-phalloidin). Figure 23B shows the 3D-MLE reconstruction in a darkly colored diagram. Figure 23C shows the horizontal cross-section of the 3D-MLE reconstruction at z position -50 nm. Figure 23D shows the corresponding rFRC map for Figure 23C. Figure 23E shows the horizontal cross-section of the 3D-MLE reconstruction at z position +300 nm. Figure 23F shows the corresponding rFRC map for Figure 23E.

[0290] The Hessian denoising algorithm (Hessian SIM) can be used over conventional Wiener SIM to decouple random discontinuous artifacts from the actual structure (see Figures 21E and 29A-29I). Figure 29A shows an image of Wiener SIM. Figure 29B shows the corresponding rFRC map. Figure 29C shows the corresponding RSM. Figure 29D shows an image of Hessian SIM. Figure 29E shows the corresponding rFRC map. Figure 29F shows the corresponding RSM. Figure 29G shows TIRF images of Wiener SIM and Hessian SIM. Figure 29H shows a TIRF image convolved with a large Gaussian kernel and encoded with an inverted sJet colormap. Figure 29I shows a combined image of PANEL (light gray channel) and Wiener SIM (dark gray channel). However, because conversion to TIRF mode is required, the RSM may lose sight of such artifacts, as can be seen from Figure 21E where the RSE value remains 0.27. Conversely, when using the rFRC map, it is possible to detect the improvement of the Hessian SIM compared to the conventional Wiener SIM, as can be seen in Figure 21F, where the corresponding rFRC value improves significantly from 1.36 to 1.24.

[0291] Richardson-Lucy (RL) deconvolution has been actively studied for many reasons, including its potential to improve the resolution and contrast of RAW images. However, conventional RL algorithms can generate artifacts when performing excessive iterations, severely limiting their applications. In typical uses of RL, tedious visual inspection is required to determine the optimal number of iterations. Here, to verify the readout of rFRC values ​​that leads to such a determination of RL iteration count, RL is applied to process TIRF images (Figure 21G, Figures 30A-30I), and then the associated rFRC values ​​are calculated for each iteration case (Figure 21H, and the right panel of Figure 21J). Figure 30A shows the TIRF results. Figures 30B and 30C show the RL deconvolution results for 80 and 200 iterations, respectively. Figure 30D shows the corresponding TIRF-SIM images. Figures 30E and 30F show the rFRC maps for Figures 30B and 30C. Figure 30G shows a TIRF image convolved with a large Gaussian kernel and color-coded with an inverted sJet colormap. The inverted irradiation intensity map (Figure 30G) is proportional to the rFRC map (Figure 30E), indicating that the degree of error in the RLD results is highly correlated with the SNR. Figures 30H and 30I show the panels for Figures 30B and 30C. Interestingly, when the corresponding TIRF-SIM image is compared with the curve distribution of the peak signal-to-noise ratio (PSNR in the left panel of Figure 21J) used as the ground truth, the curve of the rFRC value was found to exhibit a similar distribution (quadratic form, minimum value appearing at 80 iterations). In contrast, the curve of the resolution-scaled Pearson coefficient (RSP) clearly did not exhibit such a useful distribution (middle panel of Figure 21J). As shown in Figure 21I, RL deconvolution with 200 iterations produced snowflake-like artifacts, indicated by the white arrows, which were verified to be absent in the referenced TIRF-SIM image. Overall comparison demonstrated that RL with 80 iterations optimally improved image contrast while minimizing artifacts due to noise amplification.

[0292] In addition to typical fluorescence modalities, coherent computational imaging techniques, such as Fourier ptichography microscopy (FPM), can also be effectively evaluated by developing them using PANEL and fully realizing their practical potential (see methods in Figures 24A-24F). Figure 24A shows an image showing the simulated ground truth. Figure 24B shows a wide-field image of Figure 24A. Figure 24C shows an image showing the corresponding FPM reconstruction. Figure 24D shows the rFRC map of the FPM. Figure 24E shows an image showing the RSM of the FPM. Figure 24F shows an integrated image of PANEL (light gray channel) and FPM (dark gray channel). Furthermore, the proposed PANEL framework was extended to a single-frame version with the aim of accessing the image quality of modalities when limited to forming statistically independent image subsets such as 3D-SIM (see Figure 25). Figure 25A shows an image of the color-coded volume of HUVEC transfected with LifeAct-EGFP and imaged with a Nikon 3D-SIM microscope. Figures 25B-25C show the 3D-rendered rFRC maps displayed in the sJet jet colormap (Figure 25B) and after Otsu thresholding in the green channel (Figure 25C). Figures 25D-25F show the 3D-SIM image at an axial position of 500 nm (Figure 25D), the corresponding rFRC map (Figure 25E), and its PANEL result (Figure 25F). Figures 25G-25I show the 3D-SIM image at an axial position of 700 nm (Figure 25G), the corresponding rFRC map (Figure 25H), and its PANEL result (Figure 25I). As indicated by the white arrows (Figure 25I), it was found that large errors may exist in regions containing thick actin stress filaments (or strong stress filaments) that scatter and distort structured incident light. (Example 4)

[0293] PANEL validation through simulation using a learning-based approach Deep learning algorithms can learn effective representations that map high-dimensional data into output arrays. However, these mappings often lack a rational explanation and are therefore blindly assumed to be accurate. Characterizing the uncertainty of corresponding network predictions and representations is crucial for further profiling, especially in safety-critical applications. Generally, seeking uncertainty for such learning-based approaches requires significant modifications to existing learning procedures within the traditional Bayesian neural network framework, by learning weight distributions. However, a potential drawback of BNNs is that they are considerably more complex to apply and computationally expensive than the original learning-based techniques. To mitigate this dilemma, the concept of PANEL is intended to be adapted to learning-based models, allowing for the identification of subtle and irregular errors generated by black-box deep neural networks.

[0294] First, four types of simple structures are created and sparsely sampled (see Figure 26). Figure 26 shows an image illustrating a fully sparsely sampled simulation, where the rectangle (top) is used as the training dataset and the other geometric shapes (square, circle, triangle) (from top to bottom) are used as the test dataset. "Input" refers to a representative sparsely sampled input of the corresponding geometric shape, "Prediction 1" refers to the network prediction of "Input", "Ground Truth" refers to the state before sparse sampling, "GT+Prediction" refers to the combined image of the Ground Truth in the green channel and the prediction result in the red channel, "Prediction 1+Prediction 2" refers to the combined image of the two prediction results, and "Combined" refers to the combined image of the PANEL in the green channel and the prediction result in the white channel. The corresponding input images for "Prediction 1" and "Prediction 2" are sampled independently. To investigate the generalization effect, acquired structures that are sparsely sampled (as model input) or not sparsely sampled (as ground truth) are used as data pairs to verify the applicability of PANEL in a learning-based approach. Rectangular shapes were used as the training dataset, while squares, triangles, and circles were used as the test dataset. As expected, squares could be considered a subset of rectangles, and thus the network trained on rectangular shapes performed well on the square dataset. Interestingly, as can be seen from Figure 27A, when the network was faced with data not presented in the training set (out-of-distribution data), such as triangular and circular shapes, the returned results still resembled rectangular shapes. Figure 27A shows representative combined images (prediction 1: cyan channel, prediction 2: magenta channel) of rectangular, square, triangular, and circular structures, from left to right. Figure 27B shows an image showing the mean IoU value between predictions and ground truth. Figure 27C shows an image showing the mean IoU value between two predictions. Figure 27D shows an image showing the mean rFRC value between two predictions. Figure 27E shows images illustrating the corresponding results after pixel super-resolution ("PSR") (top left) and the undersampled input (bottom right).Figure 27F shows an image illustrating the ground truth ("GT") reference in Figure 27E. Figure 27G shows the rFRC map of the PSR results. Figure 27H shows an integrated image of the PANEL (green channel) and PSR results (gray channel). Figure 27I shows an image illustrating a typical result of the deconvolutional network. Figure 27J shows an image illustrating the corresponding ground truth ("GT") in Figure 27I. Figure 27K shows an enlarged view from the white boxes in Figures 27I and 27J. Figure 27L shows an integrated image of the PANEL (green channel) and deconvolution results (gray channel).

[0295] To validate PANEL's ability to detect such structural artifacts (model errors) introduced by deep learning frameworks, the same structure was sampled twice individually to generate two predictions (labeled Prediction 1 and Prediction 2 in Figure 27A). To directly evaluate the difference between the two predictions, a metric intersection was chosen instead of a set operation (IoU, also known as the Jacquard exponent). As can be seen from Figures 27B and 27C, the mean IoU values ​​between Prediction 1 and Prediction 2 showed the same distribution as between the ground truth and Prediction 1. Furthermore, Figure 27D clearly shows that the pattern of the averaged rFRC values ​​is remarkably similar to one of the IoU values, verifying that the proposed concept and rFRC metric have the ability to quantitatively analyze learning-based approaches.

[0296] Next, to verify PANEL in more detail, another simulation verification, namely pixel super-resolution (PSR in Figures 27E and 31A-31G), is also performed to evaluate its usability for learning-based applications. Figure 31A shows the result after pixel super-resolution (PSR) (top left) and the undersampled input (bottom right). Figure 31B shows a magnified view of the undersampled input in the white box of Figure 31A. Figure 31C shows the network prediction of Figure 31B. Figure 31D shows the corresponding ground truth ("GT") of Figure 31B. Figure 31E shows the rFRC map of Figure 31C. Figure 31F shows the estimated RSM of Figure 31C. Figure 31G shows an integrated image of PANEL (light gray channel) and PSR result (dark gray channel). In this case, the corresponding deep neural network was trained on data pairs where the RAW image was used as the ground truth and the corresponding downsampled version was considered the input image. Compared to the ground truth (Figure 27F), the rFRC map (Figure 27G) is thoroughly verified to accurately detect errors in the network prediction (Figure 27H) and provide a quantitative evaluation at the pixel level, as indicated by the white arrows where the artifact regions in question are located.

[0297] Additionally, for convenient applications, a special single-frame version of the rFRC computation strategy is also involved in learning-based applications. Another learning-based task, namely deconvolution, is used to evaluate such strategies (Figure 27I, Figures 32A-32J). Figure 32A shows the input image. Figure 32B shows the DNN result. Figure 32C shows the ground truth (GT) of Figure 32A. Figure 32D shows a magnified view from the white boxes of Figures 32A-32C. Figure 32E shows the RSM of Figure 32A. Figure 32F shows an integrated image of the PANEL (white) and DNN result (gray). Figure 32G shows the inverted SSIM maps of Figures 32B and 32C. Figure 32H shows the deconvolution of Figure 32B. Figures 32I and 32J show the rFRC maps of the DNN result from input images with 1% and 1.5% additive Gaussian noise. TN represents a true negative, FN represents a false negative, and FP represents a false positive. Intuitively, in contrast to the ground truth (Figures 27J-27K), the single-frame strategy also demonstrated superior quantitative performance for accessing image quality, as indicated by the white arrows (Figure 27L). (Example 5)

[0298] Reliability of PANEL in various learning-based imaging methods After analysis and consideration, the usability of PANEL is then applied to evaluate the performance of three typical open-source learning-based models, including artificial neural network acceleration PALM17 (ANNA-PALM in Figure 28A), content recognition image reconstruction (CARE in Figure 28E), and TIRF conversion to TIRF-SIM (TIRF2SIM in Figure 28I). Figure 28A shows an image showing the ANNA-PALM output (MLE result with 25 frames of tubulin as input). Figure 28B shows an image showing MLE reconstruction with a full 500 frames. Figure 28C shows the rFRC map of Figure 28A. Figure 28D shows an integrated image of PANEL (green channel) and ANNA-PALM result (gray channel). Figure 28E shows an image showing the CARE output of GFP-tagged microtubules in raw HeLa cells (raw TIRF image as input). Figure 28F shows an image showing CARE deconvolved to the original low-resolution scale (top) and its RSM (bottom). Figure 28G shows the corresponding TIRF image. Figure 28H shows the combined image of PANEL (green channel) and CARE results (gray channel). Figure 28I shows the TIRF2SIM result of CCP (gene-edited SUM159 cells expressing AP2-eGFP). Figure 28J shows the TIRF-SIM image. Figure 28K shows the image showing the TIRF input. Figure 28L shows the combined image of PANEL (green channel) and TIRF2SIM results (gray channel). Figure 28M shows the Noise2Noise result of EGFP-labeled Tg (sox10:megfp) zebrafish 2 days post-fertilization. Figure 28N shows the ground truth reference image generated by averaging 50 noise images with identical content. Figure 28O shows the image showing a representative noise input. Figure 28P shows the combined image of PANEL (green channel) and Noise2Noise results (gray channel).

[0299] In the case of ANNA-PALM, sparse MLE reconstruction (including 25 frames) is used as the input image for the network (Figures 28A, 33A-33I), and rFRC is used in combination with RSM for complementarity. Figures 33A-33C show MLE reconstruction with 25 frames (Figure 33A) and MLE reconstruction with a full 500 frames (Figure 33B), and the corresponding ANNA-PALM output (MLE result with 25 frames as input) (Figure 33C). Figures 33D-33I show the combined results of MLE (gray) and ANNA-PALM (white) (Figure 33D), PANEL results (Figure 33E), TIRF image (Figure 33F), deconvolutional ANNA-PALM (Figure 33G), RSM (Figure 33H), and rFRC map (Figure 33I). Compared to the ground truth high-density MLE reconstruction results (including the full 500 frames in Figure 28B), the rFRC map (white channel in Figures 28C and 28D) successfully partitions minute errors, as indicated by the cyan arrows. On the other hand, the RSM (white channel in Figure 28D) discovered large missing structures, as indicated by the white arrows. Based on these practical results, the integral PANEL map can effectively detect all types of errors at different scales included in the ANNA-PALM method.

[0300] In the CARE20 deconvolution application, we consider that the network output (Figure 28E) and input (Figure 28G) do not satisfy the Gaussian convolution hypothesis. In this case, the RSM cannot accurately convert the output to its low-resolution scale (Figure 28F, Figures 34A-34F), and possible false negatives are shown as purple circles. Figure 34A shows the input of the TIRF image. Figure 34B shows the CARE prediction from Figure 34A. Figure 34C shows the rFRC result color-coded in full color. Figure 34D shows the combined image of the PANEL (pale gray channel) and the CARE result (white channel). Figure 34E shows the CARE prediction deconvolved by the estimated RSF. Figure 34F shows the RSM result from Figure 34B. RSF is the resolution scaling function. RSM is the resolution scaling error map. Conversely, obvious non-biological structures occurring near the field of view boundary, as indicated by the cyan circles, are effectively detected by the rFRC map (green channel in Figure 28H). Applying the single-frame rFRC calculation strategy, it can be seen that the TIRF2SIM19 application (Figures 28I-28L) can also be evaluated by PANEL. As shown in the insets in Figures 28I-28K, two adjacent CCPs in TIRF-SIM (indicated by the two green circles in Figure 28J) are incorrectly reconstructed by the network into a single large CCP (indicated by the single white circle in Figure 28I), and the rFRC map approach (white channel in Figure 28L) accurately divides such refined errors.

[0301] PANEL's performance will be further tested to access the performance of non-super-resolution learning-based applications, such as denoising tasks. Noise2Noise38 is a widely used unsupervised learning-based approach that has the advantage of denoising noisy images without sharp images. In such an unsupervised learning model, the network only sees pairs of noisy images during training, and two images with independent noise share identical detail content; in other words, one image is used as input and the other as output. In this part, 59 fluorescence microscopy denoising (FMD) datasets were used to train the Noise2Noise network (see methods in Figures 28M-28P and 35A-35E), and RSM is removed from the PANEL framework in this evaluation example because it does not satisfy the hypothesis. Reconstruction and evaluation of Figures 28M to 28P clearly show that Noise2Noise works effectively except for the areas highlighted in the insets of Figures 28M to 28P (suspicious areas are indicated by white arrows), demonstrating that the rFRC map (white channel in Figure 28P) can accurately detect this type of error. Figure 35A shows the results after Noise2Noise (N2N) (top left) and the noise input (bottom right, Noise 1). Figure 35B shows the Noise2Noise result from the white box in Figure 35A. Figure 35C shows a reference image averaged from 50 noise images with identical content. Figure 35D shows a combined image of the PANEL (light gray channel) and the Noise2Noise result (dark gray channel). Figure 35E shows the rFRC map in Figure 35B.

[0302] As described above, a comprehensive, reliable, and universal quantitative mapping of errors at the super-resolution scale may be decisive as computational and learning-based super-resolution techniques emerge in bioimaging science, independent of arbitrary reference information. Tracing the quantitative error map down to the super-resolution scale allows for evaluation of reconstruction quality at the pixel level, enabling further design of computational processes such as STORM image fusion, adaptive low-pass filters, and automated iterative decision-making. Furthermore, the proposed rFRC map may also be an excellent option for accessing quality metrics against ground truth and estimating local resolution maps of images.

[0303] In principle, reconstruction errors in computational microscopy imaging can be broadly categorized into two types: model errors and data errors. The main cause of model errors is the distance between the artificially created estimated model and the real model in the physical world. In the case of learning-based microscopy, such a distance may stem from ignoring the network of out-of-distribution data. The main cause of data errors is the combined effect of noise conditions and sampling capabilities of hardware equipment such as sensors and cameras in the microscope system. Based on the theory of the corresponding model, model-related errors can be discovered and reduced in physical imaging systems through careful system calibration, and in learning-based applications, they can be explained by a specifically designed strategy and sufficient training data. On the other hand, data errors can be fundamentally model-free, are unavoidable with artificial system calibration methods, and are difficult to explain with larger training datasets. Taken together, it is suggested that in biological analysis, the estimation of such data errors may be more decisive. In some embodiments, with the aim of developing an absolutely model-free quantitative method, PANEL may be based on only one hypothesis: that the model for reconstruction is unbiased. This means that the proposed PANEL method can accurately detect data errors present in various methods, but it should also be noted that it is limited to estimating model errors.

[0304] In particular, considering learning-based applications, model error can be simply estimated by the inconsistency of ensembles that independently and repeatedly train models on the same dataset through multiple random initialization and optimization processes. Furthermore, because it is a purely data-driven approach that learns representations of training data, model error and data error in such learning-based applications are not necessarily mutually exclusive. As shown in Figure 27A, the PANEL framework demonstrates the potential to detect both model error and data error. Model error from predictions of out-of-distribution test samples (rectangles: training data, triangles / circles: test data) can be effectively detected by the PANEL concept and rFRC metrics (Figures 27B-27D). Alternatively, data uncertainty and model uncertainty can also be estimated by applying rFRC maps to twin predictions from two inputs (from two data samplings) and two models (from two network trainings), respectively.

[0305] The PANEL method, as a model-independent and reference-slot metric, can segment the local reconstruction quality of images from various modalities without requiring additional prior knowledge. PANEL may focus more on detecting data error than model error. However, when considering data-driven approaches, data error may generally be intertwined with a portion of model error, leading PANEL to evaluate model error to some extent. With careful consideration of the capabilities and boundaries of the corresponding approaches, PANEL may provide a universal and reliable local reconstruction quality assessment framework, which may be useful not only for image-based biological profiling but also for further advancements in the rapidly developing field of computational microscopy imaging.

[0306] Examples 6 to 11 below describe new insights and extended applications of PANEL. (Example 6)

[0307] Errors in SMLM, SIM, and deconvolution Errors in SMLM: The accuracy of SMLM was found to be largely related to the molecular activation density in each frame. In the case of a small field of view, uniform irradiation (uniform activation density) can be assumed, and errors are more likely to occur in complex structures, such as common assemblies of filaments (Figures 18A to 18L). In the case of a large field of view, since the molecular activation density is related to the irradiation intensity (the higher the intensity, the lower the activation density), errors are more likely to occur in the low-intensity irradiation region (Figures 17G, 17H, and 17P).

[0308] Errors in SIM and deconvolution: SIM and deconvolution cannot detect high-frequency errors (smaller than the resolution), such as artifacts like snowflakes. It was found that the magnitude of errors in SIM and deconvolution is related to the emission intensity of the fluorescence signal (Figures 29A-29I, 30A-30I), and the quality of the reconstruction results of SIM and deconvolution is related to the SNR of the image, and it is reasonable that the SNR is somehow proportional to the magnitude of the signal. (Example 7)

[0309] Adaptive low-pass filter based on rFRC FRC determines the high-reliability cutoff frequency (COF) of an image, indicating that noise and errors are concentrated in frequency components above the cutoff frequency. Since rFRC calculates local cutoff frequencies in different regions of the image, here we apply local cutoff frequencies to different block box regions of the entire image to adaptively apply a low-pass filter.

[0310]

number

[0311] Here, I x,y This represents a subset image of the input image with a central pixel at spatial position (x,y), and OTF(F x,y ) is the cutoff frequency F x,yThe optical transfer function (OTF) has a value of 0 outside the cutoff frequency and 1 inside the cutoff frequency. When considering Richardson-Lucy deconvolution (RLD), the image quality of the reconstruction result is highly related to the corresponding local SNR, so such reconstruction results usually have a spatially variable cutoff frequency. Figure 36A shows a simulated wide-field image. Figure 36B shows the image after running RLD on the simulated wide-field (WF) image of Figure 36A for 500 iterations. Figure 36C shows the workflow of adaptive filtering on the image, with the filter block size set to 64×64 pixels and the overlap between adjacent blocks set to 4 pixels. Figure 36D shows the image after running a global estimated cutoff frequency filter on the image of Figure 36C. Figure 36E shows the image after running an adaptive local filter on the image of Figure 36A. Figure 36F shows the ground truth (GT) image of Figure 36A. The ground truth image is convolved with PSF (FWHM=240nm), involving Poisson noise and 10% Gaussian noise. A global FRC filter may not achieve optimal results (see SSIM=0.32, PSNR=19.30 in Figure 36D), and in comparison, the designed adaptive rFRC filter is a superior choice for filtering the image after Richardson-Lucy deconvolution (see SSIM=0.42, PSNR=21.90 in Figure 36E).

[0312] Global FRC values ​​were used to estimate the OTF of the entire image, and such estimated OTF was applied to blind Richardson-Lucy deconvolution. However, FRC can estimate the effective OTF (reliable frequency components) rather than the intrinsic resolution of the system. In other words, FRC may be heavily influenced by noise amplitude. If the noise amplitude is large, frequency components will become dominant even within the OTF, which can affect FRC's estimation of the system's intrinsic resolution (higher resolution). RLD requires static OTF instead of the effective OTF of the system. Thus, this behavior introduces the natural assumption that the image processed by RLD is under a sufficiently high SNR. (Example 8)

[0313] rFRC-based SMLM fusion By developing such rFRC quality metrics, different localization results may be merged according to the weights of the rFRC maps to combine the respective advantages of different models.

[0314]

number

[0315] Here, L n This is the result of the nth localization model, and G(σ) represents a Gaussian kernel with a standard variance of σ. max(F 1~n ) is the maximum FRC value of N localization results,

number

[0316] Determining the number of iterations for deconvolution using rFRC The optimal number of iterations for Richardson-Lucy deconvolution (RLD) is generally determined by manual inspection, observing the reconstruction results and convergence. However, manual inspection is usually cumbersome and limits automated image analysis. On the other hand, under low SNR conditions, RLD may not converge, in which case it is necessary to stop the iterations early (before convergence) to avoid excessive sharpness of structures and the occurrence of undesirable artifacts. The reported rFRC is used to estimate a local reliable cutoff frequency and automatically determine the optimal number of iterations. Specifically, the optimal iterations are those with the smallest rFRC value, as the goal is to obtain a more reliable cutoff frequency equal to the rFRC value, which is smaller with higher resolution and RLD processing.

[0317] Images acquired by TIRF and TIRF-SIM are used as data pairs as shown in Figures 21G-21J and 30A-30I. The TIRF-SIM image is used as the ground truth for the RLD on the TIRF image. As can be seen from Figure 21J, as the iterations progress, the PSNR curve is expressed in quadratic form, indicating that a higher number of iterations does not necessarily yield better results. In fact, excessive iterations can induce artifacts (white arrows in Figures 21I and 30B-30D) and lower the PSNR value. In this embodiment, considering the optimal PSNR, it is found that "80" iterations is the optimal number of iterations. Observations for "80" and "200" iterations in Figures 30B and 30C show that "80" iterations improve contrast without inducing artifacts, while "200" iterations generate snowflake-like artifacts (indicated by white arrows), which can be verified by the referenced TIRF-SIM image (Figure 30D). Furthermore, the rFRC value in Figure 21J can determine the optimal number of iterations, and its curve is identical to the PSNR curve calculated for ground truth. Conversely, the resolution-scaled Pearson coefficient (RSP) metric could not represent such a useful distribution. (Example 10)

[0318] rFRC as an error mapping method for data with ground truth rFRC was initially proposed for mapping errors in the absence of ground truth in actual bioimaging applications, and can be considered a superior option for error mapping compared to SSIM for data with ground truth, allowing for more rational evaluation.

[0319] As can be seen from Figures 39A to 39F, filaments were generated at different distances and convolved in a wide-field PSF (NA=1.4). Gradually increasing noise (along the arrow in Figure 39B) was involved in the image in Figure 39B. The resulting image was acquired using Richardson-Lucy deconvolution, as shown in Figure 39C. Compared to the ground truth shown in Figure 39A, the largest errors appear in the white boxes in Figure 39B, and the rFRC map was found to accurately detect such errors, as can be seen from Figures 39D and 30E. Conversely, as shown in Figure 39F, spatial distance estimation approaches such as SSIM induced high false negatives, making it difficult to distinguish or even obscure true negatives. It should be noted that the rFRC maps formed by Reconstruction 1 and Reconstruction 2, or Reconstruction 1 and the ground truth, are in perfect agreement, demonstrating that PANEL can find errors without requiring the ground truth. Figure 39A shows the ground truth. Figure 39B shows the image with gradually increasing noise. Figure 39C shows the reconstructed image. Figure 39D shows the rFRC maps generated based on Reconstruction 1 and Reconstruction 2. Figure 39E shows the rFRC maps generated based on Reconstruction 1 and the ground truth. Figure 39F shows the inverted SSIM. (Example 11)

[0320] Resolution as an additional metric In addition to detecting the precise degree of error in reconstruction, rFRC maps also provide a quantitative metric: resolution. To construct an accurate and practical quantification tool, three key contributions were realized: the absence of false negatives due to background, the sJet color map which is better suited to visualization, and pixel-level resolution mapping. For this reason, rFRC maps can be used not only for reconstruction results but also as an excellent option for evaluating the resolution of raw data acquired by photoelectric detectors, such as stimulated emission suppression (STED) microscopy. Due to the absence of false negatives due to background, the average resolution of STED (Figure 40A) is clearly "92 nm" in the rFRC map in Figure 40C, which is more reasonable than "146 nm" in the FRC map in Figure 40B. Furthermore, the "sJet" color map can visualize the resolution distribution with higher contrast compared to the conventionally used "SQUIRREL-FRC" color map. Figure 40A shows SiR-tubulin-labeled microtubules observed under gSTED. Figure 40B shows a 64-pixel block-size FRC map color-coded with the SQUIRREL-FRC colormap. Figure 40C shows an rFRC map color-coded with the sJet colormap.

[0321] Based on a 3σ curve rather than a 1 / 7 threshold, rFRC is more stable and accurate for local FRC resolution mapping to determine an effective cutoff frequency. As shown in Figures 41A and 41B, the "64nm" resolution determined by the 1 / 7 threshold in the region indicated by the white arrow in the inset appears overconfident for such poorly resolving filaments, while the "160nm" resolution obtained by the 3σ curve was clearly milder and more reasonable. While the so-called 1 / 7 threshold may be considered unstable for local resolution mapping, it should be emphasized that the PANELJ FIJI / ImageJ plugin provides the features of both 1 / 7 threshold and 3σ curve-based resolution mapping for further potential applications. This is because the 1 / 7 threshold is common and widely used, thus providing the same local resolution mapping for a wider user community.

[0322] Although the fixed-value threshold method has been argued to be based on flawed statistical assumptions, the 1 / 7 threshold has been widely applied in the field of super-resolution. Interestingly, regardless of the criterion, the final resolution obtained in SMLM and stimulated emission suppression microscopy (STED) experiments is nearly identical. The FRC curve mapped the local SR error calculated and included in the reconstruction between small image blocks. As shown in Figures 40C and 40D, for small images, the 1 / 7 threshold becomes overconfident, i.e., smaller than all correlation values ​​in the FRC curve, making it impossible to determine the corresponding image cutoff frequency (cross in Figure 40C). In contrast, for large images, the 1 / 7 threshold yielded similar results to the 3σ curve. This indicates that the 1 / 7 threshold is suitable for determining conventional global resolution but not for determining local cutoff frequencies (local resolution). On the other hand, instead of avoiding conservative threshold selections such as the 1 / 7 threshold in the resolution determination described above, a milder threshold for error mapping is intended to reduce false positives. Therefore, considering robustness and accuracy for small image blocks, the 3σ curve is used as the threshold for calculating the FRC value of PANEL.

[0323] [Table 1]

[0324] [Table 2]

[0325] Having thus explained the basic concepts, it will be clear to those skilled in the art, after reading the detailed disclosure herein, that the aforementioned detailed disclosure is intended to be merely an example and not limiting. While not explicitly stated herein, various changes, improvements, and modifications are possible and intended for those skilled in the art. These changes, improvements, and modifications are intended to be suggested by this disclosure and fall within the spirit and scope of the exemplary embodiments herein.

[0326] Furthermore, specific terms are used to describe embodiments of this disclosure. For example, the terms “one embodiment,” “one embodiment,” and / or “several embodiments” mean that certain features, structures, or characteristics described in relation to an embodiment are included in at least one embodiment of this disclosure. Therefore, it should be emphasized and understood that two or more references to “one embodiment,” “one embodiment,” or “alternative embodiment” in various parts of this specification do not necessarily all refer to the same embodiment. Furthermore, certain features, structures, or characteristics can be suitably combined in one or more embodiments of this disclosure.

[0327] Furthermore, as will be understood by those skilled in the art, aspects of the disclosure may be illustrated and described herein in any of several patentable types or situations, including any novel and useful process, machine, manufacturing process, or composition of a substance, or any novel and useful improvement thereof. Thus, aspects of the disclosure may be implemented entirely by hardware or software (including firmware, resident software, microcode, etc.), or by a combination of software and hardware, all of which may be generally referred to herein as “units,” “modules,” or “systems.” Furthermore, aspects of the disclosure may take the form of computer program products implemented on one or more computer-readable media in which computer-readable program code is embodied.

[0328] A computer-readable signal medium may include, for example, a propagating data signal in which computer-readable program code is embodied, as part of a baseband or carrier wave. Such a propagating signal may take any of various forms, including electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium but is used by or in connection with an instruction execution system, device, or apparatus, and is capable of communicating, propagating, or transmitting a program. The program code embodied in the computer-readable signal medium may be transmitted using any suitable medium, including wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0329] Computer program code for performing the actions for the aspects of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; traditional procedural programming languages ​​such as the C programming language, Visual Basic, Fortran 2103, Perl, COBOL 2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. The program code may be executed entirely on a user computer, partially on a user computer as a standalone software package, partially on a user computer and partially on a remote computer, or fully on a remote computer or server. In the latter case, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), and this connection may be to an external computer (for example, via the Internet using an Internet Service Provider) or in a cloud computing environment, or provided as a service such as Software as a Service (SaaS).

[0330] Furthermore, the order of the described processing elements or sequences, or the use of numbers, letters, or other names, is not intended to limit the processes and methods described in the claims to any order other than those that may be specified in the claims. While the above disclosure is discussed through various specific examples that are currently considered to be various useful embodiments of the disclosure, such details are for illustrative purposes only, and it should be understood that the attached claims are not limited to the disclosed embodiments, but rather are intended to encompass variations and equivalent arrangements within the spirit and scope of the disclosed embodiments. For example, the implementation of the various components described above may be embodied in hardware devices, but may also be implemented solely as software implementations, such as existing servers or portable devices.

[0331] Similarly, in the foregoing description of embodiments of this disclosure, various features may be summarized in a single embodiment, drawing, or description in order to simplify the disclosure and aid in understanding one or more of the various embodiments. However, the methods of this disclosure should not be interpreted as reflecting an intention that the subject matter described in the claims requires features beyond those expressly described in each claim. Rather, embodiments of this disclosure do not include all the features of the embodiments described above.

[0332] In some embodiments, numerical values ​​representing quantities or characteristics used to describe specific embodiments of the present application are understood to be modified in some cases by the terms “approximately,” “approximately,” or “substantial.” For example, “approximately,” “approximately,” or “substantial” may indicate a variation of ±20% of the described value unless otherwise specified. Thus, in some embodiments, the numerical parameters described herein and in the appended claims are approximations that may vary depending on the desired characteristics obtained by a particular embodiment. In some embodiments, numerical parameters should be interpreted by applying common rounding techniques in light of the number of significant figures reported. Although the numerical ranges and parameters defining a wide range of some embodiments of the present application are approximations, the numerical values ​​defined in specific examples are reported as accurately as possible.

[0333] Each of the patents, patent applications, published patent gazettes, and other materials referenced herein, such as articles, books, specifications, publications, documents, and objects, is incorporated herein in its entirety for all purposes by this reference, except for any related application history documents, any of which are inconsistent with or contradictory to this document, or which may have a limiting effect on the broadest scope of claims that are now or later related to this document. For example, if there is any inconsistency or conflict between the use of descriptions, definitions, and / or terms relating to any of the incorporated materials and the use of descriptions, definitions, and / or terms relating to this document, the use of descriptions, definitions, and / or terms in this document shall prevail.

[0334] Finally, it should be understood that the embodiments of the Application disclosed herein are illustrative of the principles of the embodiments. Other variations that may be used may be within the scope of the Application. Therefore, alternative configurations of the embodiments of the Application may be used, without limitation, as examples, in accordance with the teachings herein. Thus, the embodiments of the Application are not limited to those specifically illustrated and described. [Explanation of symbols]

[0335] 100 Image Processing Systems 110 Image acquisition device 120 Networks 130 devices 131 Mobile devices 132 Tablet Computers 133 Laptop Computers 140 Processing Units 150 Storage device 200 Computing equipment 210 processors 220 storage 230 I / O Section 240 communication ports 300 mobile devices 310 Communication Unit 320 display units 330 Graphics Processing Unit 340 Central Processing Unit 350 I / O section 360 memory 361 Operating Systems 362 applications 370 memory units 402 Retrieval Module 404 Expansion Module 406 Rolling Module 408 Map Generation Module 410 Filtering Module 412 Display Module 414 Image Quality Determination Module 416 Disassembly Module 418 Neural Network Processing Modules

Claims

1. A method for image processing, which is performed on at least one machine, each having at least one processor and at least one memory device, A step of acquiring a first image and a second image associated with the same object, wherein the first image and the second image are two consecutive images acquired from two image acquisitions of the same object, or images acquired by decomposing an image acquired from one image acquisition of the same object; A step of determining a plurality of first blocks of the first image by performing a rolling motion on the first image, and determining a plurality of second blocks of the second image by performing a rolling motion on the second image, wherein the plurality of second blocks and the plurality of first blocks correspond one-to-one. A step of determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks, A step of generating a first target map associated with the first image and the second image based on the plurality of first characteristic values, A method for image processing, including...

2. The rolling motion of the first image is performed by sliding a window on the first image and / or by skipping a predetermined number of pixels on the first image. The method according to claim 1, wherein the rolling motion of the second image is performed by sliding a window on the second image and skipping a predetermined number of pixels on the second image.

3. The method according to claim 1 or 2, wherein at least one of the first characteristic values ​​is determined based on the correlation between one of the plurality of first blocks and its corresponding second block.

4. The at least one first characteristic value is, A step of determining a first intermediate image in the frequency domain based on the first block, A step of determining a second intermediate image in the frequency domain based on the second block, A step of determining a target frequency based on the first intermediate image and the second intermediate image, A step of determining the at least one first characteristic value based on the target frequency, The method according to claim 3, determined by a process including the following:

5. The step of determining the at least one first characteristic value based on the target frequency is: The step of specifying the target frequency as the at least one first characteristic value, or The step of determining the at least one first characteristic value based on the reciprocal of the target frequency. The method according to claim 4, including the method described in claim 4.

6. The step of determining the target frequency based on the first intermediate image and the second intermediate image is: A step of determining a first relationship between a plurality of correlation values ​​and a plurality of frequencies based on the first intermediate image and the second intermediate image, A step of determining a second relationship between the plurality of correlation values ​​and the plurality of frequencies based on a predetermined function, A step of determining the target frequency based on the first relationship and the second relationship, wherein the target frequency corresponds to the intersection of the first curve representing the first relationship and the second curve representing the second relationship. The method according to claim 4 or 5, including the method described in claim 4 or 5.

7. The step of determining a first relationship between a plurality of correlation values ​​and a plurality of frequencies based on the first intermediate image and the second intermediate image is: The step of determining the plurality of correlation values, each corresponding to one of the plurality of frequencies. Including, each of the aforementioned correlation values ​​is A step of determining a first ring image from the first intermediate image, based on the frequency, which includes pixels in a ring having the same diameter as the frequency, A step of determining a second ring image from the second intermediate image, based on the frequency, which includes pixels in a ring having the same diameter as the frequency, The steps include determining each of the correlation values ​​based on the first ring image and the second ring image, The method according to claim 6, determined according to a process including a process.

8. The step of obtaining a first image and a second image associated with the same object is: Steps to obtain the initial image, The steps include: decomposing the initial image into the first image, the second image, the third image, and the fourth image; The method according to any one of claims 1 to 7, including

9. The steps include determining a plurality of third blocks in the third image and a plurality of fourth blocks in the fourth image that correspond one-to-one, The steps include determining a plurality of second characteristic values ​​based on the plurality of third blocks and the plurality of fourth blocks, A step of generating a second target map associated with the third image and the fourth image based on the plurality of second characteristic values, A step of generating a third target map based on the first target map and the second target map, The method according to claim 8, further comprising:

10. The step of generating a third target map based on the first target map and the second target map is: A step of averaging the first target map and the second target map to generate the third target map, The steps include resizing the third target map to the size of the initial image, The method according to claim 9, including the method described in claim 9.

11. The step of determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks is: For each of the first blocks or each of the second blocks, A step of determining the average value of the pixels in the central region of each of the first block or each of the second block, The steps include comparing the average value with a threshold, In response to a determination that the average value is greater than the threshold, a first characteristic value is determined based on a first intermediate image corresponding to each of the first blocks and a second intermediate image corresponding to each of the second blocks, or in response to a determination that the average value is less than the threshold, a first characteristic value is determined based on a predetermined value. The method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

12. The steps include determining a reference map associated with the first image, The steps include determining a fourth target map by fusing the first target map and the reference map, The method according to any one of claims 1 to 11, further comprising:

13. The step of determining the reference map associated with the first image is: The steps include obtaining a reference image having a lower resolution than the first image, The steps include determining a convolution kernel based on the first image and the reference image, A step of determining an intermediate image based on the first image and the convolution kernel, The steps include determining the reference map based on the difference between the intermediate image and the reference image, The method according to claim 12, including the method described in claim 12.

14. A system for image processing, An acquisition module configured to acquire a first image and a second image associated with the same object, wherein the first image and the second image are two consecutive images acquired from two image acquisitions of the same object, or images acquired by decomposing an image acquired from one image acquisition of the same object; A rolling module configured to determine a plurality of first blocks of a first image by performing a rolling operation on the first image, and to determine a plurality of second blocks of a second image by performing a rolling operation on the second image, wherein the plurality of second blocks and the plurality of first blocks correspond one-to-one. It is a map generation module, Based on the plurality of first blocks and the plurality of second blocks, a plurality of first characteristic values ​​are determined. Based on the plurality of first characteristic values, a first target map associated with the first image and the second image is generated. A map generation module configured as follows: A system for image processing, including...

15. A non-temporary computer-readable medium comprising at least one instruction set for image processing, wherein the at least one instruction set is executed by one or more processors of an arithmetic unit. A step of acquiring a first image and a second image associated with the same object, wherein the first image and the second image are two consecutive images acquired from two image acquisitions of the same object, or images acquired by decomposing an image acquired from one image acquisition of the same object; A step of determining a plurality of first blocks of the first image by performing a rolling motion on the first image, and determining a plurality of second blocks of the second image by performing a rolling motion on the second image, wherein the plurality of second blocks and the plurality of first blocks correspond one-to-one. A step of determining a plurality of first characteristic values ​​based on the plurality of first blocks and the plurality of second blocks, A step of generating a first target map associated with the first image and the second image based on the plurality of first characteristic values, A non-temporary computer-readable medium that causes the computing device to execute a method including the above.

Images