Slide-free histological imaging method and system

Abstract

A computer-implemented method (530), system (100, 201), and computer-readable medium for generating a slide-free histology image in the form of a reconstructed ultraviolet-based autofluorescence microscopy (UV-AtuoM) image from a plurality of speckle-illuminated low-resolution images. A computer-implemented method (1150), system (100, 201), and computer-readable medium for generating a slide-free histology image in the form of a pseudo-hematoxylin and eosin (H&E)-stained image (1120) from a grayscale input image, such as a UV-AutoM image or an ultraviolet-based photoacoustic microscopy (UV-PAM) image (1110).

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 973,101, filed on September 19, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to a slide-free histological imaging method and system. Background Art

[0004] Histological examination remains the gold standard for surgical margin assessment of malignant tumors. However, conventional histological analysis involves a lengthy and costly sample preparation process that produces toxic reagent waste, depletes small specimens, and extends the time to generate histopathology reports by hours to days. This lengthy and costly process involves formalin fixation and paraffin embedding (FFPE), followed by high-quality sectioning, staining, and mounting of the specimen on slides. These unavoidable steps take days to complete, resulting in delays in generating accurate diagnostic reports ranging from hours to days. Although intraoperative frozen sections offer a faster alternative to FFPE histology by freezing fresh tissue prior to physical sectioning, intraoperative frozen sections still require 20 to 30 minutes of preparation and turnaround time. In addition, frozen sectioned specimens have inherent problems with freezing artifacts, especially when dealing with lipid-rich tissues, which can lead to intraoperative misinterpretations and diagnostic pitfalls.

[0005] The huge demand of histopathology has inspired many efforts to realize the rapid and non-invasive diagnosis of unstained fresh tissue.Some microscopy techniques for non-sectioned tissue imaging, including ultraviolet (UV) surface excitation microscopy, confocal laser scanning microscopy and light sheet microscopy, have reduced the heavy task and processing cost involved in preparing a large number of slides in conventional FFPE histology.However, these methods all require specific fluorescent labels to improve molecular specificity.Although fluorescence imaging is undoubtedly very efficient for providing morphological and kinetic information of different biomolecules in relevant cells, it will lead to the use of exogenous labels or gene transfection, which will interfere with cell metabolism and have an adverse effect on subsequent clinical implementation.In addition, long-term monitoring of cells with fluorescent labels will lead to phototoxicity to cells and photobleaching of the fluorophore itself.

[0006] Stimulated Raman scattering (SRS) and coherent anti-Stokes Raman scattering (CARS), which characterize structure through the intrinsic molecular vibrations of specific chemical bonds, offer label-free alternatives for examining C-H stretching in lipid-rich structures. Furthermore, nonlinear processes originating at noncentrosymmetric interfaces, including second harmonic generation (SHG), third harmonic generation (THG), and their combined modes, have shown great potential for intrinsic characterization of collagen and microtubule structures. However, these methods require high-power ultrafast lasers to maintain detection sensitivity and molecular contrast, which may not be readily available in most cases. Spectral confocal reflectance microscopy allows label-free, high-resolution in vivo imaging of myelinated axons, but due to its low molecular specificity, confocal microscopy with tunable wavelength capabilities remains a requirement. Quantitative phase imaging techniques also offer great potential for rapid refractive index mapping by measuring phase changes in unstained samples. However, these techniques are mostly integrated into delivery systems and are severely limited by sample thickness. Furthermore, reflectance-based imaging techniques, such as optical coherence tomography, have been translated into intraoperative diagnostic tools for label-free imaging of human breast tissue; however, they are not designed to achieve subcellular resolution and are not suitable for detecting molecular targets desired in standard-of-care clinical pathology. Summary of the Invention

[0007] It is an object of the present invention to address one or more of the disadvantages above or described herein, or to at least provide a useful alternative.

[0008] In a first aspect, a computer-implemented method for generating a pseudo hematoxylin and eosin (H&E) stained image is provided, wherein the method comprises: receiving an input image, which is an ultraviolet-based autofluorescence microscopy (UV-AutoM) image or an ultraviolet-based photoacoustic microscopy (UV-PAM) image of an unlabeled sample, wherein the input image is a grayscale image; transforming the input image into a pseudo H&E stained image of the input image using a generative adversarial network; and outputting the pseudo H&E stained image.

[0009] In some embodiments, the generative adversarial network is a cycle-consistent generative adversarial network.

[0010] In certain embodiments, the method comprises training a generative adversarial network using unpaired input and an H&E stained image.

[0011] In certain embodiments, the generative adversarial network includes four deep convolutional neural networks, including: a first generator deep convolutional neural network configured to transform an input image into a generated H&E image; a second generator deep convolutional neural network configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator deep convolutional neural network configured to distinguish between the H&E images of a training set and the generated H&E images generated by the first generator deep convolutional neural network; and a second discriminator deep convolutional neural network configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

[0012] In some embodiments, the first and second generator deep convolutional neural networks are ResNet-based or U-Net-based generator networks.

[0013] In certain embodiments, the first and second discriminator deep convolutional neural networks are PatchGAN discriminator networks.

[0014] In certain embodiments, an input image received in the form of a UV-PAM image is generated by: controlling a galvanometer scanner of a focusing assembly to focus ultraviolet light on a sample according to a scanning trajectory; receiving, via at least one transducer, photoacoustic waves emitted by the sample in response to the ultraviolet light; and generating a UV-PAM image based on the photoacoustic waves.

[0015] In some embodiments, the input image received as a UV-AutoM image is an estimated UV-AutoM image generated from a sequence of speckle-illuminated images captured according to a scanning trajectory, wherein the estimated UV-AutoM image has a higher resolution than each speckle-illuminated image in the sequence.

[0016] In certain embodiments, an estimated UV-AutoM image is generated by: a) initializing a high-resolution image object based on interpolating an average of a sequence of speckle-illuminated images; b) for each speckle-illuminated image in the sequence: i) generating an estimated speckle-illuminated image by computationally shifting the high-resolution image object to a specific position in the scan trajectory; ii) determining a filtered object-pattern composite in the frequency domain based on the estimated speckle-illuminated image in the frequency domain and an optical transfer function; iii) filtering the filtered object-pattern composite in the frequency domain based on the estimated speckle-illuminated image in the frequency domain, the corresponding captured speckle-illuminated image in the frequency domain, and the optical transfer function. and an optical transfer function in the frequency domain; iv) updating a high-resolution object based on the updated estimated speckle illumination image, the estimated speckle illumination image in the spatial domain, and the speckle pattern; v) updating the speckle pattern based on the updated estimated speckle illumination image, the estimated speckle illumination image, and the high-resolution image object; vi) applying Nesterov momentum acceleration to the high-resolution image object and the speckle pattern; and c) iteratively performing step b) until convergence is detected to reconstruct a high-resolution image object that is an estimated UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area.

[0017] In a second aspect, a computer system configured to generate a pseudo hematoxylin and eosin (H&E) stained image is provided, wherein the computer system includes one or more memories having executable instructions stored therein, and one or more processors, wherein the processor executes the executable instructions to cause the processor to: receive an input image, which is an ultraviolet autofluorescence microscopy (UV-AutoM) image or an ultraviolet photoacoustic microscopy (UV-PAM) image of an unlabeled sample, wherein the input image is a grayscale image; transform the input image into a pseudo H&E stained image of the input image using a generative adversarial network; and output the pseudo H&E stained image.

[0018] In some embodiments, the generative adversarial network is a cycle-consistent generative adversarial network.

[0019] In certain embodiments, the one or more processors are configured to train a generative adversarial network using unpaired input grayscale images and H&E stained images.

[0020] In certain embodiments, the generative adversarial network includes four deep convolutional neural networks, including: a first generator deep convolutional neural network configured to transform an input image into a generated H&E image; a second generator deep convolutional neural network configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator deep convolutional neural network configured to distinguish between the H&E images of a training set and the generated H&E images generated by the first generator deep convolutional neural network; and a second discriminator deep convolutional neural network configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

[0021] In some embodiments, the first and second generator deep convolutional neural networks are ResNet-based or U-Net-based generator networks.

[0022] In certain embodiments, the first and second discriminator deep convolutional neural networks are PatchGAN discriminator networks.

[0023] In certain embodiments, an input image received in the form of a UV-PAM image is generated by: controlling a galvanometer scanner of a focusing assembly to focus ultraviolet light on a sample according to a scanning trajectory; receiving, via at least one transducer, photoacoustic waves emitted by the sample in response to the ultraviolet light; and generating a UV-PAM image based on the photoacoustic waves.

[0024] In certain embodiments, an input image received in the form of an estimated UV-AutoM image is generated from a sequence of speckle-illuminated images captured according to a scanning trajectory, wherein the estimated UV-AutoM has a higher resolution than each speckle-illuminated image in the sequence.

[0025] In certain embodiments, an estimated UV-AutoM image is generated by: a) initializing a high-resolution image object based on interpolating an average of a sequence of speckle-illuminated images; b) for each speckle-illuminated image in the sequence: i) generating an estimated speckle-illuminated image by computationally shifting the high-resolution image to a specific position in the scan trajectory; ii) determining a filtered object-pattern composite in the frequency domain based on the estimated speckle-illuminated image in the frequency domain and an optical transfer function; iii) generating a filtered object-pattern composite in the frequency domain based on the estimated speckle-illuminated image in the frequency domain, the corresponding captured speckle-illuminated image in the frequency domain, the filtered object-pattern composite in the frequency domain, and the optical transfer function. The invention relates to a method for reconstructing an estimated UV-AutoM image object having enhanced subcellular resolution over a centimeter-scale imaging area, the method comprising: determining an updated estimated speckle-illuminated image in the frequency domain based on a mathematical transfer function; iv) updating a high-resolution object based on the updated estimated speckle-illuminated image, the estimated speckle-illuminated image in the spatial domain, and the speckle pattern; v) updating the speckle pattern based on the updated estimated speckle-illuminated image, the estimated speckle-illuminated image, and the high-resolution image object; vi) applying Nesterov momentum acceleration to the high-resolution image object and the speckle pattern; and c) iteratively performing step b) until convergence is detected to reconstruct a high-resolution image object that is an estimated UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area.

[0026] In a third aspect, one or more non-transitory computer-readable media are provided, comprising executable instructions for configuring a computer system to generate a pseudo-hematoxylin and eosin (H&E) stained image, wherein the computer system has one or more processors, wherein the one or more processors execute the executable instructions to configure the computer system to: receive an input image, which is an ultraviolet-based autofluorescence microscopy (UV-AutoM) image or an ultraviolet-based photoacoustic microscopy (UV-PAM) image of an unlabeled sample, wherein the input image is a grayscale image; transform the input image into a pseudo-H&E stained image of the input image using a generative adversarial network; and output the pseudo-H&E stained image.

[0027] In some embodiments, the generative adversarial network is a cycle-consistent generative adversarial network.

[0028] In certain embodiments, one or more processors execute executable instructions to configure the computer system to train a generative adversarial network using unpaired input grayscale images and H&E stained images.

[0029] In certain embodiments, the generative adversarial network includes four deep convolutional neural networks, including: a first generator deep convolutional neural network configured to transform an input image into a generated H&E image; a second generator deep convolutional neural network configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator deep convolutional neural network configured to distinguish between the H&E images of a training set and the generated H&E images generated by the first generator deep convolutional neural network; and a second discriminator deep convolutional neural network configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

[0030] In some embodiments, the first and second generator deep convolutional neural networks are ResNet-based or U-Net-based generator networks.

[0031] In certain embodiments, the first and second discriminator deep convolutional neural networks are PatchGAN discriminator networks.

[0032] In certain embodiments, an input image received in the form of a UV-PAM image is generated by: controlling a galvanometer scanner of a focusing assembly to focus ultraviolet light on a sample according to a scanning trajectory; receiving, via at least one transducer, photoacoustic waves emitted by the sample in response to the ultraviolet light; and generating a UV-PAM image based on the photoacoustic waves.

[0033] In certain embodiments, an input image received in the form of an estimated UV-AutoM image is generated from a sequence of speckle-illuminated images captured according to a scanning trajectory, wherein the estimated UV-AutoM has a higher resolution than each speckle-illuminated image in the sequence.

[0034] In certain embodiments, an estimated UV-AutoM image is generated by: a) initializing a high-resolution image object based on interpolating an average of a sequence of speckle-illuminated images; b) for each speckle-illuminated image in the sequence: i) generating an estimated speckle-illuminated image by computationally shifting the high-resolution image to a specific position in the scan trajectory; ii) determining a filtered object-pattern composite in the frequency domain based on the estimated speckle-illuminated image in the frequency domain and an optical transfer function; iii) generating a filtered object-pattern composite in the frequency domain based on the estimated speckle-illuminated image in the frequency domain, the corresponding captured speckle-illuminated image in the frequency domain, the filtered object-pattern composite in the frequency domain, and the optical transfer function. The invention relates to a method for reconstructing an estimated UV-AutoM image object having enhanced subcellular resolution over a centimeter-scale imaging area, the method comprising: determining an updated estimated speckle-illuminated image in the frequency domain based on a mathematical transfer function; iv) updating a high-resolution object based on the updated estimated speckle-illuminated image, the estimated speckle-illuminated image in the spatial domain, and the speckle pattern; v) updating the speckle pattern based on the updated estimated speckle-illuminated image, the estimated speckle-illuminated image, and the high-resolution image object; vi) applying Nesterov momentum acceleration to the high-resolution image object and the speckle pattern; and c) iteratively performing step b) until convergence is detected to reconstruct a high-resolution image object that is an estimated UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area.

[0035] Other aspects and embodiments will become apparent through the description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings.

[0037] Figure 1A and 1B is a schematic diagram of an example of a general-purpose computer system on which the various arrangements described herein may be implemented.

[0038] Figure 2A and 2B is a schematic diagram of an example of an embedded system on which the various arrangements described herein are implemented.

[0039] Figure 3 is a schematic diagram of an example UV-PAM system.

[0040] Figure 4A is an example of a UV-PAM image of a gold nanoparticle with a diameter of 200 nm, where the contour along the white dashed line is extracted and averaged.

[0041] Figure 4B is an example of the line profile of the average of four gold nanoparticles, where the FWHM (full width at half maximum) of the Gaussian fit (solid line) is about 613 nm, indicating Figure 3The lateral resolution of the UV-PAM system.

[0042] Figure 4C yes Figure 4A An example of the A-line signal at the center of the gold nanoparticle in FIG. 1 , where the FWHM of the envelope of the A-line signal is 38 ns, which corresponds to 58 μm, indicating Figure 3 The axial resolution of the UV-PAM system.

[0043] Figure 5A is a schematic diagram of an example UV-AutoM system.

[0044] Figure 5B and 5C is a flow chart representing an example computer-implemented method for reconstructing a UV-AutoM image from a sequence of speckle-illuminated low-resolution images.

[0045] Figure 5D is an example of pseudo-code representing a computer-implemented method for reconstructing a UV-AutoM image from a sequence of low-resolution images of speckle illumination.

[0046] Figure 6 A graphical representation of an example of an SI reconstruction method and system for reconstructing UV-AutoM images is shown.

[0047] Figure 7A is an example of a low-resolution UV-AutoM image captured by a 4X / 0.1NA objective.

[0048] Figure 7B is through Figure 5B and 5C Example of a high-resolution UV-AutoM image reconstructed using the proposed method.

[0049] Figure 7C are respectively Figure 7A and 7B Line profiles of the lines marked 700 and 710 in FIG.

[0050] Figure 8A and 8B Shown are examples of UV-AutoM images of two leaf samples with rough surfaces captured by a 4X / 0.1NA objective.

[0051] Figure 8C and 8D Shows the use Figure 5B and 5C Example of the method to reconstruct a high-resolution UV-AutoM image from multiple speckle-illuminated low-resolution images.

[0052] Figure 8E and 8FThe 10X / 0.3NA objective lens is used to Figure 8A and 8B High-resolution reference images of two leaf samples captured in .

[0053] Figure 9A UV-AutoM images of whole mouse brain (FFPE sections after dewaxing, 4 μm thickness), scale bar: 500 μm.

[0054] Figure 9B and 9C yes Figure 9A Magnified views of boxes 910 and 920 in FIG. 3 , each with a scale bar of 50 μm.

[0055] Figure 9D and 9E corresponds to Figure 9B and 9C Bright field H&E staining images of the cells, each with a 50 μm scale bar.

[0056] Figure 10A is an example image of a top view of a mouse brain.

[0057] Figure 10B and 10C are examples of reconstructed UV-AutoM images, each depicting a section through Figure 10A Cross-section of the mouse brain at lines AA and BB.

[0058] Figures 10D to 10H Shown Figure 10C High-pass quantum view of five functional areas of the mouse brain depicted in the reconstructed UV-AutoM image.

[0059] Figure 11A is a functional block diagram representing an example computer-implemented system for generating pseudo-stained histological images (ie, virtually stained histological images).

[0060] Figure 11B is a flow chart representing an example computer-implemented method 1150 for generating pseudo-hematoxylin and eosin (H&E) stained images.

[0061] Figure 12 This is a functional block diagram showing the detailed workflow of the forward loop and backward loop of Cycle-GAN.

[0062] Figure 13 11 is a functional block diagram of an example generator of the Cycle-GAN of FIG.

[0063] Figure 14A is used Figure 3An example of a grayscale UV-PAM image of a mouse brain slice generated by the system, which is provided as input to a trained Cycle-GAN configured according to Figures 11 to 13.

[0064] Figure 14B is an example of a virtually stained histology image generated as an output image by Cycle-GAN using a grayscale UV-PAM image as an input image.

[0065] Figure 14C was obtained using bright field microscopy after H&E staining. Figure 14A Example of histological images of the same sample imaged in .

[0066] Figure 15A An example of a UV-AutoM image of a 7 μm thick dewaxed FFPE mouse brain section.

[0067] Figure 15B It uses a GAN-based network Figure 15A A virtually stained version of the UV-AutoM image.

[0068] Figure 15C is relative to Figure 15A Brightfield H&E image of an imaged mouse brain section.

[0069] Figure 16A and 16B is the high-resolution UV-AutoM image reconstructed by SI, which is used Figure 5B and 5C The method was used to generate mouse brain samples with thicknesses of 100 μm and 200 μm, respectively.

[0070] Figure 16C and 16D are examples of virtually stained H&E images generated using the computer-implemented methods and systems of Figures 11 to 13.

[0071] Figure 17A An example of a label-free UV-PAM image of the hippocampus from a dewaxed FFPE mouse brain sample with a thickness of 7 μm is shown, where the UV-PAM image was acquired using Figure 3 generated by the system.

[0072] Figure 17B Shown in Figure 17A Example of a UV-AutoM image of the hippocampus imaged in .

[0073] Figure 17C Shown is the corresponding Figure 17A and 17B Example of a brightfield H&E-stained image of a mouse brain sample imaged in . DETAILED DESCRIPTION

[0074] Where steps and / or features are referenced in any one or more of the drawings with the same reference number, those steps and / or features have the same function or operation for purposes of this description unless otherwise intended.

[0075] It should be noted that the discussion contained in the "Background Art" section and the discussion above relating to prior art arrangements relate to discussions of documents or devices that have become public knowledge through their respective disclosure and / or use. This should not be interpreted as a representation that such documents or devices in any way form part of the common general knowledge in the art of the present inventors or patent applicants.

[0076] refer to Figure 1A and 1B , which shows a schematic diagram of an example of a general-purpose computer system 100 on which the various arrangements described herein are implemented.

[0077] like Figure 1A As shown, computer system 100 includes: a computer module 101; input devices such as keyboard 102, mouse pointer device 103, scanner 126, camera 127, and microphone 180; and output devices including printer 115, display device 114, and speakers 117. Computer module 101 can use an external modulator-demodulator (modem) transceiver device 116 to communicate with a communication network 120 via connection 121. Communication network 120 can be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a dedicated WAN. If connection 121 is a telephone line, modem 116 can be a traditional "dial-up" modem. Alternatively, if connection 121 is a high-capacity (e.g., cable) connection, modem 116 can be a broadband modem. Wireless modems can also be used to connect wirelessly to communication network 120.

[0078] The computer module 101 typically includes at least one processor unit 105 and a memory unit 106. For example, the memory unit 106 may have a semiconductor random access memory (RAM) and a semiconductor read-only memory (ROM). The computer module 101 also includes a plurality of input / output (I / O) interfaces, including: an audio-video interface 107 connected to a video display 114, a speaker 117, and a microphone 180; an I / O interface 113 connected to a keyboard 102, a mouse 103, a scanner 126, a camera 127, and an optional joystick or other human interface device (not shown) or a projector; and an interface 108 for an external modem 116 and a printer 115. In some embodiments, the modem 116 may be incorporated into the computer module 101, such as within the interface 108. The computer module 101 also has a local area network interface 111, which allows the computer system 100 to be connected to a local area communication network 122, known as a local area network (LAN), via a connection 123. As shown in FIG. Figure 1A As shown, the local area communication network 122 can also be connected to the wide area network 120 via a connection 124, which will typically include a so-called "firewall" device or a device with similar functionality. The local area network interface 111 may include an Ethernet circuit card, A wireless device or an IEEE 802.11 wireless device; however, many other types of interfaces may be implemented for interface 111.

[0079] I / O interfaces 108 and 113 may provide either or both serial and parallel connections, the former typically being implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 109 is provided and typically comprises a hard disk drive (HDD) 110. Other storage devices such as floppy disk drives and tape drives (not shown) may also be used. An optical disk drive 112 is typically provided to act as a source of non-volatile data. Portable memory devices such as optical disks (e.g., CD-ROMs, DVDs, Blu-ray Discs) may also be used. TM ), USB-RAM, portable external hard drive and floppy disk, for example, can be used as appropriate data sources for system 100.

[0080] The components 105 to 113 of the computer module 101 typically communicate via an interconnecting bus 104 and in a manner that results in conventional operating modes of the computer system 100 known to those skilled in the relevant art. For example, the processor 105 is coupled to the system bus 104 using connection 118. Similarly, the memory 106 and the optical drive 112 are coupled to the system bus 104 via connection 119. Examples of computers on which the described arrangements may be implemented include IBM-PCs and compatibles, Sun Sparcstations, Apple Macs, and the like. TMor similar computer systems.

[0081] The methods described herein may be implemented using a computer system 100, wherein the processes described herein may be implemented as one or more software applications 133 executable within the computer system 100. In particular, the steps of the methods described herein are executed by instructions 131 in the software 133 executed within the computer system 100 (see Figure 1B The software instructions 131 may be formed into one or more code modules, each of which is configured to perform one or more specific tasks.

[0082] The software can be stored on a computer-readable medium, such as the storage device described below. The software is loaded from the computer-readable medium into computer system 100 and then executed by computer system 100. A computer-readable medium having such software or a computer program recorded thereon is a computer program product. Using a computer program product in computer system 100 preferably implements an advantageous means for detecting and / or sharing write actions.

[0083] The software 133 is typically stored in the HDD 110 or the memory 106. The software is loaded into the computer system 100 from a computer-readable medium and executed by the computer system 100. Thus, for example, the software 133 may be stored on an optically readable disk storage medium (e.g., a CD-ROM) 125 that is read by the optical disk drive 112. A computer-readable medium having such software or a computer program recorded thereon is a computer program product.

[0084] In some cases, the application 133 may be provided to the user encoded on one or more CD-ROMs 125 and read via a corresponding drive 112, or alternatively may be read by the user from a network 120 or 122. Further, the software may also be loaded into the computer system 100 from other computer-readable media. A computer-readable storage medium is any non-transitory, tangible storage medium that provides recorded instructions and / or data to the computer system 100 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray discs, and the like. TM A disk, hard drive, ROM or integrated circuit, USB memory, magneto-optical disk, or computer readable card, such as a PCMCIA card, etc., whether such device is internal or external to the computer module 101. Examples of transitory or intangible computer-readable transmission media that may also be involved in providing software, applications, instructions and / or data to the computer module 101 include radio or infrared transmission channels and a network connection to another computer or networked device, as well as the Internet or an intranet, including electronic mail transmissions and information recorded on a website, etc.

[0085] The second portion of the application program 133 and the corresponding code modules described above can be executed to implement one or more graphical user interfaces (GUIs) for presentation or other representation on the display 114. Typically, by manipulating the keyboard 102 and the mouse 103, users of the computer system 100 and the applications can manipulate the interface in a functionally adaptable manner to provide control commands and / or input to the applications associated with the GUIs. Other forms of functionally adaptable user interfaces can also be implemented, such as an audio interface utilizing voice prompts output through the speaker 117 and user voice commands input through the microphone 180.

[0086] Figure 1B is a detailed schematic block diagram of the processor 105 and the "memory" 134. The memory 134 represents the memory that can be used by Figure 1A A logical aggregation of all memory modules (including HDD 109 and semiconductor memory 106) accessed by computer module 101 in.

[0087] When the computer module 101 is initially powered on, a power-on self-test (POST) program 150 is executed. The POST program 150 is typically stored in Figure 1A 149 of the semiconductor memory 106. Hardware devices such as ROM 149 that store software are sometimes referred to as firmware. POST program 150 checks the hardware within computer module 101 to ensure proper operation and typically checks processor 105, memory 134 (109, 106), and basic input-output system software (BIOS) module 151, which is also typically stored in ROM 149, for proper operation. Once POST program 150 successfully runs, BIOS 151 activates Figure 1A Activation of the hard disk drive 110 causes the boot loader 152 residing on the hard disk drive 110 to be executed by the processor 105. This loads the operating system 153 into the RAM memory 106, where the operating system 153 begins operating. The operating system 153 is a system-level application that can be executed by the processor 105 to implement various high-level functions, including processor management, memory management, device management, storage management, software application interfaces, and a general user interface.

[0088] The operating system 153 manages the memory 134 (109, 106) to ensure that each process or application running on the computer module 101 has enough memory to execute in it without conflicting with the memory allocated to another process. Figure 1AThe aggregate memory 134 is not intended to illustrate how specific memory segments are allocated (unless otherwise specified), but rather to provide a general view of the memory accessible to the computer system 100 and how the memory is used.

[0089] like Figure 1B As shown, the processor 105 includes multiple functional modules, including a control unit 139, an arithmetic logic unit (ALU) 140, and local or internal memory 148, sometimes referred to as a cache memory. The cache memory 148 typically includes a plurality of storage registers 144-146 in a register section. One or more internal buses 141 functionally interconnect these functional modules. The processor 105 also typically has one or more interfaces 142 for communicating with external devices via the system bus 104 using a connection 118. The memory 134 is coupled to the bus 104 using a connection 119.

[0090] Application program 133 includes an instruction sequence 131, which may include conditional branch and loop instructions. Program 133 may also include data 132 used when program 133 is executed. Instructions 131 and data 132 are stored in memory locations 128, 129, 130 and 135, 136, 137, respectively. Depending on the relative sizes of instructions 131 and memory locations 128-130, a particular instruction may be stored in a single memory location, as depicted by the instruction shown in memory location 130. Alternatively, an instruction may be segmented into multiple parts, each of which is stored in a separate memory location, as depicted by the instruction segments shown in memory locations 128 and 129.

[0091] Typically, the processor 105 is given a set of instructions to execute therein. The processor 105 awaits subsequent input, to which the processor 105 reacts by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more input devices 102, 103, data received from an external source over one of the networks 120, 102, data retrieved from one of the storage devices 106, 109, or data retrieved from a storage medium 125 inserted into a corresponding reader 112, all in Figure 1A In some cases, execution of a set of instructions may result in the output of data. Execution may also involve storing data or variables in memory 134.

[0092] The disclosed write detection and sharing arrangement uses input variables 154, which are stored in corresponding memory locations 155, 156, 157 in memory 134. The write detection and sharing arrangement produces output variables 161, which are stored in corresponding memory locations 162, 163, 164 in memory 134. Intermediate variables 158 may be stored in memory locations 159, 160, 166, and 167.

[0093] refer to Figure 1B The processor 105, registers 144, 145, 146, arithmetic logic unit (ALU) 140, and control unit 139 work together to execute the sequence of micro-operations required to perform a "fetch, decode, and execute" cycle for each instruction in the instruction group that makes up the program 133. Each fetch, decode, and execute cycle includes a fetch operation, which fetches or reads an instruction 131 from a memory location 128, 129, 130; a decode operation, in which the control unit 139 determines which instruction has been fetched; and an execute operation, in which the control unit 139 and / or ALU 140 executes the instruction.

[0094] Thereafter, a further fetch, decode, and execute cycle of the next instruction may be performed.Similarly, a store cycle may be performed by which the control unit 139 stores or writes a value to the storage location 162.

[0095] Each step or sub-process in the process described herein is associated with one or more segments of program 133 and is performed by register sections 144, 145, 147, ALU 140, and control unit 139 in processor 105, which work together to perform a fetch, decode, and execute cycle for each instruction in the instruction group of the segment of program 133.

[0096] The methods described herein may alternatively be implemented in dedicated hardware, such as one or more integrated circuits that perform the functions or sub-functions of the write detection and sharing methods. Such dedicated hardware may include a graphics processor, a digital signal processor, or one or more microprocessors and associated memory.

[0097] Figure 2A and 2B Together, they form a schematic block diagram of a general electronic device 201 including embedded components, on which the write detection and / or sharing methods to be described are desirably practiced. The electronic device 201 may be, for example, a mobile phone, a portable media player, virtual reality glasses, or a digital camera, where processing resources are limited. However, the methods to be described may also be performed on higher-level devices, such as desktop computers, server computers, and other such devices with significantly more processing resources.

[0098] like Figure 2AAs shown, electronic device 201 includes an embedded controller 202. Therefore, electronic device 201 can be referred to as an "embedded device". In this example, controller 202 has a processing unit (or processor) 205 bidirectionally coupled to an internal storage module 209. Storage module 209 can be formed by a non-volatile semiconductor read-only memory (ROM) 260 and a semiconductor random access memory (RAM) 270, as shown in FIG. Figure 2B RAM 270 may be volatile, nonvolatile, or a combination of volatile and nonvolatile memory.

[0099] The electronic device 201 includes a display controller 207 connected to a display 214, such as a liquid crystal display (LCD) panel, etc. The display controller 207 is configured to display graphical images on the display 214 according to instructions received from the embedded controller 202 to which the display controller 207 is connected.

[0100] The electronic device 201 also includes a user input device 213, which is typically formed of keys, a keypad, or similar controls. In some embodiments, the user input device 213 can include a touch-sensitive panel physically associated with the display 214 to form a touch screen. Such a touch screen can therefore be operated as a form of graphical user interface (GUI), rather than the prompt or menu-driven GUI typically used with a keypad-display combination. Other forms of user input devices can also be used, such as a microphone (not shown) for voice commands or a joystick / thumb wheel (not shown) for easy navigation about menus.

[0101] like Figure 2A As shown, the electronic device 201 also includes a portable memory interface 206 that is coupled to the processor 205 via a connection 219. The portable memory interface 206 allows a complementary portable memory device 225 to be coupled to the electronic device 201 to act as a source or destination for data or to supplement the internal storage module 209. Examples of such interfaces allow for coupling with portable memory devices such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, Personal Computer Memory Card International Association (PCMIA) cards, optical and magnetic disks.

[0102] The electronic device 201 also has a communication interface 208 to allow the device 201 to be connected to a computer or communication network 220 via a connection 221. The connection 221 can be wired or wireless. For example, the connection 221 can be radio frequency or optical. Examples of wired connections include Ethernet. In addition, examples of wireless connections include Bluetooth. TM Type local interconnection, Wi-Fi (including protocols based on IEEE802.11 family standards), Infrared Data Association (IrDa), etc.

[0103] Typically, electronic device 201 is configured to perform some special function. An embedded controller 202, possibly in combination with another special function component 210, is provided to perform the special function. For example, if device 201 is a digital camera, component 210 may represent the camera's lens, focus control, and image sensor. Special function component 210 is connected to embedded controller 202. As another example, device 201 may be a mobile phone handset. In this case, component 210 may represent those components required for communication in a cellular phone environment. If device 201 is a portable device, special function component 210 may represent multiple encoders and decoders of types including Joint Photographic Experts Group (JPEG), (Moving Picture Experts Group) MPEG, MPEG-1 Audio Layer 3 (MP3), etc.

[0104] The methods described below may be implemented using embedded controller 202 , where the processes described herein may be implemented as one or more software applications 233 executable within embedded controller 202 . Figure 2A The electronic device 201 implements the described method. In particular, referring to Figure 2B The steps of the described method are implemented by instructions in software 233 executed within controller 202. The software instructions can be formed into one or more code modules, each of which is configured to perform one or more specific tasks. The software can also be divided into two separate parts, wherein a first part and corresponding code modules perform the described method, and a second part and corresponding code modules manage the user interface between the first part and the user.

[0105] The software 233 of the embedded controller 202 is typically stored in the non-volatile ROM 260 of the internal storage module 209. The software 233 stored in the ROM 260 can be updated from a computer-readable medium when needed. The software 233 can be loaded into the processor 205 and executed by the processor 205. In some cases, the processor 205 can execute the software instructions located in the RAM 270. The software instructions can be loaded into the RAM 270 by the processor 205 adding a copy of one or more code modules from the ROM 260 to the RAM 270. Alternatively, the software instructions of one or more code modules can be pre-installed in the non-volatile area of the RAM 270 by the manufacturer. After the one or more code modules are located in the RAM 270, the processor 205 can execute the software instructions of the one or more code modules.

[0106] The application programs 233 are typically pre-installed by the manufacturer and stored in the ROM 260 before the electronic device 201 is distributed. However, in some cases, the application programs 233 may be provided to the user encoded on one or more CD-ROMs (not shown) and downloaded by the manufacturer before being stored in the internal storage module 209 or the portable memory 225. Figure 2A 201 . The portable storage interface 206 reads the software application 233. In another alternative, the software application 233 can be read by the processor 205 from the network 220 or loaded into the controller 202 or portable storage medium 225 from other computer-readable media. A computer-readable storage medium refers to any non-transitory, tangible storage medium that participates in providing instructions and / or data to the controller 202 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, hard drives, ROMs or integrated circuits, USB memory sticks, magneto-optical disks, flash memory, or computer-readable cards, such as PCMCIA cards, whether such devices are internal or external to the device 201. Examples of transitory or intangible computer-readable transmission media that can also participate in providing software, applications, instructions, and / or data to the device 201 include radio or infrared transmission channels, network connections to another computer or networked device, and the Internet or intranet, including email transmissions and information recorded on websites. A computer-readable medium having such software or a computer program recorded thereon is a computer program product.

[0107] The second part of the application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to display the Figure 2A 214 or otherwise represented. By manipulating the user input device 213 (e.g., a keypad), a user of the device 201 and the application 233 can manipulate the interface in a functionally adaptable manner to provide control commands and / or input to an application associated with the GUI. Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing voice prompts output through a speaker (not shown) and user voice commands input through a microphone (not shown).

[0108] Figure 2BThe embedded controller 202 is shown in detail with a processor 205 and internal storage 209 for executing application programs 233. The internal storage 209 includes read-only memory (ROM) 260 and random access memory (RAM) 270. The processor 205 is capable of executing application programs 233 stored in one or both of the connected memories 260 and 270. When the electronic device 201 is initially powered on, the system program resident in ROM 260 is executed. Application programs 233 permanently stored in ROM 260 are sometimes referred to as "firmware." The execution of firmware by the processor 205 can implement various functions, including processor management, memory management, device management, storage management, and user interface.

[0109] The processor 205 typically includes a plurality of functional modules, including a control unit (CU) 251, an arithmetic logic unit (ALU) 252, a digital signal processor (DSP) 253, and a local or internal memory 254 including a set of registers that typically contain atomic data elements 256, 257, and an internal buffer or cache 255. One or more internal buses 259 interconnect these functional modules. The processor 205 also typically has one or more interfaces 258 for communicating with external devices via a system bus 281 using a connection 261.

[0110] The application program 233 includes a sequence of instructions 262 to 263, which may include conditional branching and looping instructions. The program 233 may also include data used when the program 233 is executed. This data may be stored as part of the instructions or in a separate location 264 within the ROM 260 or RAM 270.

[0111] Typically, the processor 205 is given a set of instructions that are executed in the processor. This set of instructions can be organized into blocks that perform specific tasks or handle specific events that occur in the electronic device 201. Generally, the application 233 waits for an event and then executes the code block associated with the event. The event can be responded to by Figure 2A The user input device 213 is triggered by input from the user, which is detected by the processor 205. Events can also be triggered in response to other sensors and interfaces in the electronic device 201.

[0112] Execution of a set of instructions may require reading and modifying numerical variables. Such numerical variables are stored in RAM 270. The disclosed method uses input variables 271 stored in known locations 272, 273 in memory 270. Input variables 271 are processed to produce output variables 277, 279 stored in known locations 278 in memory 270. Intermediate variables 274 may be stored in additional memory locations such as locations 275, 276 in memory 270. Alternatively, some intermediate variables may exist only in registers 254 of processor 205.

[0113] Execution of an instruction sequence is accomplished in processor 205 through repeated application of a fetch-execute cycle. The control unit 251 of processor 205 maintains a register called a program counter, which contains the address of the next instruction to be executed in ROM 260 or RAM 270. At the beginning of a fetch-execute cycle, the contents of the memory address indexed by the program counter are loaded into control unit 251. The instructions thus loaded control subsequent operations of processor 205, such as causing data to be loaded from ROM memory 260 into processor register 254, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to a location stored in another register, and so on. At the end of the fetch-execute cycle, the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed, this may involve incrementing the address contained in the program counter or loading the program counter with a new address to implement a branch operation.

[0114] Each step or sub-process in the method described below is associated with one or more segments of the application 233 and is performed by repeatedly executing instruction fetch-execute cycles in the processor 205 or similar program operations of other independent processor blocks in the electronic device 201.

[0115] Various aspects provide high-throughput, label-free, and slide-free imaging methods and systems based on the intrinsic light absorption contrast under ultraviolet illumination to directly detect histologically stained biomolecules. Two methods are disclosed, namely, ultraviolet-based (i) photoacoustic microscopy (UV-PAM) and (ii) autofluorescence microscopy (UV-AutoM). To achieve high throughput of UV-PAM, a high-speed optical scanning configuration can be used. In conjunction with UV-AutoM, speckle illumination (SI) is utilized, which allows the estimation of high-resolution images using low-magnification objectives, providing subcellular resolution images over centimeter-scale imaging areas, while allowing high tolerance to image blur caused by tissue surface morphology, slide placement errors, and thickness.

[0116] For both UV-PAM and UV-AutoM image types, a deep learning-based virtual staining method and system are disclosed that can be used to generate histological-like images of large, unprocessed fresh / fixed tissues at subcellular resolution. The virtual staining method and system utilize a generative adversarial network (GAN) that is configured to transform UV-PAM or UV-AutoM images of unlabeled tissues into histologically stained images through paired / unpaired training examples. The disclosed method and system can streamline the workflow of standard-of-care histopathology from several days to less than ten minutes, enabling intraoperative surgical margin assessment, thereby reducing or eliminating the need for a second surgery due to positive margins.

[0117] Ultraviolet-based photoacoustic microscopy (UV-PAM)

[0118] Unlike conventional light microscopy, PAM takes advantage of highly specific optical absorption contrast. By using a UV pulsed laser (wavelength range of ∼240–280 nm) as the excitation beam, cell nuclei can be highlighted, providing label-free histology-like images.

[0119] refer to Figure 3 , an example of a UV-PAM system 300 is shown. In particular, a nanosecond pulsed UV laser 302 (e.g., WEDGE-HF 266nm, available from Bright Solution, LLC) is focused by a focusing assembly 303. In particular, the light emitted by the UV laser 302 is expanded by a pair of lenses 304, 308 (e.g., LA4647-UV and LA4663-UV, available from Thorlabs). The quality of the UV beam can be improved by a pinhole 306 located between the pair of lenses 304, 308. The size of the pinhole 306 can be 10μm to 100μm in diameter (e.g., P25C, available from Thorlabs). The beam is then reflected by a 1D galvanometer scanner 310 and then by an objective lens 310 (e.g., MicroSpot TM A focusing objective (LMU-20X-UVB) is focused on the bottom of a sample 312. The sample 312 is placed at the bottom of a water tank 314, which is held by a sample holder 315 attached to an XYZ translation stage 318. The water tank 314 is filled with water to allow photoacoustic waves to propagate upward and be detected by a water-immersed ultrasonic transducer 316 (e.g., V324-SU, available from Olympus NDT).

[0120] The received sound pressure is converted into an electrical signal, which is then amplified by amplifier 320 (e.g., two ZFL-500LN-BNC+, available from Mini-Circuits) and finally received by computer system 100 or 201 via data acquisition system 322 (e.g., ATS9350, available from Alazar Technologies). To generate a two-dimensional image, the maximum amplitude projection (MAP) of each A-line signal is first identified. The MAPs are then rearranged according to the order in which they were scanned to generate a grayscale image.

[0121] In operation, the galvanometer scanner 310 of the focusing assembly can be controlled to focus the ultraviolet light on the sample 312 according to a scanning trajectory. Control of the galvanometer scanner 310 can be performed by a portion of the computer system 100 or the computerized embedded device 201. The transducer 316 is configured to receive the photoacoustic waves emitted by the sample 312 in response to the ultraviolet light. The computer system 100 or the embedded device 201 generates a UV-PAM image based on the photoacoustic waves.

[0122] To measure the lateral and axial resolution of the UV-PAM system, gold nanoparticles (200 nm in diameter) were imaged with a step size of 0.15 μm on the x-axis and y-axis (Figure 4(a)). The data points of four gold nanoparticles were selected and averaged to measure the lateral resolution by Gaussian curve fitting, as shown in Figure 4(b). The full width at half maximum (FWHM) of the Gaussian fitting curve is ~0.6 μm. To evaluate the axial resolution, the envelope of the A-line signal at the center position can be extracted. Figure 4(c) shows the A-line signal at the center position of the gold nanoparticle in (a). The FWHM of the envelope of the A-line signal is 38 ns, which corresponds to 58 μm, indicating the axial resolution of the UV-PAM system.

[0123] Ultraviolet-based autofluorescence microscopy (UV-AutoM)

[0124] In histopathology, objectives with a 20X-40X magnification factor are typically required to achieve subcellular resolution to observe cell morphology and metabolic activity. However, such magnification factors limit the field of view (FOV) to 1 mm. 2 Furthermore, high-magnification objectives are more susceptible to spatially varying aberrations and have a shallow depth of field (DOF), which results in a lower tolerance for microscope slide placement errors and specimen roughness. For these reasons, capturing large tissue surfaces by image stitching using high-magnification objectives is suboptimal.

[0125] A speckle illumination (SI) method is disclosed to alleviate the inherent tradeoff between large FOV and high resolution (HR) in digital microscopy, enabling high-throughput visualization of different regions of interest with subcellular resolution.

[0126] like Figure 5A FIGURE 5 illustrates an exemplary UV-AutoM system 500. Fresh, unstained tissue 312 is placed on an open-top sample holder attached to an XYZ motorized stage 318 (e.g., three L-509s, available from PI miCos). A UV laser 302 (e.g., WEDGE-HF 266 nm, available from Bright Solution, LLC) is collimated and projected onto a fused silica diffuser (e.g., DGUV10-600, available from Thorlabs) to produce a speckle pattern. The resulting wave is focused by an aspheric UV condenser lens 512 (e.g., #33-957, available from Edmund Optics, Inc.) with a numerical aperture (NA) of 0.69, which is part of a focusing assembly 303. Sample 312 is obliquely illuminated through a UV transparent window 514 (e.g., having a transmittance greater than 90% from 200 nm to 1500 nm). The excited autofluorescence signal (primarily from NADH and FAD, with peak emission at 450 nm) is then collected by an inverted microscope 517 equipped with a 4X objective lens 518 (e.g., NA = 0.1, plan achromatic objective lens, available from Olympus NDT) and an infinity-corrected tube lens 520 (TTL180-A, available from Thorlabs), and finally imaged by a monochrome scientific complementary metal oxide semiconductor (sCMOS) camera 522 (e.g., PCOedge 4.2, 6.5 μm pixel pitch, available from PCO).

[0127] Low-magnification objectives are less affected by spatially varying aberrations over a large FOV and have a larger DOF and longer working distance, which allows for high tolerance to slide placement errors and enables flexible manipulation on the sample stage. However, their spatial resolution is largely limited by their low NA value, which is a factor that determines the achievable resolution of an imaging system according to the Rayleigh criterion (i.e., the minimum distance that an imaging system can resolve is 0.61λ / NA, where λ is the fluorescence emission wavelength and NA is the numerical aperture of the objective). To this end, SI reconstruction is used to circumvent the resolution limitation imposed by low-NA objectives in this configuration.

[0128] A computational imaging method for autofluorescence microscopy based on speckle illumination is disclosed. In a preferred embodiment, the method implements high-throughput microscopy. In particular, "high-throughput microscopy" refers to the use of automated microscopy and image analysis to visualize and quantitatively capture cellular features at large scales. More specifically, due to the application of speckle illumination, the spatial bandwidth product (i.e., field of view / resolution) of the high-throughput output is 2) is approximately or equal to 10 times that of conventional fluorescence microscopy, which is typically limited to the megapixel level. Low-magnification objectives are more advantageous for imaging large tissue surfaces because they are less affected by spatially varying aberrations. In addition, due to their large depth of field, out-of-focus image blur caused by surface irregularities, tissue thickness or slide placement errors can be minimized by implementing low-magnification lenses. However, the low numerical aperture (NA) values of such lenses largely limit the achievable resolution, hindering their application to subcellular level imaging targets. The proposed method performs iterative reconstruction through a sequence of low-resolution autofluorescence images of speckle illumination to circumvent the resolution limitation set by the NA objective, thereby facilitating fast high-resolution imaging over large imaging areas with arbitrary surface morphology.

[0129] refer to Figure 5B and 5C , a flow chart showing an example computer-implemented method 530 for reconstructing a UV-stimulated autofluorescence image (UV-AutoM) from a sequence of speckle-illuminated images is shown. The computer-implemented method 530 may be implemented by Figure 1A and 1B The computer system 100 described herein may be composed of Figure 2A and 2B The described embedded device 201 is executed. Figure 5D Pseudo code is provided that further illustrates a more specific embodiment of reconstructing a high-throughput UV-excited autofluorescence image (UV-AutoM) from a sequence of speckle-illuminated images. Figure 5D Pseudocode to describe Figure 5B and 5C Flowchart of the process.

[0130] In particular, at step 530-1, the method 530 includes recording a sequence of images of speckle illumination of a sample translated to a corresponding sequence of positions in a plane along a scanning trajectory. j (j=1, 2, ..., N). In the sense that the output of the method 530 is a high-resolution image, the sequence I of speckle-illuminated images is a low-resolution image, and each I of the low-resolution speckle-illuminated images j In one example, a sequence of images of speckle illumination can be captured using a 4X / 0.1NA objective.

[0131] At step 530-2, method 530 includes initializing an image object o(x, y) and a speckle pattern. Image object o(x, y) is referred to herein as a high-resolution image object. As shown in line 3 of the pseudocode, a sequence of speckle-illuminated images is averaged and the averaged speckle-illuminated images are interpolated, wherein the high-resolution image object o(x, y) is set to the interpolated result of the averaged speckle-illuminated images. The speckle pattern is initialized to a matrix of 1s.

[0132] In step 530-3, the current position is set as the first position in the position sequence. Figure 5D The current position in the pseudocode is represented by (x j ,y j )express.

[0133] by Figure 5B and 5C The inner loop of the flowchart shown is in the form of executing steps 530-4 to 530-11 for each speckle-illuminated image captured in the sequence. Figure 5D The pseudo code of FIG6 is shown in FIG6 , where the inner loop is represented by lines 6 to 14 and the outer loop is represented by lines 5 to 17. For the inner loop, the current position variable is effectively incremented to the next position in the position sequence and the corresponding speckle-illuminated image at the corresponding current position is used in the image processing step to modify the high-resolution image object and the speckle pattern.

[0134] More specifically, at step 530-4, method 530 includes computationally shifting the high-resolution object to the current position o(xx j ,yy j ), and then multiplied by the speckle pattern p(x, y) to generate an estimated speckle-illuminated image This is Figure 5D is shown in line 7 of the pseudocode.

[0135] At step 530-5, method 530 includes: It is multiplied by the optical transfer function OTF(k x , k y ), determine the filtered object-pattern composite ψ in the frequency domain j (k x , k y ), where k x and k y is the spatial coordinate in the frequency domain. The optical transfer function is a known optical transfer function of an apparatus for capturing a sequence of images of speckle illumination of a sample.

[0136] It should be noted that the shift operations in steps 530 - 4 and 530 - 5 are collectively referred to as application of the angular spectrum.

[0137] Steps 530 - 6 to 530 - 8 described below are a reconstruction process based on a phase retrieval algorithm called a ptychographic iterative engine (PIE).

[0138] At step 530-6, method 530 includes determining an updated estimated speckle illumination image in the frequency domain based on the estimated speckle illumination image in the frequency domain, the captured speckle illumination image in the frequency domain at the current position, the filtered object-pattern composite in the frequency domain, the optical transfer function, and the adaptive learning rate parameter α. More specifically, Figure 5D Line 9 of the pseudo code shows the specific calculation of the updated estimated speckle-illuminated image in the frequency domain, which is shown in Equation 1 below:

[0139]

[0140] In particular, an updated estimated image of the speckle illumination in the frequency domain is calculated to be equal to: the estimated image of the speckle illumination in the frequency domain plus an adaptive learning rate parameter times the conjugate of the optical transfer function times the difference between the image of the speckle illumination captured in the frequency domain at the current position and the filtered object-pattern composite in the frequency domain divided by the square of the absolute maximum of the optical transfer function.

[0141] At step 530-7, method 530 includes updating the high-resolution object based on the updated estimated speckle-illuminated image and speckle pattern in the spatial domain. Figure 5D Line 10 of the pseudocode and is represented by Equation 2 below:

[0142]

[0143] In particular, the high-resolution object is set equal to: the high-resolution object plus the conjugate of the speckle pattern multiplied by the difference between the updated estimated speckle-illuminated image in the spatial domain and the estimated speckle-illuminated image in the spatial domain divided by the square of the absolute value of the maximum value of the speckle pattern.

[0144] At step 530-8, method 530 includes updating the speckle pattern based on the updated estimated speckle illuminated image, the estimated speckle illuminated image, and the high-resolution object. This is shown in Figure 5D Line 11 of the pseudocode and is represented by Equation 3 below:

[0145]

[0146] In particular, the speckle pattern is set equal to the speckle pattern plus the conjugate of the high-resolution object multiplied by the difference between the updated estimated speckle-illuminated image in the spatial domain and the estimated speckle-illuminated image in the spatial domain divided by the square of the absolute value of the maximum value of the high-resolution object.

[0147] In step 530-9, based on the captured speckle-illuminated image I j Composite ψ with the filtered object-pattern in the frequency domain j The absolute value of the difference between the inverse Fourier transform and the total loss parameter loss of the current cycle are calculated. j This is Figure 5D is shown in line 12 of the pseudocode and is represented by Equation 4 below:

[0148] loss j =∑ j |I j -F -1 (ψ j )| Equation 4

[0149] At step 530-10, method 530 includes applying Nesterov momentum acceleration to the high-resolution object and the speckle pattern. This step is performed to accelerate the gradient descent of the reconstruction process to speed up the convergence speed.

[0150] At step 530-11, method 530 includes determining whether the current position is the last position in the position sequence. In response to a positive determination (i.e., "yes"), the method proceeds to step 530-13. In response to a negative determination (i.e., "no"), the method then proceeds to step 530-12.

[0151] At step 530-12, method 530 includes setting the current position to the next position in the position sequence.Method 530 then proceeds back to step 530-4 to perform one or more additional iterations of the inner loop represented by steps 530-4 through 530-11 until the last speckle-illuminated image is processed.

[0152] At step 530-13, method 530 includes determining whether convergence has been detected based on the total loss parameter. This is determined by determining a loss ratio based on the difference between the total loss parameters calculated for the previous and current iterations of the inner loop (i.e., steps 530-4 to 530-11) divided by the total loss parameter for the previous iteration of the inner loop. The loss ratio is then compared to a loss threshold, and the loss ratio is calculated based on the difference between the total loss parameters calculated for the previous and current iterations of the inner loop (i.e., steps 530-4 to 530-11). Figure 5DIn the example pseudocode, the loss threshold is set to 0.01. If the loss ratio is less than or equal to the loss threshold, the adaptive learning rate parameter α is reduced, and in this example, the adaptive learning rate parameter α is halved. The adaptive learning rate parameter α is adjusted to suppress oscillations of the loss function near the convergence point to minimize artifacts in the reconstructed image. Once the learning rate α is reduced to zero, the reconstruction process ends, so that a high-resolution object is determined and output in step 530-14. The output high-resolution image object is a high-resolution UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area, which has improved resolution compared to the low-resolution image of each speckle shot provided as input to the reconstruction method 530. The speckle pattern, which is initially unknown at the beginning of the method 530, is also determined in step 530-14.

[0153] This computational imaging method, represented by method 530, synthesizes a sequence of low-resolution speckle-illuminated images to reconstruct a high-resolution autofluorescence image (UV-AutoM). This method is implemented through a series of updates in both the spatial and frequency domains. The method begins with an initial guess of a high-resolution object. The object is first multiplied by the speckle pattern, Fourier transformed to the frequency domain, and low-pass filtered using an optical transfer function. The filtered spectrum is then inverse-transformed to the spatial domain, and its intensity is replaced by the corresponding low-resolution speckle-illuminated image. Finally, this updated autofluorescence image is transformed to the frequency domain and further updated. One iteration is completed until all captured low-resolution images have been included, and Nesterov momentum acceleration is implemented for faster gradient descent. After multiple iterations, a high-resolution UV-AutoM image is output, which has enhanced subcellular resolution over centimeter-scale image areas. A priori knowledge of the speckle pattern is not required; only the relative shift between each low-resolution image is known.

[0154] After several iterations, a high-resolution UV-AutoM image is output with enhanced subcellular resolution over a centimeter-scale image area. No prior knowledge of the speckle pattern is required; only the relative shift (xj, yj) between each captured image is known. This suggests that sufficient scanning range (greater than ~2 low-NA diffraction-limited spot sizes) and finer scanning steps (smaller than the target resolution) can reduce distortion in the reconstruction, and the final achievable NA is the sum of the objective NA and the speckle NA.

[0155] Figure 6 Shows the previous Figures 5A to 5C Graphical representation of an example of the described SI reconstruction method and system. Figure 6The physical constraints set by the 4X / 0.1NA objective in Fourier space and the corresponding low-resolution raw image of a mouse brain sample (100 μm thickness) are shown at 610. SI dataset 620 was acquired using the 4X / 0.1NA objective, as shown in FIG. Figure 6 As shown, in this example, the dataset includes 49 low-resolution measurements of speckle illumination, captured by translating the sample to 49 different positions in the XY plane with a scan step size of 500 nm. The SI dataset can then be used to reconstruct a high-throughput UV-AutoM image 630 with an extended passband up to NA = 0.3, which corresponds to a 3-fold resolution enhancement compared to images acquired using a 4X / 0.1NA objective.

[0156] Fluorescent nanoparticles with a diameter of 500 nm (excitation / emission: 365 nm / 445 nm, B500, available from Thermo Fisher) can be used to quantify the resolution performance of the above-mentioned SI reconstruction methods and systems. Figure 7A , shows an example of a low-resolution fluorescence image captured using a 4X / 0.1NA objective under uniform illumination, while Figure 7B The high-resolution fluorescence image was reconstructed from a low-resolution image of 49 speckle illuminations raster-scanned with a step size of 500 nm. Figure 7A and 7B All were captured using a 266 nm ultraviolet laser. Figure 7C It is along Figure 7A and 7B Intensity graphs of the solid lines 700, 710 indicated in FIG, from which we can quantify the resolution enhancement by the disclosed SI reconstruction method and system.

[0157] Advantageously, the SI reconstruction method and system can be highly tolerant to rough surfaces. To demonstrate the high tolerance of our system to rough surfaces, Figures 8A-8C UV-AutoM images of two untreated leaf samples are shown. Figure 8A and 8B shows a low-resolution autofluorescence image captured using a 4X / 0.1NA objective lens, while Figure 8C and 8D Shown is a high-resolution SI-reconstructed UV-AutoM image reconstructed from 49 low-resolution images of speckle illumination scanned with a step size of 500 nm. Figure 8E and 8F The corresponding high-resolution reference image captured using a 10X / 0.3NA objective is shown, with Figure 8C and 8DIn comparison, due to the shallow DOF of the high-NA objective, there is a clear out-of-focus area, which proves that the high-resolution UV-AutoM image reconstructed by the SI method via the low-NA objective can be far superior to that of the high-NA objective, especially when dealing with rough surfaces.

[0158] refer to Figure 9A to 9F, showing the structural match between UV-AutoM and brightfield H&E staining images. Figure 9A This is a UV-AutoM image of a whole mouse brain section (FFPE section after dewaxing, thickness of 4μm). Figure 9B and 9C They are Figure 9A The enlarged images of boxes 910 and 920 are shown in FIG. Figure 9D and 9E The corresponding bright field H&E histological images are captured by a digital slide scanner (NanoZoomerS Q, Hamamatsu). It can be seen that the cells concentrated in the cutting edge (frame 910) and hippocampus (frame 920) regions are clearly resolved in the UV-AutoM image with negative contrast, that is, they appear black on the image and show a basically perfect structural match (cell morphology and distribution) with the H&E stained image.

[0159] refer to Figures 10A to 10G , showing high-throughput label-free visualization of thick mouse brain (100 μm thick, cut by a Leica vibratome). Due to the shallow penetration depth of UV light, the excited autofluorescence surface is located within a few microns below the surface, allowing label-free and slide-free imaging of thick samples. In addition, the above-mentioned SI reconstruction method and system were applied to achieve rapid high-resolution visualization of the entire mouse brain using a 4X objective lens, which has fewer imaging aberrations and allows higher tolerance to rough tissue surfaces compared to a 10X objective lens. Figure 10A Example images showing a top view of a mouse brain. Figure 10B and 10C are examples of reconstructed UV-AutoM images, each depicting a section through Figure 10A Cross-sections of the corresponding locations (lines AA and BB) of the mouse brain, where each reconstructed UV-AutoM image was generated using the SI dataset, which includes 441 low-resolution images of speckle illumination. The exposure time of each low-resolution raw image was 200 ms, enabling fast imaging, with a total acquisition time of less than 3 minutes. Figures 10D to 10H Shown Figure 10C High-pass quantum views of five functional regions, including (a) corpus callosum 1010, (b) hippocampus 1020, (c) globus pallidus 1030, (d) caudate putamen 1040, and (e) parietal cortex 1050.

[0160] Generation of pseudo-stained histological images

[0161] Pathologists are typically trained to examine histologically stained tissue samples to make diagnostic decisions. However, UV-PAM and UV-AutoM images are grayscale. To address or alleviate this issue, a deep learning-based virtual staining method using a generative adversarial network (GAN) is disclosed for transforming UV-PAM or UV-AutoM images of unlabeled tissue into pseudo-hematoxylin and eosin (H&E)-stained images.

[0162] GANs allow for virtual staining of unpaired UV-PAM / UV-AutoM and H&E images, significantly simplifying the image preprocessing process, which is difficult because the tissue may be rotated or deformed during the staining process. Paired training methods can also be used on UV-PAM / UV-AutoM images and their corresponding H&E-stained images to perform pseudo-colorization.

[0163] In some embodiments, a method for using Figure 3 Describes methods and systems for generating UV-PAM images and using Figure 5A 、 5B SI reconstructed UV-AutoM images generated by the systems and methods of FIG5C are used as input to the trained cycle-GAN to generate pseudo-stained histological images. Additionally or alternatively, UV-PAM images generated using the systems and methods disclosed in U.S. Patent Application No. 2014 / 0356897 (the contents of which are incorporated herein by reference in their entirety) can be used as input to the GAN to generate pseudo-stained histological images.

[0164] refer to Figure 11A , a block diagram showing an example computer-implemented system for generating a pseudo-stained histological image (ie, a virtual stained histological image) is shown. The computer-implemented system can be implemented using the computer system 100 or the embedded system 201.

[0165] The computer-implemented system utilizes a generative adversarial network 1100 (GAN), which can be provided in the form of a cycle-consistent generative adversarial network (Cycle-GAN). Cycle-GAN 1100 includes four deep convolutional neural networks, namely a first generator module G, a second generator module F, a first discriminator module X, and a second discriminator module Y.

[0166] like Figure 11AAs shown, Generator G is configured to learn to transform a UV-AutoM / UV-PAM input image 1110 (illustrated as a UV-PAM image) into a brightfield H&E image 1120 (illustrated as a BR-HE image), while Generator F is configured to learn to transform the brightfield H&E image 1120 into the UV-AutoM / UV-PAM image 1110. Discriminator X is configured to distinguish between genuine UV-PAM images and fake UV-PAM images generated by Generator F, as shown in output 1130. Meanwhile, Discriminator Y is configured to distinguish between genuine brightfield H&E images and fake brightfield H&E images generated by Generator G, as shown in output 1140. Once Generator G can generate H&E images that Discriminator Y cannot distinguish from the input genuine H&E images, Generator G has learned the transformation from UV-AutoM / UV-PAM images to H&E images. Similarly, once the generator F can produce UV-AutoM / UV-PAM images that the discriminator X cannot distinguish from real H&E images, the generator F has learned the transformation from H&E images to UV-AutoM / UV-PAM images.

[0167] refer to Figure 11B , a flowchart representing an example computer-implemented method 1150 for generating a pseudo-hematoxylin and eosin (H&E) stained image is shown. The computer-implemented method can be performed by the computer system 100 or the embedded system 201.

[0168] At step 1160 , the method 1150 includes receiving an input image. The input image is an ultraviolet-based autofluorescence microscopy (UV-AutoM) image or an ultraviolet-based photoacoustic microscopy (UV-PAM) image of an unlabeled sample. The input image is a grayscale image.

[0169] At step 1170 , method 1150 includes transforming the input image into a pseudo H&E stained image of the input image using a generative adversarial network.

[0170] At step 1180 , method 1150 includes outputting a pseudo-H&E stained image.

[0171] Preferably, the generative adversarial network is a generative adversarial network with cycle consistency.

[0172] In certain embodiments, method 1150 includes training a generative adversarial network using unpaired input and H&E stained images.

[0173] refer to Figure 12, shows a functional block diagram of a detailed workflow 1200 of the forward cycle 1202 and the backward cycle 1204 of the Cycle-GAN 1100 of Figure 11. The cyclic generative adversarial network includes four deep convolutional neural networks, including: a first generator G, which is configured to transform an input image into a generated H&E image; a second generator F, which is configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator Y, which is configured to distinguish between the H&E images of the training set and the generated H&E images generated by the first generator deep convolutional neural network; and a second discriminator X, which is configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

[0174] More specifically, after a UV-AutoM / UV-PAM image 1210 (herein referred to as the input image) is input to the generator G, as shown in FIG. Figure 12 The forward loop 1202 shown in the top row of the schematic diagram is shown, where the generator G outputs a generated H&E image 1220. The discriminator Y is configured to determine whether the generated H&E image 1220 is real or fake (i.e., the discriminator Y is configured to identify whether the H&E image received from the generator G is derived from the input real H&E image or from the generator G). In turn, the generated H&E image 1220 of the generator G is provided as input to the generator F to transform it back into a UV-AutoM / UV-PAM image 1230, which is called a cycle image 1230. The loss between the input image 1210 and the cycle image 1230 is called the cycle consistency loss in Cycle-GAN. Figure 12 As shown in the bottom row 1204 of the schematic diagram, the backward loop is symmetrical with the forward loop. The backward loop starts with an input H&E image 1240, where the backward loop learns to transform the BR-H&E image 1240 into a UV-AutoM / UV-PAM image 1250. Similarly, the discriminator X is configured to determine whether the generated UV-AutoM / UV-PAM image is real or fake by comparing the generated UV-AutoM / UV-PAM image with the input image.

[0175] refer to Figure 13 , which shows the representation Figure 11AFunctional block diagram of an example generator 1300 of a Cycle-GAN. In one form, the first and second generators, generators G and F, may be configured as Resnet-based generator networks. In an alternative embodiment, generators G and F may be configured as U-Net-based generator networks. Each Resnet-based generator may be composed of several downsampling layers 1320A, 1330A, 1330B, 13330C, residual blocks 1310A-1310I, and upsampling layers 1330D, 1330E, and 1320B. Figure 13 In the example shown, nine residual blocks 1310A-1310I are used for training UV-PAM and HE images, each with a pixel size of 256x256. Spatial reflection padding (3x3) is added at the beginning of the neural network to ensure that the input and output of the corresponding generator neural network have the same size. After the padding layer 1320A, each Resnet-based generator 1300 includes a downsampling path consisting of three Convolution-InstanceNorm-ReLU layers 1330A, 1330B, and 1330C. In particular, the first Convolution-InstanceNorm-ReLU downsampling layer 1330A has a larger receptive field with a kernel size of 7x7, while the other two layers 1330B and 1330C have smaller receptive fields with a kernel size of 3x3. The image size is restored to the original image size, and the number of channels increases to 64 after the first layer 1330A. As the image passes through two more layers 1330B and 1330C, the image size is reduced by a factor of 2 while the number of channels is increased by a factor of 2. This is followed by the downsampling layers 1320A, 1330A, 1330B, and 1330C, which are long residual neural networks consisting of nine residual blocks 1310A-1310I. As the image passes through each residual block, the image size and number of channels remain constant (256x64x64). After the nine residual blocks 1310A-1310I, the generator 1300 includes an upsampling path consisting of two Convolution-InstanceNorm-ReLU layers 1330D and 1330E with a kernel size of 3x3, and a reflection padding layer 1320B (3x3) with a kernel size of 7x7, connected to the Convolution-InstanceNorm-ReLU layer 1340. After each upsampling layer, the image size is increased by a factor of 2 while the number of channels is reduced by half. After the two upsampling layers, the image size is restored to the original image size and the number of channels is reduced to 64. The last connected layer 1340 is configured to keep the image size unchanged while reducing the number of channels to 3.

[0176] In a specific configuration, the discriminator networks Dx and Dy can be fed by a 70x70 PatchGAN discriminator, which consists of four 4x4 Convolution-InstanceNorm-LeakyReLU layers. PatchGAN will produce a one-dimensional output (real or fake) after the last layer.

[0177] The described GAN was implemented using Python version 3.7.3 with PyTorch version 1.0.1. The software was implemented on a desktop computer with a 3.7GHz Core i7-8700K CPU and 32GB of RAM, running the Ubuntu 18.04.2LTS operating system. Training and testing of the Cycle-GAN neural network were performed using a GeForce GTX1080Ti GPU with 11GB of RAM. However, it should be understood that other computer systems or embedded systems 201 can be utilized.

[0178] This article will discuss examples of virtually stained histology images generated using the aforementioned Cycle-GAN.

[0179] refer to Figures 14A to 14C , a virtual staining image of a UV-PAM image of a mouse brain slice was generated using the above-described UV-PAM method and system, and transformed using the above-described Cycle-GAN 1100. The UV-PAM image and the virtual staining image of the UV-PAM image were compared with a histological image of the same sample after H&E staining. In particular, a 7 μm thick sample slice cut from an FFPE mouse brain was imaged, wherein the disclosed UV-PAM method and system was operated using a raster scan with a step size of 0.63 μm. Since the UV-PAM method and system are configured to generate grayscale images, the grayscale UV-PAM image (such as Figure 14A As shown) is transformed into a virtual dyed image, Figure 14B To evaluate whether the generated virtual stained images can provide similar information as conventional histological images, histological images of the same samples were obtained using bright field microscopy after H&E staining, as shown in Figure 2. Figure 14C When examining the colors of cell nuclei and other connective tissues in the virtual stained images generated by the UV-PAM system, it was determined that the virtual stained images generated by the UV-PAM system were substantially similar to conventional histological images. Overall, the UV-PAM system, combined with the deep learning system provided in the form of the publicly available Cycle-GAN 1100, can generate substantially accurate virtual stained histological images without the need for conventional staining techniques.

[0180] Figures 15A to 15CAn example of virtual coloring of a UV-AutoM image using the paired training method is shown. Figure 15A is a UV-AutoM image of a 7 μm thick dewaxed FFPE mouse brain section, and Figure 15B is a virtually tinted version of the UV-AutoM image using a GAN model in the form of a paired pix2pix-based network, while Figure 15C is the corresponding brightfield H&E image for comparison purposes. Color transformation via paired datasets enables accurate and reliable generation of histology-like images. However, any paired training approach requires rigorous data preprocessing procedures for image alignment and registration, and is difficult to apply to virtual staining of thick samples. We found that a Cycle-GAN-based network allows color mapping without the need for paired training examples and demonstrates great potential on biological tissues of any thickness. The Cycle-GAN network 1100 can be fed with unpaired UV-AutoM and H&E images from dewaxed FFPE mouse brain sections. The well-trained Cycle-GAN 1100 network is able to transform UV-AutoM images of unlabeled tissue into virtually H&E-stained versions of the unlabeled tissue, allowing pathologists to easily interpret the UV-AutoM images.

[0181] Figure 16A and 16B Results involving testing of the trained Cycle-GAN network 1100 on mouse brain samples with different thicknesses. Figure 16A and 16B SI reconstructed high-resolution UV-AutoM images of mouse brain samples with thicknesses of 100 μm and 200 μm, respectively. Figure 16C and 16D The corresponding virtually stained H&E image is shown in Figure 2. The successful color mapping from the UV-AutoM contrast to the H&E-stained version has greatly facilitated the development of the UV-AutoM imaging modality as a practical intraoperative diagnostic tool for physicians and pathologists in the operating room.

[0182] It should be understood that there is a complementary contrast between UV-PAM and UV-AutoM images, thereby realizing the method and system for generating UV-PAM and UV-AutoM images. Photons (or fluorescence) are generated by radiative relaxation, while heat is generated by non-radiative relaxation, wherein PA waves are released by the thermally induced pressure / temperature increase of the sample. Therefore, PA and autofluorescence images are expected to show complementary contrast according to energy conservation. This contrast is Figure 17A and 17B Using UV laser (266nm) excitation, Figure 17A shows the label-free PA contrast (UV-PAM), while Figure 17BShown is label-free autofluorescence contrast (UV-AutoM) of the hippocampus from a 7 μm thick dewaxed FFPE mouse brain sample. Figure 17C The corresponding brightfield H&E-stained image shows structural similarities with the UV-PAM and UV-AutoM images. Due to strong absorption of cell nuclei in the ultraviolet range, cell nuclei concentrated in the hippocampus appear bright in the UV-PAM image and dark in the UV-AutoM image. These complementary imaging contrast mechanisms enable dual-modality label-free imaging to provide more structural and functional information in fresh, unprocessed tissue.

[0183] Throughout the description, brain samples were extracted from Swiss Webster mice and subsequently fixed in 10% neutral buffered formalin for 24 hours at room temperature. For thin sections (2-8 μm), samples were processed through the FFPE workflow and sliced by a microtome. FFPE tissue sections were dewaxed using xylene and mounted on quartz slides for imaging by the described UV-PAM and UV-AutoM systems, followed by H&E staining procedures. For thick sections (20-200 μm), samples were cut directly by a vibratome at varying target thicknesses.

[0184] Although the invention has been described with reference to preferred embodiments, it will be appreciated by those skilled in the art that the invention may be embodied in other forms.

[0185] The advantageous embodiments and / or further developments disclosed above—except, for example, in the case of explicit dependencies or inconsistent alternatives—can be used individually or also in any desired combination with one another.

Claims

1. A computer-implemented method for generating a pseudo-hematoxylin and eosin (H&E) stained image, wherein: The method comprises: receiving an input image, wherein the input image is an ultraviolet-based autofluorescence microscopy UV-AutoM image or an ultraviolet-based photoacoustic microscopy UV-PAM image of an unlabeled sample, wherein the input image is a grayscale image; transforming the input image into a pseudo H&E stained image of the input image using a generative adversarial network; and Output the pseudo H&E staining image, wherein the input image received in the form of a UV-AutoM image is an estimated UV-AutoM image generated from a sequence of speckle-illuminated images captured according to a scanning trajectory, wherein the estimated UV-AutoM image has a higher resolution than each speckle-illuminated image in the sequence, The estimated UV-AutoM image is generated by: a) initializing a high-resolution image object based on interpolating an average of a sequence of speckle-illuminated images; b) For each speckle-illuminated image in the sequence: i) generating an image of the estimated speckle illumination by computationally shifting the high-resolution image object to a specific position in the scanning trajectory; ii) determining a filtered object-pattern composite in the frequency domain based on the image of said estimated speckle illumination and the optical transfer function in the frequency domain; iii) determining an updated estimated image of speckle illumination in the frequency domain based on the estimated image of speckle illumination in the frequency domain, the corresponding captured image of speckle illumination in the frequency domain, the filtered object-pattern composite in the frequency domain and the optical transfer function; iv) updating said high resolution image object based on said updated estimated image of speckle illumination, said estimated image of speckle illumination in the spatial domain and the speckle pattern; v) updating a speckle pattern based on the updated estimated image of speckle illumination, said estimated image of speckle illumination and said high resolution image object; and vi) applying Nesterov momentum acceleration to the high resolution image object and the speckle pattern; and c) iteratively performing step b) until convergence is detected to reconstruct said high-resolution image object, said high-resolution image object being the estimated UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area.

2. The computer-implemented method of claim 1 , wherein: The generative adversarial network is a generative adversarial network with cycle consistency.

3. The computer-implemented method of claim 1 or 2, wherein: The method includes training the generative adversarial network using unpaired input and H&E stained images.

4. The computer-implemented method of claim 1 or 2, wherein: The generative adversarial network includes four deep convolutional neural networks, including: a first generator deep convolutional neural network configured to transform the input image into a generated H&E image; a second generator deep convolutional neural network configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator deep convolutional neural network configured to distinguish between H&E images of a training set and generated H&E images generated by said first generator deep convolutional neural network; and a second discriminator deep convolutional neural network configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

5. The computer-implemented method of claim 4, wherein: The first generator deep convolutional neural network and the second generator deep convolutional neural network are ResNet-based or U-Net-based generator networks.

6. The computer-implemented method of claim 4, wherein: The first discriminator deep convolutional neural network and the second discriminator deep convolutional neural network are PatchGAN discriminator networks.

7. The computer-implemented method of claim 1 or 2, wherein: The input image received in the form of a UV-PAM image is generated in the following manner: controlling a galvanometer scanner of the focusing assembly to focus the ultraviolet light on the sample according to a scanning trajectory; receiving, by at least one transducer, photoacoustic waves emitted by the sample in response to the ultraviolet light; and The UV-PAM image is generated based on the photoacoustic wave.

8. A computer system configured to generate a pseudo-hematoxylin and eosin (H&E) stained image, wherein: The computer system includes one or more memories having executable instructions stored therein, and one or more processors, wherein execution of the executable instructions by the processors causes the one or more processors to: receiving an input image, wherein the input image is an ultraviolet-based autofluorescence microscopy UV-AutoM image or an ultraviolet-based photoacoustic microscopy UV-PAM image of an unlabeled sample, wherein the input image is a grayscale image; transforming the input image into a pseudo H&E stained image of the input image using a generative adversarial network; and Output the pseudo H&E staining image, wherein the input image received in the form of an estimated UV-AutoM image is generated from a sequence of speckle-illuminated images captured according to a scanning trajectory, wherein the estimated UV-AutoM has a higher resolution than each speckle-illuminated image in the sequence, The UV-AutoM image is generated by the one or more processors in the following manner: a) initializing a high-resolution image object based on interpolating an average of a sequence of speckle-illuminated images; b) For each speckle-illuminated image in the sequence: i) generating an image of the estimated speckle illumination by computationally shifting the high-resolution image to a specific position in the scanning trajectory; ii) determining a filtered object-pattern composite in the frequency domain based on the image of said estimated speckle illumination and the optical transfer function in the frequency domain; iii) determining an updated estimated image of speckle illumination in the frequency domain based on the estimated image of speckle illumination in the frequency domain, the corresponding captured image of speckle illumination in the frequency domain, the filtered object-pattern composite in the frequency domain and the optical transfer function; iv) updating said high resolution image object based on said updated estimated image of speckle illumination, said estimated image of speckle illumination in the spatial domain and the speckle pattern; v) updating a speckle pattern based on the updated estimated image of speckle illumination, said estimated image of speckle illumination and said high resolution image object; and vi) applying Nesterov momentum acceleration to the high resolution image object and the speckle pattern; and c) iteratively performing step b) until convergence is detected to reconstruct a high-resolution image object, which is an estimated UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area.

9. The computer system according to claim 8, wherein: The generative adversarial network is a generative adversarial network with cycle consistency.

10. The computer system according to claim 8 or 9, wherein: The one or more processors are configured to train the generative adversarial network using unpaired input grayscale images and H&E stained images.

11. The computer system according to claim 8 or 9, wherein: The generative adversarial network includes four deep convolutional neural networks, including: a first generator deep convolutional neural network configured to transform the input image into a generated H&E image; a second generator deep convolutional neural network configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator deep convolutional neural network configured to distinguish between H&E images of a training set and generated H&E images generated by said first generator deep convolutional neural network; and a second discriminator deep convolutional neural network configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

12. The computer system according to claim 11, wherein: The first generator deep convolutional neural network and the second generator deep convolutional neural network are ResNet-based or U-Net-based generator networks.

13. The computer system according to claim 11, wherein: The first discriminator deep convolutional neural network and the second discriminator deep convolutional neural network are PatchGAN discriminator networks.

14. The computer system according to claim 8 or 9, wherein: The input image received in the form of a UV-PAM image is generated in the following manner: controlling a galvanometer scanner of the focusing assembly to focus the ultraviolet light on the sample according to a scanning trajectory; receiving, by at least one transducer, photoacoustic waves emitted by the sample in response to the ultraviolet light; and The UV-PAM image is generated based on the photoacoustic wave.

15. One or more non-transitory computer-readable media comprising executable instructions for configuring a computer system to generate a pseudo-hematoxylin and eosin (H&E) stained image, wherein: The computer system has one or more processors, wherein the one or more processors execute the executable instructions to configure the computer system to: receiving an input image, wherein the input image is an ultraviolet-based autofluorescence microscopy UV-AutoM image or an ultraviolet-based photoacoustic microscopy UV-PAM image of an unlabeled sample, wherein the input image is a grayscale image; transforming the input image into a pseudo H&E stained image of the input image using a generative adversarial network; and Output the pseudo H&E staining image, wherein the input image received in the form of an estimated UV-AutoM image is generated from a sequence of speckle-illuminated images captured according to a scanning trajectory, wherein the estimated UV-AutoM has a higher resolution than each speckle-illuminated image in the sequence, The UV-AutoM image is generated by the one or more processors in the following manner: a) initializing a high-resolution image object based on interpolating an average of a sequence of speckle-illuminated images; b) For each speckle-illuminated image in the sequence: i) generating an image of the estimated speckle illumination by computationally shifting the high-resolution image to a specific position in the scanning trajectory; ii) determining a filtered object-pattern composite in the frequency domain based on the image of said estimated speckle illumination and the optical transfer function in the frequency domain; iii) determining an updated estimated image of speckle illumination in the frequency domain based on the estimated image of speckle illumination in the frequency domain, the corresponding captured image of speckle illumination in the frequency domain, the filtered object-pattern composite in the frequency domain and the optical transfer function; iv) updating said high resolution image object based on said updated estimated image of speckle illumination, said estimated image of speckle illumination in the spatial domain and the speckle pattern; v) updating a speckle pattern based on the updated estimated image of speckle illumination, said estimated image of speckle illumination and said high resolution image object; and vi) applying Nesterov momentum acceleration to the high resolution image object and the speckle pattern; and c) iteratively performing step b) until convergence is detected to reconstruct a high-resolution image object, which is an estimated UV-AutoM image with enhanced subcellular resolution over a centimeter-scale imaging area.

16. One or more non-transitory computer-readable media according to claim 15, wherein: The generative adversarial network is a generative adversarial network with cycle consistency.

17. The one or more non-transitory computer-readable media of claim 15, wherein: The one or more processors execute the executable instructions to configure the computer system to train the generative adversarial network using unpaired input grayscale images and H&E stained images.

18. One or more non-transitory computer-readable media according to any one of claims 15 to 17, wherein: The generative adversarial network includes four deep convolutional neural networks, including: a first generator deep convolutional neural network configured to transform the input image into a generated H&E image; a second generator deep convolutional neural network configured to transform the H&E image into a generated UV-AutoM or UV-PAM image; a first discriminator deep convolutional neural network configured to distinguish between H&E images of a training set and generated H&E images generated by said first generator deep convolutional neural network; and a second discriminator deep convolutional neural network configured to distinguish between the UV-AutoM or UV-PAM images of the training set and the generated UV-AutoM or UV-PAM images generated by the second generator deep convolutional neural network.

19. One or more non-transitory computer-readable media according to any one of claims 15 to 17, wherein: The first generator deep convolutional neural network and the second generator deep convolutional neural network are ResNet-based or U-Net-based generator networks.

20. One or more non-transitory computer-readable media according to any one of claims 15 to 17, wherein: The first discriminator deep convolutional neural network and the second discriminator deep convolutional neural network are PatchGAN discriminator networks.

21. One or more non-transitory computer-readable media according to any one of claims 15 to 17, wherein: The input image received in the form of a UV-PAM image is generated in the following manner: controlling a galvanometer scanner of the focusing assembly to focus the ultraviolet light on the sample according to a scanning trajectory; receiving, by at least one transducer, photoacoustic waves emitted by the sample in response to the ultraviolet light; and The UV-PAM image is generated based on the photoacoustic wave.

Images