Method and apparatus for Fourier ptychographic microscopy with coded illumination

Abstract

A method for determining a coding pattern for a light source in an FPM system is provided. The method includes determining an encoding matrix that specifies a light source pattern for a multiplexed low-resolution image. The encoding matrix is ​​used to generate the multiplexed low-resolution image. High-resolution amplitude and phase reconstruction is performed using the FPM algorithm and the multiplexed low-resolution image. A total loss function is calculated that includes an exclusive combination regularizer term that promotes sparse grouping of light sources and diversity of light source patterns among the light source patterns in the encoding matrix. The method also includes determining whether optimization of the encoding matrix is ​​complete. If optimization of the encoding matrix is ​​complete, the method includes storing the optimized encoding matrix for use by the FPM system and applying the optimized encoding matrix during use of the FPM system.

Description

[Technical Field]

[0001] This application relates to sample imaging, and more particularly to methods and apparatus for Fourier ptychography microscopy (or microscopy) using coded illumination. [Background technology]

[0002] Fourier ptychographic microscopy (FPM) is a microscopy technique that allows for high-resolution imaging over a large field of view. In FPM, an array of light sources may be used to illuminate a sample while capturing a series of low-resolution images. Each low-resolution image is illuminated by a different light source or set of light sources from the array. The captured low-resolution images are then stitched together in the Fourier domain to generate a high-resolution image.

[0003] FPM offers numerous advantages over traditional microscopy, including significantly higher spatial-bandwidth production, a simple and low-cost setup with fewer mechanical movements, and a small footprint. However, due to the number of images captured, FPM suffers from long image acquisition times, which limits its application to imaging moving samples and videography. Reconstructing a high-resolution image from the captured low-resolution images can also be prohibitively time-consuming in some applications. Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, there is a need for improved methods and apparatus for FPM. [Means for solving the problem]

[0005] Some embodiments provide a method for determining a light source coding pattern for an FPM system. The method includes determining an encoding matrix that specifies a light source pattern for a multiplexed low-resolution image, generating the multiplexed low-resolution image using the encoding matrix, and performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution image. The method also includes calculating a total loss function, which includes an exclusive combination regularization term that promotes sparse grouping of light sources and diversity of light source patterns among the light source patterns of the encoding matrix. The method also includes determining whether optimization of the encoding matrix is ​​complete. If optimization of the encoding matrix is ​​complete, the method includes storing the optimized encoding matrix for use by the FPM system and / or applying the optimized encoding matrix during use of the FPM system.

[0006] Some embodiments also provide a method for determining a coding pattern for a light source in an FPM system. The method includes determining an initial coding matrix that specifies an LED pattern for a multiplexed low-resolution image, generating a single-LED low-resolution image, and generating a multiplexed low-resolution image using the coding matrix and the single-LED low-resolution image. The method also includes performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution image, and calculating a total loss function that includes an exclusive combination regularization term that promotes sparse grouping of light sources and diversity of light source patterns among the light source patterns of the coding matrix. The method also includes determining whether optimization of the coding matrix is ​​complete. If the optimization of the encoding matrix is ​​not complete, the method includes updating the encoding matrix based on the calculated total loss function; generating an updated multiplexed low-resolution image using the updated encoding matrix and the single-LED low-resolution image; performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the updated multiplexed low-resolution image; calculating an updated total loss function including an exclusive combination regularization term; and determining whether the optimization of the encoding matrix is ​​complete based on the updated total loss function.

[0007] Some embodiments also provide a Fourier ptychography imaging system. The Fourier ptychography imaging system includes a plurality of light sources configured to emit light at a sample location, an optical system configured to capture (or image) at least a portion of a sample disposed at the sample location, and an image capture device configured to capture images of the sample through the optical system under different lighting conditions provided by the plurality of light sources. The Fourier ptychography imaging system also includes a processor and a memory coupled to the processor. The memory includes an encoding matrix specifying light source patterns to be used during capture of the low-resolution images by the imaging device, the encoding matrix including light source patterns optimized using a loss function, the loss function including an exclusive combination regularization term that promotes sparse grouping of light sources and diversity of light source patterns among the light source patterns of the encoding matrix. The memory also includes computer-executable instructions stored therein, which, when executed by the processor, cause the processor to acquire images of the sample disposed at the sample location. Each of the multiple images is illuminated using a different light source pattern specified in the encoding matrix. The computer-executable instructions, when executed by the processor, also cause the processor to store the image and initiate FPM reconstruction to generate a reconstructed image based on the stored image.

[0008] A system of one or more computers may be configured to have software, firmware, hardware, or a combination thereof installed on the system, causing the system to execute operations during operation to perform specified operations or behaviors. In some embodiments, one or more computers may include one or more graphics processing units (GPUs). One or more computer programs may include instructions that, when executed by a data processing device, are configured to perform specified operations or behaviors.

[0009] Other features and aspects of the present invention will become more fully apparent from the following detailed description, the appended claims, and the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1A is a diagram illustrating an example of an FPM system provided in accordance with an embodiment of the present disclosure; FIG. 1B is a diagram illustrating an example of a light source array that may be used with the FPM system of FIG. 1A in accordance with an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of a method for determining an encoding matrix for an FPM system according to an embodiment of the present disclosure. [Figure 3] FIG. 3A is a diagram illustrating an LED pattern for an exemplary initial encoding matrix according to an embodiment of the present disclosure; FIG. 3B is a diagram illustrating an updated LED pattern for the exemplary initial encoding matrix described in FIG. 3A after being optimized by using an exclusive combination regularization term according to an embodiment of the present disclosure. [Figure 4] FIG. 4A is a diagram illustrating an example of a simulated ground truth image from which multiple low-resolution single LED images may be generated according to an embodiment of the present disclosure; FIG. 4B is a diagram illustrating an example of a low-resolution single LED image generated from the simulated ground truth of FIG. 4A according to an embodiment of the present disclosure; FIG. 4C is a diagram illustrating an example of a multiplexed low-resolution image generated from the single LED image of FIG. 4B based on an encoding matrix according to an embodiment of the present disclosure; and FIG. 4D is a diagram illustrating an example of a high-resolution reconstructed image that may be generated from the multiplexed low-resolution image of FIG. 4C according to an embodiment of the present disclosure. [Figure 5AB] 5A and 5B are diagrams illustrating example LED patterns after optimization of the encoding matrix without and with an exclusive combination regularization term, respectively, according to an embodiment of the present disclosure. [Figure 5CD] 5C and 5D are diagrams illustrating encoding matrices corresponding to the LED patterns of FIGS. 5A and 5B, respectively, according to an embodiment of the present disclosure. [Figure 6]FIG. 6A is a graphical comparison of the mean squared error (MSE) observed in a loss function during training with and without an exclusive combination regularizer in accordance with an embodiment of the present disclosure; FIG. 6B is a graphical comparison of the MSE during optimization of an encoding matrix versus training iterations with and without an exclusive combination regularizer in accordance with an embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates a first exemplary method for determining a coding pattern for a light source of an FPM system according to an embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates a second exemplary method for determining a coding pattern for a light source of an FPM system according to an embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates a flow diagram of a system and process for converting an initial encoding matrix to an optimized encoding matrix and using the optimized encoding matrix according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Regardless of grammatical term usage, terms may include male, female, or other gender designations.

[0012] As described above, while FPM has many advantages, the substantial time required to obtain results using this technique has limited its use in some applications. The primary delays associated with FPM include the time required to capture multiple low-resolution images and the time required to reconstruct a high-resolution image from the captured low-resolution images. According to the embodiments provided herein, the FPM image capture time can be significantly reduced, thereby enabling FPM to be used in a wider range of applications (e.g., imaging moving samples, clinical trials for medical diagnosis / treatment, or any application where faster results are beneficial).

[0013] During FPM, a sample is illuminated by an array of light sources, each of which (e.g., light-emitting diodes (LEDs)) emits light toward the sample from a different angle and / or position. Low-resolution images of the sample captured using different LEDs from the LED array are processed in the Fourier domain (e.g., via amplitude and phase reconstruction based on the low-resolution images) to generate a high-resolution image.

[0014] Using a single LED each time to illuminate the sample for each low-resolution image is time-consuming. To shorten image capture time during FPM, multiplexing or "coded illumination" techniques have been developed, in which different patterns (e.g., combinations) of LEDs in an LED array are used to illuminate the sample during low-resolution image capture. However, it has been difficult to select a specific pattern of LEDs to use for each low-resolution image without affecting the quality of the reconstructed image. One approach to determining LED patterns for image capture during FPM can be found in the following non-patent document: M. Kellman, E. Bostan, M. Chen, and L. Waller, "Data-Driven Design for Fourier Ptychographic Microscopy," IEEE International Conference on Computational Photography 2019 (ICCP), 2019, pp. 1-8, doi:10.1109 / ICCPHOT.2019.8747339 Hereafter, the above document will be abbreviated as "Kellman et al." Kellman et al. describe using a neural network trained to learn LED multiplexing patterns to reduce the number of low-resolution images required for FPM image reconstruction. While effective, such an approach can require significant optimization and can converge slowly. Examples of other approaches to coded / multiplexed illumination include the following non-patent literature: Lei Tian, ​​Ziji Liu, Li-Hao Yeh, Michael Chen, Jingshan Zhong, and Laura Waller, “Computational illumination for high-speed in vitro Fourier ptychographic microscopy,” Optica 2, 904–911 (2015) Lei Tian, ​​Xiao Li, Kannan Ramchandran, and Laura Waller, “Multiplexed coded illumination for Fourier ptychography with an LED array microscope,” Biomed. Opt. Express 5, 2376–2389 (2014) The coded / multiplexed illumination techniques of these and other publications can benefit from improved LED pattern selection for low-resolution images during FPM.

[0015] In the embodiments provided herein, a regularization term referred to as the exclusivity coupling (EC) regularization term is added to the FPM loss function used to determine the FPM encoded illumination pattern (e.g., the light source pattern used for each low-resolution image). The EC adjustment term may be employed to encourage diversification of light source patterns, the use of fewer light sources in a light source pattern, and the use of patterns with more spatially separated light sources, which may be shown in the following mathematical equation (1):

[0016] (Number 1) EC Regularization Term = λ coupling * Norm(CC T -diag(CC T ) * I)

[0017] In the above equation, "C" represents the encoding matrix that defines the encoding pattern of the light array during image capture. coupling " represents an adjustable hyperparameter that can be used to determine the strength of the exclusivity coupling. "I" represents the identity matrix. As mentioned above, "λ coupling A larger value of λ encourages greater diversity in light source patterns, the use of fewer light sources within the light source pattern, and the use of patterns with light sources that are more spatially separated. coupling The selection of " may be based on trial and error, past experience in selecting light source patterns for FPM (e.g., promoting pattern diversity and sparsity with image quality during coded illumination), etc.

[0018] The exclusive combination regularizer in Equation (1) can be used with any suitable loss function used in FPM (e.g., a differentiable FPM loss function). With the addition of the exclusive combination (EC) regularizer, the total loss function (LF) becomes the sum of the FPM loss function used and the EC regularizer. This can be shown in Equation (2) below.

[0019] (Number 2) Total LF = FPM Loss Function + λ coupling * Norm(CC T -diag(CC T ) * I)

[0020] In some embodiments, the FPM loss function used may be the FPM loss function described in Kelman et al. However, other FPM loss functions can be used. The FPM loss function described in Kelman et al. is referenced in Equation (3) below.

[0021]

number

[0022] In the above number 3, "N" represents the total number of training images. * n " represents the reconstructed image. "x ~ n " represents the nth ground truth image. "γ" represents the loss function weight between the phase (γ=0) and amplitude (γ=1) loss functions. "|-|" represents the amplitude of the reconstructed image. "∠" represents the phase of the reconstructed image.

[0023] As will be described below with reference to FIGS. 1A-9, the use of exclusive joins in the total loss function can improve coded illumination LED pattern selection by improving convergence (as well as the rate of convergence) during coded illumination pattern generation.

[0024] 1A illustrates an exemplary Fourier ptychography microscopy (FPM) system 100 provided in accordance with an embodiment of the present disclosure. Referring to FIG. 1A, the FPM system 100 includes a light source array 102 having a plurality of light sources 102 configured to emit light onto a sample location 104. a-n have

[0025] The optical system (or optical system) 106 is configured to image at least a portion of a sample 108 disposed at a sample position 104. As shown in FIG. 1A, the imager 110 is configured to image the plurality of light sources 102 of the light source array 102. a-n Images (e.g., low-resolution images 112) of the sample 108 are taken through the optical system 106 under different lighting conditions provided by a-n) In some embodiments, the different lighting conditions may be selected based on an encoding matrix that defines which illuminant patterns are used for each low-resolution image. In some embodiments, the encoding matrix may be determined by training a neural network with a loss function that includes a regularization term for exclusive connections that promotes diversity and sparsity in the illuminant patterns (e.g., as described above with reference to Equation (1) and further below).

[0026] A computer 114 having a processor 116 can be coupled to the imaging device 110 and can receive and store in memory images (e.g., low-resolution images) captured by the imaging device 110. In some embodiments, the images can be stored in memory 118 (e.g., RAM, a hard drive, and / or another type of memory) associated with the processor 116. Alternatively or additionally, the image data can be stored in external memory 120 (e.g., local external memory, remote storage, cloud storage, or any combination thereof). A display 122 having a user interface 124 can be coupled to the processor 116, which may display, for example, low-resolution images, reconstructed high-resolution images, etc.

[0027] The light source array 102 includes light sources 102 a-n , which may be controlled by the processor 116 or another suitable processor, microprocessor, controller, microcontroller, digital signal processor (DSP), or field programmable gate array (FPGA) configured to function as a microcontroller, etc.

[0028] In some embodiments, the plurality of light sources 102 in the light source array 102 a-n can be individually controlled, either alone or in conjunction with one or more light sources 102 a-n(e.g., as determined by the illustrated encoding matrix 126 stored in memory 118, although other storage locations may be used, such as in external memory 120 or in the memory of another processor used to control the light source array 102).

[0029] For example, light source 102 a-n The light sources 102 may include light emitting diodes (LEDs), monochromatic or single bandwidth light sources, multi-bandwidth light sources (e.g., RGB LEDs), superluminescent LEDs, laser diodes, particularly semiconductor laser diodes, thermal emitters, fiber-based light sources, etc. a-n may be the same or may be one or more light sources 102 a-n may differ with respect to at least one of the following characteristics: wavelength, spectral bandwidth, spatial emission characteristics, temporal emission characteristics such as continuous or pulsed operation, temporal and / or spatial coherence, coherence parameters as a measure of brightness or range, etc.

[0030] In some embodiments, the light source array 102 may have approximately 80 to 280 individually controllable LEDs arranged in an xy grid, e.g., a 16x16 LED array, with each LED separated by approximately 1 to 10 mm and emitting at approximately 0.4 to 0.7 micrometers, as shown in FIG. 1B. In one specific embodiment, the LEDs may be separated by approximately 2.5 to 3.5 mm and may emit at any one or any combination of wavelengths of 0.45, 0.51, and 0.62 micrometers. Other light source array configurations, number of light sources, light source types, and / or emission wavelengths may be employed. As noted above, although the processor 116 is shown in FIG. 1B controlling the light source array 102, other embodiments may employ different processors or other control mechanisms to control the operation of the light source array 102.

[0031] The optical system 106 (FIG. 1A) may include, for example, an optical objective (or optical objective lens) 106a and a focusing lens 106b. Other optical components may be used. As described above, one advantage of FPM is that it allows for the use of low-cost, low-resolution optical components. In some embodiments, the optical objective 106a may have a numerical aperture (NA) of approximately 0.05 to 0.9. Optical objectives with other NAs may be used. In one or more embodiments, the focusing lens 106b may be a tube lens, such as an achromatic tube lens or another suitable lens. The imager 110 may include any suitable imager capable of imaging a sample through the optical system 106, such as a CMOS sensor. An exemplary pixel size may range from approximately 1 micrometer to approximately 10 micrometers, although other pixel sizes may be used.

[0032] In some embodiments, processor 116 may be a central processing unit (CPU). In other embodiments, processor 116 may include and / or be implemented as one or more other computing resources, such as a microprocessor, a microcontroller, an embedded microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA) configured to function as a microcontroller, etc. Computer 114 may include any suitable computing device, such as a tablet computer, a laptop computer, a desktop computer, a server, etc.

[0033] Memory 118 and / or 120 may be any suitable type of memory, such as, but not limited to, one or more of volatile memory and / or non-volatile memory (e.g., RAM, DRAM, SRAM, cache, hard drive, combinations thereof, etc.). In other words, memory 118 and / or 120 may include two or more types of memory. Memory 118 and / or 120 may have instructions stored therein that, when executed by processor 116, cause processor 116 to perform various operations specified by the one or more stored instructions. A first type of memory (e.g., a hard drive) may store instruction code and data that may be transferred to a second type of memory (e.g., RAM) for execution. In some embodiments, memory 118 and / or 120 may include one or both types of memory.

[0034] Display 122 may be any suitable display, including, for example, a light-emitting diode (LED) display, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, etc. User interface 124 may include, for example, one or more of a display screen, a touch panel / touch screen, an audio speaker, and a microphone. In some embodiments, user interface 124 may be controlled by processor 116, and functionality of user interface 124 may be implemented, at least in part, by computer-executable instructions (e.g., program code or software) stored in memory 118 and / or executed by processor 116.

[0035] FIG. 2 illustrates an example method 200 for determining an encoding matrix for an FPM system (e.g., encoding matrix 126 shown in FIG. 1A ) in accordance with embodiments provided herein. Referring to FIG. 2 , at block 202, an initial encoding matrix is ​​determined. In some embodiments, the encoding matrix may include multiple vectors, each specifying an LED pattern for each low-resolution image captured using the FPM system. In other words, each vector may specify which LEDs are illuminated (and / or the brightness of each illuminated LED) during image capture of each low-resolution image. For example, if 50 low-resolution images are captured for use during FPM reconstruction, the encoding matrix may include 50 vectors to identify 50 LED patterns used to illuminate the sample during capture of the low-resolution images. In some embodiments, the initial encoding matrix may be randomly initialized. In other embodiments, the initial encoding matrix may be determined based on a best guess of the encoding matrix (based on factors such as the LED array used, the type of sample being imaged, the optical system used in the FPM system, etc.). In at least one embodiment, the LED pattern can be divided into a bright-field LED pattern and a dark-field LED pattern, where only a subset of the inner LEDs are illuminated in the bright-field LED pattern and only a subset of the outer LEDs are illuminated in the dark-field LED pattern. In some embodiments, the initial encoding matrix can include a random bright-field LED pattern (e.g., having a randomized pattern of only bright-field LEDs) and a random dark-field LED pattern (e.g., having a randomized pattern of only dark-field LEDs). Any other suitable method can be used to determine the initial encoding matrix.

[0036] FIG. 3A illustrates LED patterns for an exemplary initial encoding matrix according to embodiments provided herein. Referring to FIG. 3A, eight LED patterns 302a through 302h are shown (e.g., corresponding to eight vectors in the initial encoding matrix). As described above, in some embodiments, the LED patterns can be divided into bright-field LED patterns (e.g., LED patterns 302a and 302b) and dark-field LED patterns (e.g., LED patterns 302c through 302h), where only a subset of the inner LEDs are illuminated in the bright-field LED patterns and only a subset of the outer LEDs are illuminated in the dark-field LED patterns. Other initial LED patterns may be specified by the initial encoding matrix. FIG. 3B illustrates updated LED patterns 302a′ through 302h′ following optimization using the above-described exclusiveness-combination regularization term (e.g., Equation (1)) according to embodiments provided herein, as further described below, for the exemplary initial encoding matrix of FIG. 3A.

[0037] 2, following determination of the initial encoding matrix (at block 202), a single-source, low-resolution image is generated at block 204 (e.g., a single light source, such as an LED, is used to illuminate the sample during image capture). The single-source, low-resolution image may be an experimentally determined image (e.g., captured using an FPM system, such as FPM system 100 of FIG. 1A) or a simulated image. As an example, for a 256 LED array, 256 low-resolution images may be generated, each illuminated by a different one of the LEDs in the LED array.

[0038] FIG. 4A illustrates a simulated ground truth 402 from which multiple low-resolution single LED images 404 are derived. a-n (FIG. 4B) may be generated. In some embodiments, the ground truth 402 may be a combination of stock images representing any suitable phase and / or amplitude at high resolution. a-nA forward model of the FPM reconstruction process may be used to generate a low-resolution image from a ground truth (e.g., ground truth 402), thereby simulating the low-resolution image. Both the bright-field and dark-field low-resolution images may be simulated using appropriate sections of the Fourier transform(s) of the ground truth (e.g., ground truth 402). For example, the low-resolution image may be a low-pass filtered image of a shifted Fourier transform (e.g., different shift magnitudes based on different illumination angles) of the ground truth. The simulated low-resolution image may be further magnified based on the applied optical system.

[0039] Referring again to FIG. 2 , at block 206, a multiplexed image is generated from the single-LED images and an encoding matrix (e.g., an initial encoding matrix described below or a subsequently determined encoding matrix). For example, for each LED pattern specified in the encoding matrix, multiple single-LED images may be combined (e.g., multiplexed) into a single image to simulate an image captured with that LED pattern. In some embodiments, multiple images may be combined by adding or averaging, such as using a weighted average of the images using the encoding matrix values ​​(e.g., LED luminance or brightness) as weighting factors. For example, multiple images may be added together pixel-by-pixel, with each pixel weighted by the LED luminance value specified in the encoding matrix. In one or more embodiments, all single-LED images may be generated (e.g., experimentally or by simulation), where each LED is fully bright. The encoding matrix may specify a percentage luminance (brightness ratio) for each LED in the LED pattern. The multiplexed image for the LED pattern may then be generated by adding or averaging the associated single-LED images weighted by the LED luminance specified in the encoding matrix. In one or more other embodiments, single-LED images can be generated where the LEDs used have different brightnesses, and the multiplexed image is generated by an encoding matrix that takes into account the differences in LED brightness among the single-LED images.

[0040] FIG. 4C shows the superimposed low-resolution image 406 a-m , which are generated based on the encoding matrix for the single LED image 404 in FIG. 4B. a-n For example, once generated, a low-resolution single LED image 404 a-n (FIG. 4B) are added and / or averaged to produce a multiplexed low-resolution image 406 a-m , which corresponds to the LED pattern specified in the encoding matrix as described above.

[0041] In block 208, FPM reconstruction is performed on the encoded image (e.g., generated in block 206). Specifically, when a high-resolution image is generated, high-resolution amplitude and phase reconstruction using the FPM algorithm and a multiplexed image generated from the encoding matrix are used (block 206). In some embodiments, the FPM algorithm described in Kelman et al. can be used. Any suitable FPM algorithm can be used for FPM reconstruction (e.g., any differentiable FPM algorithm can be used). Another exemplary FPM algorithm that can be used is the alternating projection method described in the following non-patent document: R.W. Gerchberg and W.O. Saxton, "A practical algorithm for the determination of phase from image and diffraction plane pictures," Optik, Bd. 35, pp. 227-246 (1972). Further, the construction of maximum likelihood estimation described in the following non-patent document can be mentioned. L. Bian, J. Suo, G. Zheng, K. Guo, F. Chen, and Q. Dai, "Fourier ptychographic reconstruction using Wirtinger flow optimization," Optics Express, Bd. 23, NR. 4, pp. 4856-4866 (2015). L. Bian, J. Suo, J. Chung, X. Ou, C. Yang, F. Chen, and Q. Dai, "Fourier ptychographic reconstruction using Poisson maximum likelihood and truncated Wirtinger gradient," Scientific Reports, Volume 6, Number 1, No. 27384, p. 7 (2016).

[0042] FIG. 4D illustrates a high-resolution reconstructed image 408, which may be, for example, a multiplexed low-resolution image 406 of FIG. 4C. a-m can be generated from

[0043] Referring again to FIG. 2, at block 210, a total loss function (e.g., including an exclusive joint regularization term) is determined based on the high-resolution images generated at block 208. As shown in equation (2), the total loss function includes the FPM loss function of the FPM algorithm used plus the exclusive joint regularization term of equation (1). For example, the loss function of equation (3) of Kelman et al. can be calculated and added to the exclusive joint regularization term (equation (1) above). Other FPM loss functions can be applied.

[0044] In general, the FPM loss function may be calculated based on the FPM-reconstructed image and the ground truth image. In some embodiments, a simulated low-resolution image may be obtained from the FPM-reconstructed high-resolution image and compared (e.g., pixel-by-pixel) with one or more multiplexed low-resolution images or one or more single-light source low-resolution images to determine whether the reconstructed image accurately depicts the details of the low-resolution image. In some embodiments, this may involve intentionally reducing the detail in the high-resolution image to approximate the level of detail in the low-resolution image. For example, a forward model of the FPM reconstruction process may be used to simulate a low-resolution image from the FPM-reconstructed high-resolution image, which may be derived from the ground truth 402 (FIGS. 4A-4B) and the low-resolution single-LED image 404. a-n This is similar to the method described above for generating

[0045] The exclusive combination regularization term is the current encoding matrix according to Equation (1) and the hyperparameter λ coupling and the selected values ​​of . As mentioned above, the exclusive-combination regularization term promotes the diversity and sparsity of light sources (e.g., multiple LEDs) in the light source pattern. As a simplified example, assume that an LED array uses two LEDs, LED1 and LED2. Two low-resolution images, Image 1 and Image 2, are generated. The luminance of LED1 in Image 1 is C11, the luminance of LED1 in Image 2 is C21, the luminance of LED2 in Image 1 is C12, and the luminance of Image 2 is C22 (see Table 1 below).

[0046] [Table 1]

[0047] The above information can be expressed by an encoding matrix C as shown in the following equation (4).

[0048]

number

[0049]

number

[0050]

number

[0051]

number

[0052] (Number 8) norm(CC T -diag(CC T ) I) = 2 (C 11 C 21 + C 12 C 22 ) 2

[0053] (Number 9) EC Regularization Term = λ coupling * 2 (C 11 C 21 + C 12 C 22 ) 2

[0054] Equation (9) shows that the exclusive coupling (EC) regularizer is maximized when both LEDs (e.g., LED1 and LED2) are on during image 1 or image 2 (e.g., when Cu, C21, C12, and C22 are nonzero). It also shows that the EC regularizer is minimized when exclusivity is maximized (e.g., when LED1 is on and LED2 is off for image 1, and when LED1 is off and LED2 is on for image 2). Thus, the exclusive coupling regularizer discourages the use of the same LED in multiple images, and the hyperparameter λ couplingadjusts the contribution of the exclusive combination regularizer. More generally, the exclusive combination regularizer encourages the use of different LEDs per image, thereby promoting sparsity and diversity.

[0055] At block 212, a determination is made as to whether the optimization of the encoding matrix is ​​complete. If the optimization of the encoding matrix is ​​complete, the optimized encoding matrix may be stored and / or used (at block 214) as described below. Otherwise, at block 216, the encoding matrix is ​​updated. In some embodiments, the optimization of the encoding matrix may be considered complete when the overall loss function (e.g., calculated at block 210) flattens over time and / or when the gradient of the loss function falls below a predetermined threshold (e.g., approaches zero). Alternatively, the optimization of the encoding matrix may be considered complete after a predetermined number of iterations of the encoding matrix training / optimization steps (e.g., a predetermined number of iterations of blocks 206, 208, 210, 212, and 216). In some embodiments, tens to hundreds of iterations may be performed before the optimization of the encoding matrix is ​​determined to be complete (e.g., assuming the encoding matrix has not yet been optimized). However, fewer or more iterations may also be performed.

[0056] Returning to block 214, assuming optimization of the encoding matrix is ​​complete, at block 214 the optimized encoding matrix may be stored (e.g., stored in memory 118 of processor 116 or another suitable location) and / or may be utilized during subsequent image acquisition and / or reconstruction operations using FPM system 100.

[0057] Returning to block 216, if it is assumed that optimization of the encoding matrix is ​​not complete (e.g., determined in block 212), the encoding matrix may be updated. For example, the encoding matrix for the light array may be updated based on the results of the total loss function in block 210 (e.g., the encoding matrix values ​​for each LED pattern specifying which LEDs are on or off and / or the brightness of the LEDs that are on may be updated based on the gradient of the loss function). In other words, the encoding matrix may be updated based on the gradient of the total loss function (e.g., determined in block 210). As a specific example, the encoding matrix values ​​specifying LED brightness may be increased or decreased by an amount proportional to the gradient of the loss function. Any other suitable method may be used to update (e.g., optimize) the encoding matrix by applying the proposed total loss function (e.g., with an exclusive combination regularization term).

[0058] Following the updating of the encoding matrix (at block 216), method 200 returns to block 206, where a new set of multiplexed images is generated using the updated encoding matrix, followed by high-resolution amplitude and phase reconstruction using the newly generated multiplexed images (at block 208). A total loss function is then calculated (at block 210), and a determination is made (at block 212) as to whether optimization of the encoding matrix is ​​complete. Blocks 216, 206, 208, 210, and 212 are repeated until optimization of the encoding matrix is ​​complete, after which the encoding matrix is ​​stored and / or applied (used) at block 214, as described above.

[0059] 3B shows an example of updated LED patterns 302a′-302h′ for the exemplary initial encoding matrix described with reference to FIG. 3A, following optimization using the above-described exclusive coupling regularization term (e.g., Equation (1)) and method 200. As shown in FIG. 3B, following optimization of the encoding matrix, significantly less clustering of adjacent LEDs is observed, increasing LED diversity and making the patterns more spatially separated. Furthermore, by including the exclusive coupling regularization term (as described below), optimization of the encoding matrix is ​​achieved more quickly.

[0060] 5A and 5B show exemplary LED patterns after optimization of the encoding matrix without and with an exclusive combination regularization term, respectively, in accordance with embodiments provided herein. Referring to FIG. 5A, LED patterns 502a-502h are shown without the exclusive combination regularization term (e.g., λ coupling 5B, the LED patterns 502a′-502h′ are determined from an encoding matrix optimized for the case with an exclusive-combination regularization term (e.g., λ coupling >0). As shown in FIG. 5B, with the exclusive combination regularization term, the resulting light source pattern is more diverse, contains fewer illumination LEDs, and is overall more spatially separated. By way of further example, FIGS. 5C and 5D show encoding matrices 504a and 504b corresponding to the LED patterns of FIGS. 5A and 5B, respectively. As shown in FIG. 5C, without the regularization term, significant clustering of LEDs can be found in both the bright-field and dark-field LED patterns. Specifically, without the exclusive combination regularization term, the loss function tends to form redundant LED groupings, such as LED groupings 506a through 506e, compared to the LED groupings of FIG. 5D formed by the use of the exclusive combination regularization term in the loss function.

[0061] The use of an exclusive combination regularizer during optimization of the encoding matrix can provide faster and more desirable convergence, as shown in Figures 6A and 6B. For example, Figure 6A graphically illustrates a comparison of the mean square error (MSE) observed in the loss function during training with and without the exclusive combination regularizer. As shown in Figure 6A, the MSE observed at loss function convergence with the exclusive combination regularizer is approximately 32% lower than the MSE observed at loss function convergence without the exclusive combination regularizer.

[0062] Regarding the convergence rate, Fig. 6B shows a graph of MSE versus training iterations during optimization of the encoding matrix with and without the exclusive combination regularizer. As shown in Fig. 6B, when the loss function includes the exclusive combination regularizer, the loss function converges faster and to a lower level.

[0063] Therefore, it includes the exclusive combination regularization term in Equation (1) and the hyperparameter λ coupling By appropriately selecting , the LED pattern becomes more diverse, the LED pattern contains fewer light sources, and the LED pattern has more spatially separated light sources. Furthermore, the optimization of the encoding matrix can be achieved faster and to a more desirable level.

[0064] FIG. 7 illustrates a first exemplary method 700 for determining a coding pattern for a light source in an FPM system, according to embodiments provided herein. Referring to FIG. 7 , at block 702, the method 700 includes determining an encoding matrix (e.g., an initial encoding matrix, such as initial encoding matrix 902 of FIG. 9 ) that specifies a light source pattern for a multiplexed low-resolution image. At block 704, the method 700 includes generating a multiplexed low-resolution image using the encoding matrix. At block 706, the method 700 includes performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the multiplexed low-resolution image. At block 708, the method 700 includes calculating a total loss function. The total loss function includes an exclusive-combination regularization term (e.g., the total loss function of Equation (2)) that promotes sparse grouping of light sources and diversity of light source patterns among the light source patterns of the encoding matrix. At block 710, the method 700 includes determining whether optimization of the encoding matrix is ​​complete. If optimization of the encoding matrix is ​​complete, method 700 includes storing the optimized encoding matrix for use by the FPM system (e.g., as encoding matrix 126 of FPM system 100 of FIG. 1A) and / or applying the optimized encoding matrix during use of the FPM system, at block 712. If optimization of the encoding matrix is ​​not complete, method 700 may include updating the encoding matrix (e.g., based on a total loss function), at block 714, and blocks 704, 706, 708, 710, and 714 may be repeated until optimization of the encoding matrix is ​​complete.

[0065] FIG. 8 illustrates a second exemplary method 800 for determining a coding pattern for a light source in an FPM system, according to embodiments provided herein. Referring to FIG. 8 , at block 802, the method 800 includes determining an initial coding matrix (e.g., the initial coding matrix 902 of FIG. 9 ) that specifies an LED pattern for a multiplexed low-resolution image. At block 804, the method 800 includes generating single-LED low-resolution images. At block 806, the method 800 includes generating multiplexed low-resolution images using the coding matrix and the single-LED low-resolution images. At block 808, the method 800 includes performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the multiplexed low-resolution images. At block 810, the method 800 includes calculating a total loss function, where the total loss function includes an exclusive-combination regularization term that promotes sparse grouping of light sources and diversity of light source patterns among the light source patterns in the coding matrix. At block 812, method 800 includes determining whether optimization of the encoding matrix is ​​complete. If optimization of the encoding matrix is ​​not complete, at block 814, method 800 includes updating the encoding matrix based on the calculated total loss function. Blocks 806, 808, 810, and 812 are then repeated for the updated encoding matrix. In some embodiments, if optimization of the encoding matrix is ​​not yet complete, blocks 806, 808, 810, 812, and 814 may be repeated until optimization of the encoding matrix is ​​complete. If optimization of the encoding matrix is ​​complete, at block 816, method 800 stores the optimized encoding matrix for use by the FPM system and / or uses the optimized encoding matrix during use of the FPM system. For example, the optimized encoding matrix may be stored in memory 118 of computer 114 as encoding matrix 126 (FIG. 1A).

[0066] In some embodiments, using an optimized encoding matrix (e.g., encoding matrix 126) may reduce the number of samples (e.g., samples 108) placed at sample locations (e.g., sample location 104) in an image (e.g., image 112). a-n ), where each of the images is illuminated using a different light source pattern specified in an encoding matrix (e.g., encoding matrix 126), and further may include initiating FPM reconstruction to generate a reconstructed image based on the acquired images.

[0067] FIG. 9 illustrates a flow diagram of a process and system 900 for converting an initial encoding matrix into an optimized encoding matrix and using the optimized encoding matrix according to embodiments provided herein. Referring to FIG. 9 , an initial encoding matrix 902 is determined (e.g., as described above with reference to method 200) and processed by a computer system 904, which is programmed to perform encoding matrix optimization using the exclusive combination regularization term of Equation (1), for example, as described above with reference to method 200 of FIG. 2, method 700 of FIG. 7, or method 800 of FIG. 8. This generates an optimized encoding matrix 906 that can be used, for example, in the FPM system 100 of FIG. 1A to illuminate a sample with the LED pattern defined in the optimized encoding matrix 906. The FPM system can capture and process a low-resolution image to generate a high-resolution image 910 (e.g., using the imager 110 and processor 116 of FIG. 1A). However, other process flows and / or systems are possible.

[0068] One or more embodiments provided herein describe optimizing an encoding matrix that specifies light source patterns used during capture of a low-resolution image (e.g., used during FPM reconstruction of a high-resolution image). The encoding matrix may be optimized using a loss function with an exclusive-combination regularizer that promotes sparse grouping of light sources and diversification of light source patterns among the light source patterns of the encoding matrix. In some embodiments, optimizing the encoding matrix may include reconstructing an image based on the encoding matrix, calculating a loss function based on the reconstructed image, and updating the encoding matrix based on the calculated loss function. In other embodiments, multiple images may be reconstructed and multiple loss functions may be calculated (e.g., calculating a loss function for each reconstructed image). The encoding matrix may then be updated based on multiple loss functions (e.g., comparing the loss functions and selecting one of the multiple loss functions for updating the encoding matrix, or combining multiple loss functions, such as averaging, for updating the encoding matrix, etc.). In other words, batch reconstruction (e.g., serial or parallel) may be performed to generate multiple loss functions, and multiple loss functions may be used alone or in combination during optimization of the encoding matrix. In yet other embodiments, multiple encoding matrices may be optimized (e.g., in parallel) and used to determine the optimal encoding matrix for FPM reconstruction (e.g., via manifold optimization). Other optimization procedures are possible.

[0069] The foregoing description discloses only some example embodiments of the present invention. Modifications to the above-described apparatus and methods that fall within the scope of the present invention will be readily apparent to those skilled in the art. Accordingly, while the present invention has been described in connection with exemplary embodiments, it will be understood that other embodiments may be included within the spirit and scope of the present invention as defined by the appended claims.

Claims

1. 1. A method for determining a coding pattern for a light source of a Fourier ptychographic microscope (FPM) system, comprising: determining an encoding matrix specifying a light source pattern for the multiplexed low-resolution images; generating a multiplexed low-resolution image using the encoding matrix; performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution images; Calculating a total loss function including an exclusive combination regularizer that promotes sparse grouping of light sources and diversification of light source patterns among the light source patterns of the encoding matrix; determining whether the optimization of the encoding matrix is ​​complete; Based on the determination of whether the optimization of the encoding matrix is ​​completed, if the optimization of the encoding matrix is ​​completed, storing the optimized encoding matrix for use by the FPM system; using the optimized encoding matrix during use of the FPM system; or Do a combination of the above two method.

2. Furthermore, based on the determination of whether the optimization of the encoding matrix is ​​completed, if the optimization of the encoding matrix is ​​not completed, updating the encoding matrix based on the calculated total loss function; generating an updated multiplexed low-resolution image using the updated encoding matrix; and performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the updated multiplexed low-resolution image; Calculating an updated total loss function that includes the exclusive combination regularization term; determining whether the optimization of the encoding matrix is ​​complete; The method of claim 1 , further comprising:

3. 3. The method of claim 2, further comprising updating the encoding matrix, generating updated multiplexed low-resolution images, performing high-resolution amplitude and phase reconstruction, calculating an updated total loss function, and determining whether encoding matrix optimization is complete.

4. The method of claim 3 , wherein updating the encoding matrix comprises updating the encoding matrix based on a gradient of a most recently calculated total loss function.

5. 4. The method of claim 3, wherein generating an updated multiplexed low-resolution image includes generating an updated multiplexed low-resolution image using the single-LED low-resolution image and the updated encoding matrix.

6. 6. The method of claim 5, wherein generating an updated multiplexed low-resolution image using the updated encoding matrix includes combining single-LED low-resolution images based on the updated encoding matrix.

7. The method of claim 1 , wherein determining the encoding matrix comprises determining an initial encoding matrix.

8. The method of claim 7 , wherein determining an initial encoding matrix comprises randomly selecting values ​​for the initial encoding matrix.

9. generating the multiplexed low resolution images includes: generating a single LED low resolution image; generating a multiplexed low-resolution image using the single-LED low-resolution image and the encoding matrix; The method of claim 1 , comprising:

10. The exclusive combination regularization term is λ coupling * Norm(CC T -diag(CC T ) * I) Including, in this case, C represents the encoding matrix, C T represents the transpose of the encoding matrix, and λ coupling represents the tunable hyperparameters, The method of claim 1.

11. 1. A method for determining a coding pattern for a light source of a Fourier ptychographic microscope (FPM) system, comprising: determining an initial encoding matrix that specifies LED patterns for the multiplexed low-resolution images; generating a single LED low resolution image; generating a multiplexed low-resolution image using the initial encoding matrix and the single-LED low-resolution image; performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution images; Calculating a total loss function including an exclusive combination regularizer that promotes sparse grouping of light sources and diversification of light source patterns among the light source patterns of the initial encoding matrix; determining whether optimization of the encoding matrix is ​​complete; If, based on the determination of whether optimization of the encoding matrix is ​​completed, optimization of the encoding matrix is ​​not completed, updating the encoding matrix based on the calculated total loss function; generating an updated multiplexed low-resolution image using the updated encoding matrix and the single-LED low-resolution image; performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the updated multiplexed low-resolution image; Calculating an updated total loss function that includes the exclusive combination regularization term; determining whether optimization of the encoding matrix is ​​complete based on the updated total loss function; and how to do it.

12. 12. The method of claim 11 , further comprising: iteratively updating the encoding matrix, generating updated multiplexed low-resolution images, performing high-resolution amplitude and phase reconstruction, and calculating an updated total loss function until optimization of the encoding matrix is ​​complete.

13. 13. The method of claim 12, wherein updating the encoding matrix comprises updating the encoding matrix based on a gradient of a most recently calculated total loss function.

14. 12. The method of claim 11, wherein determining an initial encoding matrix comprises randomly selecting values ​​for the initial encoding matrix and separating an initial light source pattern into a bright field pattern and a dark field pattern.

15. The exclusive combination regularization term is λ coupling * Norm(CC T -diag(CC T ) * I) Including, in this case, C represents the encoding matrix, C T represents the transpose of the encoding matrix, and λ coupling represents the tunable hyperparameters, The method of claim 11.

16. Furthermore, based on the determination of whether the optimization of the encoding matrix is ​​completed, if the optimization of the encoding matrix is ​​completed, The method of claim 11 , comprising storing the optimized encoding matrix for use by the FPM system.

17. Furthermore, based on the determination of whether the optimization of the encoding matrix is ​​completed, if the optimization of the encoding matrix is ​​completed, The method of claim 11 , comprising using the optimized encoding matrix during use of the FPM system.

18. Using the optimized encoding matrix acquiring an image of a sample located at a sample location for each image illuminated using a different light source pattern specified in the optimized encoding matrix; 18. The method of claim 17, further comprising generating a reconstructed image based on the acquired image, wherein generating the reconstructed image includes initiating a Fourier ptychographic microscope (FPM) reconstruction.

19. 1. A Fourier ptychography imaging system, comprising: a plurality of light sources configured to emit light at the sample location; an optical system configured to image at least a portion of a sample disposed at the sample location; an imaging device configured to capture images of the sample through the optical system under different light conditions provided by the plurality of light sources; a processor; a memory coupled to the processor containing an encoding matrix specifying a light source pattern to be used during capture of a low resolution image by the image capture device; Including, In this case, the encoding matrix includes a light source pattern optimized using a loss function including an exclusive combination regularization term that promotes sparse grouping of light sources and diversification of light source patterns among the encoding matrix, In this case, the memory includes computer-executable instructions stored therein, which, when executed by a processor, cause the processor to: acquiring images of a sample disposed at the sample location, each image illuminated with a different light source pattern specified in the encoding matrix; storing the image; and initiating a Fourier ptychographic microscope (FPM) reconstruction to generate a reconstructed image based on the stored image; system.

20. The exclusive combination regularization term is λ coupling * Norm(CC T -diag(CC T ) * I) Including, in this case, C represents the encoding matrix, C T represents the transpose of the encoding matrix, and λ coupling represents the tunable hyperparameters, 20. The system of claim 19.

Images