Method and apparatus for Fourier lamination microimaging using coded illumination

Abstract

A method of determining a coding mode of a light source of an FPM system is provided that includes determining a coding matrix specifying a light source mode for multiplexing a low resolution image. A multiplexed low resolution image is generated using the coding matrix. High resolution amplitude and phase reconstruction is performed using an FPM algorithm and multiplexed low resolution images. A total loss function is calculated that includes an exclusive coupling regularization term that facilitates diversity of light source modes of the encoding matrix and sparse grouping of light sources within the light source modes. The method further includes determining whether the encoding matrix optimization is complete. If the encoding matrix optimization is complete, the method includes storing the optimized encoding matrix for use by the FPM system and / or employing the optimized encoding matrix during use of the FPM system.

Description

Technical Field

[0001] The present application relates to sample imaging, and more particularly, to methods and apparatus for Fourier ptychography using coded illumination. Background Art

[0002] Fourier stacking microscopy (FPM) is a microscopy technique that allows for high-resolution imaging across a wide field of view. FPM employs an array of light sources to illuminate the sample while capturing a set of low-resolution images. Each low-resolution image is illuminated by a different light source or group of light sources from the array. The captured low-resolution images are then stitched together in the Fourier domain to generate a high-resolution image.

[0003] FPM offers many advantages over conventional microscopy, such as a significantly higher space-bandwidth product, a simple, low-cost setup with minimal mechanical actuation, and a small footprint. However, due to the number of images to be captured, FPM suffers from long image acquisition times, which limits its applicability to imaging moving samples and video capture. In some applications, reconstructing a high-resolution image from the captured low-resolution images can also be very time-consuming.

[0004] Therefore, there is a need for improved methods and apparatus for FPM. Summary of the Invention

[0005] In some embodiments, a method for determining a coding pattern of a light source for an FPM system is provided. The method includes determining a coding matrix that specifies a light source pattern for a multiplexed low-resolution image, generating the multiplexed low-resolution image using the coding matrix, and performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution image. The method also includes calculating a total loss function, wherein the total loss function includes an exclusive coupling regularization term that promotes diversity of light source patterns in the coding matrix and sparse grouping of light sources within the light source pattern. The method also includes determining whether coding matrix optimization is complete. If coding matrix optimization is complete, the method includes storing the optimized coding matrix for use by the FPM system and / or employing the optimized coding matrix during use of the FPM system.

[0006] In some embodiments, a method for determining a coding pattern for a light source of an FPM system is provided. The method includes determining an initial coding matrix that specifies an LED pattern for a multiplexed low-resolution image, generating a single-LED low-resolution image, and generating the multiplexed low-resolution image using the coding matrix and the single-LED low-resolution image. The method also includes performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution image, and calculating a total loss function, wherein the total loss function includes an exclusive coupling regularization term that promotes diversity in the light source pattern of the coding matrix and sparse grouping of light sources within the light source pattern. The method also includes determining whether coding matrix optimization is complete. If coding matrix optimization is not complete, the method includes updating the coding matrix based on the calculated total loss function; generating an updated multiplexed low-resolution image using the updated coding matrix and the single-LED low-resolution image; performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the updated multiplexed low-resolution image; calculating an updated total loss function, wherein the updated total loss function includes an exclusive coupling regularization term; and determining whether coding matrix optimization is complete based on the updated total loss function.

[0007] In some embodiments, a Fourier ptychography system is provided. The Fourier ptychography system includes a plurality of light sources configured to emit light onto a sample location, an optical system configured to image at least a portion of a sample positioned at the sample location, and an image capture device configured to capture images of the sample through the optical system under different light conditions provided by the plurality of light sources. The Fourier ptychography system also includes a processor and a memory coupled to the processor. The memory includes an encoding matrix that specifies a light source pattern used during low-resolution image capture by the image capture device, wherein the encoding matrix includes the light source pattern optimized using a loss function including an exclusive coupling regularization term that promotes diversity in the light source pattern of the encoding matrix and sparse grouping of light sources within the light source pattern. The memory also includes computer-executable instructions stored therein that, when executed by the processor, cause the processor to obtain images of the sample positioned at the sample location. Each of the images is illuminated using a different light source pattern specified in the encoding matrix. When executed by the processor, the computer-executable instructions further cause the processor to store the images and initiate FPM reconstruction to generate reconstructed images based on the stored images.

[0008] A system of one or more computers may be configured to perform specific operations or actions by means of software, firmware, hardware, or a combination thereof installed on the system, which, in operation, causes the system to perform the actions. In some embodiments, one or more computers may include one or more graphics processing units (GPUs). One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0009] Other features and aspects of the present invention will become more fully apparent from the following detailed description, the appended claims and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A An example FPM system provided in accordance with an embodiment of the present disclosure is shown.

[0011] Figure 1B It shows that the embodiment provided herein can be used with Figure 1A An example light source array employed in conjunction with an FPM system.

[0012] Figure 2 An example method of determining a coding matrix for an FPM system according to embodiments provided herein is shown.

[0013] Figure 3A Shown are LED patterns for an example initial encoding matrix according to embodiments provided herein.

[0014] Figure 3B ] shows the reference to the embodiment provided herein after optimization by using an exclusive coupling regularization term. Figure 3A An example initial encoding matrix is ​​described with updated LED patterns.

[0015] Figure 4A An example simulated ground truth image is shown from which multiple low-resolution single-LED images can be generated according to embodiments provided herein.

[0016] Figure 4B According to the embodiment provided herein, Figure 4A Example low-resolution single-LED image generated from simulated ground-truth images.

[0017] Figure 4C The embodiment provided herein shows the encoding matrix from Figure 4B Example of single LED image generation to multiplex low-resolution image.

[0018] Figure 4D It is shown that according to the embodiments provided herein, Figure 4C Example high-resolution reconstructed image generated by multiplexing low-resolution images.

[0019] Figure 5A and Figure 5B Example LED patterns after encoding matrix optimization without and with an exclusive coupling regularization term according to embodiments provided herein are shown, respectively.

[0020] Figure 5C and Figure 5D The embodiments provided herein respectively correspond to Figure 5A and Figure 5B The LED pattern coding matrix.

[0021] Figure 6A Shown is a graphical comparison of the mean squared error (MSE) observed in the loss function during training with and without the inclusion of an exclusive coupling regularization term according to embodiments provided herein.

[0022] Figure 6B A plot of MSE versus training iterations during encoding matrix optimization with and without an exclusive coupling regularization term is shown according to embodiments provided herein.

[0023] Figure 7 A first example method of determining a coding pattern of a light source of an FPM system according to embodiments provided herein is shown.

[0024] Figure 8 A second example method of determining a coding pattern of a light source of an FPM system according to embodiments provided herein is shown.

[0025] Figure 9 A flowchart illustrating a process and system for converting an initial coding matrix into an optimized coding matrix and for using the optimized coding matrix according to embodiments provided herein is shown. DETAILED DESCRIPTION

[0026] Independent of grammatical term usage, individuals of male, female, or other gender identities are included in the term.

[0027] As previously mentioned, while FPM offers many advantages, its use can be limited in some applications due to the considerable time required to obtain results using this technique. The primary delays associated with FPM include the time required to capture a large number of low-resolution images and the time required to reconstruct a high-resolution image from the captured low-resolution images. The embodiments provided herein can significantly reduce FPM image capture time, thereby allowing FPM to be used in a wider range of applications (e.g., any application that benefits from faster results, such as imaging moving samples, clinical testing for medical diagnosis / treatment, etc.).

[0028] During FPM, a sample is illuminated by an array of light sources, where each light source (e.g., a light-emitting diode (LED)) emits light toward the sample from a different angle and / or position. Low-resolution images of the sample captured using different LEDs from the LED array are processed in the Fourier domain to generate a high-resolution image (e.g., via amplitude and phase reconstruction based on the low-resolution image).

[0029] Using a single LED to illuminate the sample each time to obtain each low-resolution image is time consuming. To reduce image capture time during FPM, multiplexing or "coded illumination" techniques have been developed, in which different patterns (e.g., combinations) of LEDs within an LED array are employed to illuminate the sample during low-resolution image capture. However, selecting a specific pattern of LEDs to employ for each low-resolution image without compromising the quality of the reconstructed image is challenging. One method for determining the LED pattern to use for image capture during FPM is described in "Data-Driven Design for Fourier Ptychographic Microscopy," 2019 IEEE International Conference on Computational Photography (ICCP), 2019, pp. 1-8, doi:10.1109 / ICCPHOT.2019.8747339 by M. Kellman et al. describe reducing the number of low-resolution images required for FPM image reconstruction by using a neural network trained to learn LED multiplexing patterns. While effective, this approach requires extensive optimization and can be slow to converge. Other example methods for coded / multiplexed illumination are described in "Computational illumination for high-speed in vitro Fourier ptychographic microscopy," Optica 2, 904-911 (2015) by Lei Tian, ​​Ziji Liu, Li-Hao Yeh, Michael Chen, Jingshan Zhong, and Laura Waller, and "Multiplexed coded illumination for Fourier Ptychography with an LED array microscope," Biomed. Opt. Express 5, 2376-2389 (2014) by Lei Tian, ​​Xiao Li, Kannan Ramchandran, and Laura Waller. These and other coded / multiplexed illumination techniques would benefit from improved LED pattern selection for low-resolution imaging during FPM.

[0030] According to embodiments provided herein, a regularization term called an exclusive coupling (EC) regularization term is added to the FPM loss function used to determine the FPM encoded illumination pattern (e.g., the light source pattern for each low-resolution image). The EC adjustment term can be used to promote diversity in light source patterns, use fewer light sources within a light source pattern, and use patterns with more spatially separated light sources, as shown in Equation (1):

[0031] (1) EC regularization term = λ coupling *Norm(CC T -diag(CC T )*I)

[0032] where C represents the coding matrix that defines the coding pattern of the light array during image capture, λ coupling is a tunable hyperparameter that can be used to determine the strength of the exclusive coupling, and I is the identity matrix. As mentioned above, larger λ coupling Values ​​of λ promote diversity in light source patterns, use of fewer light sources within a light source pattern, and use of patterns with more spatially separated light sources. coupling The selection of can be based on trial and error, past experience with light source pattern selection for FPM, etc. (e.g., to promote pattern diversity and sparsity while promoting image quality during coded illumination).

[0033] The exclusive coupling regularization term of Equation (1) can be used with any suitable loss function for FPM (e.g., a differentiable FPM loss function). By adding the exclusive coupling (EC) regularization term, the total loss function (LF) becomes the sum of the adopted FPM loss function and the EC regularization term:

[0034] (2) Total LF = FPM loss function + λ coupling *Norm(CC T -diag(CC T )*I)

[0035] In some embodiments, the FPM loss function used may be the FPM loss function described by Kellman et al., but other FPM loss functions may also be used. The FPM loss function of Kellman et al. is listed below as formula (3):

[0036] (3)

[0037] Where N = total number of training images, γ = the loss function weight between the phase (γ = 0) loss function and the amplitude (γ = 1) loss function, |·| = the amplitude of the reconstructed image, and ∠ = the phase of the reconstructed image.

[0038] As will be referred to below Figure 1A -to Figure 9 As described, using exclusive coupling within the overall loss function can improve coded lighting LED pattern selection by improving the convergence (and convergence rate) during coded lighting pattern generation.

[0039] Figure 1A An example Fourier plyometric microscopy (FPM) system 100 is shown, provided in accordance with an embodiment of the present disclosure. Figure 1A , the FPM system 100 includes a light source array 102 having a plurality of light sources 102 a to 102 n configured to emit light onto a sample location 104 .

[0040] The optical system 106 is configured to image at least a portion of a sample 108 located at the sample position 104. Figure 1A As shown, the image capture device 110 is configured to capture images of the sample 108 (e.g., low-resolution images 112a to 112n) through the optical system 106 under different light conditions provided by the plurality of light sources 102a to 102n of the light source array 102. In some embodiments, the different light conditions can be selected based on an encoding matrix that defines the light source pattern to be used for each low-resolution image. In some embodiments, the encoding matrix can be determined by training a neural network whose loss function includes a regularization term for exclusive coupling that promotes diversity of light source patterns and sparsity within the light source pattern (e.g., as previously described with reference to equation (1) and further described below).

[0041] A computer 114 having a processor 116 can be coupled to the image capture device 110 and can receive images (e.g., low-resolution images) captured by the image capture device 110 for storage in a memory. In some embodiments, the images can be stored in a memory 118 (e.g., RAM, a hard drive, and / or another memory type) associated with the processor 116. Alternatively or additionally, the image data can be stored in an external memory 120 (e.g., local external memory, remote storage, cloud storage, or any combination thereof). A display 122 having a user interface 124 can be coupled to the processor 116, e.g., for displaying the low-resolution image, the reconstructed high-resolution image, etc.

[0042] The light source array 102 may include a uniform or non-uniform array of light sources 102a to 102n, which may be controlled by a processor 116 or another suitable processor, microprocessor, controller, microcontroller, digital signal processor (DSP), or field programmable gate array (FPGA) configured to act as a microcontroller, etc.

[0043] In some embodiments, the light sources 102a to 102n of the light source array 102 can be individually controlled and can operate individually or in combination with one or more light sources 102a to 102n (e.g., as defined by the encoding matrix 126 shown as being stored within the memory 118, although other storage locations may be used, such as within the external memory 120 or within the memory of another processor used to control the light source array 102).

[0044] Example light sources 102a-102n may include light emitting diodes (LEDs), monochromatic or single-bandwidth emitting light sources, multi-bandwidth light sources (e.g., RGB LEDs), superluminescent LEDs, laser diodes (particularly semiconductor laser diodes), thermal emitters, fiber-based light sources, etc. All of the light sources 102a-102n may be identical, or one or more of the light sources 102a-102n may differ in at least one of the following characteristics: wavelength, spectral bandwidth, spatial emission characteristics, temporal emission characteristics (e.g., continuous or pulsed operation), coherence parameters (e.g., degree of temporal and / or spatial coherence), brightness or range, etc.

[0045] In some embodiments, a light source array 102 having approximately 80 to 280 individually controllable LEDs in an xy grid may be employed, such as a 16x16 LED array. Figure 1B As shown, each LED is spaced approximately 1 mm to 10 mm apart and emits at approximately 0.4 microns to 0.7 microns. In a specific embodiment, the LEDs may be spaced approximately 2.5 mm to 3.5 mm apart and utilize wavelengths of 0.45 microns, 0.51 microns, and / or 0.62 microns. Other light source array arrangements, numbers of light sources, types of light sources, and / or emission wavelengths may be utilized. As described above, although the processor 116 Figure 1B 1 is shown as controlling the light source array 102 , but in other embodiments, a different processor or other control mechanism may be used to control the operation of the light source array 102 .

[0046] For example, the optical system 106 ( Figure 1A) may include an optical objective lens 106a and a focusing lens 106b. Other optical components may be used. As described above, one of the benefits of FPM is that it allows the use of low-cost, low-resolution optical components. In some embodiments, the optical objective lens 106a may have a numerical aperture (NA) of approximately 0.05 to 0.9. Other NA optical objective lenses may be used. In one or more embodiments, the focusing lens 106b may be a tube lens, such as an achromatic tube lens, or another suitable lens. The image capture device 110 may include any suitable imaging device capable of imaging a sample through the optical system 106, such as a CMOS sensor or the like. Example pixel sizes may be in the range of about 1 micron to about 10 microns, but other pixel sizes may be used.

[0047] In some embodiments, the processor 116 may be a central processing unit (CPU). In other embodiments, the processor 116 may include and / or be implemented as one or more other computing resources, such as, but not limited to, a microprocessor, a microcontroller, an embedded microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA) configured to perform as a microcontroller, etc. The computer 114 may include any suitable computing device, such as a tablet computer, a laptop computer, a desktop computer, a server, etc.

[0048] The memory 118 and / or 120 can be any suitable type of memory, such as, but not limited to, one or more of volatile memory and / or non-volatile memory (e.g., RAM, DRAM, SRAM, cache, hard drive, combinations thereof, etc.). In other words, the memory 118 and / or 120 can include more than one memory type. The memory 118 and / or 120 can have a plurality of instructions stored therein that, when executed by the processor 116, cause the processor 116 to perform various actions specified by one or more of the stored instructions. Code and data can be stored in a first type of memory (e.g., a hard drive) and transferred to a second type of memory (e.g., RAM) for execution. In some embodiments, the memory 118 and / or 120 can include either or both of the memory types.

[0049] The display 122 may include any suitable display, such as a light emitting diode (LED) display, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, etc. The user interface 124 may include, for example, one or more of a display screen or a touch panel and / or screen, an audio speaker, and a microphone. In some embodiments, the user interface 124 may be controlled by the processor 116, and the functions of the user interface 124 may be implemented at least in part by computer-executable instructions (e.g., program code or software) stored in the memory 118 and / or executed by the processor 116.

[0050] Figure 2 The coding matrix (eg, Figure 1A Example method 200 of encoding matrix 126). Figure 2 In block 202, an initial encoding matrix is ​​determined. In some embodiments, the encoding matrix may include multiple vectors, each vector specifying an LED pattern for each low-resolution image to be captured using the FPM system. In other words, each vector may specify which LEDs to illuminate (and / or the brightness of each LED to illuminate) during image capture of the corresponding low-resolution image. For example, if 50 low-resolution images are to be captured for use during FPM reconstruction, the encoding matrix may include 50 vectors identifying the 50 LED patterns to be used to illuminate the sample during low-resolution image capture. In some embodiments, the initial encoding matrix may be randomly initialized. In other embodiments, the initial encoding matrix may be determined based on a best guess for the encoding matrix (e.g., based on factors such as the LED array employed, the type of sample being imaged, the optical system employed within the FPM system, etc.). In at least one embodiment, the LED patterns may be categorized as brightfield LED patterns and darkfield LED patterns, where only a subset of the inner LEDs are illuminated for the brightfield LED pattern and only a subset of the outer LEDs are illuminated for the darkfield LED pattern. In some embodiments, the initial encoding matrix can include a randomized bright field LED pattern (e.g., only a randomized bright field LED pattern) and a randomized dark field LED pattern (e.g., only a randomized dark field LED pattern). Any other suitable method for determining the initial encoding matrix can be used.

[0051] Figure 3A FIG. 1 shows an example initial coding matrix LED pattern according to an embodiment provided herein. Figure 3A, eight LED patterns 302a to 302h are shown (e.g., corresponding to eight vectors within the initial encoding matrix). As mentioned, in some embodiments, the LED patterns can be divided into brightfield LED patterns (e.g., such as LED patterns 302a and 302b) and darkfield LED patterns (e.g., such as LED patterns 302c to 302h), where only a subset of inner LEDs are illuminated for the brightfield LED patterns and only a subset of outer LEDs are illuminated for the darkfield LED patterns. Other initial LED patterns can be specified by the initial encoding matrix. Figure 3B It is shown that after optimization using the exclusive coupling regularization term described above (eg, Equation (1)) according to the embodiments provided herein, Figure 3A 1 and 2. The updated LED patterns 302a' to 302h' (described further below) of the example initial encoding matrix of FIG.

[0052] Reference Figure 2 After determining the initial encoding matrix (in block 202), in block 204, a single light source low resolution image is generated (e.g., using a single light source such as an LED to illuminate the sample during image capture). The single light source low resolution image can be an experimentally determined image (e.g., using a single light source such as an LED to illuminate the sample during image capture). Figure 1A The FPM system 100 may capture a low-resolution image or simulate an image captured by the FPM system 100. As an example, for a 256-LED array, 256 low-resolution images may be generated, each low-resolution image being generated by illumination by a different LED in the LED array.

[0053] Figure 4A An example simulated ground truth image 402 is shown from which a plurality of low-resolution single LED images 404a to 404n can be generated ( Figure 4B ). In some embodiments, the true value image 402 can be a combination of stock images that represent any suitable phase and / or amplitude at high resolution. In order to generate the low-resolution images 404a to 404n, a forward model of the FPM reconstruction process can be used to simulate low-resolution images from the true value image (e.g., the true value image 402). Both bright field and dark field low-resolution images can be simulated using appropriate portions of the Fourier transform of the true value image (e.g., the true value image 402). For example, the low-resolution image can be a low-pass filtered image of a shifted Fourier transform of the true value image (e.g., with different shift amounts depending on different illumination angles). Based on the optical system used, the simulated low-resolution image can be further amplified.

[0054] Refer again Figure 2In block 206, a multiplexed image is generated from the single LED images and an encoding matrix (e.g., an initial encoding matrix or a subsequently determined encoding matrix as described below). For example, for each LED pattern specified in the encoding matrix, multiple single LED images can be combined (e.g., multiplexed) into a single image to simulate an image captured using the LED pattern. In some embodiments, the images can be combined by adding or averaging, such as by weighted averaging the images using the encoding matrix values ​​(e.g., for LED brightness) as weighting factors. For example, the images can be added together pixel by pixel, where each pixel is weighted by the LED brightness value specified in the encoding matrix. In one or more embodiments, all single LED images can be generated (e.g., experimentally or via simulation) where each LED is at full brightness. The encoding matrix can specify a percentage brightness for each LED in the LED pattern, and the multiplexed image for the LED pattern can be generated by adding or averaging the associated single LED images weighted by the LED brightness specified in the encoding matrix. In one or more other embodiments, single LED images may be generated where the employed LEDs have varying brightness, and where the multiplexed image is generated by a coding matrix that accounts for LED brightness differences between the single LED images.

[0055] Figure 4C shows the encoding matrix from Figure 4B Example multiplexed low-resolution images 406a to 406m generated from the single LED images 404a to 404n. For example, once generated, the low-resolution single LED images 404a to 404n ( Figure 4B ) may be summed and / or averaged to form multiplexed low resolution images 406a to 406m corresponding to the LED patterns specified in the encoding matrix as described above.

[0056] In block 208, FPM reconstruction is performed on the coded image (e.g., generated in block 206). Specifically, a high-resolution image is generated using high-resolution amplitude and phase reconstruction that utilizes an FPM algorithm and a multiplexed image generated from the coding matrix (in block 206). In some embodiments, the FPM algorithm described in Kellman et al. may be employed. Any suitable FPM algorithm may be used for FPM reconstruction (e.g., any differentiable FPM algorithm). Other example FPM algorithms that may be employed include the alternating projection method described in R. W. Gerchberg and W. O. Shaxton, “A practical algorithm for the determination of phase from image and diffraction plane pictures”, Optik, Bd. 35, pp. 227-246, (1972) and L. Bian, J. Suo, G. Zheng, K. Guo, F. Chen and Q. Dai, “Fourier ptychographic reconstruction using Wirtinger flow optimization”, Optics Express, Bd. 23, Nr. 4, pp. 4856-4866, 2 (2015) and L. Bian, J. Suo, J. Chung, X. Ou, C. Yang, F. Chen and Q. Dai, “Fourier ptychographic reconstruction using Poisson maximum likelihood and truncated Wirtinger flow optimization”, Optics Express, Bd. 23, Nr. 4, pp. 4856-4866, 2 (2015). gradient", the maximum likelihood estimation formula described in Scientific Reports, Bd.6, Nr.1, p.27384, 7 (2016).

[0057] Figure 4D It is shown that, for example, Figure 4C An example high-resolution reconstructed image 408 is generated by multiplexing the low-resolution images 406a to 406m.

[0058] Refer again Figure 2In block 210, a total loss function (e.g., including an exclusive coupling regularization term) is determined based on the high-resolution image generated in block 208. As shown in equation (2), the total loss function includes the FPM loss function of the employed FPM algorithm plus the exclusive coupling regularization term of equation (1). For example, the loss function of equation (3) of Kellman et al. can be calculated and added to the exclusive coupling regularization term (equation (1) above). Other FPM loss functions can be employed.

[0059] In general, an FPM loss function can be calculated based on an FPM reconstructed image and a ground truth image. In some embodiments, a simulated low-resolution image can be obtained from the FPM reconstructed high-resolution image and compared to one or more or one or more single-light source low-resolution images in the multiplexed low-resolution image (e.g., pixel by pixel) to determine whether the reconstructed image accurately depicts the details of the (one or more) low-resolution images. In some embodiments, this can include intentionally reducing the details within the high-resolution image to approximate the level of detail within the low-resolution image. For example, a forward model of the FPM reconstruction process can be used to simulate a low-resolution image from the FPM reconstructed high-resolution image in a manner similar to that described above with reference to the method for reconstructing a low-resolution image from the ground truth image 402 ( Figures 4A to 4B ) generates low resolution single LED images 404a to 404n in the manner described.

[0060] Based on the current encoding matrix according to Equation (1) and the selected hyperparameter λ coupling The exclusive coupling regularization term is determined by the value of . As described above, the exclusive coupling regularization term promotes the sparsity of light sources (e.g., LEDs) within the light source pattern and the diversity of the light source pattern. As a simplified example, assume that the LED array uses two LEDs, LED1 and LED2. Two low-resolution images, Image 1 and Image 2, are generated. The brightness of LED1 in Image 1 is C 11 , and the brightness of LED1 in image 2 is C 21 , the brightness of LED2 in image 1 is C 12 , and the brightness of LED2 in image 2 is C 22 (as shown in the table below).

[0061] LED 1 LED 2 Image 1 <![CDATA[C 11 ]]> <![CDATA[C 12 ]]> Image 2 <![CDATA[C 21 ]]> <![CDATA[C 22 ]]>

[0062] This information can be represented in the encoding matrix C as follows:

[0063] (4) Make

[0064] (5)

[0065] (6)

[0066] (7)

[0067] (8)norm(CCT-diag(CC T )I)=2(c 11 c 21 +c 12 c 22 ) 2

[0068] (9)EC regularization term = λ coupling *2(c 11 c 21 +c 12 c 22 ) 2

[0069] Equation (9) shows that when two LEDs (eg, LED1 and LED2) are turned on during Image 1 or Image 2 (eg, when C 11 、C 21 、C 12 and C 22 is non-zero), the exclusive coupling (EC) regularization term is maximized. Furthermore, the EC regularization term is minimized when exclusivity is maximized (e.g., when for image 1, LED1 is on and LED2 is off, and for image 2, LED1 is off and LED2 is on). Therefore, the exclusive coupling regularization term discourages the use of the same LED in multiple images, and the effect of the exclusive coupling regularization term is determined by the hyperparameter λ. coupling More generally, the exclusive coupling regularization term promotes both sparsity and diversity by encouraging the use of different LEDs for each image.

[0070] In block 212, a determination is made as to whether encoding matrix optimization is complete. If encoding matrix optimization is complete, the optimized encoding matrix may be stored and / or used as described below (in block 214); otherwise, the encoding matrix is ​​updated in block 216. In some embodiments, encoding matrix optimization may be considered complete when the total loss function (e.g., calculated in block 210) has plateaued over time and / or the gradient of the loss function has dropped below a predetermined threshold (e.g., close to zero). Alternatively, encoding matrix optimization may be considered complete after a predetermined number of iterations of the encoding matrix training / optimization steps (e.g., a predetermined number of iterations of blocks 206, 208, 210, 212, and 216). In some embodiments, tens to hundreds of iterations may be performed before determining that encoding matrix optimization is complete (e.g., assuming the encoding matrix is ​​not yet considered optimized), although fewer or more iterations may be performed.

[0071] Returning to block 214 , assuming encoding matrix optimization is complete, in block 214 the optimized encoding matrix may be stored (e.g., in memory 118 of processor 116 or another suitable location) and / or employed during subsequent image acquisition and / or reconstruction operations using the FPM system 100 .

[0072] Returning to block 216, assuming that the encoding matrix optimization is not complete (e.g., as determined in block 212), the encoding matrix can be updated. For example, based on the results of the total loss function in block 210, the encoding matrix for the light array can be updated (e.g., the encoding matrix values ​​for each LED pattern specifying which LEDs are on or off and / or the brightness of the on LEDs can be updated based on the gradient of the loss function). In other words, the encoding matrix can be updated based on the gradient of the total loss function (e.g., determined in block 210). As a specific example, the encoding matrix values ​​specifying the brightness of the LEDs can be increased or decreased by an amount proportional to the gradient of the loss function. Any other suitable method can be used to update (e.g., optimize) the encoding matrix by employing the proposed total loss function (e.g., using an exclusive coupling regularization term).

[0073] After the encoding matrix is ​​updated (in block 216), method 200 returns to block 206, where a new set of multiplexed images is generated using the updated encoding matrix, and high-resolution amplitude and phase reconstruction is subsequently performed using the newly generated multiplexed images (in block 208). The total loss function is then calculated (in block 210), and a determination is made as to whether the encoding matrix optimization is complete (in block 212). Blocks 216, 206, 208, 210, and 212 are repeated until the encoding matrix optimization is complete; thereafter, the encoding matrix is ​​stored and / or used in block 214 as previously described.

[0074] Figure 3B It is shown that after optimization using the above-mentioned exclusive coupling regularization term (e.g., equation (1)) and method 200, reference Figure 3A The example updated LED patterns 302a'-302h' of the example initial encoding matrix are described. Figure 3B As shown in , after optimizing the encoding matrix, significantly lower clustering of adjacent LEDs is observed, LED diversity is increased, and the modes are more spatially separated. In addition, by including an exclusive coupling regularization term (described below), the optimization of the encoding matrix can be achieved faster.

[0075] As another example, Figure 5A and Figure 5B , respectively, show the example LED patterns after encoding matrix optimization with and without exclusive coupling regularization terms according to the embodiments provided herein. Figure 5A, the LED patterns 502a to 502h are determined based on the encoding matrix optimized without using the exclusive coupling regularization term (e.g., where λ coupling =0), and Figure 5B The LED patterns 502a' to 502h' are determined based on the encoding matrix optimized using the exclusive coupling regularization term (eg, where λ coupling >0). Figure 5B As shown, with the exclusive coupling regularization term, the resulting light source pattern is more diverse, includes fewer illuminated LEDs, and is generally more spatially separated. To further illustrate, Figure 5C and Figure 5D The corresponding Figure 5A and 5B The coding matrix 504a and 504b of the LED pattern. Figure 5C As shown, without the regularization term, significant clustering of LEDs can be found in both bright-field and dark-field LED modes. Specifically, compared with the exclusive coupling regularization term formed in the loss function, Figure 5D Compared to the LED groups 506a to 506e, without using the exclusive coupling regularization term, the loss function tends to form redundant LED groups, such as LED groups 506a to 506e.

[0076] like Figure 6A and Figure 6B As shown, using an exclusive coupling regularization term during encoding matrix optimization can provide faster and more desirable convergence. For example, Figure 6A A graphical comparison of the mean squared error (MSE) observed in the loss function during training with and without the exclusive coupling regularization term is shown. Figure 6A As shown, the MSE value observed during the convergence of the loss function with the exclusive coupling regularization term is approximately 32% lower than the MSE value observed during the convergence of the loss function without the exclusive coupling regularization term.

[0077] Regarding the convergence rate, Figure 6B Figure 2 shows the relationship between MSE and training iterations during encoding matrix optimization with and without the exclusive coupling regularization term. Figure 6B As shown, when the loss function includes the exclusive coupling regularization term, the loss function converges faster and converges to a lower level.

[0078] Therefore, by including the exclusive coupling regularization term in Equation (1), and the hyperparameter λ coupling With appropriate selection of , the LED patterns are diversified, the LED patterns contain fewer light sources, and the LED patterns have more spatially separated light sources. In addition, the optimization of the encoding matrix can be achieved faster and to a more desired level.

[0079] Figure 7 A first example method 700 of determining a coding pattern for a light source of an FPM system according to embodiments provided herein is shown. Figure 7 In block 702, method 700 includes determining an encoding matrix (eg, an initial encoding matrix, eg, Figure 9 902 of the initial encoding matrix). In box 704, the method 700 includes generating a multiplexed low-resolution image using the encoding matrix. In box 706, the method 700 includes performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the multiplexed low-resolution image. In box 708, the method 700 includes calculating a total loss function, wherein the total loss function (e.g., the total loss function of equation (2)) includes an exclusive coupling regularization term that promotes diversity of light source patterns of the encoding matrix and sparse grouping of light sources within the light source pattern. In box 710, the method 700 includes determining whether the encoding matrix optimization is complete. If the encoding matrix optimization is complete, then in box 712, the method 700 includes: storing the optimized encoding matrix for use by the FPM system (e.g., as Figure 1A The coding matrix 126 of the FPM system 100 of the embodiment of the present invention may be used to determine the coding matrix 126 of the FPM system 100 and / or to employ the optimized coding matrix during use of the FPM system. If the coding matrix optimization is not complete, then in block 714, the method 700 may include updating the coding matrix (e.g., based on the total loss function), and blocks 704, 706, 708, 710, and 714 may be repeated until the coding matrix optimization is complete.

[0080] Figure 8 A second example method 800 for determining a coding pattern for a light source of an FPM system according to embodiments provided herein is shown. Figure 8 In block 802, method 800 includes determining an initial encoding matrix (eg, Figure 9802 ). In box 804, method 800 includes generating a single LED low resolution image. In box 806. Method 800 includes using the encoding matrix and the single LED low resolution image to generate a multiplexed low resolution image. In box 808, method 800 includes using the FPM algorithm and the multiplexed low resolution image to perform high resolution amplitude and phase reconstruction. In box 810, method 800 includes calculating a total loss function, wherein the total loss function includes an exclusive coupling regularization term that promotes diversity of light source patterns of the encoding matrix and sparse grouping of light sources within the light source pattern. In box 812, method 800 includes determining whether encoding matrix optimization is complete. If encoding matrix optimization is not complete, then in box 814, method 800 includes updating the encoding matrix based on the calculated total loss function. Thereafter, boxes 806, 808, 810, and 812 are repeated for the updated encoding matrix. In some embodiments, if the encoding matrix optimization is not yet complete, blocks 806, 808, 810, 812, and 814 may be repeated until the encoding matrix optimization is complete. If the encoding matrix optimization is complete, then in block 816, method 800 may include storing the optimized encoding matrix for use by the FPM system and / or employing the optimized encoding matrix during use of the FPM system. For example, the optimized encoding matrix may be used as encoding matrix 126 ( Figure 1A ) is stored in the memory 118 of the computer 114.

[0081] In some embodiments, employing an optimized encoding matrix (e.g., encoding matrix 126) can include obtaining images (e.g., images 112a to 112n) of a sample (e.g., sample 108) located at a sample position (e.g., sample position 104), each of the images being illuminated using a different light source pattern specified within the encoding matrix (e.g., encoding matrix 126), and initiating FPM reconstruction to generate a reconstructed image based on the obtained images.

[0082] Figure 9 Flowchart showing a process and system 900 for converting an initial coding matrix into an optimized coding matrix and for using the optimized coding matrix according to the embodiments provided herein. Figure 9 , an initial encoding matrix 902 is determined (e.g., as described above with reference to method 200) and processed by a computer system 904 that is programmed to perform encoding matrix optimization using the exclusive coupling regularization term of equation (1), e.g., as described above with reference to Figure 2 Method 200, Figure 7 Method 700 or Figure 8 This produces an optimized encoding matrix 906, which can be Figure 1AThe FPM system 100 is used, for example, to illuminate a sample with an LED pattern defined within an optimized encoding matrix 906. The FPM system can capture and process low-resolution images to produce high-resolution images 910 (e.g., using Figure 1A 10 and processor 116). Other process flows and / or systems may be used.

[0083] One or more embodiments provided herein describe the optimization of an encoding matrix that specifies a light source pattern used during the capture of a low-resolution image (e.g., employed during FPM reconstruction of a high-resolution image). The encoding matrix can be optimized by employing a loss function with an exclusive coupling regularization term that promotes diversity in the light source pattern of the encoding matrix and sparse grouping of light sources within the light source pattern. In some embodiments, the optimization of the encoding matrix can include reconstructing an image based on the encoding matrix, calculating a loss function based on the reconstructed image, and updating the encoding matrix based on the calculated loss function. In other embodiments, multiple images can be reconstructed and multiple loss functions can be calculated (e.g., a loss function for each reconstructed image). Thereafter, the encoding matrix can be updated based on the multiple loss functions (e.g., by comparing the loss functions and selecting one of the loss functions for updating the encoding matrix, by combining multiple loss functions (e.g., averaging) for updating the encoding matrix, etc.). In other words, batch reconstruction can be performed (e.g., serially or in parallel) to generate multiple loss functions, and the multiple loss functions can be used individually or in combination during encoding matrix optimization. In other embodiments, multiple encoding matrices may be optimized (eg, in parallel) and used to determine the optimal encoding matrix for FPM reconstruction (eg, via manifold optimization).Other optimization procedures may be employed.

[0084] The foregoing description discloses only exemplary embodiments of the present invention; modifications of the above-described apparatus and method that fall within the scope of the present invention will be apparent to those skilled in the art. Therefore, while the present invention has been disclosed in conjunction with exemplary embodiments thereof, it should be understood that other embodiments may fall within the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A method for determining a coding pattern of a light source for a Fourier ptychography (FPM) system, the method comprising: determining a coding matrix that specifies a light source pattern for multiplexing the low-resolution image; generating the multiplexed low-resolution image using the encoding matrix; performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution image; Calculating a total loss function, wherein the total loss function includes an exclusive coupling regularization term, wherein the exclusive coupling regularization term promotes the diversity of the light source pattern of the encoding matrix and the sparse grouping of light sources within the light source pattern; determining whether encoding matrix optimization is complete; and When, based on the determination of whether the encoding matrix optimization is completed, the encoding matrix optimization is completed: Storing the optimized encoding matrix for use by the FPM system; employing the optimized coding matrix during use of the FPM system; or Its combination.

2. The method according to claim 1, further comprising: When, based on the determination of whether the encoding matrix optimization is completed, the encoding matrix optimization is not completed: updating the encoding matrix based on the calculated total loss function; generating an updated multiplexed low-resolution image using the updated encoding matrix; performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the updated multiplexed low-resolution image; Calculating an updated total loss function, wherein the updated total loss function includes the exclusive coupling regularization term; as well as Determines whether encoding matrix optimization is complete.

3. The method of claim 2 , further comprising repeatedly updating the encoding matrix, generating an updated multiplexed low-resolution image, performing high-resolution amplitude and phase reconstruction, calculating an updated total loss function, and determining whether the encoding matrix optimization is complete.

4. The method according to claim 3, wherein: Updating the encoding matrix includes updating the encoding matrix based on a gradient of a most recently calculated total loss function.

5. The method according to claim 3, wherein Generating an updated multiplexed low-resolution image includes generating an updated multiplexed low-resolution image using the single-LED low-resolution image and the updated encoding matrix.

6. The method according to claim 5, wherein: Generating an updated multiplexed low-resolution image using the updated encoding matrix includes combining the single-LED low-resolution images based on the updated encoding matrix.

7. The method according to claim 1, wherein Determining the encoding matrix includes determining an initial encoding matrix.

8. The method according to claim 7, wherein: Determining an initial encoding matrix includes randomly selecting a value of the initial encoding matrix.

9. The method according to claim 1, wherein: Generating the multiplexed low-resolution image comprises: Generate a single LED low-resolution image; and The multiplexed low-resolution image is generated using the single-LED low-resolution image and the encoding matrix.

10. The method according to claim 1, wherein The exclusive coupling regularization term includes λ coupling *Norm(CC T -diag(CC T )*I), where C is the encoding matrix, C T is the transpose of the encoding matrix, and λ coupling is a tunable hyperparameter.

11. A method for determining a coding pattern of a light source for a Fourier ptychography (FPM) system, the method comprising: determining an initial encoding matrix specifying an LED pattern for multiplexing a low-resolution image; Generate a single LED low-resolution image; generating a multiplexed low-resolution image using the initial encoding matrix and the single-LED low-resolution image; performing high-resolution amplitude and phase reconstruction using an FPM algorithm and the multiplexed low-resolution image; Calculating a total loss function, wherein the total loss function includes an exclusive coupling regularization term, wherein the exclusive coupling regularization term promotes the diversity of the light source pattern of the initial encoding matrix and the sparse grouping of light sources within the light source pattern; determining whether encoding matrix optimization is complete; and When, based on the determination of whether the encoding matrix optimization is completed, the encoding matrix optimization is not completed: Updating the initial encoding matrix based on the calculated total loss function; generating an updated multiplexed low-resolution image using the updated encoding matrix and the single-LED low-resolution image; performing high-resolution amplitude and phase reconstruction using the FPM algorithm and the updated multiplexed low-resolution image; calculating an updated total loss function, wherein the updated total loss function includes the exclusive coupling regularization term; and Determining whether encoding matrix optimization is completed based on the updated total loss function.

12. The method of claim 11, further comprising repeating the following operations until the encoding matrix optimization is completed: updating the encoding matrix, generating an updated multiplexed low-resolution image, performing high-resolution amplitude and phase reconstruction, and calculating an updated total loss function.

13. The method according to claim 12, wherein: Updating the encoding matrix includes updating the encoding matrix based on a gradient of a most recently calculated total loss function.

14. The method according to claim 11, wherein: Determining an initial encoding matrix includes randomly selecting values ​​for the initial encoding matrix and separating an initial light source pattern into a bright field pattern and a dark field pattern.

15. The method according to claim 11, wherein The exclusive coupling regularization term includes λ coupling *Norm(CC T -diag(CC T )*I), where C is the encoding matrix, C T is the transpose of the encoding matrix, and λ coupling is a tunable hyperparameter.

16. The method of claim 11, further comprising, when the encoding matrix optimization is completed based on the determination of whether the encoding matrix optimization is completed: The optimized encoding matrix is ​​stored for use by the FPM system.

17. The method according to claim 11, further comprising: When the coding matrix optimization is completed based on the determination of whether the coding matrix optimization is completed: employing the optimized coding matrix during use of the FPM system.

18. The method according to claim 17, wherein: The optimized coding matrix includes: obtaining images of the sample at the sample location, each of the images being illuminated using a different light source pattern specified within the optimized encoding matrix; and A reconstructed image is generated based on the acquired image, and the generating of the reconstructed image includes initiating Fourier stack microscopy (FPM) reconstruction.

19. A Fourier stack imaging system comprising: a plurality of light sources configured to emit light onto a sample location; an optical system configured to image at least a portion of a sample positioned at the sample location; an image capture device configured to capture images of the sample through the optical system under different light conditions provided by the plurality of light sources; processor; as well as a memory coupled to the processor and comprising an encoding matrix specifying an illuminant pattern used during capture of a low-resolution image by the image capture device, wherein the encoding matrix comprises an illuminant pattern optimized using a loss function including an exclusive coupling regularization term that promotes diversity in the illuminant pattern of the encoding matrix and sparse grouping of illuminants within the illuminant pattern; and wherein the memory includes computer executable instructions stored therein, which, when executed by the processor, cause the processor to: acquiring images of the sample at the sample location, each image being illuminated using a different light source pattern specified within the encoding matrix; storing the image; and Fourier stack microscopy (FPM) reconstruction is initiated so that a reconstructed image is generated based on the stored images.

20. The system of claim 19, wherein: The exclusive coupling regularization term includes λ coupling *Norm(CC T -diag(CC T )*I), where C is the encoding matrix, C T is the transpose of the encoding matrix, and λ coupling is a tunable hyperparameter.

Images