Microscope-based system and method for determining beam processing paths

Abstract

A microscope-based system is provided, including an illumination assembly including an illumination source and a pattern illuminator, and a processing module coupled to the illumination source and the pattern illuminator. The processing module is configured to identify regions of interest in a sample, generate a two-dimensional illumination mask for each of a plurality of fields of view, determine an illumination sequence for the regions of interest by minimizing a sum of a plurality of inter-region travel distances between a series of regions of interest for each field of view, determine an illumination path within each of the regions of interest that follows the illumination sequence, and control the illumination source and the pattern illuminator to illuminate the regions of interest based on the illumination sequence and the illumination path for each of the plurality of fields of view. Methods of use are also provided.

Description

[Technical Field]

[0001] (Priority Claim) This application claims priority to U.S. Provisional Patent Application No. 63 / 374,931, filed September 8, 2022, entitled "MICROSCOPE-BASED SYSTEM AND METHOD OF DETERMINING BEAM PROCESSING PATH," the entire contents of which are incorporated herein by reference.

[0002] (Incorporated by reference) All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. [Background technology]

[0003] (background) Hypothesis-free, highly sensitive subcellular proteomics is challenging due to the limited sensitivity of mass spectrometry and the lack of protein amplification tools. Without such techniques, bulk protein discovery at specific locations of interest in cell and tissue samples is not possible.

[0004] Spatial proteomics enables protein mapping of biological samples, revealing the underlying geometric framework of protein-protein interactions. Cell biologists and histologists have benefited greatly from recent advances in spatial proteomics, enabling, for example, mapping disease-associated microenvironment proteins, the distribution of structural proteins in structured histological samples, or the identification of proteins in specific organelles. While the goal of targeted spatial proteomics is to identify the location of known proteins, de novo spatial proteomics requires spatial protein identification without prior knowledge of the proteins being investigated. Unlike transcriptomics, which uses PCR to amplify signals, enabling de novo transcriptomics such as RNA sequencing, PCR-equivalent technologies are not yet available for proteomics.

[0005] De novo spatial proteomics can be performed using two main techniques: microscopy and mass spectrometry (MS). Strictly speaking, microscopy is a targeted approach that relies on fluorescent protein or fluorescent dye labeling. The recent large-scale immunostaining of the Protein Atlas Project has mapped thousands of protein species, thus equating to de novo spatial proteomics databases. The limitation of this approach is its applicability to specific biological questions, which requires a new, exhaustive multi-year process to be performed on biological samples with specific mutations.

[0006] MS has long been implemented to identify novel proteomes. Both immunoprecipitation (IP) and MS are widely used biochemical approaches to identify proteomes associated with bait proteins. Recent proximity labeling (PL) approaches provide better spatial accuracy closer to the bait protein. IP and PL results sometimes suffer from low specificity, potentially due to nonspecific interactions during the pull-down process.

[0007] Laser capture microdissection (LCM) allows for the isolation of proteins in specific regions of interest and the subsequent identification of novel spatial proteomes. However, the beam size of the cutting laser is too large to achieve spatial precision. Its indiscriminate axial cutting introduces nonspecific noise and reduces specificity.

[0008] Recent advances in spatially targeted optical microproteomics (STOMP) and its derivative approaches provide another novel spatial proteomics tool for identifying the proteome in a specific region of interest under a microscope. However, this novel spatial proteomics tool lacks the fundamental scale-up requirements for achieving the sensitivity and specificity required for MS, making it difficult to identify low-abundance proteins. Summary of the Invention [Means for solving the problem]

[0009] (Disclosure Summary) In consideration of the above challenges, U.S. Patent No. 11,265,449 disclosed an image-guided system and method that allows for the illumination of samples in various patterns. Through a unique integration of optics, photochemistry, image processing, and mechatronic design, such systems and methods have the ability to process high-abundance proteins, lipids, nucleic acids, or biochemical species for modulation, conversion, isolation, or identification in regions of interest based on user-defined microscopic image features, making them widely useful in cell or tissue sample experiments. More specifically, this technology labels proteins in a region of interest (ROI) of a biological sample (e.g., using biotinylation) and then applies proximity photolabeling to precisely tag proteins in the target region. After photolabeling, the biotinylated proteins are extracted from the sample and subjected to mass spectrometry proteomic analysis. Photo-induced labeling results in low background, making microscopy-guided proteomics feasible. However, illuminating tens of thousands of FOVs typically requires at least one day. The present specification recognizes the need for improved methods for photolabeling proteins in a single FOV within a reasonable time.

[0010] In one aspect, the present invention provides a microscope-based system for rapid illumination of multiple regions of interest in multiple fields of view of a biological sample, the system including a light source, a pattern illumination device, and a processing module coupled to the light source and the pattern illumination device, the processing module configured to identify a region of interest for each of the multiple fields of view; determine, for each field of view, an illumination sequence for the regions of interest by minimizing a sum of multiple inter-region travel distances between a series of regions of interest; and control the light source and the pattern illumination device to illuminate the regions of interest based on the illumination sequence, the illumination sequence differing among the multiple fields of view.

[0011] In another aspect, the present invention also provides a computer-implemented method for performing on a computer processor rapid illumination of multiple regions of interest in multiple fields of view of a biological sample, the method including: identifying a region of interest for each of the multiple fields of view; determining, for each field of view, an illumination sequence for the regions of interest by minimizing a sum of multiple inter-region travel distances between a series of regions of interest; determining, for each field of view, an illumination path that follows the illumination sequence within each of the regions of interest; and controlling the light source and the pattern illumination device to illuminate the regions of interest based on the illumination sequence and the illumination path, wherein the illumination sequence differs between the multiple fields of view.

[0012] In some implementations, the inter-region movement distance is the sum of the straight-line distances between the center points of each of the regions of interest.

[0013] In some implementations, the processing module is further configured to determine an illumination path within each of the regions of interest according to the illumination sequence.

[0014] In some embodiments, the processing module is further configured to control the light source and the pattern illuminator to illuminate the region of interest according to the illumination path and to prevent illumination outside the region of interest.

[0015] In some embodiments, the illumination path extends from a starting point located at the boundary of the first region of interest in the sequence.

[0016] In some embodiments, each of the regions of interest does not overlap or connect with other regions of interest in one of the fields of view.

[0017] In some implementations, the illumination path includes a plurality of illumination stop points and illumination restart points, each of the illumination stop points indicating respective coordinates for switching to each of the restart points.

[0018] In some embodiments, one of the restart points is located within and surrounded by the boundary of one of the regions of interest.

[0019] In some embodiments, one of the restart points is located at one boundary of the region of interest.

[0020] In some implementations, the processing module determines the illumination path by minimizing the number of stopping points and restarting points so as to minimize the total distance of the illumination path.

[0021] In some embodiments, the illumination path includes an end point of a first field of view of the plurality of fields of view, and the processing module is further configured to stop illumination of the illumination path of the first field of view at the end point.

[0022] In some embodiments, the processing module is further configured to control the light source and pattern illumination device to illuminate the region of interest from the start point or each restart point to each stop point, to prevent illumination of the region of interest from each stop point to each restart point, and to stop illumination of the region of interest at the end point.

[0023] A method is provided that includes identifying at least one region of interest in a plurality of fields of view of a biological sample; generating a two-dimensional illumination mask for each of the plurality of fields of view; determining, for each of the plurality of fields of view, an illumination sequence for the at least one region of interest by minimizing a sum of a plurality of inter-region travel distances between a series of regions of interest; determining, for each field of view, an illumination path that follows the illumination sequence within each of the regions of interest; and controlling an illumination source and a pattern illuminator of a microscope-based system to illuminate the region of interest for each of the plurality of fields of view based on the illumination sequence and the illumination path.

[0024] In one aspect, the inter-region movement distance is the sum of the straight-line distances between the center points of each of the regions of interest.

[0025] In another aspect, the method includes controlling the illumination source and the pattern illuminator to illuminate the region of interest according to an illumination path and to prevent illumination outside the region of interest.

[0026] In some embodiments, the illumination path extends from a starting point located at a boundary of a first region of interest in the sequence.

[0027] In one embodiment, each of the regions of interest does not overlap or connect with other regions of interest in one of the fields of view.

[0028] In some aspects, the illumination path includes a plurality of stop and restart points, each of the stop points indicating respective coordinates for switching to each of the restart points.

[0029] In one embodiment, one of the restart points is located within and surrounded by the boundary of one of the regions of interest.

[0030] In another embodiment, one of the restart points is located at one boundary of the region of interest.

[0031] In some embodiments, determining the illumination path includes minimizing the number of stop points and restart points so as to minimize the total distance between every two regions of interest in the illumination path.

[0032] In one aspect, the illumination path includes an end point of a first field of view of the plurality of fields of view, and the method further includes ceasing illumination of the illumination path of the first field of view at the end point.

[0033] In some aspects, the method further includes controlling the illumination source and the pattern illumination device to begin illuminating the region of interest at the start point or each restart point, temporarily stopping illumination of the region of interest from each stop point to each restart point, and stopping illumination of the region of interest at an end point for each of the multiple fields of view. [Brief explanation of the drawings]

[0034] BRIEF DESCRIPTION OF THE DRAWINGS The embodiments are provided by way of example only and will therefore become more fully understood from the non-limiting detailed description of the invention and the accompanying drawings, in which:

[0035] [Figure 1] FIG. 1 depicts a schematic diagram of a microscope-based system according to one embodiment of the present invention.

[0036] [Figure 2] FIG. 2 is a flowchart of a computer-implemented method performed by a processing module for rapid illumination determination for multiple regions of interest in multiple fields of view of a biological sample.

[0037] [Figure 3A] FIG. 3A represents an image of one field of view of sample S obtained by the imaging assembly.

[0038] [Figure 3B] FIG. 3B depicts an exemplary illumination mask for the image of FIG. 3A.

[0039] [Figure 3C] FIG. 3C represents an exemplary lighting sequence determined by a processing module according to the present invention.

[0040] [Figure 3D-3F] 3D, 3E, and 3F show the illumination paths of the corresponding regions of interest.

[0041] [Figure 4] FIG. 4 shows illumination paths for two regions of interest according to another embodiment of the present invention.

[0042] [Figure 5] FIG. 5 shows a schematic diagram of a microscope-based system according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0043] (Detailed explanation) Embodiments of the present invention will become apparent from the following detailed description, which proceeds with reference to the accompanying drawings, in which like reference numerals refer to like elements and in which:

[0044] The terms "first" and "second" may be used herein to describe various features / elements (including steps), but these features / elements should not be limited by these terms unless the context dictates otherwise. These terms may be used to distinguish one feature / element from another. Thus, a first feature / element discussed below could be referred to as a second feature / element, and similarly, a second feature / element discussed below could be referred to as a first feature / element, without departing from the teachings of the present invention.

[0045] The term "beam" as used herein refers to a laser beam used as an illumination source in the present invention. In one embodiment, a femtosecond laser can be used as the illumination source to provide a two-photo effect for high axial illumination precision.

[0046] The term "region of interest" as used herein is user-defined. A region of interest can be the location of a cell nucleus, nucleolus, mitochondria, or any organelle or subcellular compartment. A region of interest can be the location of a protein of interest or a morphological signature. A region of interest can also be a feature defined by two-color imaging, such as the colocation site of proteins A and B, or actin filaments near the centrosome.

[0047] The term "illuminating" as used herein refers to irradiating a point or region with photosensitizing light to achieve localized photolabeling, where the molecule may be a protein, amino acid, lipid, or nucleic acid. Photolabeling is achieved by including a photosensitizer such as riboflavin, rose bengal, or a photosensitizing protein (e.g., miniSOG and Killer Red), and a chemical reagent such as phenol, aryl azide, benzophenone, Ru(bpy)32+, or their derivatives for labeling purposes.

[0048] Examples of microscope-based systems and illumination methods of the present invention include those described in U.S. Patent No. 11,265,449, the entire contents of which are incorporated herein by reference for all purposes. In one embodiment shown in FIG. 1 , a microscope-based system 10 of the present invention may include, for example, but not limited to, a microscope 11, an imaging assembly 12, an illumination assembly 13, and a processing module 14. The microscope 11 includes an objective lens (not shown in FIG. 1 ) and a high-precision microscope stage 15 configured to mount a sample S. The imaging assembly 12 may include a camera 121 and an imaging light source 122. The illumination assembly 13 may include an illumination light source 131 and a patterned illumination device 132.

[0049] In this embodiment, the illumination light source 131 is different from the imaging light source 122 used to image the sample, such as an LED light. The illumination light source 131 here is used only to illuminate the region of interest determined by image processing, and is achieved by point scanning. That is, the illumination light source 131 can be a laser, and the point scanning is achieved by scanning a mirror, such as a galvanometer mirror. For example, a femtosecond laser can be used as the illumination light source 131.

[0050] In this embodiment, processing module 14 is coupled to microscope 11, imaging assembly 12, and illumination assembly 13. In another embodiment, microscope-based system 10 can include a first processing module that independently controls imaging assembly 12 and a second processing module that independently controls illumination assembly 13. Processing module 14 can be a computer, workstation, or computer CPU capable of executing programs designed to operate the system.

[0051] In some embodiments, processing module 14 employs a series of four steps that are repeated tens of thousands of times: Step 1: processing module 14 controls imaging assembly 12 to cause camera 121 to acquire at least one image of sample S in a first field of view (FOV); Step 2: one or more images are automatically sent to processing module 14 in real time based on predefined criteria, so that image processing identifies a region of interest (ROI) and generates an illumination mask for the image of biological sample S; Step 3: processing module 14 controls illumination assembly 13 to illuminate the ROI of sample S according to the illumination mask; and Step 4: after the ROI is fully illuminated, processing module 14 controls stage 15 to move to a second field of view subsequent to the first FOV.

[0052] This rapidly performed iterative process provides enough target protein (e.g., found within a target cellular structure) to overcome the fundamental problem of the lack of viable protein amplification techniques. Prior art techniques have not been optimized to perform such a process with a large number of iterations within a few hours. Without such speed, only high-abundance proteins, most of which are already known, can be identified.

[0053] To improve illumination performance, the present invention provides a microscope-based system for rapid illumination of multiple regions of interest among multiple fields of view of a biological sample, the microscope-based system including a processing module configured to employ algorithms to plot efficient illumination sequences and shortest illumination paths within and between regions of interest in each field of view. See Figure 2. The processing module of the present invention is configured to execute a computer-implemented method including steps 201, 202, 203, and 204.

[0054] In step 201, the processing module is configured to identify regions of interest and generate a two-dimensional illumination mask for each of the multiple fields of view. As described above, the biological sample S is mounted on the stage, and the processing module controls the imaging assembly to obtain images of the biological sample S in each of the multiple fields of view. The images may be fluorescent stained images or bright-field images. The processing module or a connected computer then automatically performs image processing on the images using image processing techniques such as binarization, erosion, filtering, or trained artificial intelligence methods to identify regions of interest based on criteria set by the user. After image processing, the processing module generates a two-dimensional illumination mask that simply indicates all desired regions of interest in each of the multiple fields of view for subsequent illumination. According to the present invention, each identified region of interest exists separately. In other words, each region of interest does not overlap or connect with other regions of interest in one of the fields of view. If two or more regions of interest overlap or connect with each other, these regions of interest are considered to be "one" region of interest.

[0055] In step 202, the processing module is configured to determine an illumination sequence for the regions of interest in each field of view by minimizing the sum of multiple inter-region travel distances between the series of regions of interest. The illumination sequence here refers to sorting the regions of interest in a sequence based on the distribution of the regions of interest, where the total distance between each two regions of interest in the sequence is shortest. In other words, if the illumination time of the regions of interest can be reduced, the sum of multiple inter-region travel distances between the series of regions of interest will be shortest. In one embodiment of the present invention, the inter-region travel distance is the sum of the straight-line distances between the center points of each region of interest.

[0056] In step 203, the processing module is configured to determine an illumination path according to an illumination sequence within each of the regions of interest. The illumination source provides illumination light through the illumination path to illuminate the region of interest of the sample. Thus, illumination path guidance can be used to avoid illumination outside the region of interest while accurately performing photochemical reactions within the region of interest. As mentioned above, the distribution of the regions of interest affects the sequence within several fields of view, and the sequence affects the path. Therefore, the paths within multiple fields of view are varied.

[0057] In step 204, the processing module is configured to control the illumination source and the pattern illuminator to illuminate the region of interest based on the illumination sequence and illumination path for each of the plurality of fields of view.

[0058] In general, an objective of the present invention is to provide a highly efficient algorithm that reduces illumination time while still achieving the maximum area of ​​photoreaction within a region of interest. As described, a processing module controls the illumination assembly to illuminate each ROI location. The illumination sequence provides a minimum distance between each two regions of interest to reduce illumination device travel time. In addition, the illumination path can be guided by conventional algorithms, such as a flooding algorithm. The illumination path of the present invention provides a method for reading as few pixels as possible and uses a minimal amount of memory allocation to increase illumination progression speed. Specific exemplary embodiments according to the present disclosure are described as follows.

[0059] See Figures 3A-3F. As shown in Figure 3A, a field of view 300 of a sample S includes cells 301a-301e, non-cellular material 303, and regions of interest 302a-302e within the cells, e.g., cell nuclei, which can be identified by the processing module 14 by their morphology, e.g., using an artificial intelligence model. In some embodiments, the artificial intelligence model incorporated in the processing module 14 is configured to provide or predict an illumination mask to be used to control the illumination of the cell nuclei, i.e., regions of interest 302, of each cell 301. Due to the variability and diversity of biological samples, the size, shape, and location of the illuminated regions of interest vary from field to field. Therefore, the processing module 14 provides a different illumination mask for each field of view of the biological sample S.

[0060] An exemplary illumination mask 304 for the field of view 300 is shown in Figure 3B. Regions of interest 302a-302e correspond to the coordinates of cell nuclei in the field of view 300 identified by processing module 14. Each of the regions of interest 302a-302e is separate from the other regions of interest; the regions of interest 302a-302e do not overlap or connect with each other in any of the illumination masks 304.

[0061] An exemplary illumination sequence 311 is shown in FIG. 3C. To begin the process of determining the illumination sequence, the raster of the processing module 14 scans the field of view 300 from the edge of the field of view 300. For a global minimum distance strategy, the processing module 14 is configured to calculate the distance between each region of interest 302a-302e and all other regions of interest 302a-302e in the field of view and sort the regions of interest 302a-302e into a scan sequence, which is also an illumination sequence, based on the distribution of the regions of interest 302a-302e. For example, when the scan path of the raster reaches region of interest 302a, the region of interest 302a is the first region to scan and determine illumination path 302-1. The first region of interest 302a can then be used as the basis for sorting all of the regions of interest 302b-302e to define an illumination sequence. 3C, the illumination sequence 311 is marked by dashed lines 312 arranged in the order 302a, 302b, 302c, 302d, and 302e. After determining the illumination sequence 311, the processing module sequentially scans the regions of interest 302a, 302b, 302c, 302d, and 302e to determine the corresponding illumination paths 302-1, 302-2, 302-3, 302-4, and 302-5.

[0062] As described above, the illumination source 131 is a point light source, such as a laser, and illumination of the region of interest 302 is achieved by moving the light source and / or light along an illumination path. If a moving point of light is scanned across the region of interest 302 during the illumination process, the overall illumination time of each field of view may depend, at least in part, on the order in which the region of interest 302 is scanned. One aspect of the present invention is a method and system for identifying and implementing a scanning approach that minimizes the time spent illuminating the region of interest in each field of view. In other words, the present invention provides a method for determining the minimum path for illuminating each entire region of an ROI using a filling algorithm, e.g., a flood filling method.

[0063] See Figures 3C and 3D, which are enlarged views of illumination paths 302-1 and 302-2 for the regions of interest. After determining the illumination sequence 311, the navigation module begins calculating and determines the illumination path 302-1 for the first region of interest 302a. As shown in Figure 3D, the raster scan path reaching the edge of the first region of interest 302a indicates the location of the starting point 320, and the illumination path within the region of interest may be a spiral that begins at the periphery of the region of interest and extends toward the center at the first stopping point 330-1. Thus, according to the illumination path 320, the navigation module controls the illumination assembly to illuminate the first region of interest 302s from the starting point 320 and temporarily stop illumination at the first stopping point 330-1. The dashed line 312 represents the path of the illumination assembly moving to the next region of interest, e.g., 302b, without illumination.

[0064] According to the illumination sequence 311, the progression module then calculates and determines the illumination path 302-2 for the second region of interest 302b. As shown in FIG. 3E, the raster scan path that reaches the edge of the second region of interest 302b indicates the location of the restart point 320-2, and the illumination path within the region of interest may be a spiral that starts from the periphery of the region of interest and extends toward the center of the region of interest at the second stop point 330-2. The area between the first stop point 330-1 and the restart point 320-2 is a non-illuminated portion that is not illuminated by the illumination source 131 and the pattern illuminator 132. Each stop point 330-n indicates a separate coordinate for switching to each restart point 320-n+1. The illumination paths 302-3 and 302-4 are calculated and determined by the process module based on the same rules disclosed by the present invention.

[0065] 3F shows the illumination path 302-5 of the last region of interest 302e of the first field of view according to the determined illumination sequence 311. As shown in FIG. 3F, the raster scan path reaching the edge of the second region of interest 302e indicates the location of restart point 320-5, and the illumination path within the region of interest may be spiral, starting from the periphery of the region of interest and extending toward the center at termination point 340. After reaching termination point 240, all regions of interest are fully illuminated, and processing module 14 controls stage 15 to move to the next field of view and begin imaging, identify the region of interest, determine the illumination sequence and illumination path, and repeat the illumination process.

[0066] Therefore, as described above, the present invention provides a novel algorithm in which the distance between each two regions of interest 302 is minimized and the total scan distance through regions of interest 302a, 302b, 302c, 302d, and 302e in sequences 302-1, 302-2, 302-3, 302-4, and 302-5, respectively, is minimized.

[0067] According to the present invention, each of the regions of interest does not overlap or connect with other regions of interest in one of the fields of view. In some embodiments, if two or more regions of interest are very close to each other, the illumination paths of these adjacent regions of interest can be combined to form a "combined illumination path." To define whether two or more regions of interest are close enough to be "adjacent," one skilled in the art can use a "4-neighbor graph model" or an "8-neighbor graph model" to determine which pixels are adjacent to a given pixel.

[0068] FIG. 4 shows an example of a combined illumination path. As shown in FIG. 4, regions of interest 401a and 401b are close to each other, and dotted lines represent the boundaries of the regions of interest. In this embodiment, it is assumed that regions of interest 401a and 401b are the only two regions of interest in a single field of view. When a raster scan path reaches the boundary of region of interest 401a, indicating the location of start point 420, illumination path 412 within the region of interest can extend along the boundary of regions of interest 401a and 401b. Then, illumination path 412 spirals to the center of first stop point 430-1. Next, illumination path 412 jumps to restart point 420-1 within the boundary of region of interest 401a and extends toward the center of end point 440.

[0069] A "combined illumination path" is a method of achieving a "local minimum" of the illumination paths of two adjacent regions of interest. The combined illumination path may illuminate small areas outside the regions of interest. If the user does not want to illuminate areas outside the regions of interest in any event, the processing module can be trained to not use the combined illumination path.

[0070] In yet another embodiment, the combined illumination path algorithm can be applied even when the region of interest is irregularly shaped rather than generally round. Similar to the example of Figure 4, there can be multiple restart / stop points in the illumination path within an irregularly shaped region of interest.

[0071] In certain embodiments, the illumination paths of the present invention are calculated or determined by a filling algorithm, for example, a flood filling method. The filling algorithm can be coded based on a self-defined numerical control code, as shown in Table 1.

[0072] Table 1. Self-defined numerical control codes [Table 1]

[0073] In some embodiments, the self-defined numerical control code can be implemented in an FPGA, MCU, CPLD, or PLC as an encoder to convert the illumination path into two-dimensional point coordinates. The point coordinates on the solid line defined by each code, d10000-d10008, are exposed to illumination energy once. This method allows the system to conserve the amount of data transferred. Additionally, the self-defined numerical control code can be transferred as a one-dimensional array structure, occupying less memory, due to the FIFO (first-in, first-out) implemented from the host computer to the processing module 14.

[0074] In the embodiment shown in Table 1, the order of the control codes in the fill algorithm determines the illumination path proceeding in a clockwise direction, as shown in Figures 3D-3F. However, in other embodiments, the order of the control codes can be changed to illustrate the illumination path in a counterclockwise direction.

[0075] In some embodiments, as shown in FIG. 3C, the total distance between every two regions of interest 302 in the illumination sequence is minimized, and therefore the number of stop and restart points is minimized to minimize the total distance between every two regions of interest in the path.

[0076] After determining the illumination path, the processing module is further configured to control the illumination light source and pattern illumination device to start illuminating the region of interest at the start point or each restart point, temporarily stop illuminating the region of interest from each stop point to each restart point, and stop illuminating the region of interest at the end point for each of the multiple fields of view. Because the entire biological sample S can be divided into multiple fields of view, the distribution of the regions of interest 302 will be different under different fields of view. The different distribution of the regions of interest 302 will affect the illumination sequence, and therefore the sequence between different fields of view will be different. The total time to photolabel proteins in a 2 cm x 2 cm sample well using a 40x objective lens can range, for example, from 2 hours to 15 hours, depending on the number of ROIs to be illuminated.

[0077] In one embodiment, a detailed microscope-based system for rapid illumination of multiple regions of interest within multiple fields of view of a biological sample according to the present disclosure is shown in FIG. 5. The microscope-based system 500 according to the present disclosure includes a motorized inverted epifluorescence microscope 501 (e.g., a Nikon® Ti2-E microscope) with drift-free focus settings, a controller 506 (e.g., a desktop computer with a field-programmable gate array), and an illumination subsystem 503. The software-firmware integration program in the controller 506 precisely controls imaging, image segmentation, photochemical illumination, and field-of-view changes. The controller 506 controls an LED light source 502 to acquire multicolor fluorescence images (e.g., 488 nm, 568 nm, 647 nm) from a sample on the microscope stage 505, and an sCMOS camera 504 to acquire images of the sample. Wide-field imaging of each color can require, for example, a 100-millisecond exposure time, with a 10-millisecond color switch via the LED's electronic shutter.

[0078] Images can be analyzed in real time by the controller 506 to identify and segment regions of interest in the sample using either traditional image processing or deep learning integrated into the system. This step can take anywhere from 0.1 seconds to 1 second, depending on the complexity of the processing and image quality. In some embodiments, deep learning-based image segmentation can be used to identify regions of interest and generate masks to cover complex images or images of poor image quality. For example, hundreds of annotated images can be used to train a semantic segmentation model using a U-Net convolutional neural network. Pre- and / or post-processing can also be implemented to improve training results, allowing the trained system to perform image segmentation and mask generation more efficiently. In some embodiments, the system uses an integrated software-firmware program to control and precisely coordinate image capture, image segmentation into regions of interest, photochemical illumination of the regions of interest, and stage movement to change the field of view.

[0079] After image capture and processing by the system's controller 506, a mask is generated so that a desired region of interest in the field of view can be illuminated, for example, with two-photo labeling of the region of interest. The mask can be a set of coordinates in the field of view of the sample that correspond to the region of interest. The illumination subsystem uses a 780 nm femtosecond light source 508 (e.g., a Coherent® Chameleon Vision I laser) for two-photon illumination, which induces photochemical reactions (chemical labeling) in the x, y, and z directions. Two-photon illumination allows for good chemical labeling accuracy in the z direction.

[0080] The laser output is adjusted by rotating a half-wave plate 510, which can change the linear polarization of the laser, so that the output can be attenuated by passing it through a polarizing beam splitter cube 512. An acousto-optic modulator (AOM) 514 (e.g., a Gooch & Housego AOMO 3080-125 acousto-optic modulator) under the control of the controller 506 acts as a femtosecond optical shutter to switch the laser light on and off. A quarter-wave plate 516 further converts the polarization of the laser beam to circular polarization. Lenses 518 and 520 expand the size of the laser beam to meet the requirements of a microscope objective lens 522.

[0081] Controller 506 controls a pair of galvanometer scan mirrors (galvo mirrors) 524 and 526 (Cambridge Technology® 6215H mirrors with 671 drivers) to transmit femtosecond light through the microscope's scan lens 528 and tube lens 530, then through objective lens 522, and onto the sample on stage 505. To avoid any slowdown due to mechanical movement, multiband dichroic mirrors 532 and 534 (e.g., mirrors described in U.S. Patent Application No. 63 / 354,806, filed June 23, 2022, the disclosure of which is incorporated herein by reference) are used to enable multicolor imaging and femtosecond light illumination without the movement of mechanical elements such as turrets or shutters. After imaging, region-of-interest identification, mask generation, and two-photon illumination in the sample field of view, controller 506 moves stage 505 to allow imaging, region-of-interest identification, mask generation, and illumination in the next field of view. This process continues until all fields of the sample have been imaged. The only mechanical movements required in this process were a fast galvo scan and a relatively slow stage movement to the next field of view.

[0082] While various exemplary embodiments have been described above, any of several modifications can be made to the various embodiments without departing from the scope of the invention as set forth in the claims. For example, the order in which the various method steps described are performed can often be changed in alternative embodiments, and in other alternative embodiments, one or more method steps can be skipped entirely. Optional features of the various apparatus and system embodiments may be included in some embodiments but not in other embodiments. Accordingly, the foregoing description has been provided primarily for illustrative purposes and should not be construed as limiting the scope of the invention as set forth in the claims.

Claims

1. 1. A computer-implemented method executed on a processor of a computer for rapid illumination of multiple regions of interest in multiple fields of view of a biological sample, the method comprising: for each field of view, identifying the plurality of regions of interest to generate a two-dimensional illumination mask; determining, for each field of view, an illumination sequence for a plurality of regions of interest by minimizing a sum of a plurality of inter-region travel distances between the series of regions of interest; For each field of view, determining an illumination path within each of said regions of interest; and For each field of view, the method includes controlling an illumination source and a pattern illuminator of a microscope-based system to illuminate the plurality of regions of interest based on the illumination sequence and the illumination path.

2. 2. The computer-implemented method of claim 1, wherein the inter-region movement distance is a sum of straight-line distances between center points of each of the plurality of regions of interest.

3. 2. The computer-implemented method of claim 1, wherein the processor is further configured to control the illumination source and the pattern illumination device to illuminate the plurality of regions of interest according to the illumination path and to prevent illumination outside each of the regions of interest.

4. The computer-implemented method of claim 1 , wherein the illumination path extends in a spiral fashion from a starting point located at a boundary of a first region of interest of the sequence toward a center.

5. 10. The computer-implemented method of claim 1, wherein each of the regions of interest does not overlap or connect with other regions of interest in each of the plurality of fields of view.

6. 10. The computer-implemented method of claim 1, wherein the illumination path includes a plurality of stop points and restart points, each of the stop points indicating a respective coordinate for switching to a next restart point.

7. 7. The computer-implemented method of claim 6, wherein one of the restart points is located within, surrounded by a boundary of, or at a boundary of one of the regions of interest.

8. 2. The computer-implemented method of claim 1, wherein determining the illumination path comprises minimizing the number of the stop points and the restart points so as to minimize a total distance between each two or more adjacent regions of interest in the illumination path or a total distance within a region of interest.

9. 2. The computer-implemented method of claim 1, wherein the illumination path includes an end point for each of the fields of view, the method further comprising, for each of the fields of view, stopping illumination of the illumination path at the end point.

10. 2. The computer-implemented method of claim 1, wherein the processing module is further configured to control the illumination source and the pattern illumination device to start illuminating the region of interest at the start point or each restart point, temporarily stop illuminating the region of interest from each stop point to each restart point, and stop illuminating the region of interest at the end point for each of the plurality of fields of view.

11. 10. The computer-implemented method of claim 1, wherein the illumination path can be steered to perform photochemical reactions within multiple regions of interest within multiple fields of view of the biological sample.

Images