Microscope-based system and method for determining beam processing path

Abstract

A microscope-based system is provided. The microscope-based system includes: an irradiation assembly including an irradiation light source and a pattern irradiation device; and a processing module coupled to the irradiation light source and the pattern irradiation device. The processing module is configured to identify a region of interest in the sample to generate a two-dimensional illumination mask for each of a plurality of fields of view, and for each field of view, determine an illumination order for the region of interest by minimizing a sum of a plurality of region-to-region travel distances between consecutive regions of interest, determining an illumination path within each of the regions of interest following the illumination order; and controlling the irradiation light source and the pattern irradiation device to irradiate the region of interest based on the irradiation order and the irradiation path for each of the plurality of fields of view. A method of use is also provided.

Description

Priority claim

[0001] This application requests priority to the U.S. Provisional Patent Application filed on September 8, 2022, with application number 63 / 374,931, and invention name "Microscope-based system and method for determining a beam processing path," the entire contents of which are incorporated into this application by reference. Incorporation into this application by reference

[0002] All publications and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Background Art

[0003] Highly sensitive, hypothesis-free subcellular proteomics is challenging due to the limited sensitivity of mass spectrometry and the lack of protein amplification tools. Without such techniques, it is impossible to discover proteins of interest at specific locations in cell and tissue samples in high abundance.

[0004] Spatial proteomics allows protein profiling of biological samples to reveal the geometric framework of potential protein-protein interactions. Cell biologists and histologists have greatly benefited from recent developments in spatial proteomics, enabling, for example, disease-associated microenvironmental protein profiling, structural protein distribution on structured histological samples, or protein identification of specific organelles. Targeted spatial proteomics aims to locate known proteins, while denovo spatial proteomics requires spatial protein identification without prior knowledge of the proteins to be sought. Unlike transcriptomics, which uses PCR to amplify the signal, thus enabling de novo transcriptomics such as RNAseq, there is no equivalent to PCR for proteomics.

[0005] For nascent proteomics, two main techniques are available: microscopy and mass spectrometry (MS). Strictly speaking, microscopy is a targeted approach that relies on fluorescent proteins or fluorescent dyes. The recent large-scale immunostaining of the Protein Atlas Project has mapped thousands of protein species, making it equivalent to a nascent proteomics database. The limitation of this approach is that it can be applied to specific biological questions, and a comprehensive process of many years must be implemented again for biological samples with specific mutations.

[0006] Mass spectrometry techniques have long been used to identify nascent protein bodies. Immunoprecipitation (IP) and mass spectrometry are both widely used biochemical approaches to identify protein bodies associated with bait proteins. The nearest proximity labeling (PL) method provides better spatial precision in approaching the bait protein. The results of immunoprecipitation and proximity labeling are sometimes less specific, which may be due to nonspecific interactions during the pull-down process.

[0007] Laser capture microdissection (LCM) enables protein isolation in specific regions of interest and subsequent spatial proteome identification. However, the beam size of the cutting laser is too large to achieve spatial precision. Its non-discriminatory axial cutting introduces nonspecific noise and reduces specificity.

[0008] Recent developments in spatially targeted optical microproteomics (STOMP) and its derivatives provide another de novo spatial proteomics tool to identify proteomes in specific regions of interest under the microscope. However, it lacks the fundamental amplification requirements to achieve the sensitivity and specificity required of mass spectrometry, which poses a challenge in identifying low-abundance proteins. Summary of the invention

[0009] In view of the above objectives, U.S. Patent No. 11,265,449 discloses an image-guided system and method capable of irradiating a changing pattern on a sample. Through the unique integration of optics, photochemistry, image processing and mechatronic design, such systems and methods are capable of processing high-content proteins, lipids, nucleic acids or biochemical substances for regulation, conversion, separation or identification in regions of interest based on user-defined microscopic image features, and are widely used in cell or tissue sample experiments. More specifically, the present technology labels (using, for example, biotinylation) proteins at regions of interest (ROIs) of biological samples, and then applies proximity photolabeling to accurately label proteins in the target area. After photolabeling, the biotinylated proteins are extracted from the sample and subjected to mass spectrometry proteomics analysis. Photoinduced labeling ensures low background, making microscope-guided proteomics feasible. However, it usually takes at least one day to irradiate tens of thousands of fields of view (FOV). It is recognized here that there is a need for an improved method for photolabeling proteins in a field of view within a reasonable duration.

[0010] In one aspect, the present invention provides a microscope-based system for rapidly illuminating multiple regions of interest in multiple fields of view of a biological sample, comprising a light source, a pattern illumination device, and a processing module coupled to the light source and the pattern illumination device, wherein the processing module is configured to identify a region of interest for each of the multiple fields of view; for each field of view, determine an illumination order for the regions of interest by minimizing the sum of multiple region-to-region travel distances between consecutive regions of interest; and control the light source and the pattern illumination device to illuminate the regions of interest based on the illumination order, wherein the illumination order in the multiple fields of view is different.

[0011] On the other hand, the present invention also provides a computer-implemented method for rapidly irradiating multiple regions of interest in multiple fields of view of a biological sample, the computer-implemented method being executed in a processor of a computer, comprising: identifying a region of interest for each of the multiple fields of view; for each field of view, determining an irradiation order for the regions of interest by minimizing the sum of multiple region-to-region travel distances between consecutive regions of interest; for each field of view, determining an irradiation path within each of the regions of interest following the irradiation order; and controlling the light source and the pattern irradiation device to irradiate the region of interest based on the irradiation order and the irradiation path, wherein the irradiation order in the multiple fields of view is different.

[0012] In some embodiments, the region-to-region travel distance is the sum of the straight-line distances between the center points of each of the regions of interest.

[0013] In some embodiments, the processing module is further configured to determine an illumination path within each of the regions of interest following the illumination sequence.

[0014] In some embodiments, the processing module is further configured to control the light source and the pattern illumination device to illuminate the region of interest according to the illumination path and prevent illumination outside the region of interest.

[0015] In some embodiments, the illumination path extends from a starting point located at a boundary of a first region of interest in the sequence.

[0016] In some embodiments, each region of interest does not overlap or connect with any other region of interest in the field of view.

[0017] In some embodiments, the irradiation path includes a plurality of irradiation stop points and irradiation resume points, and each of the irradiation stop points indicates an independent coordinate for switching to each of the resume points.

[0018] In some embodiments, one of the recovery points is located within one of the regions of interest and is surrounded by a boundary of the region of interest.

[0019] In some embodiments, one of the recovery points is located at a boundary of one of the regions of interest.

[0020] In some embodiments, the processing module determines the illumination path by minimizing the number of the stopping points and the number of the resuming points, thereby minimizing the total distance of the illumination path.

[0021] In some embodiments, for a first field of view among the multiple fields of view, the illumination path includes a termination point, and the processing module is further configured to terminate the illumination of the illumination path for the first field of view at the termination point.

[0022] In some embodiments, the processing module is further configured to control the light source and the pattern illumination device to illuminate the area of ​​interest from the starting point or each recovery point to each stop point, prevent irradiation of the area of ​​interest from each stop point to each recovery point, and terminate irradiation of the area of ​​interest at the end point.

[0023] A method is provided, comprising: identifying at least one region of interest in a plurality of fields of view of a biological sample; generating a two-dimensional illumination mask for each of the plurality of fields of view; determining, for each of the plurality of fields of view, an illumination sequence for the at least one region of interest by minimizing a sum of a plurality of region-to-region travel distances between consecutive regions of interest; determining, for each field of view, an illumination path within each of the regions of interest following the illumination sequence; and controlling an illumination light source and a pattern illumination device of a microscope-based system to illuminate the region of interest based on the illumination sequence and the illumination path for each of the plurality of fields of view.

[0024] In one aspect, the region-to-region distance is the sum of the straight-line distances between the center points of each of the regions of interest.

[0025] In another aspect, the method includes controlling the illumination light source and the pattern illumination device to illuminate the region of interest according to the illumination path and to prevent illumination outside of the region of interest.

[0026] In certain aspects, the illumination path extends from a starting point located at a boundary of a first region of interest in the sequence.

[0027] In one aspect, each region of interest does not overlap or connect with any other region of interest in the field of view.

[0028] In certain aspects, the illumination path includes a plurality of stop points and resume points, and each of the stop points indicates independent coordinates for switching to each of the resume points.

[0029] In one aspect, one of the recovery points is located within one of the regions of interest and is surrounded by a boundary of the region of interest.

[0030] In another aspect, one of the recovery points is located at a boundary of one of the regions of interest.

[0031] In certain aspects, the step of determining the illumination path includes minimizing the number of the stopping points and the resuming points so as to minimize the total distance between every two regions of interest in the illumination path.

[0032] In one aspect, for a first field of view of the plurality of fields of view, the illumination path includes a termination point, and the method further includes terminating illumination of the illumination path for the first field of view at the termination point.

[0033] In some aspects, the method further includes controlling the illumination light source and the pattern illumination device to start illuminating the region of interest at the starting point or each recovery point, to temporarily stop illuminating the region of interest from each stop point to each recovery point, and to terminate the illuminating of the region of interest at the termination point for each of the multiple fields of view. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Embodiments of the present invention may be more fully understood from the detailed description and accompanying drawings, which are intended to illustrate and not to limit the present invention, wherein:

[0035] Figure 1 A schematic diagram showing a microscope-based system according to one embodiment of the present invention.

[0036] Figure 2 is a flow chart of a computer-implemented method performed by a processing module to determine rapid illumination on multiple regions of interest in multiple fields of view of a biological sample.

[0037] Figure 3A Represents an image of a field of view of a sample S acquired by the imaging component.

[0038] Figure 3B express Figure 3A An image of an exemplary illumination reticle.

[0039] Figure 3C An exemplary illumination sequence determined by the processing module of the present invention is shown.

[0040] Figure 3D , 3Eand 3F depict the illumination paths of the corresponding regions of interest.

[0041] Figure 4 Depicted are illumination paths for two regions of interest according to another embodiment of the present invention.

[0042] Figure 5 Depicted is a schematic diagram of a microscope-based system according to another embodiment of the present invention. DETAILED DESCRIPTION

[0043] Embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein like reference numerals refer to like components.

[0044] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context indicates otherwise. These terms can be used to distinguish one feature / element from another feature / element. Therefore, without departing from the teachings of the present invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0045] As used herein, the term "light beam" is a laser beam, used as the irradiation light source of the present invention. In one embodiment, a femtosecond laser can be used as the irradiation light source to produce a double light effect to achieve high axial irradiation accuracy.

[0046] As used herein, the term "region of interest" is defined by the user. They can be the location of the nucleus, nucleolus, mitochondria, or any organelle or subcellular compartment. They can be the location of a protein of interest, or a morphological feature. They can also be features defined by two color imaging, such as the co-location site of proteins A and B, or actin filaments near the centrosome.

[0047] The term "irradiation" as used herein refers to irradiating a photosensitive light (photosensitizing light) on a point or area to achieve local light marking, wherein the molecule can be a protein, an amino acid, a lipid or a nucleic acid. The light marking process is by including a photosensitizer such as riboflavin, Rose Bengal or a photosensitive protein (such as miniSOG and Killer Red, etc.) and a chemical reagent such as phenol, aryl azide, benzophenone, Ru (bpy) 32 + or their derivatives for marking purposes.

[0048] Examples of microscope-based systems and illumination methods of the present invention include those described in U.S. Pat. No. 11,265,449, which is fully incorporated herein by reference for all purposes. Figure 1In one embodiment shown, the microscope-based system 10 of the present invention may include, for example but not limited to, a microscope 11, an imaging component 12, an illumination component 13, and a processing module 14. The microscope 11 includes an objective lens ( Figure 1 The imaging component 12 may include a camera 121 and an imaging light source 122 . The irradiation component 13 may include an irradiation light source 131 and a pattern irradiation device 132 .

[0049] In this embodiment, the irradiation light source 131 is different from the imaging light source 122 used for sample imaging, such as an LED lamp. The irradiation light source 131 here is only used to irradiate the area of ​​interest determined by image processing, and is implemented by point scanning. That is, the irradiation light source 131 can be a laser, and the point scanning is implemented by a scanning mirror such as a galvanometer reflector. For example, a femtosecond laser can be used as the irradiation light source 131.

[0050] In this embodiment, the processing module 14 is coupled to the microscope 11, the imaging assembly 12, and the irradiation assembly 13. In another embodiment, the microscope-based system 10 may include a first processing module that independently controls the imaging assembly 12, and a second processing module that independently controls the irradiation device 13. The processing module 14 may be a computer, a workstation, or a CPU of a computer that is capable of executing a program designed for an operating system.

[0051] In some embodiments, the processing module 14 uses four consecutive steps and repeats tens of thousands of times. Step 1: The processing module 14 controls the imaging component 12 so that the camera 121 obtains at least one image of the sample S in the first field of view (FOV); Step 2: Automatically transmit the image or multiple images to the processing module 14 in real time according to a predetermined standard to identify the region of interest (ROI) through image processing and generate an illumination mask of the image of the biological sample S; Step 3: The processing module 14 controls the illumination component 13 to illuminate the region of interest of the sample S according to the illumination mask; and Step 4: When the region of interest is completely illuminated, the processing module 14 controls the stage 15 to move to the second field of view behind the first field of view.

[0052] This rapid, repetitive process provides enough target proteins (e.g., found in target cell structures) to overcome the fundamental problem of the lack of viable protein amplification technology. Existing technologies are not optimized to perform such a process multiple times over several hours. Without such speed, one can only identify proteins that are mostly known and abundant.

[0053] To improve the irradiation performance, the present invention provides a microscope-based system for rapidly irradiating multiple regions of interest in multiple fields of view of a biological sample, the system comprising a processing module configured to employ an algorithm to map an effective irradiation sequence and the shortest irradiation paths within and between the regions of interest in each field of view. Figure 2 The processor module of the present invention is configured to execute the computer-implemented method comprising steps 201 , 202 , 203 and 204 .

[0054] In step 201, the processing module is configured to identify the region of interest to generate a two-dimensional irradiation mask for each of the multiple fields of view. As described above, the biological sample S is loaded onto the stage, and the processing module controls the imaging component to acquire an image of the biological sample S for each of the multiple fields of view. The image can be a fluorescent staining image or a bright field image. Then, the processing module or a connected computer automatically performs image processing on the image using an image processing technique (such as threshold processing, erosion, filtering, or a trained artificial intelligence method) to identify the region of interest based on the criteria set by the user. After the image processing, the processing module generates a two-dimensional irradiation mask showing only all desired regions of interest in each of the multiple fields of view for subsequent irradiation. According to the present invention, each identified region of interest exists separately. In other words, each region of interest does not overlap or connect with any other region of interest in one of the fields of view. If two or more regions of interest overlap or connect with each other, these regions of interest are considered to be "one" region of interest.

[0055] In step 202, the processing module is configured to determine, for each field of view, an illumination order of the regions of interest by minimizing the sum of multiple region-to-region travel distances between consecutive regions of interest. The illumination order here refers to sorting the regions of interest based on the distribution of the regions of interest, wherein the total distance between each two regions of interest in the sorting is minimized. In other words, if we can shorten the illumination time of the regions of interest, the sum of multiple region-to-region travel distances between consecutive regions of interest is minimized. In an embodiment of the present invention, the region-to-region travel distance is the sum of the straight-line distances between the center points of each of the regions of interest.

[0056] In step 203, the processing module is configured to determine the illumination path in each region of interest following the light sequence. The illumination light source provides the illumination light via the illumination path to illuminate the region of interest of the sample. Therefore, with the guidance of the illumination path, the photochemical reaction can be accurately performed in the region of interest while avoiding illumination outside the region of interest. As mentioned above, the distribution of the regions of interest will affect the sequence in the multiple fields of view, and the sequence will affect the path. Therefore, the paths in the multiple fields of view are various.

[0057] In step 204, the processing module is configured to control the illumination light source and the pattern illumination device to illuminate the region of interest based on the illumination sequence and the illumination path for each of the plurality of fields of view.

[0058] In general, an object of the present invention is to provide a very effective algorithm to reduce the illumination time, but still perform the maximum area of ​​photoreaction within the region of interest. As described above, the processing module controls the illumination assembly to illuminate each region of interest. The illumination sequence provides a minimum distance between each two regions of interest to shorten the travel time of the illumination device. In addition, the illumination path can be performed using a traditional algorithm such as a flooding algorithm. The illumination path of the present invention provides a method to speed up the illumination progress by reading as few pixels as possible and using a minimum amount of memory allocation. The following describes certain exemplary embodiments according to the present disclosure.

[0059] Please refer to FIG. 3A to FIG. 3F .like Figure 3A As shown, the field of view 300 of the sample S contains cells 301a to 301e, non-cellular material 303, and subcellular regions of interest 302a to 302e, such as cell nuclei, which can be identified by their morphology by the processing module 14, for example, using an artificial intelligence model. In some embodiments, the artificial intelligence model incorporated into the processing module 14 is configured to provide or predict an illumination mask, which is used to control the illumination of the nucleus (i.e., the region of interest 302) of each cell 301. Due to the variability and diversity of biological samples, the size, shape, and position of the region of interest that needs to be irradiated will be different in each field of view. Therefore, the processing module 14 will provide a different illumination mask for each field of view of the biological sample S.

[0060] Figure 3B An exemplary illumination mask 304 is shown for the field of view 300. Regions of interest 302a-302e correspond to the coordinates of cell nuclei identified by the processing module 14 in the field of view 300. Each of the regions of interest 302a-302e is separate from the other regions of interest; the regions of interest 302a-302e do not overlap or connect to each other in any illumination mask 304.

[0061] Figure 3C An exemplary illumination sequence 311 is shown in . To begin the process of determining the illumination sequence, the processing module 14 raster scans the field of view 300 from the edge of the field of view 300. The processing module 14 is configured to calculate the distance between each region of interest 302a to 302e and all other regions of interest 302a to 302e in the field of view for a global minimum distance strategy, and sort the regions of interest 302 in a scanning sequence, which is also an illumination sequence, based on the distribution of the regions of interest 302a to 302e. For example, when the raster scanning path reaches the region of interest 302a, the region of interest 302a is the first region to be scanned and determines the illumination path 302-1. The first region of interest 302a can then be used as a basis for sorting all regions of interest 302b to 302e and defining an illumination sequence. As Figure 3C As shown, the illumination sequence 311 is marked by a dotted line 312 arranged in the order of 302a, 302b, 302c, 302d and 302e. After determining the illumination sequence 311, the processing module sequentially scans the regions of interest 302a, 302b, 302c, 302d and 302e to determine the corresponding illumination paths of 302-1, 302-2, 302-3, 302-4 and 302-5.

[0062] As described above, the illumination light source 131 is a point light source such as a laser, and illumination of the region of interest 302 is performed by moving the light source and / or light along an illumination path. When the light spot is moved to scan across the region of interest 302 during the illumination process, the total illumination time for each field of view may depend at least in part on the order in which the regions of interest 302 are scanned. One aspect of the present invention is a method and system for identifying and implementing a scanning pattern that minimizes the time spent illuminating the region of interest in each field of view. In other words, the present invention provides a method for determining a minimum route to illuminate the entire area of ​​each region of interest through a filling algorithm (e.g., a flood filling method).

[0063] Please refer to Figure 3C and 3D , which are enlarged views of the illumination paths 302-1 and 302-2 of the regions of interest. After determining the illumination sequence 311, the processing module starts to calculate and determine the illumination path 302-1 of the first region of interest 302a. Figure 3DAs shown, the raster scanning path reaches the edge of the first region of interest 302a, which is marked as the starting point 320, and the irradiation path within the region of interest can be a spiral line starting from the periphery of the region of interest and extending to the center at the initial stop point 330-1. Therefore, according to the irradiation path 320, the processing module controls the irradiation component to irradiate the first region of interest 302a from the starting point 320, and temporarily suspends the irradiation at the initial stop point 330-1. The dotted line 312 represents the path of the irradiation component moving to the subsequent region of interest (e.g., 302b) without irradiation.

[0064] Based on the illumination sequence 311, the processing module then calculates and determines the illumination path 302-2 of the second region of interest 302b. Figure 3E As shown, the edge of the raster scanning path reaching the second region of interest 302b is marked as the position of the recovery point 320-2, and the illumination path within the region of interest can be a spiral line starting from the periphery of the region of interest and extending to the center at the second stop point 330-2. Between the initial stop point 330-1 and the recovery point 320-2 is a non-irradiated portion, which passes through the illumination light source 131 and the pattern illumination device 132 without illumination. Each of the stop points 330-n indicates an independent coordinate for switching to each recovery point 320-n+1. The illumination paths 302-3 and 302-4 are calculated and determined by the processing module according to the same rules disclosed in the present invention.

[0065] Figure 3F FIG. 3 shows an illumination path 302-5 of the last region of interest 302e of the first field of view according to the determined illumination sequence 311. Figure 3F As shown, the raster scanning path reaches the edge of the second region of interest 302e marked as the position of the recovery point 320-5, and the irradiation path within the region of interest can be a spiral line starting from the periphery of the region of interest and extending to the center at the end point 340. After reaching the end point 340, all regions of interest are fully irradiated, and the processing module 14 controls the stage 15 to move to the subsequent field of view to start imaging, identify the region of interest, determine the irradiation order and irradiation path, and perform irradiation processing again.

[0066] As described above, the present invention therefore provides a novel algorithm to determine that the distance between each two regions of interest 302 is minimized, and the total scanning distance passing through the regions of interest 302a, 302b, 302c, 302d and 302e in sequence 302-1, 302-2, 302-3, 302-4 and 302-5 respectively is minimized.

[0067] According to the present invention, each of the regions of interest does not overlap or connect with any other region of interest in one of the fields of view. In some embodiments, if two or more regions of interest are very close to each other, the illumination paths of these adjacent regions of interest can be combined together to become a "joint illumination path". In order to define whether two or more regions of interest are close enough to become "neighbors", those skilled in the art can use a "4-neighbor graph model" or an "8-neighbor graph model" to know which pixels are adjacent to a given pixel.

[0068] Figure 4 An example of a combined irradiation path is depicted. Figure 4 As shown, the regions of interest 401a and 401b are close to each other, and the dotted lines represent the boundaries of their regions of interest. In this embodiment, we assume that the regions of interest 401a and 401b are the only two regions of interest in a field of view. When the raster scanning path reaches the position of the boundary marking starting point 420 of the region of interest 401a, the illumination path 412 within the region of interest can be extended along the boundary of the regions of interest 401a and 401b, and then the illumination path 412 spirally extends to the center at the initial stop point 430-1. Next, the illumination path 412 jumps to the recovery point 420-1 within the boundary of the region of interest 401a and extends to the center at the termination point 440.

[0069] A "joint illumination path" is a method of achieving a "local minima" of the illumination paths of two adjacent regions of interest. It allows illumination of a small area outside of the region of interest. If the user does not want illumination outside of the region of interest under any circumstances, they can instruct the processing module not to use the joint illumination path.

[0070] In another embodiment, if the region of interest is an irregular shape instead of the usual circular shape, the joint illumination path algorithm can still be applied. Figure 4 Similar to the example in, there may be multiple resume / stop points in the illumination path within a region of interest with an irregular shape.

[0071] In some embodiments, the irradiation path of the present invention is calculated or determined by a filling algorithm (eg, flood filling method). The filling algorithm can be encoded based on a custom numerical control code, as shown in Table 1.

[0072] Table 1 Customized NC codes

[0073] In some embodiments, the custom numerical control code can be implemented as an encoder on an FPGA, MCU, CPLD or PLC to convert the irradiation path into two-dimensional point coordinates. The point coordinates in the solid line determined by each program code d10000 to d10008 will be exposed once under the irradiation energy. This method allows the system to save the amount of data transmitted. In addition, the custom numerical control code can be transmitted in the form of a one-dimensional array structure, which takes up less memory for the first-in-first-out (FIFO) implemented from the host computer to the processing module 14.

[0074] In the embodiment shown in Table 1, the numerical control code sequence of the filling algorithm determines the irradiation path drawn clockwise, such as Figures 3D to 3F However, in other embodiments, when the order of the numerical control codes is changed, the irradiation path may be drawn in a counterclockwise direction.

[0075] In some embodiments, because Figure 3C As shown, the total distance between every two regions of interest 302 in the illumination sequence is minimized, so the number of stop points and resume points is minimized to minimize the total distance between every two regions of interest in the path.

[0076] After determining the irradiation path, the processing module is further configured to control the irradiation light source and the pattern irradiation device to start irradiating the region of interest at the starting point or each recovery point, temporarily stop irradiating the region of interest from each stop point to each recovery point, and terminate the irradiation of the region of interest at the termination point for each of the multiple fields of view. Since the entire biological sample S can be divided into multiple fields of view, the distribution of the region of interest 302 will be different in different fields of view. The different distribution of the region of interest 302 will affect the irradiation order, and the order in different fields of view will also change accordingly. Depending on the number of irradiated regions of interest, the total time for photolabeling proteins in a 2 cm x 2 cm sample chamber using a 40x objective lens can be, for example, in the range of 2 to 15 hours.

[0077] Figure 5A detailed microscope-based system according to the present invention is shown for rapidly illuminating multiple regions of interest in multiple fields of view of a biological sample. The microscope-based system 500 according to the present disclosure includes a motorized inverted epifluorescence microscope 501 (e.g., a Nikon® Ti2-E microscope) with drift-free focus settings, a controller 506 (e.g., a desktop computer with a field programmable gate array), and an illumination subsystem 503. The software-firmware integration program in the controller 506 tightly coordinates the control of imaging, image segmentation, photochemical illumination, and field of view changes. The controller 506 controls an LED light source 502 for obtaining multi-color fluorescence images (e.g., 488 nm, 568 nm, 647 nm) from a sample on a microscope stage 505 and a sCMOS camera 504 for capturing images of the sample. Wide-area imaging of each color may require (e.g., 100 ms exposure time) an LED electronic shutter for 10 ms color switching.

[0078] The controller 506 can analyze the image in real time, using traditional image processing or deep learning embedded in the system to identify and segment the area of ​​interest in the sample. This step takes 0.1 to 1 second, depending on the processing complexity and image quality. In some embodiments, image segmentation based on deep learning can be used to identify areas of interest and generate masks for complex images or low-quality images. For example, hundreds of annotated images can be used to train a semantic segmentation model using a U-Net convolutional neural network. Pre-processing and / or post-processing can also be implemented to improve the training results, and the trained system can perform image segmentation and mask generation more efficiently. In some embodiments, the system uses a software-firmware integration program to control and closely coordinate image capture, image segmentation into areas of interest, photochemical irradiation of areas of interest, and stage movement for changing the field of view.

[0079] After the image is captured and processed by the system's controller 506, a mask is generated to illuminate the desired region of interest in the field of view, such as marking the region of interest with two-photons. The mask can be a collection of coordinates on the sample field of view corresponding to the region of interest. The illumination subsystem uses a 780nm femtosecond light source 508 (e.g., Coherent® Chameleon Vision I laser) for triggering two-photon illumination of photochemical reactions (chemical labels) in the x, y, and z directions. Two-photon illumination obtains better chemical labeling accuracy in the z direction.

[0080] The laser power is adjusted by rotating the half-wave plate 510, which can change the linear polarization direction of the laser, so the power can be attenuated by passing through the polarization beam splitter cube 512. Acousto-optic modulator (AOM) 514 (e.g., Gooch & Housego AOMO 3080-125 AOM) acts as a femtosecond light shutter to turn the laser on and off under the control of controller 506. Quarter wave plate 516 further changes the polarization of the laser beam to circular polarization. Lenses 518 and 520 expand the laser beam size to meet the requirements of microscope objective 522.

[0081] Controller 506 controls a pair of galvanometer scanning mirrors (galvanometer mirrors) (Cambridge Technology® 6215H mirrors with 671 actuators) 524 and 526 to direct femtosecond light through the microscope's scanning lens 528 and tube lens 530, via objective 522 to the sample on stage 505. To avoid any slowdown due to mechanical movement, multi-band dichroic mirrors 532 and 534 (such as those described in the mirrors described in U.S. Application No. 63 / 354,806 filed on June 23, 2022, the disclosure of which is incorporated herein by reference) are used to achieve multicolor imaging and femtosecond light irradiation without moving mechanical components such as turrets or shutters. After imaging, identifying, creating a mask, and two-photon irradiation of the region of interest in the sample field of view, controller 506 moves stage 505 so that the next field of view can be imaged, identified, created, and irradiated. This process continues until all fields of view of the sample have been imaged. The only mechanical motion required in the process is the rapid scanning of the galvanometer and the relatively slow movement of the stage to the next field of view.

[0082] Although various illustrative embodiments are described above, any of a variety of changes may be made to the various embodiments without departing from the scope of the invention as described in the claims. For example, in alternative embodiments, the order in which the various described method steps are performed can often be changed, and in other alternative embodiments, one or more method steps may be skipped altogether. Optional features of the various device and system embodiments may be included in some embodiments and not in other embodiments. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention as described in the claims.

Claims

1. A microscope-based system for rapidly illuminating multiple regions of interest in multiple fields of view of a biological sample, comprising: An irradiation assembly including an irradiation light source and a pattern irradiation device; as well as A processing module is connected to the illumination light source and the pattern illumination device, wherein the processing module is configured to: identifying the region of interest to generate a two-dimensional illumination mask for each of the plurality of fields of view; For each field of view, determining an illumination order for the regions of interest by minimizing a sum of a plurality of region-to-region travel distances between consecutive regions of interest; For each field of view, determining an illumination path following the illumination sequence within each of the regions of interest; and The illumination light source and the pattern illumination device are controlled to illuminate the region of interest based on the illumination sequence and the illumination path for each of the plurality of fields of view.

2. The system of claim 1, wherein: The region-to-region travel distance is the sum of the straight-line distances between the center points of each of the regions of interest.

3. The system of claim 1, wherein: The processing module is further configured to control the illumination light source and the pattern illumination device to illuminate the region of interest according to the illumination path and prevent illumination outside the region of interest.

4. The system of claim 1, wherein: The illumination path extends from a starting point located at a boundary of a first region of interest in the sequence.

5. The system of claim 1, wherein: Each region of interest does not overlap or connect with any other region of interest in the field of view.

6. The system of claim 1, wherein: The irradiation path includes a plurality of irradiation stop points and irradiation resumption points, and each of the irradiation stop points indicates independent coordinates for switching to each of the resumption points.

7. The system of claim 6, wherein: The starting point and one of the recovery points are located within one of the regions of interest and are surrounded by a boundary of the region of interest.

8. The system of claim 6, wherein: The starting point and one of the recovery points are located at a boundary of one of the regions of interest.

9. The system of claim 6, wherein: The processing module determines the illumination path by minimizing the number of the stopping points and the number of the resuming points to minimize a total distance of the illumination path.

10. The system of claim 1, wherein: For a first field of view among the multiple fields of view, the illumination path includes a termination point, and the processing module is further configured to terminate the illumination of the illumination path for the first field of view at the termination point.

11. The system of claim 10, wherein: The processing module is further configured to control the illumination light source and the pattern illumination device to start irradiating the area of ​​interest at the starting point or each recovery point, to temporarily stop irradiating the area of ​​interest from each stop point to each recovery point, and to terminate the irradiation of the area of ​​interest at the termination point for each of the multiple fields of view.

12. A computer-implemented method for rapidly illuminating a plurality of regions of interest in a plurality of fields of view of a biological sample, the computer-implemented method being executed in a processor of a computer, comprising: identifying the region of interest to generate a two-dimensional illumination mask for each of the plurality of fields of view; For each field of view, determining an illumination order for the regions of interest by minimizing a sum of a plurality of region-to-region travel distances between consecutive regions of interest; For each field of view, determining an illumination path in each of the regions of interest following the illumination sequence; as well as An illumination light source and a pattern illumination device of the microscope-based system are controlled to illuminate the region of interest based on the illumination sequence and the illumination path for each of the plurality of fields of view.

13. The computer-implemented method of claim 12, wherein: The region-to-region travel distance is the sum of the straight-line distances between the center points of each of the regions of interest.

14. The computer-implemented method of claim 12, wherein: The processor is further configured to control the illumination light source and the pattern illumination device to illuminate the region of interest according to the illumination path and to prevent illumination outside the region of interest.

15. The computer-implemented method of claim 12, wherein: The illumination path extends from a starting point located at a boundary of a first region of interest in the sequence.

16. The computer-implemented method of claim 12, wherein: Each region of interest does not overlap or connect with any other region of interest in the field of view.

17. The computer-implemented method of claim 12, wherein: The illumination path includes a plurality of stop points and resume points, and each of the stop points indicates independent coordinates for switching to each of the resume points.

18. The computer-implemented method of claim 17, wherein: One of the recovery points is located within one of the regions of interest and is surrounded by a boundary of the region of interest.

19. The computer-implemented method of claim 17, wherein: One of the recovery points is located at a boundary of one of the regions of interest.

20. The computer-implemented method of claim 17, wherein: The step of determining the illumination path includes minimizing the number of the stopping points and the resuming points so as to minimize the total distance between every two regions of interest in the illumination path.

21. The computer-implemented method of claim 12, wherein: For a first field of view among the plurality of fields of view, the illumination path includes a termination point, and the method further includes terminating illumination of the illumination path for the first field of view at the termination point.

22. The computer-implemented method of claim 21, wherein: The processing module is further configured to control the illumination light source and the pattern illumination device to start irradiating the area of ​​interest at the starting point or each recovery point, to temporarily stop irradiating the area of ​​interest from each stop point to each recovery point, and to terminate the irradiation of the area of ​​interest at the termination point for each of the multiple fields of view.

23. A method comprising: identifying at least one region of interest in a plurality of fields of view of a biological sample; generating a two-dimensional illumination mask for each of the plurality of fields of view; for each of the plurality of fields of view, ordering illumination of the at least one region of interest by minimizing a sum of a plurality of region-to-region travel distances between consecutive regions of interest; For each field of view, determining an illumination path in each of the regions of interest following the illumination sequence; as well as An illumination light source and a pattern illumination device of the microscope-based system are controlled to illuminate the region of interest based on the illumination sequence and the illumination path for each of the plurality of fields of view.

24. The method of claim 23, wherein: The region-to-region travel distance is the sum of the straight-line distances between the center points of each of the regions of interest.

25. The method of claim 23, further comprising controlling the illumination light source and the pattern illumination device to illuminate the region of interest according to the illumination path and prevent external illumination of the region of interest.

26. The method of claim 23, wherein: The illumination path extends from a starting point located at a boundary of a first region of interest in the sequence.

27. The method of claim 23, wherein: Each of the regions of interest does not overlap or connect with any other regions of interest in the field of view.

28. The method of claim 23, wherein: The irradiation path includes a plurality of stop points and resume points, each of the stop points indicating independent coordinates for switching to each of the resume points.

29. The method of claim 28, wherein: One of the recovery points is located within one of the regions of interest and is surrounded by a boundary of the region of interest.

30. The method of claim 28, wherein: One of the recovery points is located at a boundary of one of the regions of interest.

31. The method of claim 28, wherein: The step of determining the illumination path includes minimizing the number of the stopping points and the resuming points so as to minimize the total distance between every two regions of interest in the illumination path.

32. The method of claim 23, wherein: For a first field of view among the plurality of fields of view, the illumination path includes a termination point, and the method further includes terminating illumination of the illumination path for the first field of view at the termination point.

33. The method as claimed in claim 32 further includes controlling the illumination light source and the pattern illumination device to start irradiating the area of ​​interest at the starting point or each recovery point, to temporarily stop irradiating the area of ​​interest from each stop point to each recovery point, and to terminate the irradiation of the area of ​​interest at the termination point for each of the multiple fields of view.

Images