Artificial intelligence model for training light illumination system for microscope-based patterns

Abstract

A method for training an artificial intelligence model for microscope-based pattern light illumination includes selecting an AI model architecture having a predetermined number of convolutional layers between input and output layers. Then, one or more images captured by a microscope-based system having an image processing system are introduced into the AI model, the one or more images marked with a known region of interest for processing by the predetermined number of convolutional layers. The output value is evaluated from a known region of interest and the validated output value is accepted for identifying the region of interest. The output value may be further implemented in the generation of a mask pattern.

Description

Priority claim

[0001] This case claims priority to U.S. Provisional Patent Application No. 63 / 368,704, filed on July 18, 2022, entitled “TRAINING AI MODEL FOR AMICROSCOPE-BASED PATTERN PHOTOILLUMINATION SYSTEM,” which is incorporated herein by reference in its entirety. Incorporated herein by reference

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Background Art

[0003] Artificial intelligence models and systems improve the accuracy, efficiency, and capabilities of image processing systems. Artificial neural networks are widely used in fields ranging from transportation to medicine because they provide the opportunity to transform complex data to produce relevant and useful output information.

[0004] In particular, the biomedical field has benefited greatly from the advances of these two AI models. Disease diagnosis, treatment strategies, drug development, research, etc. are increasingly relying on and incorporating AI to accelerate the understanding of anatomy and physiology. However, current systems associated with digital image processing of biological specimens have deficiencies in identifying important cellular structures and functions. These deficiencies in current technology have prompted people to continue to rely on empirical methods to understand complex structures and associated functions. For example, genome sequencing may be used instead of visual confirmation of specific structures or events, and comprehensive studies are often required to justify the quantitative measurement results. This is laborious and may reduce the accuracy of the results.

[0005] Advances in microscopy may allow for more detailed visualization of cell biology, but require extensive education and training by the user; and still result in reduced accuracy and efficiency. User-based microscopy is subjective, and even with relevant dyes and probes, can lead to misinterpretations or missed opportunities. Precise identification of targetable abnormalities could aid diagnosis and treatment.

[0006] For these reasons, there is a need to provide improved methods, systems and tools for training artificial intelligence models and associated artificial neural networks. In particular, there is a need to provide simplified systems and methods associated with computer-implemented image processing artificial intelligence models that can be used to generate mask patterns for actinic irradiation. At least some of these objectives will be achieved by the following various embodiments. Summary of the invention

[0007] Described herein are systems and methods for training an AI model for generating mask patterns for light illuminating a sample associated with a microscope-based system.

[0008] A method for training an artificial intelligence model for microscope-based pattern light illumination is provided, the method comprising the following steps: obtaining a sample image captured by a microscope-based system, wherein the image has a target pattern corresponding to a region of interest; inputting the image into an input layer of an artificial intelligence model architecture, the artificial intelligence model architecture having multiple layers between the input layer and an output layer; outputting a proposed pattern corresponding to the region of interest using the output layer of the artificial intelligence model architecture; comparing the proposed pattern with the target pattern to calculate a loss between the proposed pattern and the target pattern; updating the weights of the multiple layers of the artificial intelligence model architecture based on the calculated loss; and repeating the steps of inputting, outputting, comparing, and updating until the calculated loss is below a loss threshold.

[0009] In one aspect, the method includes the step of preprocessing the image to normalize image pixel values ​​across the image.

[0010] In some aspects, multiple regions of interest are annotated on an image before being introduced into an AI model architecture.

[0011] In some aspects, the region of interest is marked with one or more probes configured to bind to the region of interest.

[0012] In some aspects, multiple regions of interest are labeled with probes.

[0013] In one aspect, each of the plurality of regions of interest is labeled with a different probe.

[0014] In some embodiments, the microscope-based system is configured to illuminate a region of interest.

[0015] In some aspects, the selected AI model architecture includes a network of AI models that communicate with each other.

[0016] In some aspects, the method includes the step of performing a segmentation of one or more features within an image captured by the microscope-based system by a first artificial intelligence model.

[0017] In some embodiments, the method includes training other artificial intelligence models within the network.

[0018] In some aspects, each of the one or more images obtained from the microscope-based system is captured with a different field of view.

[0019] A method for generating a mask for pattern illumination of a sample using a microscope-based system is provided, the method comprising: using an image capture system of the microscope-based system to capture an image of a target field of view (FOV) of the sample; inputting the image into a trained artificial intelligence model; using the trained artificial intelligence model to output a mask pattern corresponding to one or more regions of interest in the sample; and controlling an illumination component of the microscope-based system to illuminate the one or more regions of interest of the sample using the mask pattern.

[0020] In some aspects, the method includes normalizing image pixel values ​​of the image.

[0021] In some aspects, the step of inputting the image also includes inputting the standardized image pixel values ​​into the artificial intelligence model.

[0022] In one aspect, the mask pattern includes a plurality of regions of interest.

[0023] In another aspect, the trained artificial intelligence model is configured to segment one or more regions of interest from an input image.

[0024] In one aspect, the sample is a biological sample.

[0025] In another aspect, a microscope-based system is configured to capture images of multiple target FOVs of a sample.

[0026] In some aspects, the method includes combining the sample with one or more probes configured to bind one or more elements within the sample, and wherein the one or more probes are responsive to light from an illumination assembly.

[0027] A method for generating a mask pattern using an artificial intelligence model in a microscope-based system is provided, the method comprising: introducing one or more photoactivatable probes into a sample of the microscope-based system, the one or more photoactivatable probes being configured to bind to one or more regions of interest of the sample; capturing an image of the sample using an illumination system of the microscope-based system; inputting the image into an input layer of a trained artificial intelligence model; outputting a mask pattern corresponding to one or more regions of interest in the sample using the trained artificial intelligence model; and controlling an illumination component of the microscope-based system to illuminate the one or more regions of interest of the sample using the mask pattern, and the illumination component being configured to activate the one or more photoactivatable probes.

[0028] In one aspect, the mask pattern is based on the distribution of one or more photoactivatable probes within the sample.

[0029] In another aspect, the mask pattern includes a plurality of regions of interest.

[0030] In some aspects, the trained AI model has been trained using validated examples of regions of interest from training images.

[0031] In another aspect, the trained artificial intelligence model has been trained using microscope-based system inputs, and wherein the trained artificial intelligence model is configured to segment one or more regions of interest from pixel value data of a training image.

[0032] In some aspects, the sample is a biological sample.

[0033] In some aspects, the microscope-based system is configured to capture multiple images of a sample.

[0034] In another aspect, a microscope-based system captures each of the one or more images with a different field of view (FOV).

[0035] In some aspects, the one or more photoactivatable probes are responsive to light from an illumination assembly. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The novel features of the present invention are particularly described in the following claims. The features and advantages of the present invention will be better understood by referring to the following detailed description of exemplary embodiments utilizing the principles of the present invention and the accompanying drawings, in which:

[0037] Figure 1 An embodiment of a process associated with training an artificial intelligence model for microscopy-based pattern illumination according to the methods described herein is shown.

[0038] Figure 2 is a flowchart describing the method of training an AI model.

[0039] Figure 3-4 is a flow chart describing a method for patterning a sample using a microscope-based system.

[0040] Figure 5 is an embodiment of a microscope-based system for patterning illumination of a sample. DETAILED DESCRIPTION

[0041] Computer-implemented image processing typically involves one or more image processing algorithms used in a neural network to evaluate and transform inputs (e.g., images obtained from a microscope-based system). For example, an artificial intelligence (AI) model may employ one or more image processing algorithms to generate masks for use in photochemical irradiation of target objects or regions of interest within an image. Image processing of biological samples can provide efficient and specific identification of structures and associated functions at the intracellular level. However, training AI models is uniquely associated with AI functions and capabilities.

[0042] The AI ​​model may include one or more artificial neural networks (ANN) (e.g., neural networks) that operate by an activation algorithm configured to modify or transform data across one or more hidden layers. For example, a convolutional neural network (CNN) may be configured in a computer-based system having an input layer and an output layer, with one or more hidden layers between the input layer and the output layer. The hidden layers of a CNN may be associated with an activation algorithm that is associated with the desired operation and transformation of the data therethrough. For example, input data (e.g., an image obtained from a microscope-based system). The input image may then be processed by an input layer that may have different segments of the layer. Each layer segment may be associated with different weights or coefficients of a numerical representation of a segment of the input data. The weights and numerical representations of the data segment may then be processed by an activation algorithm in a hidden layer adjacent to the input layer. The activation algorithms of adjacent hidden layers may merge the weights and numerical representations of the data segments so that they may continue to be processed by the remaining layers of the CNN. Likewise, a weight may be associated with the convolution from the first hidden layer, and the weight may be incorporated into one or more subsequent activation algorithms at various hidden layers throughout the CNN. The last layer of the CNN may be an output layer, where final adjustments to the data may be performed and output values ​​associated with the input data or a fragment thereof may be produced.

[0043] In some embodiments, the AI ​​model relies on one or more algorithms to process the input data into output values. The algorithm may take into account data related to the input image and certain coefficients or weights and bias values ​​incorporated into the AI ​​model. Reducing the number of algorithms can reduce the amount of time required for the AI ​​model to process data. For example, each hidden layer may include multiple algorithms, each of which takes time to calculate (e.g., convolution). Therefore, preprocessing the image to standardize the image pixels can reduce the number of algorithms required to perform the processing and thus reduce the amount of processing time within the AI ​​model. Preprocessing of the image can also reduce the amount of errors or the need for additional verification within the AI ​​model. In some embodiments, the data is processed by the AI ​​model without verification. For example, a data set (e.g., a data set associated with an image) to be processed by the AI ​​model may not be verified, or may need to be verified before or after the AI ​​model is processed. In some embodiments, the data set to be processed by the AI ​​may be a small enough amount of data, and verification will not be required based on the size of the data set.

[0044] An example of an AI model may include a convolutional neural network associated with U-net. According to any of the methods described herein, U-net may be a convolutional neural network for image processing. According to any of the methods described herein, a microscope-based system may have a computer-based system that operates an AI model. The AI ​​model may process one or more image inputs or data inputs to the first layer of a U-net convolutional neural network (e.g., U-net). U-net may process images through a series of layers. The processing layers of U-net may be viewed as being in one or more stages or paths of image processing. For example, one stage may be an encoding (convolution) process of an image. The down-conversion path may include one or more pooling layers (e.g., a maximum pooling layer) between each down-conversion hidden layer. In some embodiments, the pooling layer processes data associated with image pixel values, and the pooling layer may sample the maximum image pixel value of the previous layer. These layers may alternate between pooling layers and down-conversion layers. Each down-conversion layer may have different pixel values ​​than the previous down-conversion layer. In some embodiments, the second stage of U-net may be a decoding (deconvolution) stage that begins between the input layer and the up-conversion layer. In some embodiments, the up-conversion layers are separated by at least one up-sampling layer. Each up-conversion layer may include additional data added during the up-sampling layer, the up-sampling layer being configured to provide additional information related to the image data input into the system.

[0045] In some embodiments, preprocessing an image to normalize image pixel values ​​can reduce the number of pooling layers and convolutional layers required for image processing. For example, an image input to the system that has not been preprocessed may require additional pooling layers and additional convolutional layers to compensate for the unnormalized image pixel values. Alternatively, if the image has been preprocessed, fewer pooling layers and convolutional layers may not be required to process the image.

[0046] In some embodiments, preprocessing the image to standardize the image pixel values ​​can reduce or completely eliminate the algorithms used when processing the image through the hidden layer of the AI ​​model. For example, an image with standardized image pixel values ​​can have a pixel value distribution recognizable by a trained AI model, allowing the AI ​​model to process the image by predicting a mask pattern associated with the image input described herein. In some embodiments, the AI ​​model can be fully trained to recognize aspects or attributes of the image data input, which can allow the AI ​​model to predict an appropriate mask used with a microscope-based system illumination system for photochemical irradiation of unmasked fragments or portions of the sample. When an image has standardized image pixel values ​​associated with the pixel intensity contained within the image, the arrangement and distribution of these pixels can allow the AI ​​model to identify the components and features of the image based on the distribution, intensity and other attributes of the pixel values ​​distributed throughout the input image. For example, a tissue sample with multiple cells distributed thereon can be input into a microscope-based system. Cells can be in different stages of mitotic activity, and morphological changes are observed, which may be related to a unique pattern or distribution of pixel intensities including cells undergoing mitosis. Therefore, an image of this sample can be captured and input into a trained AI model. The AI ​​model can identify inputs and inputs related to cell properties associated with mitosis. The AI ​​model can then predict a mask associated with cells in mitosis to quickly identify those cells in the sample that are undergoing mitosis. The predicted mask may be sufficient to highlight or identify cells undergoing mitosis and exclude other characteristic objects in the image.

[0047] In some embodiments, the AI ​​model undergoes one or more stages of training, such as Figure 1 For example, the first phase can be a training phase, in which the AI ​​model runs training data, evaluates output based on the training data, and adjusts variables (e.g., weights of a layer) based on the evaluation.

[0048] Still refer to Figure 1 The second phase may be a validation phase, which includes running validation data, evaluating output based on the validation data, and optionally adding one or more new variables within the AI ​​model based on the evaluation; or testing validation data within the AI ​​model. When evaluating validation data, the AI ​​model may be ready for sample data that is not test data or validation data.

[0049] like Figure 1 As shown, two phases are shown. In the training phase, the method can operate in a selectively openable loop, including running training data, evaluating the training data after running the training data through the AI ​​model, and adjusting variables according to the needs, requirements or expectations of the operator and / or for a specific purpose. The second verification phase (e.g., an optional verification phase) shows that the training phase continues and verification data can be introduced for processing in the verification phase. Specifically, the verification phase can include processing verification data, which can include evaluated training data from the training phase. The verification data can then be evaluated, and the verification phase can continue to adjust one or more variables of the AI.

[0050] The evaluated verification data can alternatively be introduced into the testing phase within the AI ​​model. The testing phase can continue with additional verification of the test phase data. After evaluating the test data, the AI ​​model can be identified as fully trained and ready to process data for light irradiation. In some embodiments, if the evaluation of the test data fails, the test data can be reintroduced into the verification phase. New variables or adjustments to AI-associated variables can be included in the verification phase. If the AI-associated variables are adjusted, the data can be reintroduced into the training phase for preliminary processing and evaluation. In some embodiments, the output values ​​processed by the AI ​​model may include data that is subject to evaluation. For example, the training phase may include output values ​​of data processed by the AI ​​model. The output value may refer to data submitted or evaluated at any stage according to any method described herein.

[0051] A method for training an AI model in the process of generating a mask pattern for photochemical irradiation used with a microscope-based system may include training the AI ​​model to apply a map to a sample introduced into a microscope-based system. The map may be coordinated with a sample (e.g., a general reference map of a sample within a microscope-based system). The reference map may be improved using image pixel values ​​of an image obtained from a sample by a microscope-based system. The map may have data related to the sample from which it originates, and the map may be input into an AI model (e.g., a neural network). The neural network may convolve the data via one or more hidden layers between the input layer and the output layer of the neural network. Based on the output, the AI ​​model may identify an area of ​​interest or an object to generate a mask. For example, the structure or features of a sample that has been incorporated into the map may generate an output value related to an area of ​​interest or an object. When an acceptable output is reached, the AI ​​model may use the characteristics of the map related to the acceptable output to generate a mask related to the accepted map. A mask related to the map may then be generated for use with a microscope-based system and a related irradiation system.

[0052] In some embodiments, a method for training an AI model may include comparing the output of the AI ​​model with an expected output and updating the weights in the AI ​​model based on the comparison (e.g., loss). For example, in some embodiments, new data (e.g., a sample image obtained by a microscope-based system) may be input into the AI ​​model. The AI ​​model may perform forward propagation to encode the image through a convolutional layer. The AI ​​model may then generate an output, which may be compared to the expected output to calculate a loss, which is a value representing the difference between the output and the expected output. In a specific embodiment, the input image may be a target field of view (FOV) of the sample, and the output may include a mask pattern corresponding to an area of ​​interest in the target field of view of the sample. The mask pattern may be compared to an expected mask pattern to calculate the loss.

[0053] In some embodiments, the method of training an AI model in the process of generating a mask pattern for photochemical irradiation may further include backward propagation from the output layer back to the input layer through the hidden layer algorithm of the AI ​​model. Backward propagation can update the weights in the AI ​​model for additional training. In a specific embodiment, the first output result can trigger the reverse or backward propagation of information processed by the system, so that the resulting backward propagation output can trigger or initiate the input of additional data from the sample (e.g., a graph). The new or additional data from the sample can be a regional map or a map that is coordinated with a smaller area or a more specific regional geometry than the initial map. Therefore, the new or additional data from the regional map is introduced into the input layer of the AI ​​model neural network and processed through one or more hidden layers until the output layer.

[0054] In some embodiments, the cycle of forward and backward propagation can continue until the output of a specific area of ​​interest, feature, or object of the sample is identifiable. For example, a regional map may continue to have a smaller surface area than a previous map of the same sample. In some embodiments of any of the methods described herein, subsequent regional maps may be rejected as invalid segmentation maps or regions of the initial map. For example, when the sample is related to a tissue or multiple cells, the initial map can be the entire sample, including different cell types, structures, and features. When the initial map is processed by the AI ​​model, an output is generated, which triggers back propagation, and additional or new regional maps are started and then processed. The new map may be rejected because it does not belong to the area of ​​interest, and on subsequent back propagation, a new regional map that does not include the previous map is established.

[0055] In some embodiments, training an AI model in the process of generating a mask pattern for light irradiation with a microscope-based system may include introducing a sample into a microscope-based system. The microscope-based system may have a computer-based system operating an AI model therein, which may generate a first image related to the sample. For example, the first image may be related to the entire sample of substantially all samples. The first image is then introduced into an AI model (e.g., a neural network), and the algorithm operating therein is processed from the input layer through one or more hidden layers in each neuron. The algorithm may include a fixed algorithm that incorporates values ​​related to input data (e.g., an image or image of a sample) and one or more attributes, such as weights or bias inputs associated with processing through hidden layers. Data may be processed to an output layer through a neural network, where one or more output categories may indicate associated values ​​associated with the output category. In some embodiments, the output may trigger or initiate a map of the sample to be reestablished. Any method described herein may include each layer having a bias and a weight, and the bias and weight may be dynamically adjusted according to a loss function or a cost function.

[0056] In some embodiments according to any method described herein, a subsequent image may be related to a previous image (e.g., a first image, a subsequent image, or a regional image). In some embodiments according to any implementation described herein, a subsequent image may be a reconstructed (de novo) image of a previous image or a naive image. For example, the first image of a sample may be processed by an AI model (e.g., a neural network), and the output may trigger the acquisition of a subsequent image. A subsequent image may be a second layer of the first image, so that the subsequent image and the first image are substantially the same. In some embodiments, a subsequent image may have different features, such as a different image area or region relative to the sample, for example, a subsequent image may substantially repeat the first image, or a subsequent image may have an offset from the position of the first image relative to the position of the sample. In any embodiment or method described herein, a subsequent image may be combinable with a previous image that provides multiple layers of information related to an image or sample. In other embodiments, a subsequent image may be processed by an AI model without considering the results of the previous image processing.

[0057] In some embodiments of any of the methods described herein, a graph may be generated in a sequence relative to a sample in a microscope-based system. For example, a graph may be generated in a sequential linear manner, wherein the microscope-based system has an imaging system configured to capture an image of a sample. The imaging system may capture multiple images related to the same sample. For example, the imaging system may scan an image. In some embodiments, data from scanned or captured image fragments may be processed periodically, wherein the image fragments may be associated with fragments or images captured at a certain point in time during image capture. For example, in the case where any of the methods described herein includes capturing an image with a microscope-based system and the image captured when the imaging system captures the image is less than the image of the entire fragment, the portion of the captured image may be input into the AI ​​model and processed. In some embodiments, the image fragments are preprocessed to achieve image pixel stabilization before being submitted to the AI ​​model. The image fragments may be processed according to any of the methods described herein.

[0058] In some embodiments, processing a segment of an image of the entire sample may include inputting the image segment into an input layer of the AI ​​model, and the image is data processed by the AI ​​model, including through one or more hidden layers between the input layer and the output layer. In some embodiments, the output associated with the processed image segment may trigger or initiate back propagation, wherein the AI ​​model further processes the image and may trigger additional image capture based on the data obtained from processing the image.

[0059] In some embodiments, multiple images of a sample may be captured by an imaging system, wherein each image captured may have a different field of view (FOV). Each FOV of a sample may include one or more regions of interest. For example, a first image with a first FOV may be captured, a second image with a second FOV may be captured, and a third image may be captured in a third FOV. In some embodiments, a first FOV may be used when capturing more than one image, thereby capturing more than one image with a FOV. A second set of images with a second FOV may be captured, and each image in the second set of images may be captured using the same second FOV. In some embodiments, an image is captured as part of a set of images, wherein each image in the set of images is captured using the same FOV for each image in the set of images. In some embodiments, an image set refers to a set of image fragments of a sample.

[0060] In some embodiments, any of the methods herein may include establishing parameters associated with AI model training or image processing. The method may include maintaining the parameters once they are established for subsequent imaging or different fields of view. In other embodiments, the parameters may be changed or adjusted to achieve the desired output. In some embodiments, the parameters do not change even for different FOVs. For example, according to any of the methods herein, the initial parameters for each FOV are not adjusted using a traditional or standard image processing algorithm before processing the image segmentation process.

[0061] Training an AI model for use with a microscope-based system may include developing an architecture of a neural network and determining some categories of an output layer. In some embodiments, the output layer includes one or more categories that are related to the number of objects expected in the image. In some embodiments, the number of output categories may be predetermined. For example, a user may establish one or more output categories before image processing according to any of the methods described herein. In some embodiments, the number of categories is provided or identified by an AI model. For example, an AI model according to any of the methods described herein may capture images from a sample using a microscope-based system. During image preprocessing, one or more segmentation methods may identify a certain initial number of suggested objects, and the number of suggested output categories may be related to the number of objects identified during preprocessing. For example, a sample with a certain type of tissue may include extracellular structures, different cell types with different morphologies, or distinguishable layers associated with sample thickness. Therefore, preprocessing of a sample image may have one or more outputs for each cell type, each type of extracellular structure, and each layer of tissue.

[0062] Training an AI model for use with a microscope-based system can include developing a set of known or validated input data (e.g., a database) that is introduced into the AI ​​model at an input layer so that each input from the database is processed by the AI ​​model neural network, and the resulting output can be an acceptable indication related to a known or expected output based on the database input. For example, images of multiple cells including epithelial cells with relevant cell connections (e.g., tight junctions and gap junctions) are selected from a database to be input into the AI ​​model for training related to epithelial cells (e.g., cell connections). The database image is input to the input layer of the neural network and processed via one or more hidden layers, where a weight is associated with each synaptic relationship from one layer to the next. The weights and input values ​​are applied to the activation algorithm and provide an output related to the identification of intercellular connections. In some embodiments, since the connexons between two cells are located in the interval or interval segment separating the cells, the output related to the identification of intercellular connections is further classified as a gap junction. Therefore, the output can provide details of the image that indicate the presence of one or more gap junctions and the demarcation of specific cell types (e.g., endothelial cells) and intercellular contacts.

[0063] In some embodiments, the database consists of images compiled from a general database. For example, training an AI model for use with a microscope-based system can include developing a database of acceptable or known images with exemplary structures, relationships, and objects within images. In some embodiments, objects within images can include known elements used to verify the output produced by AI model processing. In some embodiments, the database includes images captured by the microscope-based system. In some embodiments, the microscope-based system is in electrical communication with a network of microscope-based systems, such that the network of microscope-based systems compiles a database of images obtained by each microscope-based system in communication with the network.

[0064] In some embodiments, the database contains one or more images or input data obtained by the microscope-based system, which has been verified based on the details, features, and attributes of the one or more images. For example, a user can use a microscope-based system to obtain an image based on a sample introduced therein. The user can approve, confirm, or indicate the attributes of the image obtained for the purpose of classifying and storing the image in the database so that the image can be used in subsequent training of the AI ​​model. For example, when a user introduces a sample of epithelial tissue into a microscope-based system, the user can indicate that it is endothelial tissue, as well as additional information such as the source of the tissue, the known history or disease state of the tissue, and whether the tissue is known to have been previously exposed to any modifiers.

[0065] In some embodiments, the database may include different cell types, different tissue types, combinations of cells, tissues, extracellular material, intracellular material, genetic material, or derivatives thereof. In some embodiments, the database may include images of any sample obtained or suitable for use in combination with a microscope-based system, wherein the data (e.g., image) has been modified with a stain, probe, label, or other effector compound to convert the native tissue components into labeled, tagged, or marked tissue components. For example, a fluorescent probe may be introduced into the sample before the image is taken so that the resulting image (e.g., input data from the database) may have a visual indicator of the structure or target associated with the label used.

[0066] In some embodiments of any of the methods described herein, the sample loaded into the microscope-based system is a biological sample (e.g., a sample derived from a eukaryotic organism). In some embodiments of any of the methods described herein, the sample loaded into the microscope-based system is a biological sample (e.g., a sample derived from a prokaryotic organism). In some embodiments of any of the methods described herein, the sample loaded into the microscope-based system is a non-biological sample (e.g., a sample derived from a non-biological source). In some embodiments of any of the methods described herein, the sample loaded into the microscope-based system is any sample, sample fragment, sample portion, or combination of other samples related to those described herein.

[0067] According to any method described herein, training an AI model may include a combination and memory of AI model training based on a validated example for training. The AI ​​model may be trained using validated examples (e.g., examples from a database) for one or more elements that may be introduced into the AI ​​model from images captured by a microscope-based system. Since the AI ​​model is trained on validated examples of known areas of interest, and the AI ​​model is adjusted or developed to include modifications associated with the training of validated samples, the AI ​​model may be trained to identify relevant structures of the input image and adapt to the input image based on the identified structure. For example, in the case where the image captured by the microscope-based system includes cells, the AI ​​model may be trained to identify the presence of cells within the image, and the processing algorithm may be adjusted to improve the processing efficiency associated with the cell-related area of ​​interest. Another embodiment may be that in the case where the image captured by the microscope-based system includes a tissue layer, the AI ​​model may adjust the processing algorithm of one or more hidden layers to improve processing time when preparing to analyze a tissue sample. Another option may be to incorporate input data from a captured image into an AI model that includes training of all potential elements of the image, and the AI ​​system must process all potential outputs, rather than those associated with elements within the captured image.

[0068] In some embodiments, a microscope-based system may input images captured in the system into a first AI model that is trained for sample identification and has outputs related to the classification or identification of sample types. Then, a second AI model may be started based on the sample type identified in the output value of the first AI model. For example, the AI ​​model may identify input samples from a microscope-based system as tissue samples, and may exclude algorithms and processing related to irrelevant elements of the image. In other embodiments, the AI ​​model may be segmented so that the input layer receives images captured from a microscope-based system, and the AI ​​model fully processes the input data to identify the sample type. The rest of the same AI model may continue to be processed through hidden layers that are suitable for analyzing elements and attributes of image data related to the identified sample type. In some embodiments, the AI ​​model includes a network of sub-models trained for different classifications of sample types or attributes. Therefore, the processing of images by the AI ​​model network may include sequentially processing image input data by a network of sub-models based on output values ​​from previously processed models.

[0069] In some embodiments, any of the methods described herein can be incorporated into a data input (e.g., an image or image pixel) that has been previously modified by one or more compounds to highlight, mark, or identify a structure therein. In such an embodiment, the AI ​​model can be trained using data from a database related to one or more attributes required for AI processing. For example, in the case where a sample has been labeled with a stain that can be absorbed by cells or cell components, the data in the database used to train the AI ​​model to process new samples with similar components and manifestations can be similar stains. In this way, any of the methods described herein can be used after a sample or image (e.g., data input) has been modified.

[0070] In some embodiments, the methods described herein can be used to multiplex or multiple output recognition of more than one feature, object or area of ​​interest of input data (e.g., captured images). For example, cells to which stains are applied in advance are introduced into a microscope-based system. Images of stained cells are captured and processed by any method described herein. Depending on the attributes of the objects in the image (e.g., data input for processing), the AI ​​model can generate more than one output indicating multiple aspects of the image. For example, a stain can be associated with one output, a cell type can be associated with another output, an intercellular interaction can be associated with another output, a cell membrane morphology can be associated with another output, and a disease state or input data of a cell can be associated with another output.

[0071] In some embodiments, the number of outputs can be selected when determining the architecture of the neural network. In another embodiment, according to any of the methods described herein, the number of outputs can be selected after the image from the microscope-based system has been processed by the AI ​​model. For example, the AI ​​model can provide more than one output value, indicating a particular data attribute of the input. In such an embodiment, the number of outputs generated by the AI ​​processing may be greater than the number of outputs that the user considers desirable. Therefore, the user can select fewer than the number of available outputs.

[0072] In some embodiments, the neural network architecture is dynamic and modified based on a predetermined or selected number of outputs. For example, the number of layers (e.g., hidden layers) between the input layer and the output layer can be related to the number of outputs that the AI ​​model needs to produce. Therefore, by modifying the number of output possibilities, the number of hidden layers can be increased or decreased accordingly.

[0073] In some embodiments, input data (e.g., images obtained from a microscope-based system) may be preprocessed. Preprocessing of input data may be associated with one or more modifications or changes between acquiring data and inputting it into an AI model. For example, preprocessing may include preprocessing of the pixel values ​​of an image related to image pixel standardization before submitting the image as input to the ANN. For example, an image obtained from a microscope-based system may include an initially obtained 16-bit image. A 16-bit image may have a quantifiable value for each pixel in the image, ranging from the minimum pixel value of the image to the maximum pixel value of the image. In some embodiments, a preprocessing method for an image obtained from a microscope-based system may be associated with adjusting for unforeseen differences or aberrations in sample input. For example, differences in images obtained from a microscope-based system may be attributed to a failure in the equipment, components, or operation of a microscope-based system. Therefore, preprocessing of the image may result in standardization of image pixel values, which may compensate for differences and improve the processing efficiency of the neural network when generating a mask pattern from input data.

[0074] In some embodiments, preprocessing can reduce the amount of algorithmic processing of the neural network. Preprocessing the image obtained from the microscope-based system can standardize the image pixel values ​​of the entire image or the identified area of ​​interest. Standardized image pixel values ​​can provide input data configured to be input into the neural network. For example, each pixel of the image obtained from the microscope-based system can be quantized (e.g., related to applying a histogram to the obtained image), and the numerical representation of each pixel can be used as input data for the neural network. In the case where the image obtained from the microscope-based system has been preprocessed to standardize the image pixel values, fewer algorithms may be required within the hidden layers of the neural network to process the data to obtain a verified output. For example, the preprocessed image may require fewer propagation incidences to occur through the activation algorithm to achieve a valid output. In some embodiments, the preprocessed image can reduce the number of hidden layers required to process the input image through a trained AI model.

[0075] In some embodiments according to any of the methods described herein, the method may include an image normalization method process prior to any image segmentation processing or AI model inference (e.g., prediction) when generating a mask pattern for actinic irradiation. Image segmentation can refer to a computer-based operation to label individual pixels of one or more objects or regions of an image. For example, image semantic segmentation can be related to the classification of each pixel in an image relative to an object, target, or area of ​​interest in the image. In some embodiments, image semantic segmentation labels pixels as a particular category or set of categories that are associated with predetermined distinguishable features of the pixel and the object that includes the pixel.

[0076] In some embodiments of image segmentation described herein, one or more layers associated with a single image may be used to generate a segmentation map. Each of the one or more layers may be associated with a specific object or region of interest in the image, so that the layers may each represent the segmentation of the object in the image based on the pixel classification of the pixels including the object. For example, a single cell may have intracellular structures such as a nucleus, a Golgi body, mitochondria, vacuoles, proteasomes, transmembrane proteins, and a fibrous mesh network that is distributed everywhere. According to some methods described herein, a microscope-based system may capture a first image of a cell, including one or more cell structures within the image. The first image may then be input into a well-trained AI model, as described herein, and output values ​​may be generated based on different image pixel values ​​from the first image. A second image may be obtained and processed by a well-trained AI model, so that output values ​​may be generated based on different pixel values ​​from subsequent images. The output values ​​may be compared with each other to obtain the accuracy and consistency of the output values ​​associated with multiple images processed by the AI ​​model.

[0077] The image segmentation process may consist of one or more image processing algorithms. This may result in the image segmentation process only processing more specific cases. The input situation may be affected by many attributes, such as clarity, contrast, and brightness. According to any of the methods described herein, the image segmentation process may be able to evaluate many situations associated with manual operation, machine problems, or any other reasons. In some embodiments, before processing the image segmentation process, it may be necessary to perform an image pixel normalization process for the input image.

[0078] In some embodiments according to any method described herein, the system may include a functional component configured to obtain an image from a microscope-based system. For example, a microscope-based system as described in U.S. Patent Publication No. US20180367717 A 1 has an illumination component and an imaging component calibrated for a microscope. The camera is appropriately placed in the microscope to obtain an image of a sample (e.g., a biological sample introduced into the microscope). The microscope-based system may be configured to obtain an image with a quantifiable bit depth. For example, the bit depth of an image obtained by a microscope-based system may have a bit depth of 1 bit, 2 bits, 3 bits, 4 bits, 8 bits, 16 bits, 32 bits, 64 bits, 128 bits, or any bit depth in the range of greater than 0 to 128 bits. In some embodiments, the bit depth may be greater than 128 bits. In some embodiments, the bit depth of an image obtained from a microscope-based system may be a bit depth associated with the capabilities of an imaging component, an illumination component, a camera, or other element of a microscope-based system.

[0079] In some embodiments according to any of the methods described herein, a mask can be generated based on the output or result of an image processing algorithm when an AI model or machine learning evaluates an input according to one or more layers of a deep learning neural network. For example, a mask is generated based on an input of an image processed by one or more layers of an AI model (e.g., a neural network). The image can be processed by a developed neural network architecture, and the mask can be generated based on the image processing.

[0080] In some embodiments according to any of the methods described herein, a mask can be predicted by an AI model that has been trained for input related data. The AI ​​model can be trained using similar or related data inputs of known values ​​that have expected outputs associated with mask generation. After the AI ​​model has been fully trained and the bias or weight values ​​have been properly calibrated, new data (e.g., images captured from a microscope-based system) can be introduced into an AI model that is capable of predicting outputs based on the initially submitted data. The predicted output can then be used to develop a mask that is configured to be applied to the data (e.g., images captured from a microscope-based system). In some embodiments, the mask is associated with an area of ​​interest of the image, and the mask covers non-target areas or exposes only the area of ​​interest. The light source can then emit light onto the mask so that the unmasked portion (e.g., the area of ​​interest) is illuminated.

[0081] In some embodiments according to any of the methods described herein, the AI ​​model architecture may include multiple layers. Each of the multiple layers may perform a different function related to image processing. Some embodiments of different layers within the AI ​​model may be pooling layers, convolutional layers, fully connected layers, classification layers, normalization layers. The configuration of the layers within the overall neural network architecture may be developed based on the functions and capabilities required by the AI ​​model and the microscope-based system.

[0082] In some embodiments of any of the methods described herein, a U-net neural network may be a segmentation mechanism for processing data (e.g., images) obtained from a microscope-based system. U-net processing may include the step of developing a preprocessed image. In some embodiments, developing a preprocessed image may include preprocessing the image to standardize the image pixel values ​​from a first image obtained by a microscope-based system to become a preprocessed image with standardized image pixel values. For example, the standardized distribution or mean of the standardized image pixel values ​​on all image pixel values ​​of the image may be 0. In some embodiments, the preprocessed image is an 8-bit image, while the image originally obtained is a 16-bit image. In some embodiments, the preprocessing method may be based on input data (e.g., an image to be processed by an AI model). For example, preprocessing may be at least related to data intensity, regularization, resampling or standardization of the data to be introduced into the U-net AI model and other features of the preprocessing data. The method may also include designing or selecting a neural network architecture (e.g., an architecture of a U-net). For example, the patch size of the input (e.g., an image obtained from a microscope-based system), the number of pooling layers, the organization and number of convolutional layers. The method may also include steps of defining a loss function, a batch size, a learning rate, and other parameter functions.

[0083] In some embodiments according to any of the methods described herein, hardware considerations may aid in the design and training of an AI model. For example, the capacity of processing units and storage may be considered when developing an AI model architecture and neural network selection. The method may also include a process for validating training to analyze the AI ​​model. For example, after an AI model has been built and training has been performed, according to any of the methods described herein, the output and operation may be evaluated to ensure that the AI ​​model is optimized.

[0084] In some embodiments, in order for the AI ​​model to accurately and efficiently perform image segmentation, it may be necessary to preprocess the images captured by the microscope-based system. Optimizing the image segmentation process can improve the image segmentation capabilities. Described herein may be methods for improving image processing and preprocessing. For example, compared to brightness or contrast image analysis, the methods described herein can optimize and provide improved image processing and preprocessing.

[0085] The method for processing an image for photochemical irradiation obtained from a microscope-based system may include first obtaining an image using a microscope or a technique including using a microscope-based system. The image obtained is used to determine the location of the image or the features of a certain area on the image that needs further processing within the entire image. In some embodiments and according to any method described herein, the identification of these areas can be accomplished by using one or more image segmentation techniques (e.g., image semantic segmentation, instance segmentation, or object segmentation) using deep learning. The mask can be generated based on the output of deep learning (e.g., an AI model). The generated mask can be applied to the image, and light can pass through the mask to irradiate the target area exposed by the mask. The light can irradiate the area within the mask. For example, the light can have a wavelength sufficient to irradiate the object, structure, or feature of the image exposed by the mask.

[0086] In some embodiments according to any method described herein, the AI ​​model is configured to process an image that has been standardized. For example, a method for standardizing the pixel values ​​of an image obtained by a microscope-based system may include first capturing an image using a microscope-based system. Then, the process may include applying a histogram to the image. The pixel values ​​may then be analyzed. For example, each pixel may have a value associated with a color range within the pixel. The pixel values ​​may be quantized, and the range of pixel values ​​may have a minimum pixel value and a maximum pixel value. Then a conversion method may be selected or established. The selection of a conversion method may be based on one or more characteristics of a pixel value data set. For example, a conversion method may be based on the slope and peak of a trend defined by a range of values ​​between a minimum value and a maximum value. In some embodiments, the conversion method may be a linear conversion method (e.g., a linear conversion, a linearity conversion, a multi-order linear conversion, a Sigmoid curve fitting conversion). In some embodiments, the conversion method may be a nonlinear conversion method (e.g., a modified Sigmoid curve fitting conversion). In some embodiments, the modified Sigmoid curve fitting conversion is a hybrid conversion method of a nonlinear conversion and a linear conversion. In some embodiments, a method for normalizing pixel values ​​of an image from a microscope-based system can result in an image originally obtained from the microscope-based system being converted from a first bit depth to a second bit depth by a selected conversion method. For example, the image originally obtained from the microscope-based system can be a 16-bit image. The selected conversion method can be applied to the image originally obtained from the microscope-based system, thereby converting the 16-bit initial image into an 8-bit pre-processed image having normalized pixel values ​​produced by the selected conversion method.

[0087] In some embodiments, the preprocessing methods described herein are configured to optimize images for processing using well-trained AI models. For example, the methods described herein can optimize the converted images obtained for a specific image or a specific sample or a specific area (e.g., an area of ​​interest) of an image. In some embodiments, the image initially obtained can be a whole image obtained by a microscope-based system. In some embodiments, the image initially obtained can be a whole image that is smaller than the image initially obtained from the microscope-based system. For example, a specific area of ​​an image can be selected, and then the specific area can be evaluated, and the range of pixel values ​​in the specific area can be determined. Then, the selected conversion method can be performed on the specific area based on the range of pixel values ​​in the specific area between the minimum pixel value in the specific area and the maximum pixel value in the specific area.

[0088] According to some embodiments described herein, a microscope-based system initially captures a 16-bit image. After a histogram is drawn based on the image, the pixel values ​​obtained can be distributed within a range of values ​​between a minimum pixel value and a maximum pixel value. The distribution of pixel values ​​other than the minimum pixel value and the maximum pixel value can be less than half, less than one-third, less than one-quarter, or less than a portion of the maximum image pixel value provided by the 16-bit image initially obtained. The distribution of pixel values ​​of the initially obtained image may be due to the intensity level of the signal excited by the fluorescence. In some embodiments, the intensity level of the fluorescence excitation can be high or low, and the intensity level can affect the distribution of pixel values ​​within the pixel value range of the initially obtained image. In some embodiments, the initially obtained image captured by the microscope-based system can be observed as bright or dark, or a similar description of the image being between bright and dark.

[0089] In some embodiments, additional processing using a well-trained AI model may identify target biological activities or elements that are observable within the image. For example, an image may contain multiple cells that may interact with each other. After the image is pre-processed and the image pixel values ​​are normalized using the methods described herein, the AI ​​model can process the image to highlight or indicate interactions between cells. In some embodiments, cell-to-cell interactions may include cell-to-cell connections (e.g., gap junctions or tight junctions); in some embodiments, cell-to-cell interactions may include the transmission of cytokines between cells observed in the image; in some embodiments, the AI ​​model may process images to highlight cellular activities, such as apoptosis or cell lysis; in some embodiments, the AI ​​model may process images to highlight phagocytic acidity of bone marrow cells; in some embodiments, the image initially acquired includes one or more neurons, and the AI ​​model highlights the transmission of messages (e.g., neurotransmission) through and between one or more neurons; in some embodiments, the AI ​​model may process images to highlight individual cells transporting or endocytosing extracellular substances into cells; in some embodiments, the AI ​​model may process images to highlight the presence of extracellular or intracellular concentrations or therapeutic compounds; in some embodiments, the AI ​​model may process images to highlight intracellular activities, such as intracellular metabolism, respiration, cell signaling, protein synthesis, collagen matrix, and other target cell activities or functions.

[0090] In some embodiments, any of the methods described herein may include or relate to semantic segmentation. Validated or known examples of images, component elements, or areas of interest associated with known or expected attributes of images captured by a microscope-based system may be used to train an AI model. For example, exemplary images are first labeled with known positions, sizes, dimensions, proximity to another known complementary element, and the like. These exemplary images can then be introduced into the AI ​​model for training. The AI ​​model can also be configured or calibrated through user interaction so that areas of interest of images captured by a microscope-based system can be identified on the captured images, and those areas containing areas of interest in the captured images can be processed by the AI ​​model according to any of the methods described herein.

[0091] In some embodiments of the methods and systems described herein, an AI model can be trained to detect and segment objects in sample images obtained from a microscope-based system. For example, cell morphology associated with cell surface area or specific objects within a cell (e.g., a nucleus) can indicate the stage of a classification category of a disease type.

[0092] In one embodiment, reference Figure 2 The method 200 of training an AI model to output a mask pattern corresponding to an area of ​​interest of a sample of a microscope-based system may include obtaining one or more images of the sample using the microscope-based system. Obtaining the image may include obtaining the image using an image capture system of the microscope-based system. The image may correspond to one or more target fields of view (FOV) of the sample. The image may have a target mask pattern corresponding to one or more areas of interest of the sample in the target FOV.

[0093] At operation 202, method 200 may include inputting (one or more) images having a target mask pattern into an artificial intelligence (AI) model. At operation 204, the method may include encoding (one or more) images in a forward pass through a convolutional layer of the AI ​​model.

[0094] At operation 206, the method may include outputting a mask pattern using the AI ​​model corresponding to the region of interest of the sample. At operation 208, the method may further include comparing the output mask pattern with the target mask pattern to calculate a loss. The loss may include, for example, a difference between the output mask pattern and the target mask pattern.

[0095] At operation 210, the method may further include using the loss in back-propagation through the convolutional layers of the AI ​​model to update weights in the AI ​​model.

[0096] The AI ​​model can be further trained by repeating some or all of the above operations. For example, after the weights are updated in operation 210, the method can be repeated using additional training images. Each iteration of forward and backward propagation through the convolutional layers of the AI ​​model produces updated weights, thereby further improving the output mask pattern. Ultimately, the AI ​​model is fully trained when the output mask pattern corresponds to the target mask pattern, or until the loss calculated between the output mask pattern and the target mask pattern is below a loss threshold (e.g., less than 1% loss, less than 5% loss, less than 10% loss, etc.).

[0097] Figure 3 A flow chart 300 is shown that describes a method for generating a mask for pattern illumination of a sample using a microscope-based system. At operation 302, the method may include obtaining one or more images of the sample using the microscope-based system. Obtaining the images may include obtaining the images using an image capture system of the microscope-based system. The images may correspond to one or more target fields of view (FOV) of the sample. The images may have a target mask pattern corresponding to one or more regions of interest of the sample in the target FOV.

[0098] At operation 304, the method may include inputting one or more images into a trained artificial intelligence (AI) model, such as the one above. Figure 2 or any other AI model described in this article.

[0099] At operation 306 , the method may further include outputting a mask pattern corresponding to a region of interest of a target FOV of the sample using the trained AI model.

[0100] Next, at operation 308 , the method may include controlling a pattern illumination system of the microscope-based system to illuminate a region of interest of the sample using the output mask pattern.

[0101] Figure 4 A flow chart 400 is shown describing another method of generating a mask for pattern illumination of a sample using a microscope-based system. At operation 402, one or more photoactivatable probes can be introduced to a sample of a microscope-based system.

[0102] At operation 404, the method may include obtaining one or more images of the sample using a microscope-based system. Obtaining the images may include obtaining the images using an image capture system of the microscope-based system. The images may correspond to one or more target fields of view (FOVs) of the sample. The images may have a target mask pattern corresponding to one or more regions of interest of the sample in the target FOV.

[0103] At operation 406, the method may include inputting one or more images into a trained artificial intelligence (AI) model, such as the one above. Figure 2 or any other AI model described in this article.

[0104] At operation 408 , the method may further include outputting a mask pattern corresponding to a region of interest of a target FOV of the sample using the trained AI model.

[0105] Next, at operation 410 , the method may include controlling a pattern illumination system of the microscope-based system to illuminate a region of interest of the sample using the output mask pattern to activate one or more photo-activatable probes.

[0106] Figure 5 An embodiment of a microscope-based system is shown, which can be configured to use any AI-generated mask for pattern irradiation described herein. Additional details of the microscope system can be found in U.S. Patent No. US11,265,449, which is incorporated herein by reference in its entirety. The microscope-based system of this embodiment includes a microscope 10, an imaging component 12, an irradiation component 11, and a processing module 13a. Microscope 10 includes an objective lens 102 and a stage 101. Stage 101 is configured to load sample S. Imaging component 12 may include a (controllable) camera 121, an imaging light source, and a focusing device 123. Processing module 13a may include one or more processors configured to perform the processes and methods described herein. In some embodiments, processing module 13a may include one or more trained AI models configured to generate a mask pattern corresponding to one or more areas of interest in a sample.

[0107] The stage 101 can be moved to provide different views of the sample S. The sample S can include, for example, a fluorescent sample, a reflective sample, or a sample that can be marked by light projected from an imaging subsystem. For example, the sample mark can be bleached, activated, physically damaged, or chemically transformed. The mark can be analyzed by the imaging subsystem, and the position of the mark can be indicated by the result of the projected light.

[0108] In some embodiments, as described in U.S. Pat. No. US11,265,449, the image obtained by the camera 121 can be processed in the processing subsystem 13a to identify regions of interest in the sample. For example, when the sample contains cells, specific subcellular regions of interest can be identified based on their morphology. In some embodiments, the regions of interest identified from the image by the processing module can then be selectively illuminated with different light sources. For example, molecular photobleaching at certain subcellular regions, photoactivation of fluorophores at defined locations, optogenetics, phototriggered release of reactive oxygen species within specified organelles, or photoinduced labeling of biomolecules in defined structural features of cells all require patterned irradiation. The AI ​​model described herein can create or generate patterns or masks for patterned irradiation. Therefore, Figure 5 The embodiment has a pattern illumination component 11, which projects light corresponding to the pattern or mask generated by the AI ​​onto the sample S through the lens 3, the mirror 4, the lens 6 and the mirror 8. In some embodiments, the pattern illumination component 11 illuminates the pattern of the area of ​​interest in the sample S with laser light by moving the mirror in the pattern illumination component 11.

[0109] The microscope, stage, imaging subsystem, and / or processing subsystem may include one or more processors configured to control and coordinate the operation of the entire system described and illustrated herein. In some embodiments, a single processor may control the operation of the entire system. In other embodiments, each subsystem may include one or more processors. The system may also include hardware such as memory to store, capture, and process data captured by the system. Optionally, the memory may be accessed remotely, such as through the cloud. In some embodiments, the methods or techniques described herein may be computer-implemented methods. For example, the system disclosed herein may include a non-temporary computing device readable medium having instructions stored thereon, wherein the instructions may be executed by one or more processors to cause the computing device to perform any method described herein.

[0110] The following examples included herein illustrate, by way of illustration and not limitation, specific embodiments in which the claimed subject matter may be practiced.

[0111] Example 1: A tissue sample is loaded into a microscope-based system. The tissue sample includes cells and structures that can be seen in an image of the sample obtained by an image capture system of the microscope. The captured image is preprocessed, and the image pixel values ​​of the captured image are converted into standard values ​​relative to each other. The image pixel values ​​of the preprocessed image are then input into an already trained AI model. The AI ​​model processes input data regarding the image pixel values ​​of the preprocessed image. Output values ​​related to the identification of image pixel values ​​associated with a region of interest in the sample are generated. The output values ​​are used to generate a mask that highlights the region of interest for irradiating the sample in an exposure area of ​​the mask.

[0112] Example 2: A cell sample from the immune system is cultured with bacteria. The cultured cells are introduced into a microscope-based system, and initial images showing a variety of cells at various stages of phagocytosis are obtained. The images are preprocessed to standardize the image pixel values, and the preprocessed images are then introduced into a well-trained AI model. The AI ​​model can be trained using validated examples of similar cells and similar phagocytic activity. Based on the training of the AI ​​model, the neural network of the AI ​​model processes the preprocessed images to provide output values ​​associated with the area of ​​interest within the initial image. A mask is generated using the output values, and the mask exposes the area of ​​interest (e.g., contact between immune cells and bacteria). The microscope-based system is then used to generate light emitted from the illumination system of the microscope. The illumination system is configured to emphasize the area of ​​interest within the mask, and the image is converted to emphasize areas and cells that are undergoing stages of phagocytosis.

[0113] Example 3: Patient-derived samples are introduced into a microscope-based system having multiple cells distributed throughout the image. The microscope-based system is configured to capture images of intracellular components, including components of the ubiquitin proteasome system (UPS). The microscope-based system captures multiple images of the sample and performs preprocessing to standardize the image pixel values. Each image is used as input data, where the image pixel values ​​are processed by an AI model that has been trained using a validated example of proteasome activity. Output values ​​related to those pixels within the image that contain or present proteasome or UPS activity are generated. The output values ​​of each image captured from the same sample are evaluated, and a mask is generated based on the reconciliation of all images processed by the AI ​​model. The mask is applied to the sample, and the UPS components are identified using an illumination technique. The number of UPS components and the observed activity can be used for diagnostic purposes related to UPS activity.

[0114] Example 4: A patient does not respond to a targeted treatment regimen. A sample derived from a patient is introduced into a microscope-based system, and an image is captured therein. The image can be preprocessed and the image pixel values ​​can be normalized. The preprocessed image is then introduced into an AI model that is trained on a validated paradigm for structural targets of the treatment regimen. Output values ​​related to the structural targets are generated, and a mask can be generated to emphasize those targets for irradiation. The mask can provide a limited or absent incidence of the structural targets. Thus, false positives can be considered to be related to limited or absent structural targets of the treatment.

[0115] Example 5: A sample is introduced into a microscope-based system, and multiple images of the sample are obtained by the system. The images obtained may include images of different fields of view of the same sample. These images can be pre-processed and introduced into a well-trained AI model that has been trained based on a proven paradigm of cell-to-cell contacts (e.g., connections between two or more cells). Output values ​​are generated, and masks are developed using the generated output values ​​to emphasize areas of the image that include cell-to-cell contacts. These emphasized areas can then be illuminated within the microscope-based system so that the cell-to-cell contacts are illuminated by light of different colors or light of different intensities from the microscope-based system.

[0116] Example 6: A microscope-based system scans a biological sample and captures an image by continuously scanning the image. As the image is scanned, the pixel values ​​from the scan can be instantly introduced into an AI model as described herein with respect to the progress of the image scan. The pixel values ​​are processed by a neural network and output values ​​are generated. The AI ​​model can be trained using validated examples of regions of interest associated with the sample, and when the AI ​​model processes the pixel values, output values ​​associated with the presence of one or more regions of interest are generated. A mask can be generated based on the output values ​​associated with the regions of interest, and when the microscope-based system scans the image, the resulting processed image can be segmented to emphasize the regions of interest.

[0117] Example 7: Introducing a sample into a microscope-based system. Capture an image of the sample and preprocess to standardize the image pixel values. Introduce the standardized image pixel values ​​into the AI ​​model and generate output values. The AI ​​model can be trained using verified examples related to the area of ​​interest found in the captured image. After the first processing within the neural network, additional output values ​​related to the same image may be required. The AI ​​model can trigger subsequent evaluations of subsequent images or pixel value input data, and the image can continue to be additionally processed to generate additional output values ​​for comprehensive identification of the area of ​​interest. A mask can then be generated based on the output values. The mask can then be applied to the image or sample and illuminated by one or more beams of light to visually identify the area of ​​interest within the sample.

[0118] Example 8: A fluid sample from the knee joint of a patient with rheumatoid arthritis (RA) is loaded into a microscope-based system. The microscope-based system captures an initial image of the sample with sufficient resolution to roughly identify cells and related activities. A computer-based system within the microscope-based system is started to preprocess the image and standardize the pixel values ​​of the entire image. The preprocessed image is then input into a well-trained AI model operating on the computer-based system. The input image is subjected to image semantic segmentation to perform pixel classification on various cell types. The AI ​​model can be trained using validated examples of cells and cell activities in rheumatoid arthritis fluid. Masks are generated to identify and segment each different cell type. Thus, it can be irradiated and observed that the segmented cell types include white blood cells and high concentrations of metacarpal phalangeal joint-specific fibroblast-like synoviocytes, which are usually indicative of metacarpal phalangeal joint RA. Treatment can be adjusted based on metacarpal phalangeal joint-specific FLS from the knee joint sample to improve response.

[0119] Example 9: Patients with non-small cell lung cancer and DNA mutations in the gene encoding PDL-1 were previously evaluated by next generation sequencing (NGS) to identify the DNA mutation. Tissue samples are loaded onto a microscope-based system, and images are obtained showing details of malignant cells. The images are preprocessed according to the methods described herein to standardize the image pixel values ​​from the initially acquired images. The preprocessed images are then input into an AI artificial neural network within a computer-based system of the microscope-based system. The images are subjected to image semantic segmentation to identify malignant cells, and the AI ​​model further processes the images through multiple layers of semantic segmentation to segment and detect surface markers on the cell surface. Therefore, the AI ​​model generates a mask based on the output values ​​to emphasize the pixels of the cell surface material. Neural network processing identifies functional PDL-1 kinases on the cell surface. In addition, the AI ​​model also identifies that immune cells contact the cell surface through PD 1-PDL 1 binding interactions. The output layer emphasizes the area of ​​interest based on the training developed for the AI ​​model to identify immune checkpoint activity.

[0120] Example 10: An unknown sample is introduced into a microscope-based system and an image is captured. The image pixel values ​​of the image are standardized. The standardized image is input into a well-trained AI model that has been trained on multiple regions of interest, including the structure, function, and activity of cells and components in the image. The AI ​​model processes the image and presents output values ​​associated with the regions of interest. The user then selects one or more identified regions of interest from the generated output values, which are used to generate a mask associated with the unknown sample. The mask is applied to the sample, and the regions of interest are displayed and emphasized. Based on the properties of the illuminated regions of interest, a decision about the sample can be made.

[0121] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are considered to be part of the subject matter disclosed herein and can be used to achieve the benefits described herein.

[0122] The process parameters and step sequences described and / or illustrated herein are provided as examples only and may be changed as desired. For example, although the steps illustrated and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated herein may also omit one or more steps described or illustrated herein, or include additional steps in addition to the disclosed steps.

[0123] Any of the methods and / or systems described herein can be associated with illuminating an image, picture, photograph, etc. For example, light irradiation can refer to an irradiation unit that provides irradiation to an image of a sample so that a target, region of interest, and / or area of ​​the image is selectively irradiated. In some embodiments, the region of interest is targeted, and light irradiation of the desired area can be achieved by generating a mask or pattern (e.g., generated by a trained AI platform), which can be applied to the sample or image to selectively target and irradiate one or more regions of interest.

[0124] In some embodiments described herein, one or more methods or systems may be related to the use, interaction, training and / or development of artificial intelligence (AI) systems. AI may refer to machine learning, deep learning, neural networks, etc. AI may refer to a combination of one or more artificial intelligence processing systems. For example, an AI system may be related to artificial neural networks, such as convolutional neural networks, modular neural networks, feedforward neural networks-artificial neurons, radial basis function neural networks, Kohonen self-organizing neural networks, recurrent neural networks (RNN), long / short-term memory, etc. ANN can be described as being related to the physiological activity of neurons. For example, a natural neuron can receive information as input at a first location or end of the neuron (e.g., a neuron end having dendrites that are configured to receive information (e.g., neurotransmitters) and process the information along the body of the neuron (e.g., an axon) toward the axon end, where the neuron can produce transformed information related to the metabolism, transmission, or propagation of the chemical information received at the dendrites). The transformed information can then be released at or through the axon end. In a similar conceptual manner, an artificial neural network (e.g., ANN) can receive an input of certain data (e.g., an image obtained from a microscope-based system), which is received into the ANN and processed by one or more intermediate algorithms. The input data can be transformed by one or more intermediate algorithms so that the transformed data obtained by the ANN has one or more different but related characteristics or attributes to the input data.

[0125] According to any of the methods described herein, the layers of the AI ​​model may include one or more layers having different characteristics or properties that are related to the configuration for transforming the data processed therethrough. For example, the layer may be a convolutional layer, a maximum pooling layer, an average pooling layer, a batch normalization layer, an activation function layer. In some embodiments, the layer may include multiple layers. For example, the layer may include a combination of at least two or more layers, as described herein.

[0126] In some embodiments, according to any of the methods described herein, a data set to be processed by the AI ​​model or a data set that has been processed by the AI ​​model can be verified for model output. For example, verification of the data set can include calculating the difference between the model output and the annotated results of the AI ​​processing. In some embodiments, the training of the AI ​​model includes an evaluation process after the AI ​​model has run or processed the training data. For example, an operator (e.g., a user) can evaluate the results of the processed training data run through the AI ​​model. For example, the training data can be introduced into the AI ​​model as described herein. The training data can be processed through one or more layers of the AI ​​model, and the resulting data can be evaluated. Based on the evaluation data, the variables of the AI ​​model algorithm (e.g., one or more algorithms running within the AI) can be adjusted according to the evaluation of the processed training data.

[0127] Any method described herein may include first, obtaining images captured by a microscope-based system, each image having a corresponding target pattern annotated according to the region of interest, so that we have two images. The first image is the original image, and the second image is a mask (e.g., target pattern) annotated by an expert. The step of introducing the image into the AI ​​model architecture to obtain an output and evaluating the output of the image according to the corresponding target pattern is the evaluation step of the training loop shown in the diagram. The stage of determining the weights (e.g., variables) of the AI ​​model architecture is the variable adjustment step of the training loop shown in the diagram.

[0128] Any of the methods described herein (including user interfaces) may be implemented as software, hardware, or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions executable by a processor (e.g., a computer, tablet, smart phone, etc.), which when executed by the processor causes the processor to control the execution of any steps, including but not limited to: display, communication with a user, analysis, modification of parameters (including time, frequency, intensity, etc.), determination, alarm, etc. For example, any of the methods described herein may be performed at least in part by a device including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the process of the method.

[0129] Although various embodiments have been described and / or illustrated herein in the context of a fully functional computing system, one or more of these embodiments may be distributed as a program product in various forms, regardless of the specific type of computer-readable medium used to actually perform the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include scripts, batches, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the embodiments disclosed herein.

[0130] As described herein, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions (such as those contained within the modules described herein). In their most basic configuration, these computing devices can each include at least one storage device and at least one physical processor.

[0131] As used herein, the term "memory" or "storage device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one embodiment, a storage device can store, load and / or maintain one or more modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), optical drive, cache, one or more variations or combinations of the same, or any other suitable storage medium.

[0132] In addition, as used herein, the term "processor" or "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one embodiment, the physical processor can access and / or modify one or more modules stored in the above-mentioned storage device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), portions of one or more of the above-mentioned physical processors, variations or combinations of one or more of the above-mentioned physical processors, or any other suitable physical processor.

[0133] Those of ordinary skill in the art will recognize that any process or method disclosed herein may be modified in many ways. The process parameters and order of steps described and / or illustrated herein are provided as examples only and may be changed as desired. For example, although the steps described and / or illustrated herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order shown or discussed.

[0134] The various exemplary methods described and / or shown herein can also omit one or more steps described or shown herein, or include extra steps except the step disclosed. In addition, the step of any method disclosed herein can be combined with any one or more steps of any other method disclosed herein.

[0135] A processor as described herein may be configured to perform one or more steps of any method disclosed herein. Alternatively or in combination, a processor may be configured to combine one or more steps of one or more methods disclosed herein.

[0136] Although various illustrative embodiments are described above, any of a variety of changes may be made to the various embodiments without departing from the scope of the invention described by the claims. For example, in alternative embodiments, the order in which the various described method steps are performed may often be changed, and in other alternative embodiments, one or more method steps may be skipped. Optional features of various device and system embodiments may be included in some embodiments and not included in other embodiments. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention set forth in the claims.

Claims

1. A method for training an artificial intelligence model for microscope-based patterned light illumination, the method comprising the following steps: acquiring an image of the sample captured by the microscope-based system, wherein the image has a target pattern corresponding to the region of interest; Inputting the image into an input layer of an artificial intelligence model architecture, the artificial intelligence model architecture having a plurality of layers between the input layer and an output layer; Outputting a suggested pattern corresponding to the region of interest using the output layer of the artificial intelligence model architecture; comparing the proposed pattern with the target pattern to calculate a loss between the proposed pattern and the target pattern; Update the weights in the multiple layers of the artificial intelligence model architecture according to the calculated loss; as well as The input, output, comparison and update steps are repeated until the calculated loss is lower than the loss threshold.

2. The method of claim 1 further comprising the step of preprocessing the image to normalize image pixel values ​​across the image.

3. The method of claim 1, wherein a plurality of regions of interest are annotated on the image before the image is introduced into the artificial intelligence model architecture. The method of claim 1 , wherein the region of interest is marked using one or more probes configured to bind to the region of interest. The method of claim 1 , wherein a plurality of regions of interest are marked using probes. The method of claim 1 , wherein each of the plurality of regions of interest is labeled using a different probe.

7. The method of claim 1, wherein the microscope-based system is configured to illuminate the region of interest.

8. The method of claim 1, wherein the selected artificial intelligence model comprises a network of artificial intelligence models that communicate with each other.

9. The method of claim 8, further comprising the step of segmenting, by a first artificial intelligence model, one or more features within the image captured by the microscope-based system.

10. The method of claim 8, further comprising training other artificial intelligence models within the network.

11. The method of claim 1 , wherein each of the one or more images acquired by the microscope-based system is captured with a different field of view.

12. A method of generating a mask for patterned illumination of a sample using a microscope-based system, the method comprising: capturing an image of a target field of view (FOV) of the sample using an image capture system of the microscope-based system; Inputting the image into a trained artificial intelligence model according to the method of claim 1; outputting a mask pattern corresponding to one or more regions of interest in the sample using the trained artificial intelligence model; as well as An illumination component of the microscope-based system is controlled to illuminate the one or more regions of interest of the sample using the mask pattern.

13. The method of claim 12, further comprising normalizing image pixel values ​​of the image.

14. The method of claim 12, wherein inputting the image further comprises inputting the standardized image pixel values ​​into the trained artificial intelligence model. The method of claim 12 , wherein the mask pattern comprises a plurality of regions of interest.

16. The method of claim 12, wherein the trained artificial intelligence model is configured to segment the one or more regions of interest from the input image.

17. The method of claim 12, wherein the sample is a biological sample.

18. The method of claim 12, wherein the microscope-based system is configured to capture images of multiple target FOVs of the sample.

19. The method of claim 12, further comprising combining the sample with one or more probes configured to bind one or more elements within the sample, and wherein the one or more probes are responsive to light from the illumination system.

20. A method for generating a mask pattern using an artificial intelligence model in a microscope-based system, the method comprising: introducing one or more photoactivatable probes into a sample of a microscope-based system, the one or more photoactivatable probes configured to bind to one or more regions of interest of the sample; capturing an image of the sample using an image capture system of the microscope-based system; Inputting the image into an input layer of a trained artificial intelligence model according to claim 1; outputting a mask pattern corresponding to one or more regions of interest in the sample using the trained artificial intelligence model; as well as An illumination component of the microscope-based system is controlled to illuminate the one or more regions of interest of the sample using the mask pattern, and the illumination component is configured to activate the one or more photoactivatable probes.

21. The method of claim 20, wherein the mask pattern is based on a distribution of the one or more photoactivatable probes within the sample.

22. The method of claim 20, wherein the mask pattern comprises a plurality of regions of interest.

23. The method of claim 20, wherein the trained artificial intelligence model has been trained using validated examples of regions of interest from training images.

24. The method of claim 20, wherein the trained artificial intelligence model has been trained using microscope-based system inputs, and wherein the artificial intelligence model is configured to segment one or more regions of interest from pixel value data of a training image.

25. The method of claim 20, wherein the sample is a biological sample.

26. The method of claim 20, wherein the microscope-based system is configured to capture multiple images of the sample.

27. The method of claim 20, wherein each of the one or more images is captured by the microscope-based system with a different field of view (FOV).

28. The method of claim 20, wherein one or more photoactivatable probes respond to light from the illumination system.

Images