Image processing system, microscope, machine learning algorithm, and predictive kernel labeling method

Abstract

An image processing system configured to receive at least one reference image, where each reference image is a microscope image of a captured cell of a biological sample, where the at least one reference image includes at least one reference marker directed to a reference cell compartment of the captured cell. The image processing system is configured to process the at least one reference image using a trained deep neural network to generate a target image, where the target image includes a target marker directed to a target cell compartment of the captured cell, where the at least one reference marker includes a fluorescent marker, where the fluorescent marker is not directed to the target cell compartment of the captured cell. The reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus.

Description

Technical Field

[0001] The present disclosure relates to methods and systems for processing microscopy images. The present disclosure relates specifically to methods and systems for predicting nuclear markers using machine learning. Background Art

[0002] In fluorescence microscopy, the spatial distribution or spatiotemporal distribution of specific biomolecules is made visible by fluorescent markers that are visible against a dark background. Fluorescent markers are dyes with specific binding sites or dyes coupled to antibodies or fluorescent proteins. Fluorescent markers are usually selected so that their distribution specifically marks specific cell compartments such as organelles, such as the nucleus. The position or morphology of organelles can diagnose the cell state or cell health. Specifically, when analyzing cells in microscope images, identifying the nucleus is usually the first step. Identifying the nucleus is very important for fields such as histology and pathology, which rely on the identification of specific cell types to diagnose diseases or identify abnormalities. Traditional methods for identifying the nucleus rely on the application of DNA stains or fluorescent transgenic expression near the nucleus. Typically, DAPI or Hoechst staining is used to mark the nucleus. However, nuclear staining tends to fade, is toxic, and has the risk of changing cell behavior and especially cell division.

[0003] In recent years, deep learning methods have been applied to process microscope images of various modalities. Specifically, convolutional neural networks such as U-net (Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation", Medical Image Computing and Computer-Assisted Intervention Conference (MICCAI), Springer, LNCS, Vol. 9351: 234-241) are used for various tasks such as protein detection and image segmentation of single cells. It has also been proposed to use deep learning methods to predict fluorescent labels in transmitted light images of unlabeled fixed or living biological samples (for example, Christiansen et al., "In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images", Cell 173, 792-803). Other approaches employ convolutional neural network architectures to segment nuclei from brightfield microscopy images (e.g., Fishman et al., “Practical segmentation of nuclei in brightfield cell images with neural networks trained on fluorescently labeled samples,” Journal of Microscopy 284.1, 2021, 12-24); employ machine learning architectures that can classify, segment, and predict orthogonal imaging modalities in a variety of hyperspectral imaging techniques (e.g., Manifold et al., “A versatile deep learning architecture for classification and label-free prediction of hyperspectral images,” Nature Machine Intelligence 3.4, 2021, 306-315); train neural networks to predict images with significantly increased contrast of target structures (e.g., et al., "Label2label: training a neural network to selectively restore cellular structures in fluorescence microscopy", Journal of Cell Science 135.3, 2022); or applying deep learning-based algorithms for 2D cell segmentation (Al-Kofahi et al., "A deep learning-based algorithm for 2-D cell segmentation in microscopy images", BMC bioinformatics 19.1 (2018), 1-11). Summary of the invention

[0004] The inventors have realised that applying machine learning methods provides a solution for improving the workflow of fluorescence microscopy.In the present disclosure, a method is proposed to predict nuclear markers from fluorescent markers directed to other cellular compartments.

[0005] Disclosed herein is an image processing system comprising one or more processors and one or more storage devices. The image processing system is configured to receive at least one reference image, wherein each reference image is a microscope image of a cell capturing a biological sample, wherein the at least one reference image comprises at least one reference marker pointing to a reference cell compartment of the captured cell. The image processing system is further configured to process the at least one reference image using a trained deep neural network to generate a target image, wherein the target image comprises a target marker pointing to a target cell compartment of the captured cell, wherein the at least one reference marker comprises a fluorescent marker, wherein the reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus.

[0006] The disclosed image processing system thus enables fluorescent markers of structures distributed on the cell to be used to infer the cell nucleus, thereby avoiding nuclear staining, thereby freeing up spectral bandwidth and reducing the risk to the examined cells.

[0007] According to an embodiment, the image processing system can be further configured to process the target image to obtain the center of mass and shape of the nucleus of the captured cell. The at least one reference marker can be based on a cytoskeletal marker or a cytoplasmic marker. The at least one reference marker can include a first fluorescent marker and a second fluorescent marker different from the first fluorescent marker. In this way, the image processing system can infer the nucleus with high accuracy.

[0008] According to an embodiment, the at least one reference image may include a plurality of reference images, wherein the plurality of reference images may represent a series of optical slices parallel to an optical path of the microscope. Processing the at least one reference image includes: generating a plurality of target images, and generating a three-dimensional reconstruction of the target cell compartment from the plurality of target images.

[0009] The image processing system may also include an image analysis system configured to process the target image based on the target tag, wherein the processing may include at least one of the following: performing semantic image segmentation to obtain a class label for each pixel in the target image; applying a cell segmentation algorithm to the target image; applying a distance transform and thresholding to the target image to obtain a count of captured cells; or identifying a cell type of captured cells. Thus, the result of the processing reduces the burden on the microscope user by directly providing highly relevant data.

[0010] According to another embodiment, the image processing system can be configured to process the target image to provide a command to the microscope to at least one of: adjust the control flow of the microscope; change the imaging modality used by the microscope; or change the central field of view of the microscope based on the identified cells. Thus, the result of the processing can be used as direct feedback to the microscope to reduce the burden on the user when studying a large number of samples.

[0011] The present disclosure also relates to a microscope comprising the above-mentioned image processing system. The microscope is configured to provide at least one reference image to the image processing system and to display the target image.

[0012] The present disclosure also relates to a computer-implemented method for automatic image management of a microscope, the method comprising: using a microscope to obtain at least one reference image, sending the at least one reference image to a server hosting the above-mentioned image processing system, receiving at the microscope the target image sent from the server, and controlling the settings of the microscope based on processing the target image.

[0013] The present disclosure also relates to a computer-implemented method for training a deep neural network. The method includes: receiving training tuples, each training tuple includes at least one reference image and a target image, wherein each reference image is a microscope image of cells of a captured biological sample, wherein the at least one reference image includes at least one reference marker pointing to a reference cell compartment of the captured cell, and the target image includes a target marker pointing to a target cell compartment of the captured cell, wherein the at least one reference marker includes a fluorescent marker, wherein the reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus. The method also includes: training the deep neural network on the training tuple to predict the target image from the at least one reference image.

[0014] In many aspects, the deep neural network can be based on a fully convolutional image-to-image neural network, or can be based on a visual transformer, or can be based on a diffusion model, or can be based on an adversarial network. The training tuple can also include bright field microscopy images and phase contrast microscopy images.

[0015] The method may also include pre-training the deep neural network on the pre-trained tuple to predict another target tag from another reference tag, the another target tag and the another reference tag being different from the target tag and the reference tag.

[0016] In an embodiment where the deep neural network is based on a variational autoencoder, the method may further include: pre-training the deep neural network on the pre-training tuple in a semi-supervised manner.

[0017] The present disclosure also relates to a trained machine learning algorithm, which is trained in the following manner: receiving training tuples, each training tuple comprising at least one reference image and a target image; and adjusting the machine learning algorithm based on the training tuples to obtain the trained machine learning algorithm, wherein each reference image is a microscope image of cells captured in a biological sample, wherein the at least one reference image comprises at least one reference marker pointing to a reference cell compartment of the captured cells, and the target image comprises a target marker pointing to a target cell compartment of the captured cells, wherein the at least one reference marker comprises a fluorescent marker, wherein the reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Embodiments of the invention are shown in the accompanying drawings, in which:

[0019] Figure 1 Prediction of nuclear labeling using an image processing system according to an embodiment is shown;

[0020] Figure 2Automated image management of a microscope based on nuclear labeling predicted by an image processing system according to an embodiment is shown;

[0021] Figure 3 The training of an image processing system for predicting nuclear markers is shown;

[0022] Figure 4 A flow chart showing a method of operating a microscope based on predicted nuclear labeling according to an embodiment;

[0023] Figure 5 A flow chart showing a method of training an image processing system for predicting nuclear markers according to an embodiment;

[0024] Figure 6 A system configured according to an embodiment is shown; and

[0025] Figure 7 Microscope images and output images containing nuclear labeling predicted according to the examples are reproduced. DETAILED DESCRIPTION

[0026] Figure 1 An image processing system 100 according to an embodiment of the present invention is shown. The image processing system includes one or more processors and one or more storage devices. The storage device stores a trained deep neural network 105. The image processing system 100 is configured to receive a reference image 110. The reference image 110 is a microscope image depicting cells of a biological sample. The reference image 110 can be obtained in real time from a microscope 150, or can be retrieved from a storage device. The reference image 110 includes reference markers pointing to reference cell compartments of the depicted cells.

[0027] The reference marker can be a fluorescent marker. By targeting a specific protein located next to or within a specific cell compartment, the reference marker points to the specific compartment and makes the compartment visible in the captured fluorescence. Cell compartments include all closed parts in the cytosol of a eukaryotic cell, such as organelles, mitochondria, chloroplasts, peroxisomes, lysosomes, endoplasmic reticulum, nuclei, Golgi bodies, vesicles, and microtubules. In an embodiment, the reference image 110 can include a plurality of different fluorescent markers, each of which points to a different cell compartment.

[0028] The processor of the image processing system 100 is configured to execute the trained neural network 105 stored in the memory to generate a target image 120. Based on the parameters θ1 acquired during training, the deep neural network 105 is executed to generate the target image 120, which contains target markers of the target cell compartment.

[0029] The target image includes a target marker of a target cell compartment depicted in the biological sample. The target cell compartment is not in the marker cell compartment of the reference image. However, the reference markers included in the reference image can be selected so that they can reliably infer the target marker, thereby inferring the position and shape of the target cell compartment. Therefore, the image processing system 100 is configured to predict the target marker (i.e., the second staining of the cell nucleus) from the reference marker (i.e., the first staining), thereby converting the first staining into the second staining.

[0030] Identifying the nucleus is particularly important in cell biology. Visualization of the nucleus allows identification of cell types. In addition, labeling the nucleus in a direct way allows counting the number of cells, which can assess cell growth, for example, to determine the effectiveness of drug treatment. In addition, labeling the nucleus allows identification of the subcellular localization of proteins around the nucleus to understand the function and activity of these molecules. However, many cell lines available to researchers are designed with endogenous markers rather than nuclear markers. However, the use of additional nuclear dyes can affect the behavior of DNA and thereby affect cell division and the survival of the cell line. The disclosed invention can rely solely on endogenous markers while still locating the nucleus. In addition, in certain research areas (e.g., nuclear damage research), it is in principle impossible to use nuclear dyes.

[0031] In order to predict the position and shape of the nucleus, the reference fluorescent marker must be fully distributed along the cell body so that it conveys information pointing to the position and shape of the nucleus. For example, fluorescent markers for tubulin and / or actin can be used. Alternatively, any other cytoskeleton or cytoplasmic markers (cytoskeleton or cytosollabeling) can be used. For example, intermediate filaments can be targeted. Therefore, reference image 110 can include the marking of one or more specific cytoskeleton marker antibodies. In an embodiment, one or more fluorescent markers included in one or more reference images can be based on antibody markings.

[0032] In some embodiments, instead of using a single reference image 110, multiple reference images are provided to the image processing system 100. The multiple reference images can represent a series of optical slices parallel to the microscope optical path. Such multiple reference images can correspond to a set of microscope images, each image corresponding to a specific focal plane within a thick sample. In these embodiments, a three-dimensional reconstruction of the sample can be achieved, thereby obtaining an image perpendicular to the axial dimension. Using the generated target image, a three-dimensional reconstruction of the target cell compartment (e.g., a cell nucleus) can be obtained. In other embodiments, the series of optical slices can also include a time series.

[0033] Figure 2An image analysis pipeline for controlling a microscope 150 based on an image processing system 100 is shown. The microscope 150 is capturing a microscope image 110, which is fed to the image processing system 100. The microscope 150 may be an epifluorescence microscope. The microscope may also be based on more advanced designs, such as confocal microscopes and total internal reflection fluorescence microscopes.

[0034] In an embodiment, the image processing system 100 is included in a microscope 150. In other embodiments, the image processing system 100 is located in a remote server connected to the microscope 150 via a data network. The microscope image 110 contains a reference marker, such as one pointing to a first cell compartment. The image processing system 100 provides a target image 120 containing a predicted staining (e.g., a nuclear staining). Thus, the disclosed microscope 150 is capable of locating cell nuclei without the use of nuclear markers or nuclear markers.

[0035] Microscope 150 may include a monitor configured to display target image 120. Target image 120 is further provided to image analysis system 210. Image analysis system 210 is configured to generate output image 250 including predictions 252, such as specific cell regions identified in target image 120. Figure 2 The image analysis system 210 is shown separate from the image processing system 100 , but it should be understood that it illustrates that the image analysis system 210 may form part of the image processing system 100 .

[0036] The image analysis system 210 can calculate the centroid and shape of the nucleus of the captured cell. These data can allow the statistics of the nuclei contained in the biological sample to be obtained. The count of the captured cells can be obtained from the target image 120. In addition, the nuclei generated by the deep neural network 105 are marked to allow the cell type of the captured cells to be identified. Therefore, the image analysis system 210 can include a machine learning system that is configured to identify the cell type of the cell depicted in the sample based on the application of classification to the marked nuclei. The classification can be based on the stored nuclear shape classification. It is well known that different types of cells can have very different nuclear shapes, and these shapes have an impact on cell function. Many pathologies are related to changes in nuclear shape. For example, many white blood cells of the immune system have multi-lobed nuclei, while smooth muscle cells have spindle-shaped nuclei. Nuclear shape has a significant effect on the transcriptional activity of the cell. Granulocytes have different nuclear shapes and are usually identified and distinguished histologically by their nuclear shapes. The shape of the nuclear can be measured by the geometric measures of the marked nuclei. In addition, the image analysis system 210 can be a machine learning system that is configured to apply a cell segmentation algorithm to a target marker, such as a target marker for a cell nucleus. Therefore, the image analysis system 210 can be configured to process the target image 120 to perform semantic image segmentation. Therefore, image pixels corresponding to the cell nucleus can be identified by a first class of labels, and image pixels that do not belong to the cell nucleus can be identified by another class of labels. The semantic image segmentation of the cell can also include the segmentation of the entire cell body containing the cytoplasm (Cytoplasm). The machine learning model 210 can also be configured to process the target image 120 to perform a distance transformation on the target image. Based on the labeling of the target cell compartment, the distance from all background points to the nearest object point (such as the cell nucleus) of the target cell compartment can be determined. Processing the target image can also include applying thresholding. Therefore, the monitor of the microscope can also display the processing results of the target image 120. For example, the count of captured cells or cells of a specific type can be highlighted.

[0037] Based on prediction 252, microscope 150 can adjust its control flow, change imaging modality and / or move the center field of view (a Center Field of View). For example, if the current sample does not contain a specific cell type, the control flow of microscope 150 can be adjusted to analyze another sample. In addition, the field of view of microscope 150 can be adjusted based on the result that the current field of view does not contain a specific cell type. In a similar manner, the imaging modality used by microscope 150 can be adjusted. Therefore, the method of feedback microscopy can be applied to reduce the tedious task of manually locating samples by fully integrating the task of identifying and locating samples of interest into the image acquisition workload. The present disclosure can also be applied to correlative microscopy, which combines and utilizes information across dimensions, modes and scales.

[0038] Figure 3 The training of the deep neural network 105 of the image processing system 100 is shown. The microscope 150 generates an image pair 310. The image pair 310 includes: an image 310-1 in which a source cell compartment (e.g., an organelle) is labeled; and a matching image 310-2 in which the cell nucleus is indicated by a mask or marker. In an embodiment, both images 310-1 and 310-2 are fluorescence microscopy images. Therefore, image 310-1 includes a fluorescent marker pointing to a reference cell compartment, while image 310-2 captures the same sample but includes a marker pointing to the cell nucleus. In an alternative embodiment, image 310-2 includes additional markers or masks. In an embodiment, image 310-2 is an image based on a non-optical microscope that includes a label or mask that allows prediction of the cell nucleus.

[0039] In an embodiment, the deep neural network 105 is a convolutional neural network consisting of an image-to-image pipeline. Training the deep neural network 105 is based on adjusting the parameters θ1 of the deep neural network 105. Based on comparing the generated image 320 with the image 310-2, the deep neural network 105 becomes better at generating an output image 320 corresponding to the image 310-2. When trained correctly, the deep neural network 105 is able to perform cell nucleus inference on image scenes not seen during training. When training the deep neural network 105, a loss function is calculated, which can be a mean absolute error, a mean squared error, or a binary cross entropy error. The deep neural network 105 can be trained directly for the intended task, or can be pre-trained for a similar task and then trained for the intended task.

[0040] In an embodiment, the deep neural network 105 is based on U-net. The U-net architecture consists of the following paths: a contraction path, which is based on the typical architecture of a convolutional network; and an expansion path, in which features and spatial information are combined with high-resolution features from the contraction path through a series of up-convolutions and fusions. In other embodiments, the deep neural network is a visual transformer (VT). The VT model is an architecture based on a transformer originally designed for text-based tasks. The VT model can first apply a convolutional layer to extract low-level features from the input image. A tokenizer can then be applied to the feature map to group pixels into visual tokens. These tokens are provided to the transformer network to model the relationship between the tokens. The converted patches can be used for image-to-image conversion. In other embodiments, the deep neural network 105 implements an image-to-image diffusion model. The image-to-image diffusion model is a conditional diffusion model of the form p(y|x), where x and y are both images. Given a training output image y, a noisy version is generated Given x and the noise level indicator, train the neural network f θ To denoise In other embodiments, the deep neural network 105 implements an image-to-image generative adversarial network (GAN). GAN includes a generator network and a discriminator network, which are trained simultaneously to maximize a joint loss. The goal of the generator network is to generate realistic images in the transformed domain that cannot be distinguished from images in the original domain. In other embodiments, the deep neural network 105 implements a variational autoencoder (VAE). VAE is a probabilistic generative model that includes: a first neural network, the encoder, which maps input variables to a latent space corresponding to the parameters of a variational distribution; a second neural network, the decoder, which maps from the latent space to the input space to produce or generate data points. VAE is particularly suitable for semi-supervised learning.

[0041] In some embodiments, the deep neural network 105 is pre-trained on a similar task of predicting another target cell compartment from another source cell compartment. In embodiments such as those relying on VAE, the deep neural network can be pre-trained in a semi-supervised manner using a large corpus of microscopy images, where only a subset of the corpus contains supervised data, i.e., labels of the target cell compartment (e.g., nuclei). The deep neural network 105 can include a non-linear output layer, such as a sigmoid layer.

[0042] Figure 4 A flow chart showing a method of applying a deep neural network to control microscope operation.

[0043] In step 410, the microscope captures at least one reference image of the biological sample. The method further comprises step 420 of sending the at least one reference image to a remote server comprising the above-described image processing system 100 via an interface of the microscope. In these embodiments, the image processing system 100 corresponds to a server, and the trained deep neural network 105 forms part of the processing software running on the server. In step 430, the remote server feeds the at least one reference image to the trained deep neural network 105 in real time to generate a target image. The remote server sends the target image back to the microscope.

[0044] In step 440, the controller of the microscope evaluates the received target image and adjusts the settings of the microscope, such as the settings regarding the imaging modality and the field of view. In an embodiment, based on the evaluation of the received target image, the controller of the microscope provides a command to the microscope to provide the next sample to the sample holder. To process the target image, the microscope can employ the above-mentioned image analysis system 210.

[0045] Figure 5 A flow chart of a computer-implemented method 500 for training a deep neural network (e.g., deep neural network 105) for predicting nuclear markers is shown. The method includes receiving training tuples 520, each of which includes a reference image and a target image. The reference image and the target image are both microscope images that capture cells of a biological sample. The reference image contains reference markers pointing to a reference cell compartment of the cell. The target image contains target markers pointing to a target cell compartment (e.g., a cell nucleus). The reference markers and the target markers can be based on fluorescent markers.

[0046] The method also includes training 530, during which a parameter θ1 of the deep neural network is adjusted according to the difference between the observed output image and the reference image. The difference can be measured by a loss function. The step of training 530 includes iterating over all training samples, for example, in batches.

[0047] In an embodiment, method 500 includes: step 510, pre-training a deep neural network on a pre-training task to predict a first target cell compartment from a first reference cell compartment, which is different from the target cell compartment and the reference cell compartment trained during training 530.

[0048] In an embodiment, the pre-training step 510 may be based on semi-supervised training. In an embodiment, the deep neural network 105 is trained on a large dataset of unlabeled microscopy images, which may include bright field microscopy images, phase contrast microscopy images, or other microscopy modalities without fluorescent labels. The deep neural network is trained to recognize patterns and structures in the images, which can be used to infer the location and shape of the target cell compartment.

[0049] Figure 7 Results obtained using the image processing system 100 according to an embodiment and training according to the method 500 are shown. Column 710 shows an exemplary reference image including a label of a reference organelle. Column 720 reproduces a corresponding training target image including a label of a cell nucleus. Column 730 shows a generated target image obtained by processing the image of column 710 using the trained deep neural network 105.

[0050] Therefore, when comparing columns 720 and 730, the quality of the disclosed method can be assessed. Figure 7 As shown in the results of , the proposed method allows the shape and position of the cell nucleus to be reliably inferred from a marker pointing to another cellular compartment. Therefore, the disclosed method can save spectral bandwidth to detect more other markers / markers and avoid the need for toxic nuclear staining.

[0051] Embodiments may be based on the use of machine learning models or machine learning algorithms. Machine learning may refer to algorithms and statistical models that can be used by computer systems that do not use explicit instructions, but rely on models and inferences to perform specific tasks. For example, in machine learning, data transformations inferred from analysis of historical and / or training data may be used instead of rule-based data transformations. For example, a machine learning model may be used or a machine learning algorithm may be used to analyze the content of an image. In order for a machine learning model to analyze the content of an image, a training image may be used as input and the machine learning model may be trained using training content information as output. By training a machine learning model using a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the image, so the machine learning model may be used to recognize the content of the image not included in the training data. The same principle applies to other types of sensor data: by using training sensor data and desired outputs, a machine learning model is trained, and the machine learning model "learns" the conversion between sensor data and output, which can be used to provide outputs based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) may be preprocessed to obtain a feature vector, which is used as input to the machine learning model.

[0052] The machine learning model can be trained using training input data. The above example uses a training method called "supervised learning". In supervised learning, the machine learning model is trained using multiple training samples, each of which may include multiple input data values ​​and multiple expected output values, i.e., each training sample is associated with an expected output value. By specifying both the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples similar to the samples provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (e.g., classification algorithms, regression algorithms, or similarity learning algorithms). When the output is limited to a limited set of values ​​(categorical variables), a classification algorithm can be used, that is, the input is classified into one of the limited set of values. When the output can have any numerical value (within a certain range), a regression algorithm can be used. Similarity learning algorithms can be similar to classification and regression algorithms, but are based on learning from examples using a similarity function that measures the similarity or degree of correlation between two objects. In addition to supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, (only) input data may be provided, and an unsupervised learning algorithm may be used to find structure in the input data (e.g., find commonalities in the data by grouping or clustering the input data). Clustering is the assignment of input data comprising multiple input values ​​into subsets (clusters) such that input values ​​within the same cluster have similarities according to one or more (predefined) similarity criteria, but are dissimilar to input values ​​included in other clusters.

[0053] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Based on the actions taken, a reward is calculated. Reinforcement learning is based on training one or more software agents to select actions that increase the cumulative reward, thereby making the software agent perform better on a given task (the increase in reward is the evidence).

[0054] In addition, some techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained at least in part using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms (which can be called representation learning algorithms) can retain the information in their input, but can also transform it in a useful way, usually as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0055] In some examples, anomaly detection (i.e., outlier detection) can be used, the goal of which is to identify input values ​​that are sufficiently different from the majority of inputs or training data to raise suspicion. In other words, a machine learning model can be trained at least in part using anomaly detection, and / or a machine learning algorithm can include an anomaly detection component.

[0056] In some examples, a machine learning algorithm can use a decision tree as a prediction model. In other words, a machine learning model can be based on a decision tree. In a decision tree, an observation about an item (e.g., a set of input values) can be represented by a branch of the decision tree, and an output value corresponding to the item can be represented by a leaf of the decision tree. A decision tree can support both discrete values ​​and continuous values ​​as output values. If discrete values ​​are used, the decision tree can be represented as a classification tree, and if continuous values ​​are used, the decision tree can be represented as a regression tree.

[0057] Association rules are another technique that can be used with machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or utilize one or more relationship rules that represent knowledge derived from the data. These rules can be used to store, manipulate, or apply knowledge.

[0058] Machine learning algorithms are often based on machine learning models. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and / or a rule set that represents the learned knowledge (e.g., based on the training performed by the machine learning algorithm). In an embodiment, the use of a machine learning algorithm may mean the use of an underlying machine learning model (or multiple underlying machine learning models). The use of a machine learning model may mean that the machine learning model and / or the data structure / rule set that is the machine learning model is trained by a machine learning algorithm.

[0059] For example, a machine learning model can be an artificial neural network (ANN). ANN is a system inspired by biological neural networks, such as the neural networks in the retina or brain. ANN includes a plurality of interconnected nodes and a plurality of connections between the nodes, the so-called edges. There are usually three types of nodes: input nodes that receive input values, hidden nodes that are (only) connected to other nodes, and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can transfer information from one node to another. The output of a node can be defined as a (non-linear) function of its input (e.g. the sum of its inputs). The input of a node can be used in a function based on the "weight" of the edge or the node providing the input. The weights of the nodes and / or edges can be adjusted during the learning process. In other words, the training of an artificial neural network can include adjusting the weights of the nodes and / or edges of the artificial neural network, i.e. achieving the desired output for a given input.

[0060] Alternatively, the machine learning model can be a support vector machine, a random forest model or a gradient boosting model. A support vector machine (i.e., a support vector network) is a supervised learning model with a related learning algorithm, which can be used to analyze data (e.g., in classification or regression analysis). A support vector machine can be trained by providing an input with multiple training input values ​​belonging to one of two categories. A support vector machine can be trained to assign new input values ​​to one of two categories. Alternatively, the machine learning model can be a Bayesian network, which is a probabilistic directed acyclic graph model (Probabilistic Directed Acyclic Graphical Model). A Bayesian network can use a directed acyclic graph to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and a heuristic technique that mimics a natural selection process.

[0061] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0062] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0063] Some embodiments relate to a microscope comprising: Figures 1 to 5 Alternatively, the microscope may be Figures 1 to 5 part of one or more related systems, or Figures 1 to 5 One or more related system connections. Figure 6A schematic diagram of a system 600 configured to perform the methods described herein is shown. The system 600 includes a microscope 610 and a computer system 620. The microscope 610 is configured to capture images and is connected to the computer system 620. The computer system 620 is configured to perform at least a portion of the methods described herein. The computer system 620 can be configured to execute a machine learning algorithm. The computer system 620 and the microscope 610 can be separate entities, but can also be integrated in a common housing. The computer system 620 can be part of a central processing system of the microscope 610, and / or the computer system 620 can be part of a subcomponent of the microscope 610, such as a sensor, actuator, camera, or lighting unit of the microscope 610.

[0064] The computer system 620 may be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices, distributed in different locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 620 may include any circuit or combination of circuits. In one embodiment, the computer system 620 may include one or more processors, which may be of any type. As used herein, a processor may represent any type of computing circuit, such as, for example, a microscope or a microscope component (e.g., a camera) or any other type of processor or processing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA). Other types of circuits that may be included in the computer system 620 may be custom circuits, application specific integrated circuits (ASICs), and the like, such as one or more circuits (e.g., communication circuits) for wireless devices (e.g., mobile phones, tablet computers, laptops, two-way radios, and similar electronic systems). The computer system 620 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 620 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 620.

[0065] Part or all of the method steps can be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, one or more of some of the most important method steps can be performed by such devices.

[0066] According to certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. It may be implemented using a non-temporary storage medium (e.g., a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) in which electronically readable control signals are stored that cooperate with (or can cooperate with) a programmable computer system to perform the corresponding method. Therefore, the digital storage medium may be computer readable.

[0067] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0068] Generally, the embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is executed to perform one of the methods. The program code may, for example, be stored on a machine readable carrier.

[0069] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0070] In other words, one embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0071] Therefore, another embodiment of the present invention is a storage medium (or a data carrier or a computer-readable medium) comprising, when executed by a processor, a computer program stored thereon for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein comprising a processor and a storage medium.

[0072] Therefore, another embodiment of the invention is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.For example, the data stream or the sequence of signals may be configured to be transmitted via a data communication connection (eg via the Internet).

[0073] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0074] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0075] Another embodiment according to the invention comprises an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may comprise, for example, a file server for transferring the computer program to the receiver.

[0076] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0077] Reference numerals 100 Image Processing System 110 reference images 120 Target Image 150 Microscope 105 Deep Neural Networks 210 Image Analysis System 250 Output image 252 Predictions 310 training image pairs 310-1 Training reference image 310-2 Training target image 320 Predicted target image 410-440 Methods and steps for controlling a microscope 510-530 Methods and steps for training deep neural networks 600 System 610 Microscope 620 Computer Systems 710 Example reference image 720 Example training target image 730 Generated target image

Claims

1. An image processing system (100), comprising one or more processors and one or more storage devices, wherein: The image processing system (100) is configured as follows: Receiving at least one reference image (110), wherein each reference image (110) is a microscope image of a cell of a captured biological sample, wherein the at least one reference image (110) includes at least one reference marker pointing to a reference cell compartment of the captured cell; The at least one reference image (110) is processed using a trained deep neural network (105) to generate a target image (120), wherein the target image (120) includes a target marker pointing to a target cell compartment of the captured cell, wherein the at least one reference marker includes a fluorescent marker, wherein the reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus.

2. The image processing system (100) according to claim 1, wherein: The image processing system (100) is further configured to process the target image (120) to obtain the centroid and shape of the nucleus of the captured cell.

3. The image processing system (100) according to claim 1 or 2, wherein: The at least one reference marker is based on a cytoskeletal marker or a cytoplasmic marker.

4. The image processing system (100) according to one of claims 1 to 3, wherein: The at least one reference marker includes a first fluorescent marker and a second fluorescent marker different from the first fluorescent marker.

5. The image processing system (100) according to one of claims 1 to 4, wherein: The at least one reference image (110) comprises a plurality of reference images, wherein the plurality of reference images represent a series of optical slices parallel to an optical path of the microscope (150), and wherein processing the at least one reference image (110) comprises generating a plurality of target images, and generating a three-dimensional reconstruction of the target cell compartment from the plurality of target images.

6. The image processing system (100) according to one of claims 1 to 5, further comprising an image analysis system (210) configured to process the target image (120) based on the target label, wherein: The processing includes at least one of the following: performing semantic image segmentation to obtain a class label for each pixel in the target image (120), applying a cell segmentation algorithm to the target image (120), applying a distance transform and thresholding to the target image (120), Obtain a count of captured cells, or Identify the cell type of the captured cells.

7. The image processing system (100) according to one of claims 1 to 6, wherein: The image processing system (100) is further configured to process the target image (120) to provide at least one of the following commands to the microscope (150): adjusting the control flow of the microscope (150), changing the imaging modality used by the microscope (150), or The central field of view of the microscope (150) is altered based on the identified cells.

8. A microscope (150) comprising the image processing system (100) according to one of claims 1 to 7, wherein: The microscope (150) is configured to provide at least one reference image (110) to the image processing system (100) and to display the target image (120).

9. A computer-implemented method (400) for automatic image management of a microscope (150), the method (400) comprising: Step (410): using the microscope (150) to obtain at least one reference image (110); Step (420): sending the at least one reference image (110) to a server, the server comprising an image processing system (100) according to one of claims 1 to 7; Step (430): receiving, at the microscope (150), the target image (120) sent from the server; as well as Step (440): Controlling the settings of the microscope (150) based on the processing of the target image (120).

10. A computer-implemented method (500) for training a deep neural network, the method comprising: Step (520): receiving training tuples (310), each training tuple comprising at least one reference image (310-1) and a target image (310-2), wherein each reference image (310-1) is a microscope image of cells of a captured biological sample, wherein the at least one reference image (310-1) comprises at least one reference marker pointing to a reference cell compartment of the captured cells, and the target image (310-2) comprises a target marker pointing to a target cell compartment of the captured cells, wherein the at least one reference marker comprises a fluorescent marker, wherein the reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus; and Step (530): Training the deep neural network (100) on the training tuple (310) to predict the target image (310-2) from the at least one reference image (310-1).

11. The method according to claim 10, wherein: The deep neural network (105) is based on a fully convolutional image-to-image neural network, or based on a visual converter, or based on a diffusion model, or based on an adversarial network.

12. The method according to claim 10 or 11, wherein: The training tuple also includes bright field microscopy images and phase contrast microscopy images.

13. The method according to one of claims 10 to 12, wherein: The method further comprises the step (510) of pre-training the deep neural network (105) on the pre-trained tuples to predict another target label from another reference label, the another target label and the another reference label being different from the target label and the reference label.

14. The method according to claim 13, wherein: The deep neural network (105) is based on a variational autoencoder, wherein the method comprises a step (510): pre-training the deep neural network (105) on the pre-trained tuple in a semi-supervised manner.

15. A trained machine learning algorithm (105), wherein the training is performed by: Receiving training tuples (310), each training tuple comprising at least one reference image (310-1) and a target image (310-2), wherein: Each reference image (310-1) is a microscope image of a cell of a captured biological sample, wherein the at least one reference image (310-1) comprises at least one reference marker pointing to a reference cell compartment of the captured cell, and the target image (310-2) comprises a target marker pointing to a target cell compartment of the captured cell, wherein the at least one reference marker comprises a fluorescent marker, wherein the reference cell compartment is a distributed structure within the cell, and wherein the target cell compartment is a cell nucleus; and The machine learning algorithm is tuned based on the training tuple (310) to obtain the trained machine learning algorithm (105).

Images