Systems and methods for processing biologically relevant data, systems and methods for controlling a microscope, and microscopes

Abstract

A system (100) for processing biology-related data includes one or more processors (110) coupled to one or more storage devices (120). The system (100) is configured to receive biology-related image-based search data (103) and configured to generate a first high-dimensional representation of the biology-related image-based search data (103) by a trained visual recognition machine learning algorithm executed by the one or more processors (110). The first high-dimensional representation includes at least 3 entries each having a different value. Further, the system (100) is configured to obtain a plurality of second high-dimensional representations (105) of a plurality of biology-related image-based input data sets or a plurality of biology-related language-based input data sets. Additionally, the system (100) is configured to compare, by the one or more processors (110), the first high-dimensional representation to each of the plurality of second high-dimensional representations (105).

Description

Technical Field

[0001] Examples involve the processing of biologically relevant data and / or the control of microscopes. Background Technology

[0002] Many biological applications generate massive amounts of data. For example, images are collected from numerous biological structures and stored in databases. Manually analyzing biological data is extremely time-consuming and costly. Summary of the Invention

[0003] Therefore, there is a need to improve the concepts used for processing biologically relevant data and / or microscopic control.

[0004] This need can be satisfied through the subject matter of the claims.

[0005] Some embodiments relate to a system including one or more processors coupled to one or more storage devices. The system is configured to receive retrieval data based on biologically relevant images and to generate a first high-dimensional representation of the retrieval data based on biologically relevant images using a trained visual recognition machine learning algorithm executed by the one or more processors. The first high-dimensional representation includes at least three entries, each with a different value. Further, the system is configured to obtain multiple second high-dimensional representations of multiple input datasets based on biologically relevant images or multiple input datasets based on biologically relevant languages. Additionally, the system is configured to compare the first high-dimensional representation with each of the multiple second high-dimensional representations.

[0006] By using visual recognition machine learning algorithms, image-based retrieval requests can be mapped to high-dimensional representations. By allowing the high-dimensional representations to have entries with various distinct values ​​(as opposed to one-hot encoding), semantically similar biological search terms can be mapped to similar high-dimensional representations. By obtaining high-dimensional representations of multiple input datasets based on biologically relevant images or multiple input datasets based on biologically relevant languages, high-dimensional representations that are identical or similar to the high-dimensional representation of the retrieval request can be found. In this way, it becomes possible to find images or text corresponding to the retrieval request. In this manner, trained visual recognition machine learning algorithms can retrieve biologically relevant images from multiple biological images (e.g., biological image databases) or biologically relevant texts from multiple biological texts (e.g., collections or databases of scientific papers) based on image-based retrieval input. Retrieval can also be achieved in existing databases or from images generated by running experiments (e.g., images of one or more biological samples taken by a microscope), even if the images have not been previously labeled or tagged.

[0007] Some embodiments relate to a system including one or more processors and one or more storage devices. The system is configured to receive image-based retrieval data and to generate a first high-dimensional representation of the image-based retrieval data using a trained visual recognition machine learning algorithm executed by the one or more processors. The first high-dimensional representation includes at least three entries, each with a different value. Further, the system is configured to obtain multiple second high-dimensional representations of multiple image-based input datasets and to select a second high-dimensional representation from the multiple second high-dimensional representations based on a comparison of the first high-dimensional representation with each of the multiple second high-dimensional representations. Additionally, the system is configured to provide control signals for controlling the operation of a microscope based on the selected second high-dimensional representation.

[0008] By using visual recognition machine learning algorithms, image-based retrieval requests can be mapped to high-dimensional representations. By allowing the high-dimensional representations to have entries with various distinct values ​​(as opposed to one-hot encoded representations), semantically similar search terms can be mapped to similar high-dimensional representations. By obtaining high-dimensional representations from multiple image-based input datasets, high-dimensional representations that are identical or similar to the high-dimensional representations of the search terms can be found. In this way, images corresponding to the retrieval request can be found. Using this information, a microscope can be driven to the corresponding location to capture images, enabling the capture of other images of the location of interest (e.g., images taken using higher magnification, different lighting, or filters). In this way, samples (e.g., biological samples or integrated circuits) can first be imaged at low magnification to find the location corresponding to the retrieval request, after which the location of interest can be analyzed in more detail.

[0009] Some embodiments relate to a system including one or more processors coupled to one or more storage devices. The system is configured to determine multiple clusters of multiple second high-dimensional representations of multiple image-based input datasets using a clustering algorithm executed by the one or more processors. Further, the system is configured to determine a first high-dimensional representation of the cluster centers of the clusters within the multiple clusters, and is configured to select a second high-dimensional representation from the multiple second high-dimensional representations based on a comparison of the first high-dimensional representation with each or a subset of the multiple second high-dimensional representations. Additionally, the system is configured to provide control signals for controlling the operation of a microscope based on the selected second high-dimensional representation.

[0010] By identifying clusters of second-dimensional representations, second-dimensional representations corresponding to semantically similar content can be grouped into a single cluster. By determining cluster centers and comparing and identifying one or more second-dimensional representations closest to the cluster centers, one or more images representing typical images of that cluster can be found. For example, different clusters may include second-dimensional representations corresponding to different feature parts of a biological sample (e.g., cytoplasm, nucleus, cytoskeleton). The system can provide control signals to move the microscope to a location where typical images of one or more clusters are captured (e.g., to capture more images at that location with varying microscope parameters). Attached Figure Description

[0011] The following will describe some examples of apparatus and / or methods by way of example and with reference to the accompanying drawings, wherein:

[0012] Figure 1 This is a schematic diagram of a system used to process biologically related data;

[0013] Figure 2 This is a schematic diagram of another system used for processing biologically related data;

[0014] Figure 3 is a schematic diagram of another system for processing biologically relevant data;

[0015] Figure 4 This is a schematic diagram of a system used to control a microscope;

[0016] Figure 5 This is a schematic diagram of a system for controlling a microscope based on retrieval data from biologically relevant images;

[0017] Figure 6 This is a schematic diagram of a system used to control a microscope;

[0018] Figure 7a is a schematic diagram of a system that controls a microscope based on retrieval data from biologically relevant images using a clustering algorithm;

[0019] Figure 7b is a schematic diagram of a system for processing biologically relevant data using a clustering algorithm;

[0020] Figure 8 This is a schematic diagram of a system used for data processing;

[0021] Figure 9 This is a flowchart of a method for processing biologically related data;

[0022] Figure 10 This is a flowchart of a method for controlling a microscope; and

[0023] Figure 11This is a flowchart of a method for controlling another microscope. Detailed Implementation

[0024] A more comprehensive description of the various examples will now be given with reference to the accompanying drawings, which illustrate some of the examples. For clarity, the thickness of lines, layers, and / or regions may be exaggerated in the figures.

[0025] Accordingly, while the other examples can have various modifications and alternatives, some specific examples have been shown in the figures and will be described in detail thereafter. However, this detailed description does not limit the other examples to the specific forms described. The other examples may cover all modifications, equivalents, and alternatives falling within the scope of this disclosure. Throughout the description of the figures, the same or similar reference numerals refer to the same or similar elements, which may be implemented in the same or modified form when compared with each other, while providing the same or similar function.

[0026] It will be understood that when an element is referred to as "connected" or "linked" to another element, these elements can be directly connected or linked, or connected or linked via one or more intermediate elements. Unless otherwise defined explicitly or implicitly, if "or" is used to combine two elements A and B, this will be understood to disclose all possible combinations, i.e., only A, only B, and A and B. Alternative terms for the same combinations are "at least one of A and B" or "A and / or B". With the necessary modifications, this also applies to combinations of more than two elements.

[0027] The terminology used in this document to describe particular examples is not intended to limit other examples. Whenever the use of singular forms such as “a,” “an,” and “the,” and where the use of a single element is not explicitly or implicitly defined as mandatory, other examples may use plural elements to achieve the same functionality. Similarly, when a function is subsequently described as being implemented using multiple elements, other examples may use a single element or processing entity to achieve the same functionality. It will be further understood that, when used, the terms “comprise,” “comprising,” “includes,” and / or “including” indicate the presence of the described feature, integer, step, operation, process, action, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, processes, actions, elements, components, and / or any group thereof.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) are used herein with the common meaning in the field to which their examples belong.

[0029] Figure 1 A schematic diagram of a system 100 for processing biologically relevant data according to an embodiment is shown. System 100 includes one or more processors 110 coupled to one or more storage devices 120. System 100 is configured to receive (first) retrieval data 103 based on biologically relevant images and to generate a first high-dimensional representation of the (first) retrieval data 103 based on biologically relevant images using a trained visual recognition machine learning algorithm executed by the one or more processors 110. The first high-dimensional representation includes at least three entries, each with a different value (or includes at least 20, at least 50, or at least 100 entries with values ​​different from each other). Further, system 100 is configured to obtain multiple second high-dimensional representations 105 of multiple input datasets based on biologically relevant images or multiple input datasets based on biologically relevant languages. Additionally, system 100 is configured to compare the first high-dimensional representation with each of the multiple second high-dimensional representations 105 by the one or more processors 110.

[0030] Retrieval data 103 based on biologically relevant images can be image data (e.g., pixel data of images) of the following: biological structures including nucleotides or nucleotide sequences; biological structures including proteins or protein sequences; biomolecules; biological tissues; biological structures with specific behaviors; and / or biological structures with specific biological functions or specific biological activities. Biological structures can be molecules, viroids or viruses, artificial or natural membrane-encapsulated vesicles, subcellular structures (such as organelles), cells, spheroids, organoids, three-dimensional cell cultures, biological tissues, organ slices, or portions of organs, whether in vivo or in vitro. For example, an image of a biological structure can be an image of the location of proteins within a cell or tissue, or an image of a cell or tissue containing endogenous nucleotides (e.g., DNA) with a labeled nucleotide probe bound to it (e.g., in situ hybridization). Image data can include pixel values ​​for each pixel of the image for each color dimension of the image (e.g., three color dimensions for RGB representation). For example, depending on the imaging modality, other channels can be adapted to be associated with excitation or emission wavelengths, fluorescence lifetimes, light polarization, stage positions in the three spatial dimensions, and different imaging angles. The retrieval data 103 based on biologically relevant images can be an XY pixel image, volumetric data (XYZ), time-series data (XY+T), or a combination thereof (XYZT). Furthermore, additional dimensions depending on the type of image source can be included, such as channels (e.g., spectral emission bands), excitation wavelength, stage position, logical position as in multi-well plates or multi-position experiments, and / or mirror and / or objective lens positions as in light sheet imaging. For example, a user can input an image as a pixel image or a higher-dimensional picture, or a database can provide images as pixel images or higher-dimensional pictures. The retrieval data 103 based on biologically relevant images can be received from one or more storage devices, a database stored on the storage devices, or can be input by the user.

[0031] A high-dimensional representation (e.g., a first high-dimensional representation and a second high-dimensional representation) can be a hidden representation, a latent vector, an embedding, a semantic embedding, and / or a token embedding, and / or may also be referred to as a hidden representation, a latent vector, an embedding, a semantic embedding, and / or a token embedding.

[0032] The first high-dimensional representation and / or the second high-dimensional representation can be a numerical representation (e.g., including only numerical values). The first high-dimensional representation and / or the second high-dimensional representation can include more than 100 dimensions (or more than 300 dimensions or more than 500 dimensions) and / or less than 10,000 dimensions (or less than 3,000 dimensions or less than 1,000 dimensions). Each entry in the high-dimensional representation can be one dimension of the high-dimensional representation (e.g., a high-dimensional representation with 100 dimensions includes 100 entries). For example, a suitable representation of semantically relevant biological data can be achieved using a high-dimensional representation with more than 300 dimensions and less than 1,000 dimensions. The first high-dimensional representation can be a first vector and each second high-dimensional representation can be a corresponding second vector. If vector representations are used for the entries of the first high-dimensional representation and the entries of the second high-dimensional representation, efficient comparisons and / or other computations (e.g., normalization) can be implemented, but other representations (e.g., as a matrix representation) are also feasible. For example, the first high-dimensional representation and / or the second high-dimensional representation can be normalized vectors. The first and second high-dimensional representations can be normalized to the same value (e.g., 1). For example, the final layer of a trained speech recognition machine learning algorithm can represent a non-linear operation, which can then be normalized. The first and / or second high-dimensional representations can be generated by a trained visual recognition machine learning algorithm, which may have been trained with a loss function such that the trained visual recognition machine learning algorithm outputs a normalized high-dimensional representation. However, other methods can also be used for normalizing the first and second high-dimensional representations.

[0033] For example, compared to a one-hot encoded representation, a first high-dimensional representation and / or a second high-dimensional representation may include various entries (at least three entries) with values ​​not equal to 0. Corresponding to the first high-dimensional representation, each of the plurality of second high-dimensional representations may include at least 3 entries, each entry having a different value (or including at least 20, at least 50, or at least 100 entries with values ​​different from each other). By using high-dimensional representations that allow for various entries with values ​​not equal to 0, information about semantic relationships between high-dimensional representations can be reproduced. For example, more than 50% (or more than 70% or more than 90%) of the values ​​of the entries in the first high-dimensional representation and / or more than 50% (or more than 70% or more than 90%) of the values ​​of the entries in the second high-dimensional representation may be not equal to 0. Sometimes, a one-hot encoded representation may also have more than one entry with a value not equal to 0, but only one entry has a high value, while the values ​​of all other entries are at a noise level (e.g., less than 10% of that high value). Conversely, for example, the values ​​of 5 or more entries (or 20 or more entries or 50 or more entries) in the first high-dimensional representation can be 10% (or 20% or 30% larger) than the maximum absolute value of an entry in the first high-dimensional representation. Further, the values ​​of 5 or more entries (or 20 or more entries or 50 or more entries) in each of the plurality of second high-dimensional representations can be 10% (or 20% or 30% larger) than the corresponding maximum absolute value of an entry in the second high-dimensional representation. For example, the values ​​of 5 or more entries (or 20 or more entries or 50 or more entries) in one of the plurality of second high-dimensional representations can be 10% (or 20% or 30% larger) than the maximum absolute value of an entry in that single second high-dimensional representation. For example, each entry in the first high-dimensional representation and / or the second high-dimensional representation can include values ​​between -1 and 1.

[0034] A first high-dimensional representation can be generated by applying at least a portion (e.g., an encoder) of a trained visual recognition machine learning algorithm with a trained set of parameters to retrieval data 103 based on biologically relevant images. For example, generating the first high-dimensional representation by a trained visual recognition machine learning algorithm can mean that the first high-dimensional representation is generated by the encoder of the trained visual recognition machine learning algorithm. The trained set of parameters for the trained visual recognition machine learning algorithm can be obtained during the training of the visual recognition machine learning algorithm as described below.

[0035] The values ​​of one or more entries in the first high-dimensional representation and / or one or more entries in the second high-dimensional representation can be proportional to the probability of the presence of a specific biological function or activity. By using a mapping that generates high-dimensional representations that preserve the semantic similarity of the input datasets, semantically similar high-dimensional representations can be closer to each other than semantically less similar high-dimensional representations. Furthermore, if two high-dimensional representations represent input datasets with the same or similar specific biological functions or activities, one or more entries in these two high-dimensional representations can have the same or similar values. Because semantics are preserved, one or more entries in a high-dimensional representation can indicate the presence or absence of a specific biological function or activity. For example, the higher the value of one or more entries in a high-dimensional representation, the higher the probability of the presence of a biological function or activity associated with those one or more entries.

[0036] A trained visual recognition machine learning algorithm can also be called an image recognition model or a visual model. A trained visual recognition machine learning algorithm can be or may include a trained visual recognition neural network. A trained visual recognition neural network may include 20 or more layers (or 40 or more layers or 80 or more layers) and / or fewer than 400 layers (or fewer than 200 layers or fewer than 150 layers). A trained visual recognition neural network can be a convolutional neural network or a capsule network. Using convolutional neural networks or capsule networks can provide trained visual recognition machine learning algorithms with high accuracy for biologically relevant image data. However, other visual recognition algorithms can also be used. For example, a trained visual recognition neural network may include multiple convolutional layers and multiple pooling layers. However, for example, if a capsule network is used and / or a stride of 2 is used instead of a stride of 1 for convolution, pooling layers can be avoided. A trained visual recognition neural network may use a modified linear unit activation function. Using the modified linear unit activation function can provide high-precision trained visual recognition machine learning algorithms for input data based on biologically relevant images, but other activation functions (e.g., hard tanh, sigmoid, or tanh activation functions) can also be applied. For example, the trained visual recognition neural network can include a convolutional neural network and / or can be a residual network (ResNet) or a densely connected convolutional network (DenseNet) whose depth depends on the size of the input image.

[0037] Multiple second high-dimensional representations 105 of multiple biologically related image input datasets or multiple biologically related language input datasets can be obtained by receiving a second high-dimensional representation 105 from a database (e.g., a database stored by one or more storage devices), or by generating multiple second high-dimensional representations 105 based on multiple input datasets based on biologically related images or multiple input datasets based on biologically related languages. For example, if the multiple second high-dimensional representations are based on multiple input datasets based on biologically related images, the system 100 can be configured to generate a second high-dimensional representation among the multiple second high-dimensional representations by a trained visual recognition machine learning algorithm executed by one or more processors, thereby obtaining a second high-dimensional representation. For example, a trained visual model is capable of representing images in a semantic embedding space (e.g., represented as a second high-dimensional representation). Alternatively, if the multiple second high-dimensional representations are based on multiple input datasets based on biologically related languages, the system 100 can be configured to generate a second high-dimensional representation among the multiple second high-dimensional representations by a trained language recognition machine learning algorithm executed by one or more processors, thereby obtaining a second high-dimensional representation. Optionally, it can be combined as follows Figure 6 As described in 7a and / or 7b, the second high-dimensional representation is clustered, and then the first high-dimensional representation can be compared with each of the second high-dimensional representations of the cluster centers or with the second high-dimensional representation closest to the cluster center.

[0038] Similar to the retrieval data 103 based on biologically relevant images, each of the multiple input datasets based on biologically relevant images can be image data (e.g., pixel data of images) of the following: biological structures including nucleotides or nucleotide sequences; biological structures including proteins or protein sequences; biomolecules; biological tissues; biological structures with specific behaviors; and / or biological structures with specific biological functions or specific biological activities. A trained visual recognition machine learning algorithm can convert the image data of these images into semantic embeddings (e.g., a second high-dimensional representation). Multiple input datasets based on biologically relevant images can be received from one or more storage devices or from a database stored on the storage devices.

[0039] Each of the multiple biology-related language-based input datasets can be textual input related to biological structures, functions, behaviors, or activities. For example, a biology-related language-based input dataset could be nucleotide sequences, protein sequences, descriptions of biomolecules or biological structures, descriptions of the behavior of biomolecules or biological structures, and / or descriptions of biological functions or activities. Textual input can be natural language describing biomolecules (e.g., polysaccharides, poly / oligonucleotides, proteins, or lipids) or the behavior of biomolecules in the context of an experiment or dataset. For example, the biology-related language-based retrieval data 101 could be nucleotide sequences, protein sequences, or a coarse-grained set of biological terms.

[0040] A set of biological terms may include multiple coarse-grained search terms (or alternatively, molecular biology subject terms) belonging to the same biological subject. A set of biological terms may include catalytic activity (e.g., a reaction formula using terms for isolates and products), pathways (e.g., which pathway is involved, e.g., glycolysis), sites and / or regions (e.g., binding sites, active sites, nucleotide binding sites), GO gene ontology (e.g., molecular functions, such as nicotinamide adenine dinucleotide NAD binding, microtubule binding), GO biological functions (e.g., apoptosis, gluconeogenesis), enzyme and / or pathway databases (e.g., unique identifiers for sic functions, such as in BRENDA / EC numbers or UniPathways), subcellular localization (e.g., cytoplasm, nucleus, cytoskeleton), families and / or domains (e.g., binding sites for posttranslational modifications, motifs), open reading frames, single nucleotide polymorphisms, restriction sites (e.g., oligonucleotides recognized by restriction enzymes), and / or biosynthetic pathways (e.g., biosynthesis of lipids, polysaccharides, nucleotides, or proteins). For example, this group of biological terms could be a subcellular localization group, and coarse-grained search terms could be cytoplasm, nucleus, and cytoskeleton.

[0041] If coarse-grained search terms are used as input datasets for biology-related language, the length of multiple biology-related language input datasets can be less than 50 characters (or less than 30 characters or less than 20 characters), and / or if nucleotide sequences or protein sequences are used as input datasets for biology-related language, the length of multiple biology-related language input datasets can be more than 20 characters (or more than 40 characters, more than 60 characters, or more than 80 characters). For example, because three base pairs encode one amino acid, nucleotide sequences (DNA / RNA) are typically about three times longer than polypeptide sequences (e.g., peptides, proteins). For example, if the biology-related language input dataset is a protein sequence or amino acid, it can be more than 20 characters long. If the biology-related language input dataset is a nucleotide sequence or descriptive text in natural language, it can be more than 60 characters long. For example, an input dataset based on a biology-related language may include at least one non-numeric character (e.g., an alphabetic character).

[0042] A trained language recognition machine learning algorithm can also be referred to as a textual model, text model, or language model. A language recognition machine learning algorithm can be or may include a trained language recognition neural network. A trained language recognition neural network may include more than 30 layers (or more than 50 layers or more than 80 layers) and / or less than 500 layers (or less than 300 layers or less than 200 layers). A trained language recognition neural network can be a recurrent neural network, such as a long short-term memory network. Using a recurrent neural network, such as a long short-term memory network, a language recognition machine learning algorithm with high accuracy can be provided for data based on biologically relevant languages. However, other language recognition algorithms can also be used. For example, a trained language recognition machine learning algorithm can be an algorithm capable of handling input data of variable length (e.g., the Transformer-XL algorithm). For example, the length of the first biologically relevant language-based input dataset may differ from the length of the second biologically relevant language-based input dataset. For example, protein sequences are typically tens to hundreds of amino acids long (one amino acid is represented as a letter in a protein sequence). "Semantics" (e.g., the biological function of a substring in a sequence (called a polypeptide, motif, or domain in biology) can vary in length. Therefore, an architecture capable of accepting inputs of variable length can be used.

[0043] One or more processors 110 can be configured to compare a first high-dimensional representation with each of a plurality of second high-dimensional representations. The comparison can be performed by calculating the distance between the first and second high-dimensional representations. If the first and second high-dimensional representations are represented by vectors (e.g., normalized vectors), the distance between them (e.g., Euclidean distance or earth mover's distance) can be calculated effortlessly. This distance can be repeatedly calculated for each of the plurality of second high-dimensional representations. For example, comparing the first high-dimensional representation with each of the plurality of second high-dimensional representations can be done based on an Euclidean distance function or an earth mover's distance function. Based on the calculated distances, system 100 can select one or more second high-dimensional representations based on selection criteria (e.g., one or more second high-dimensional representations with the closest distance or within a distance threshold). For example, system 100 can be configured to select the second high-dimensional representation among the plurality of second high-dimensional representations that is closest to the first high-dimensional representation based on this comparison. System 100 can output or store one or more second high-dimensional representations that meet selection criteria, one or more biologically related image input datasets corresponding to the one or more second high-dimensional representations from multiple biologically related image input datasets, and / or one or more biologically related language input datasets corresponding to the one or more second high-dimensional representations from multiple biologically related language input datasets. For example, system 100 can output and / or store the closest second high-dimensional representation, the biologically related image input dataset corresponding to the closest second high-dimensional representation from multiple biologically related image input datasets, and / or the biologically related language input dataset corresponding to the closest second high-dimensional representation from multiple biologically related language input datasets.

[0044] Because a high-dimensional representation with several non-zero entries is used, two or more high-dimensional representations can be combined to perform retrieval for logical combinations of two or more search terms. For example, a user can input two or more search images and one or more logical operators (e.g., AND or NOT operators), and can combine the corresponding generated first high-dimensional representations based on the logical operators. For example, system 100 can be configured to receive second biologically related image-based retrieval data and information about logical operators. Further, system 100 can generate a first high-dimensional representation of the second biologically related image-based retrieval data using a trained language recognition machine learning algorithm executed by one or more processors. Additionally, system 100 can determine a combined high-dimensional representation based on a combination of the first and second high-dimensional representations of the first and second biologically related image-based retrieval data, according to logical operators. The combined high-dimensional representation can be a normalized high-dimensional representation (e.g., a normalized vector).

[0045] Furthermore, system 100 can compare the combined high-dimensional representation with each of the plurality of second high-dimensional representations. Based on the comparison between the combined high-dimensional representation and each of the plurality of second high-dimensional representations, one or more second high-dimensional representations can be selected based on selection criteria (e.g., one or more second high-dimensional representations with nearest distance or within a distance threshold).

[0046] System 100 can output or store one or more second high-dimensional representations that meet selection criteria, one or more biologically related image input datasets corresponding to the one or more second high-dimensional representations from multiple biologically related image input datasets, and / or one or more biologically related language input datasets corresponding to the one or more second high-dimensional representations from multiple biologically related language input datasets. The selected one or more biologically related image input datasets (e.g., biological images) or the selected one or more biologically related language input datasets (e.g., biological text) can illustrate or describe biological structures including logical combinations of search terms, such as those represented by first biologically related image retrieval data, second biologically related image retrieval data, and information about logical operators. In this way, retrieval for logical combinations of two or more retrieval images can be achieved. The logical operators can be AND, OR, or NOT operators. The NOT operator can suppress unwanted hits. The NOT operation can be determined by retrieval against negative search terms. For example, the embedding of the negative search term (e.g., the first high-dimensional representation) can be generated and inverted. Then, the k embeddings closest to the negative search term can be determined from multiple embeddings (multiple second high-dimensional representations) associated with the image, and said k embeddings are removed from the multiple embeddings. Optionally, the average of the remaining multiple embeddings (e.g., medoid or arithmetic mean) can be determined. This newly calculated second high-dimensional representation can be used for new queries in the embedding space to obtain more precise hits. The OR operation can be implemented by determining the closest or k closest elements (second high-dimensional representations) for each search term, where k is an integer between 2 and N. For example, all search terms connected by OR can be iterated and the closest hit or k closest hits can be output. Furthermore, several logical operators can be combined by parsing expressions and performing searches sequentially or from the inside out.

[0047] For example, the logical operator is the AND operator, and the combined high-dimensional representation is determined by adding and / or averaging a first high-dimensional representation of the first biologically relevant image-based retrieval data and a second biologically relevant image-based retrieval data. For example, the arithmetic mean of the first high-dimensional representation of the first biologically relevant image-based retrieval data and the second biologically relevant image-based retrieval data can be determined. For example, the arithmetic mean can be determined by the following formula:

[0048]

[0049] Where yi is the first high-dimensional representation, and N is the number of vectors to be averaged (e.g., the number of logically combined search terms). Determining the arithmetic mean yields a normalized high-dimensional representation. Alternatively, the geometric mean, harmonic mean, quadratic mean, or centroid can be used. Centroids can be used to avoid large errors in distributions with holes (e.g., closed regions with no data points). Centroids find the element closest to the mean. The centroid m can be defined as:

[0050]

[0051] Where y is the entire embedding (multiple second high-dimensional representations), and yi is one of the second high-dimensional representations. d is the embedding corresponding to the search term (the first high-dimensional representation), and d is a distance metric (e.g., Euclidean distance or L2 norm). For example, the element Y closest to the mean can be found, and then the k elements closest to the center can be determined (e.g., by using a quicksort algorithm).

[0052] As described above, the retrieval data 103 based on biologically relevant images can be of various types (e.g., images of biological structures including nucleotide or protein sequences or images of biological structures representing a set of coarse-grained search terms). A single visual recognition machine learning algorithm can be trained to process only one type of input. Therefore, system 100 can be configured to select a visual language recognition machine learning algorithm from multiple trained visual recognition machine learning algorithms based on the retrieval data 103 based on biologically relevant images. For example, multiple trained visual recognition machine learning algorithms can be stored by one or more storage devices 120, and system 100 can select one of the trained visual recognition machine learning algorithms based on the type of input received as the retrieval data 103 based on biologically relevant images. For example, the trained visual recognition machine learning algorithm can be selected from multiple trained visual recognition machine learning algorithms by a classification algorithm (e.g., a visual recognition machine learning algorithm) configured to classify the retrieval data 103 based on biologically relevant images.

[0053] System 100 can be implemented in a microscope, and can be connected to or include a microscope. The microscope can be configured to obtain biologically relevant image-based retrieval data 103 and / or multiple biologically relevant image-based input datasets by capturing images of one or more biological samples. The multiple biologically relevant image-based input datasets can be stored by one or more storage devices 120, and / or multiple biologically relevant image-based input datasets can be provided to generate multiple second high-dimensional representations.

[0054] Further details and aspects of System 100, combined with the proposed concepts and / or one or more examples described above or below (e.g., Figure 2 The following will be illustrated in Figure 7. System 100 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0055] Figure 2A schematic diagram of a system 200 for processing biologically relevant data according to an embodiment is shown. A user can begin a query 201 using images (e.g., retrieval data based on biologically relevant images) as images of biological structures, such as those containing specific protein or nucleotide sequences. For example, system 200 includes a visual model 220 (e.g., a CNN) trained with semantic embeddings from a textual model trained on: a large number of protein sequences (e.g., a protein sequence database); nucleotide sequences (e.g., a nucleotide sequence database); scientific publications (e.g., a database of biologically relevant publications); or other text describing the role and / or biological function of an object of interest, such as blog posts, research group homepages, online articles, forum posts, or social media posts. For example, as described below, visual model 220 has learned to predict these semantic embeddings during training, but other model training methods are also possible. User input 201 (e.g., query text) can first be classified into a corresponding category by visual model 210 (e.g., an image of a biological structure containing protein or nucleotide sequences), and system 200 can find the correct second visual model 230 for that category from a repository of such models containing one or more visual models required to process the category of the input text. The query image 201 is then converted into its corresponding embedding 260 (a first high-dimensional representation) using a forward pass through the corresponding pre-trained visual model 230 (a trained visual recognition machine learning algorithm). Image data in database 240 (e.g., a database 240 stored by one or more storage devices) or image data as part of an experiment run in a microscope can be converted into its corresponding embedding 250 (multiple second high-dimensional representations) via a forward pass through a pre-trained visual model 220. The pre-trained visual model 220 and the second visual model 230 can be the same visual model (a trained visual recognition machine learning algorithm). For example, for performance reasons, this part can be completed and stored in a suitable database 255 (e.g., a database stored by one or more storage devices) before the user query, or, for example, stored in a suitable database 255 along with the image data. Databases 240 and 255 can be equivalent or the same, but they can also be different databases. However, for a single or small number of images, such as in a running experiment, the forward propagation of the images can be completed during the run, thus bypassing the intermediate storage device 255 of the visual embedding 257. For example, the image repository 240 can represent a public or private database, or it can represent the storage medium of the microscope during the running experiment. The two generated embeddings (i.e., an embedding 260 for query text and an embedding 250 for images) can be compared 270 in the embedding space (e.g., their relative distance can be calculated).This comparison can be performed using different distance metrics, such as Euclidean distance or bulldozer distance. Other distance metrics (e.g., those used in clustering) can also be used. For example, the closest embedding 280 can be determined, and the corresponding image 290 can be found in repository 240 and returned to the user. The number of images to be returned can be predetermined by the user or calculated based on a distance threshold or other criteria. For example, retrieval of one or more closest embeddings can provide the k closest elements in multiple embeddings 250 (multiple second high-dimensional representations), where k is an integer. For example, the Euclidean distance (L2 norm) between the embedding of the retrieval query and all elements of the multiple embeddings 250 can be determined. The obtained distances (e.g., the same as the number of elements in the multiple embeddings) can be sorted, and the element with the minimum distance or the k elements with the k minimum distances can be output.

[0056] Further details and aspects of System 200, combined with the proposed concepts and / or one or more examples described above or below (e.g., Figure 1 The following is an illustration (Figures 3 to 7). System 200 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0057] Figure 3 illustrates a schematic diagram of a system 300 for processing biologically relevant data according to an embodiment. A user can begin a query 201 using an image (e.g., retrieval data based on biologically relevant images) as an image of a biological structure, for example, including a specific protein sequence or nucleotide sequence. Optionally, the query 201 can be pre-classified using a suitable classifier 210 (e.g., a neural network, statistical machine learning algorithm, depending on the input type). In some embodiments, pre-classification 315 can be skipped. The results of the pre-classification can be used to select a suitable model 230, which can convert the user query 201 into its relevant semantic embedding 260 using a pre-trained model 230 as a feature extractor.

[0058] User input and images from data source 240 are concatenated and processed within this semantic embedding space. Data source 240 can be a dedicated or public data repository or an imaging device such as a microscope. Data types can be images, text, coarse-grained search terms, or instrument-specific data recorded by the data source. For example, it can include a visual model 220 (e.g., a CNN) trained on the semantic embeddings of a textual model trained against: a large number of protein sequences (e.g., a protein sequence database); nucleotide sequences (e.g., a nucleotide sequence database); scientific publications (e.g., a database of biology-related publications); or other text describing the role and / or biological function of an object of interest, such as blog posts, research group homepages, online articles, forum posts, or social media posts. Visual model 220 may have been pre-trained to predict these semantic embeddings during training. For example, a first visual model 220 and an input feature extractor 230 (e.g., a second visual model) are both trained against the same embedding space. The first visual model 220 and the feature extractor 230 can be the same visual model (a trained visual recognition machine learning algorithm). The query 201 is then transformed into its corresponding embedding 260 using forward propagation through the input feature extractor 230. Data from a data source 240, which is either a database or part of an experiment running in a microscope, can be transformed into its corresponding embedding 250 via forward propagation through a pre-trained model 220 (visual model). For example, for performance reasons, this process can be completed before the user query, and the semantic embeddings are stored in a suitable database 255, or, for example, along with the image data. Databases 240 and 255 can be equivalent or identical, but they can also be different databases. However, for a single or small number of images, such as those used in an experiment, the forward propagation for the images can be completed during the run, bypassing the intermediate storage 255 for the visual embeddings 257. The two generated embeddings (i.e., an embedding 260 for the query and an embedding 250 for the data source) can now be compared 270 in the embedding space (e.g., their relative distance can be calculated). This comparison can be performed using different distance metrics, such as Euclidean distance or bulldozer distance. Other distance metrics can also be used. For example, distance metrics used in clustering can play a role.

[0059] System 300 can determine the closest embedding 280, locate the corresponding data (e.g., images) in repository 240 or in a run experiment, and return the corresponding data 381. Depending on the exact purpose of the embodiment, this final step may lead to different downstream processing steps. In some cases, it may be necessary to feed data (such as coordinates of objects found in a sample and stage coordinates) 383 to an image source (e.g., a microscope) that can alter the process of running the experiment. In some embodiments, the corresponding data may be output to user 385, who may decide to adjust the run experiment or further process the data. Other embodiments may archive the corresponding data in database 387 for future retrieval. Alternatively, still within the semantic embedding space, the corresponding data may be converted back to any input data type and may be used to query public database 389 to search scientific publications, social media entries or blog posts 390, images 393 of the same biomolecules, or biological sequences identified by sequence alignment 395. All found information may be returned to user 385 and / or written to database 387 as functional annotations of images recorded in the repository from which the searched data originated.

[0060] Figure 3 illustrates an example of image-to-image retrieval using image queries. In one embodiment, image repository 240 may represent a public or private database; in another embodiment, image repository 240 may represent the storage medium of a microscope used during an experiment.

[0061] Further details and aspects of System 300 are combined with the proposed concepts and / or one or more examples described above or below (e.g., Figure 1 , Figure 2 and Figure 4 This will be illustrated in Figure 7. System 300 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0062] Figure 4A schematic diagram of a system 400 for controlling a microscope according to an embodiment is shown. System 400 includes one or more processors 110 and one or more storage devices 120. System 400 is configured to receive image-based retrieval data 401 and to generate a first high-dimensional representation of the image-based retrieval data 401 by means of a trained visual recognition machine learning algorithm executed by one or more processors 110. The first high-dimensional representation includes at least three entries, each with a different value (or includes at least 20, at least 50, or at least 100 entries with values ​​different from each other). Further, system 400 is configured to obtain a plurality of second high-dimensional representations 405 of a plurality of image-based input datasets and is configured to select a second high-dimensional representation 405 from the plurality of second high-dimensional representations based on a comparison of the first high-dimensional representation performed by one or more processors 110 with each of the plurality of second high-dimensional representations 405. Additionally, system 400 is configured to provide a control signal 411 for controlling the operation of the microscope based on the selected second high-dimensional representation 405.

[0063] Image-based retrieval data 401 can be image data (e.g., pixel data of an image) of the sample to be analyzed. The sample to be analyzed can be a biological sample, an integrated circuit, or any other sample that can be imaged using a microscope. For example, if the sample is a biological sample, image-based retrieval data 401 can be images of: biological structures including nucleotides or nucleotide sequences; biological structures including proteins or protein sequences; biomolecules; biological tissues; biological structures with specific behaviors; and / or biological structures with specific biological functions or specific biological activities. For example, if the sample is an integrated circuit, image-based retrieval data 401 can be images of sub-circuits (e.g., memory cells, converter cells, ESD protection circuits), circuit elements (e.g., transistors, capacitors, or coils), or structural elements (e.g., gates, vias, pads, or gaskets).

[0064] Multiple second high-dimensional representations 405 can be obtained from a database or generated by a visual recognition machine learning algorithm. For example, system 400 can be configured to generate multiple second high-dimensional representations 405 of multiple image-based input datasets by a visual recognition machine learning algorithm executed by one or more processors 110.

[0065] The microscope can be configured to capture multiple images of a sample. Multiple image-based input datasets can represent multiple images of the sample. These multiple image-based input datasets can be image data of images of the sample captured by the microscope. For example, multiple images of the sample can be captured at different locations to cover the entire sample, or to cover a region of interest of the sample that is too large to be captured as a single image at the desired magnification. The image data of each of the multiple images can represent one of the multiple image-based input datasets. System 400 can be configured to store the location of the captured images. The location can be stored together with the corresponding image or with a corresponding second high-dimensional representation 405. System 400 can include a microscope, or a microscope can be connected to or include system 400.

[0066] System 400 can select a plurality of second high-dimensional representations that meet a selection criterion (e.g., the second high-dimensional representation closest to the first high-dimensional representation). A comparison of the first high-dimensional representation with each of the plurality of second high-dimensional representations can provide one or more second high-dimensional representations closest to the first high-dimensional representation. System 400 can be configured to select one or more second high-dimensional representations closest to the first high-dimensional representation based on this comparison.

[0067] System 400 can be configured to determine the location of a microscope target based on a selected second high-dimensional representation. The microscope target location can be the location of an image being captured, corresponding to the selected second high-dimensional representation. For example, the microscope target location can be a location stored together with the selected second high-dimensional representation or a location stored together with an image, corresponding to the selected second high-dimensional representation. Alternatively, the microscope target location can be the location of an image being captured, represented by image-based input data, and the location corresponding to the selected second high-dimensional representation.

[0068] System 400 can be configured to provide control signals to control the operation of the microscope based on a determined microscope target position. Control signal 411 can be an electrical signal provided to the microscope to control movement, magnification, light source selection, filter selection, and / or other microscope functions. For example, control signal 411 can be configured to trigger the microscope to drive to the microscope target position. For example, in response to control signal 411, the microscope optics and / or sample stage can be moved to the microscope target position. In this way, additional images of the sample can be taken at that position, which is the result of the retrieval. For example, images of the region of interest can be taken at higher magnification, different light sources, and / or different filters. For example, language-based retrieval data 405 can represent a retrieval of cell nuclei in a large biological sample, and system 400 can provide control signal 411 for driving the microscope to the cell nucleus position. If several cell nuclei are found, system 400 can be configured to provide control signal 411 to successively drive the microscope to different positions to take more images at those positions.

[0069] Further details and aspects of System 400 are combined with the proposed concepts and / or one or more examples described above or below (e.g., Figure 1 To Figure 3 and Figure 5 The following will be elaborated up to 7). System 400 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0070] Figure 5 A schematic diagram of a system 500 for controlling a microscope based on retrieval data from biologically relevant images, according to an embodiment, is shown. System 500 can be combined with... Figure 4 The described system is implemented similarly. System 500 is able to find images similar to user-provided query images and can modify the experimental setup. Microscope 501 can move the stage back to any position of the found similar images.

[0071] For example, a user can use an image as input (e.g., retrieval data based on biologically relevant images) to begin a query 550 and start an experiment. The user input can be passed through a pre-trained visual model 220 as described above or below. Forward propagation through this visual model 220 can create a semantic embedding (a first high-dimensional representation) of image 260. Microscope 501 can create a series of images 510 (e.g., a series type as defined above or below). Image 510 can be forward propagated through the same visual model 220 as before to create corresponding embeddings 250 (multiple second high-dimensional representations). 270 The distances between these subsequent embeddings and one or more embeddings from the user query can be calculated. Among the recorded embeddings 250, similar images can be found, defined by thresholding the distances or by the number of pre-determined or automatically found retrieval results. 580 Their corresponding coordinates can be found and these coordinates can be transmitted back to the microscope 590, which can then modify the experiment to record those new coordinates 595. For example, details regarding the coordinate types and changes to the experiment are described above or below. Users can send multiple images for querying at the same time, instead of querying just one image.

[0072] In a variant of this embodiment, the query image 550 may not be manually entered by the user, but may be the result of another experiment on the same or another imaging device, which automatically triggers a query for that experiment. In another variant of this embodiment, the query image 550 may come from a database (e.g., as a result of a retrieval query, which may in turn have been manually entered or entered via an imaging device or laboratory equipment) and automatically trigger a query for that experiment.

[0073] Figure 5 An example of image-to-image retrieval for running experiments based on user-defined input image queries can be shown.

[0074] Further details and aspects of System 500 are combined with the proposed concepts and / or one or more examples described above or below (e.g., Figures 1 to 4 and Figures 6 to 11 The system 500 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0075] Figure 6A schematic diagram of a system for controlling a microscope according to an embodiment is shown. System 600 includes one or more processors 110 coupled to one or more storage devices 120. System 600 is configured to determine multiple clusters of multiple second high-dimensional representations 405 of multiple image-based input datasets using a clustering algorithm executed by the one or more processors 110. Further, system 600 is configured to determine a first high-dimensional representation of the cluster centers of the clusters within the multiple clusters, and is configured to select a second high-dimensional representation 405 from the multiple second high-dimensional representations based on a comparison of the first high-dimensional representation with each second high-dimensional representation 405 or a subset of the second high-dimensional representations 405. Additionally, system 600 is configured to provide a control signal 411 to control the operation of the microscope based on the selected second high-dimensional representation.

[0076] Clusters of second high-dimensional representations 405 can represent multiple second high-dimensional representations 405 that are small in distance from each other. For example, the distance between the second high-dimensional representations 405 of a cluster can be smaller than the distance to the second high-dimensional representations 405 of other clusters, and / or the second high-dimensional representation 405 of a cluster can include a distance to the cluster center of its own cluster that is smaller than the distance to the cluster centers of any other cluster in the plurality of clusters. Each of the plurality of clusters can include at least 5 (or at least 10, at least 20, or at least 50) second high-dimensional representations 405.

[0077] Clustering algorithms can be or can include machine learning algorithms, such as k-means clustering, mean-shift clustering, k-centroid clustering, support vector machine, random forest, or gradient boosting.

[0078] System 600 can determine a first high-dimensional representation of the cluster centers for each of multiple clusters. System 600 can determine the first high-dimensional representation of the cluster centers, for example, by computing a second high-dimensional representation of the cluster, a linear combination of the second high-dimensional representations that minimize the total distance to all second high-dimensional representations of the cluster, or a nonlinear combination of the second high-dimensional representations of the cluster.

[0079] System 600 can be configured to generate multiple second high-dimensional representations of multiple image-based input datasets through a visual recognition machine learning algorithm executed by one or more processors 110.

[0080] System 600 can be configured to select one or more second high-dimensional representations from a plurality of second high-dimensional representations that are closest to the first high-dimensional representation based on the comparison.

[0081] System 600 can be configured to determine the microscope target location based on a selected second high-dimensional representation. The microscope target location can be the location of an image being captured, represented by image-based input data corresponding to the selected second high-dimensional representation. A control signal can be configured to trigger the microscope to drive it to the microscope target location.

[0082] System 600 may also include a microscope configured to capture multiple images of the sample. Multiple image-based input datasets can represent multiple images of the sample.

[0083] Further details and aspects of System 600 are combined with the proposed concepts and / or one or more examples described above or below (e.g., Figures 1 to 5 and Figure 7a to Figure 11 The system 600 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0084] Figure 7a illustrates a schematic diagram of a system 700 according to an embodiment that controls a microscope based on retrieval data from biologically relevant images using a clustering algorithm. System 700 can be combined with... Figure 6The described system is implemented similarly. Microscope 501 can generate a series of images 510, the coordinates of which are stored. A pre-trained visual model 220, as described below, can compute corresponding embeddings 250 (e.g., latent vectors, multiple second high-dimensional representations) via forward propagation. The resulting set of embeddings 250 can be clustered using a suitable clustering algorithm 740, such as k-means clustering, mean-shift clustering, or other algorithms. For each cluster, centers 750 can be determined by computing combinations of the corresponding latent vectors 250. For example, linear combinations (linear combinations of the second high-dimensional representations of the clusters) can be used. Alternatively, other combinations, including nonlinear combinations, can be applied. In this way, cluster centers 760, which are themselves latent vectors, can be obtained. By applying a suitable distance metric as described above or below, image retrieval 770 can be performed on the acquired series of images 510 to obtain those images whose embeddings are most similar to the found cluster centers. Similarity thresholds can be calculated automatically, provided by the user, and / or obtained and / or refined by showing the search results to the user and allowing the user to select desired images. The coordinates of the refined search results can be obtained at 580 and transmitted back to the microscope at 590, allowing the microscope to modify the experiment to record new images at those coordinates at 595. For any hardware parameters available to the microscope (e.g., one or more or all parameters), those new images can have the same or different instrument settings as the previous ones (e.g., different illumination or detection settings, different objectives, zoom, etc.). User interaction is possible at all steps 580, 590, and 595, where the user can optionally refine the search results, or decide which coordinates to acquire, which imaging modalities to use, which category of images to acquire, and which category of images to ignore.

[0085] Coordinates in the sense described above can be stage position (lateral position), timestamp, z-position (axial position), illumination wavelength, detection wavelength, mirror position (e.g., mirror position in a light sheet microscope), number of iterations in a loop, logical position in the sample (e.g., a hole in a well plate or a position defined in a multi-position experiment), time gate in time-gated recording, nanosecond timestamp in a fluorescence lifetime image, and / or any other hardware parameters available to the microscope that can record a series of images along its dimensions.

[0086] Figure 7a illustrates an example of image-to-image retrieval used to query the running experiment by employing unsupervised clustering with semantic embeddings.

[0087] Further details and aspects of System 700 are combined with the proposed concepts and / or one or more examples described above or below (e.g., Figures 1 to 6 and Figure 7b to Figure 11The system 700 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0088] Figure 7b illustrates a schematic diagram of a system 790 according to an embodiment for processing biologically relevant data using a clustering algorithm. System 790 can be combined with... Figure 6 And / or similarly implemented as the system described in Figure 7a.

[0089] Microscope 501 can generate a series of images 510, which are passed through a pre-trained visual model 220 to compute semantic embeddings 250. This is similar to combining... Figure 6 The semantic embeddings 250 are clustered in the manner described in and / or Figure 7a. Any new clusters or outliers, as defined by an item count threshold or a distance measurement threshold, can be identified by a suitable clustering algorithm 740 (such as k-means clustering, mean-shift clustering, or other algorithms). For example, one of four actions or a combination of these actions can be taken subsequently. The coordinates of the new clusters can be sent to the microscope to change the currently running experiment and alter the image modality 791, for example, as described in conjunction with Figure 7a. Alternatively or additionally, the image corresponding to the cluster of the newly discovered semantic embeddings can be returned to the user 792, who can then change the currently running experiment or decide on other actions to take. Alternatively or additionally, the newly discovered embeddings and their corresponding images and metadata can be stored as annotations in a repository 793 for future retrieval. Alternatively or concurrently, the semantic embeddings of newly discovered clusters can be translated into biological sequences, natural language, or coarse-grained search terms, and can be used to query public databases 794 to search for scientific publications, social media entries or blog posts 795, images of the same biomolecule 796, or biological sequences identified, such as those identified by sequence alignment 797. All found information can be returned to the user and / or written to the database as functional annotations for images recorded in currently running experiments.

[0090] System 790 can identify new structures of interest (e.g., phenotypes).

[0091] According to one aspect, clustering can be performed during recording. In this way, images corresponding to various categories of biological phenotypes can be identified. Examples of these images (e.g., images determined by the k-means of clustering by k-centroids) can be presented to the user. The user can identify which phenotypes are included in the sample. The user can save time manually retrieving these phenotypes and can also obtain descriptive statistics on the frequency of occurrence of these phenotypes. Furthermore, irrelevant categories of phenotypes or experimental artifacts can be detected and omitted in detailed recordings (e.g., at higher resolution or according to time series). In this way, time spent on recording and subsequent data analysis can be saved.

[0092] From one perspective, existing data (e.g., instead of using images from running experiments) can be analyzed through unsupervised clustering based on its stored semantic embeddings. In this way, existing categories can be detected. These categories can be added to the database as annotations and can be further used for future retrievals.

[0093] According to one aspect, the data from running the experiment can be classified and further processed through unsupervised clustering (e.g., as shown in Figure 7a).

[0094] Further details and aspects of System 790, combined with the proposed concepts and / or one or more examples described above or below (e.g., Figure 1 To Figure 7a and Figures 8 to 11 The system 790 may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0095] Combination Figure 1The system described in any one of Figures 7b may include or may be a computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) wherein one or more processors and one or more storage devices are located in the computer device, or the system may be a distributed computing system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations (e.g., local clients and one or more remote service clusters and / or data centers)). The system may include a data processing system that includes a system bus for connecting the various components of the system. The system bus can provide communication links between the various components of the system and can be implemented as a single bus, a combination of buses, or in any other suitable manner. Electronic components may be coupled to the system bus. Electronic components may contain any circuitry or combination of circuits. In one embodiment, the electronic component may contain a processor of any type. As used herein, a processor can mean any type of computing circuit, such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field-programmable gate array (FPGA) of a microscope or microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that can be included in an electronic component can be custom circuits, application-specific integrated circuits (AS1C), etc., such as one or more circuits (e.g., communication circuits) used in wireless devices like mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The system includes one or more storage devices, which may in turn include one or more storage elements suitable for a particular application (e.g., main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives that process removable media (e.g., optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). The system may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touch screen, voice recognition device, or any other device that allows system users to input information into and receive information from the system.

[0096] Additionally, the system may include a microscope connected to a computer device or a distributed computing system. The microscope can be configured to generate an input dataset based on biologically relevant images by capturing images of one or more samples.

[0097] The microscope can be an optical microscope (e.g., a diffraction-limited or sub-diffraction-limited microscope, such as a super-resolution microscope or a nanomicroscope). The microscope can be a stand-alone microscope or a microscope system with attached components (e.g., a confocal scanner, an additional camera, a laser, a climate chamber, an autoloading mechanism, a liquid handling system, attached optical components (such as an additional multiphoton beam path, optical tweezers, etc.)). For example, other image sources can also be used if they can acquire images of objects associated with biological sequences (e.g., proteins, nucleic acids, lipids) or other samples. For example, the microscope according to the embodiments described above or below can enable depth-finding microscopy.

[0098] Further details and aspects of the system are combined with the proposed concepts and / or one or more examples described above or below (e.g., Figures 1 to 11 The system may include one or more additional optional features corresponding to one or more aspects of the proposed concept and / or one or more aspects of one or more examples described above or below.

[0099] Some embodiments relate to a microscope that includes, as in combination with Figure 1 To the system described in one or more figures in Figure 7b. Alternatively, the microscope can be as combined with Figure 1 To be part of or connected to one or more of the systems described in Figure 7b. Figure 8 A schematic diagram of a system 800 for processing data according to an embodiment is shown. A microscope 810, configured to capture images of one or more samples (e.g., biological samples or integrated circuits), is connected to a computer device 820 (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) configured to process data. The microscope 810 and the computer device 820 can be combined as follows. Figure 1 Implement as shown in one or more of the figures in Figure 7b.

[0100] Figure 9 A flowchart of a method for processing retrieval data based on biologically relevant images, according to an embodiment, is shown. Method 900 includes receiving 910 retrieval data based on biologically relevant images and generating 920 a first high-dimensional representation of the retrieval data based on biologically relevant images using a trained visual recognition machine learning algorithm. The first high-dimensional representation includes at least three entries, each with a different value. Further, method 900 includes obtaining 930 multiple second high-dimensional representations of multiple input datasets based on biologically relevant images or multiple input datasets based on biologically relevant languages. Additionally, method 900 includes comparing 940 the first high-dimensional representation with each of the multiple second high-dimensional representations.

[0101] By using visual recognition machine learning algorithms, image-based retrieval requests can be mapped to high-dimensional representations. By allowing the high-dimensional representations to have entries with various distinct values ​​(compared to one-hot encoded representations), semantically similar biological search terms can be mapped to similar high-dimensional representations. By obtaining high-dimensional representations of multiple input datasets based on biologically relevant images or multiple input datasets based on biologically relevant languages, high-dimensional representations that are identical or similar to the high-dimensional representation of the retrieval request can be found. In this way, it becomes possible to find images or text corresponding to the retrieval request. In this manner, trained visual recognition machine learning algorithms can retrieve biologically relevant images from multiple biological images (e.g., a biological image database) or biologically relevant texts from multiple biological texts (e.g., a collection or repository of scientific papers) based on image-based retrieval input. Even if the images have not been previously labeled or tagged, retrieval can be achieved from existing databases or images generated by running experiments (e.g., images of one or more biological samples taken by a microscope).

[0102] Further details and aspects of Method 900 combine the proposed ideas and / or one or more examples described above or below (e.g., Figure 1 This is illustrated in Figure 7b). Method 900 may include one or more additional optional features corresponding to one or more aspects of the proposed idea and / or one or more aspects of one or more examples described above or below.

[0103] Figure 10 A flowchart of a method for controlling a microscope according to an embodiment is shown. Method 1000 includes receiving 1010 image-based retrieval data and generating 1020 a first high-dimensional representation of the image-based retrieval data using a trained visual recognition machine learning algorithm. The first high-dimensional representation includes at least three entries, each with a different value. Further, method 1000 includes obtaining 1030 multiple second high-dimensional representations of a plurality of image-based input datasets and selecting 1040 second high-dimensional representations from the plurality of second high-dimensional representations based on a comparison of the first high-dimensional representation with each of the plurality of second high-dimensional representations. Additionally, method 1000 includes controlling 1050 the operation of the microscope based on the selected second high-dimensional representation.

[0104] By using visual recognition machine learning algorithms, image-based retrieval requests can be mapped to high-dimensional representations. By allowing the high-dimensional representations to have entries with various values ​​(compared to one-hot encoded representations), semantically similar search terms can be mapped to similar high-dimensional representations. By obtaining high-dimensional representations from multiple image-based input datasets, high-dimensional representations that are identical or similar to the high-dimensional representations of the search terms can be found. In this way, images corresponding to the retrieval request can be found. Using this information, a microscope can be driven to the corresponding location to capture images, enabling the capture of other images of the location of interest (e.g., images taken using higher magnification, different lighting, or filters). In this way, samples (e.g., biological samples or integrated circuits) can first be imaged at low magnification to find the location corresponding to the retrieval request, after which the location of interest can be analyzed in more detail.

[0105] Further details and aspects of Method 1000, combined with the proposed ideas and / or one or more examples described above or below (e.g., Figure 1 This is illustrated in Figure 7b). Method 1000 may include one or more additional optional features corresponding to one or more aspects of the proposed idea and / or one or more aspects of one or more examples described above or below.

[0106] Figure 11 A flowchart of another method for controlling a microscope according to an embodiment is shown. Method 1100 includes determining multiple clusters of multiple second high-dimensional representations of a plurality of image-based input datasets using a clustering algorithm, and determining a first high-dimensional representation of the cluster center of one of the plurality of clusters. Further, method 1100 includes selecting 1130 second high-dimensional representations from the plurality of second high-dimensional representations based on a comparison of the first high-dimensional representation with each of the plurality of second high-dimensional representations or a subset of the second high-dimensional representations. Additionally, method 1100 includes providing 1140 control signals for controlling the operation of the microscope based on the selected second high-dimensional representation.

[0107] By identifying clusters of second-dimensional representations, second-dimensional representations corresponding to semantically similar content can be grouped into a single cluster. By determining cluster centers and comparing and identifying one or more second-dimensional representations closest to the cluster centers, one or more images representing typical images of that cluster can be found. For example, different clusters may include second-dimensional representations corresponding to different feature parts of a biological sample (e.g., cytoplasm, nucleus, cytoskeleton). The system can provide control signals to move the microscope to a location where typical images of one or more clusters can be captured (e.g., to capture more images at that location with varying microscope parameters).

[0108] Further details and aspects of Method 1100 combine the proposed ideas and / or one or more examples described above or below (e.g., Figures 1 to 10 The method 1100 may include one or more additional optional features corresponding to one or more aspects of the proposed idea and / or one or more aspects of one or more examples described above or below.

[0109] The following describes one or more of the above embodiments (e.g., in combination with...). Figures 1 to 11 Examples of application and / or implementation details of one or more embodiments described in the figures.

[0110] Based on one aspect, an image-to-image retrieval function is proposed for databases or running microscopy experiments. The type of image-to-image retrieval can be based on the semantic embedding of the query created through a first-stage text model. A second-stage image model can associate these semantic embeddings with the image, thus connecting the image domain to the text domain. The relevance of the hits can be scored according to a distance metric in the semantic embedding space. This allows not only searching for exact matches but also searching for similar images with relevant semantics. In the context of biologically relevant semantics, this can refer to similar biological functions. One aspect allows for retrieving the entire sample in a running experiment and searching for images similar to the query image or previously unknown objects in the sample.

[0111] Biology, and microscopy in particular, generates vast amounts of data that are often poorly or unannotated. For example, it may only become clear in retrospect which annotations might be useful or which new biological discoveries were unknown at the time of the experiment. The focus may be on image data, but the proposed ideas may not necessarily be limited to it. For instance, images may transcend 2D pixel images and encompass multidimensional image tensors with three spatial dimensions, a temporal dimension, and other dimensions related to the physical properties of the fluorescent dyes used or the characteristics of the imaging system. According to one aspect, such data can be made accessible by allowing semantic retrieval of large amounts of image data stored in databases or as part of experiments run in the microscope. These experiments can be single, one-off events or part of long-term experiments such as screening activities.

[0112] Image-to-image retrieval can be implemented not only in databases but also within running experiments (e.g., the current sample) to retrieve images similar to the input query, transforming the sample into a searchable data resource. Alternatively, image-to-image retrieval can enable automatic clustering of images during experimentation and allow searching for all relevant images from the current sample, future samples, or an image repository. This can represent a knowledge discovery tool that can find rare events or previously unknown (e.g., objects not specified by the user) within a sample.

[0113] According to one aspect, image-to-image retrieval can be used to optionally query microscopes that serve as image sources during an experiment run and to modify that experiment run.

[0114] Other implementations of image-to-image retrieval using statistical machine learning (e.g., support vector machines, random forests, or gradient boosting) may have to rely on image features curated or designed by human experts. The high dimensionality of images can reduce the accuracy of such classic machine learning methods. According to one aspect of the proposed idea, deep learning (e.g., CNNs, capsule networks) can be used to extract image features, thereby automatically allowing the use of a large number of image features across multiple scales, which can improve the accuracy of image recognition. Furthermore, images can be mapped to semantic embeddings instead of one-hot encoded vectors, which can allow the discovery of previously unseen but similar images. Since significant morphological variations are found in images of biological samples, the proposed idea can achieve a higher hit rate than other methods, whether for hit events or miss events.

[0115] An example of image-to-image retrieval can be based on the following steps:

[0116] 1. A visual model trained to predict semantic token embeddings from images can convert a query image into its associated semantic embedding.

[0117] 2. The same visual model can also create corresponding embeddings of a series of images from imaging devices or databases.

[0118] 3. Based on the distance metric between the semantic embeddings of the query and the image in the embedding space, the most relevant and closest hits can be retrieved and scored.

[0119] 4. Optionally, during the run of the experiment, the physical coordinates of the hit items can be used to modify the experiment and start alternative recordings of the images at these coordinates.

[0120] The model can be trained as described below, but it can also be trained in different ways.

[0121] For example, four alternative ways to obtain the semantic embedding of a query (e.g., step 1 above) could be:

[0122] a) Manual input by the user.

[0123] b) Experimental results using (the same or other) imaging equipment.

[0124] c) From a database (e.g., by imaging or by manual or automated querying using another laboratory device)

[0125] d) Unsupervised clustering and arithmetic combination of image embeddings generated by imaging devices and models.

[0126] In one respect, users can query a database using images instead of text. All images in the database can be converted into embeddings using one or more pre-trained visual models (e.g., CNNs) as described above or below. These embeddings can be stored in the same or different databases along with the image data. User queries can be converted into embeddings through forward propagation through the same visual model. Using a suitable distance metric, the semantically (in the embedding space) closest images can be searched and returned. This comparison can be made using different distance metrics, such as Euclidean distance or bulldozer distance, but other distance metrics are also possible. Most distance metrics used in clustering can be applied.

[0127] For example, any image, whether provided by the user or acquired solely by the microscope during the experiment, can be used to discover semantically relevant images throughout the sample. Transformation and similarity retrieval can be performed in a similar manner to those described above. The data acquired by the microscope can be arranged such that logical coordinates or, for example, physical stage coordinates of each image within a mosaic (e.g., a set of images covering a field of view larger than the current field of view) can be associated with the image data.

[0128] Image-to-image retrieval can be used to query existing databases or data from run experiments with any image. Within the context of a run experiment, any image acquired by the microscope can be used to query databases to find similar images. Additional information can be found through further annotations of the image, and new insights into the structure and function of the image in question can be gained. This transforms the microscope into a smart lab assistant that enhances image data with semantic and functional information, thereby aiding in data interpretation.

[0129] In one respect, images similar to user-provided or recorded images can be found throughout the sample. A retrievable amount of images can be recorded using pre-scanning with a microscope. Pre-scanning can cover an area or volume larger than the current field of view. Query images can be provided by the user, selected by the user from the current experiment, or automatically selected by a pre-trained visual model. This saves time because only the locations of interest can be recorded in detail under different imaging conditions and modalities (e.g., more colors, different magnifications, additional lifetime information, etc.). This also saves storage space because only images of interest can be stored. Other images can be discarded.

[0130] Alternatively, automated clustering can be performed, and the microscope can assist users in gaining new insights by indicating which distinct semantic categories exist within the sample. By automating the pre-scanning and clustering steps, users can save significant time previously spent manually searching, identifying, and characterizing all objects (e.g., single cells, organs, tissues, organoids, and their parts). Moreover, because the semantic embedding space serves as an objective measure of similarity, bias can be eliminated, and since embeddings are created based on biologically relevant textual data, this directly associates images with meaningful biological information.

[0131] In fact, the proposed microscope can be used to convert samples into searchable data resources.

[0132] The proposed image-to-image retrieval can be applied to basic biological research (e.g., to help find relevant data and reduce experimental recording time) and / or hit validation and toxicological analysis in drug discovery.

[0133] The trained language recognition machine learning algorithm and / or the trained visual recognition machine learning algorithm can be obtained through training as described below. A system for training a machine learning algorithm for processing biologically relevant data may include one or more processors and one or more storage devices. The system may be configured to receive input training data based on biologically relevant language. Additionally, the system may be configured to generate a first high-dimensional representation of the input training data based on biologically relevant language using a language recognition machine learning algorithm executed by one or more processors. The first high-dimensional representation includes at least three entries, each with a different value. Further, the system may be configured to generate output training data based on biologically relevant language based on the first high-dimensional representation using a language recognition machine learning algorithm executed by one or more processors. Additionally, the system may be configured to adjust the language recognition machine learning algorithm based on a comparison between the input training data based on biologically relevant language and the output training data based on biologically relevant language. Additionally, the system may be configured to receive input training data based on biologically relevant images associated with the input training data based on biologically relevant language. Further, the system may be configured to generate a second high-dimensional representation of the input training data based on biologically relevant images using a visual recognition machine learning algorithm executed by one or more processors. The second high-dimensional representation includes at least three entries, each with a different value. Furthermore, the system can be configured to adjust the visual recognition machine learning algorithm based on a comparison between the first high-dimensional representation and the second high-dimensional representation.

[0134] Biologically relevant language-based input training data can be textual input related to biological structures, functions, behaviors, or activities. For example, it can be nucleotide sequences, protein sequences, descriptions of biomolecules or structures, descriptions of the behavior of biomolecules or structures, and / or descriptions of biological functions or activities. The biologically relevant language-based input training data can be the first biologically relevant language-based input training dataset in a training set (e.g., input character sequences, such as nucleotide or protein sequences). A training set can include multiple biologically relevant language-based input training datasets.

[0135] The output training data based on biologically relevant language can be of the same type as the input training data based on biologically relevant language, which optionally includes predictions of the next element. For example, the input training data based on biologically relevant language can be a biological sequence (e.g., a nucleotide sequence or a protein sequence), and the output training data based on biologically relevant language can also be a biological sequence (e.g., a nucleotide sequence or a protein sequence). A language recognition machine learning algorithm can be trained such that the output training data based on biologically relevant language is equivalent to the input training data based on biologically relevant language, which optionally includes predictions of the next element of the biological sequence. In another example, the input training data based on biologically relevant language can be the biological category of a coarse-grained search term, and the output training data based on biologically relevant language can also be the biological category of a coarse-grained search term.

[0136] The input training data based on biologically relevant images can be image training data (e.g., pixel data of training images) of the following: biological structures including nucleotides or nucleotide sequences; biological structures including proteins or protein sequences; biomolecules; biological tissues; biological structures with specific behaviors; and / or biological structures with specific biological functions or activities. The input training data based on biologically relevant images can be the first input training dataset based on biologically relevant images in a training set. A training set can include multiple input training datasets based on biologically relevant images.

[0137] The input training data based on biologically relevant languages ​​can be a training dataset of biologically relevant languages ​​(e.g., input character sequences, such as nucleotide sequences or protein sequences). The training dataset can include multiple input training datasets based on biologically relevant languages. The system can repeatedly generate a first high-dimensional representation for each of the multiple input training datasets based on biologically relevant languages ​​in the training dataset. Further, the system can generate output training data based on biologically relevant languages ​​for each generated first high-dimensional representation. The system can adjust the language recognition machine learning algorithm based on each comparison between the input training data based on biologically relevant languages ​​in the multiple input training datasets based on biologically relevant languages ​​and the corresponding output training data based on biologically relevant languages. In other words, the system can be configured to repeatedly generate a first high-dimensional representation, generate output training data based on biologically relevant languages, and adjust the language recognition machine learning algorithm for each input training dataset of biologically relevant languages ​​in the training dataset. The training dataset can include sufficient input training datasets based on biologically relevant languages ​​to achieve the training objective (e.g., the output change of the loss function is below a threshold).

[0138] The multiple first-dimensional representations generated during the training of a language recognition machine learning algorithm can be called a latent space or semantic space.

[0139] This system can repeatedly generate a second high-dimensional representation for each of the multiple biologically relevant image-based input training datasets in the training group. Furthermore, the system can adjust the visual recognition machine learning algorithm based on each comparison between the first high-dimensional representation and the corresponding second high-dimensional representation. In other words, the system can repeatedly generate a second high-dimensional representation and adjust the visual recognition machine learning algorithm for each biologically relevant image-based input training dataset in the training group. The training group can include a sufficient number of biologically relevant image-based input training datasets to achieve the training objective (e.g., the output of the loss function changes below a threshold).

[0140] For example, system 100 uses a combination of speech recognition machine learning algorithms and visual recognition machine learning algorithms (e.g., also known as a visual semantic model). The speech recognition machine learning algorithms and / or visual recognition machine learning algorithms can be deep learning algorithms and / or artificial intelligence algorithms.

[0141] Training can converge quickly, and / or language recognition machine learning algorithms can be trained using the cross-entropy loss function (but other loss functions can also be used), which can provide well-trained algorithms for biologically relevant data.

[0142] The visual recognition machine learning algorithm can be trained by adjusting its parameters based on a comparison between a high-dimensional representation generated by a speech recognition machine learning algorithm and a high-dimensional representation generated by a visual recognition machine learning algorithm from the corresponding input training data. For example, the network weights of a visual recognition neural network can be adjusted based on this comparison. The adjustment of the parameters (e.g., network weights) of the visual recognition machine learning algorithm can be done with consideration of a loss function. For example, the comparison between a first and a second high-dimensional representation used to adjust the visual recognition machine learning algorithm can be based on a cosine similarity loss function. Training can converge quickly, and / or the visual recognition machine learning algorithm can be trained using a cosine similarity loss function (but other loss functions may also be used), providing a well-trained algorithm for biologically relevant data.

[0143] For example, a visual model can learn how to represent an image (e.g., as a vector) in a semantic embedding space. Therefore, a measure of the distance between two vectors can be used, which can represent the prediction A (a second high-dimensional representation) and the true situation B (a first high-dimensional representation). For example, the measure is cosine similarity as defined below:

[0144]

[0145] This involves dividing the dot product of prediction A and the actual situation B by the dot product of their respective sizes (e.g., in the L2 norm or Euclidean norm).

[0146] Further details regarding the non-training-specific aspects of systems used to train machine learning algorithms will be provided in conjunction with the proposed ideas and / or one or more examples described above or below (e.g., Figures 1 to 11 This will be explained in more detail.

[0147] Implementation examples can be based on the use of machine learning models or machine learning algorithms. Machine learning can refer to algorithms and statistical models that allow a computer system to perform a specific task without explicit instructions, relying instead on models and inference. For example, in machine learning, data transformations inferred from the analysis of historical and / or training data can be used instead of rule-based data transformations. For example, machine learning models or algorithms can be used to analyze the content of images. To enable a machine learning model to analyze image content, it can be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model “learns” to recognize the content of images, and thus can be used to recognize the content of images not included in the training data. The same principle can also be used for other kinds of sensor data: by training a machine learning model using training sensor data and a desired output, the machine learning model “learns” a transformation between sensor data and output that can be used to provide output based on non-training sensor data provided to the machine learning model.

[0148] Machine learning models can be trained using training input data. The examples detailed above use a training method known as "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, where each sample can include multiple input data values ​​and multiple expected output values; that is, each training sample is associated with an expected output value. By specifying training samples and expected output values, the machine learning model "learns" which output value to provide based on input samples similar to those provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms, such as classification algorithms, regression algorithms, or similarity learning algorithms. Classification algorithms can be used when the output is restricted to a finite set of values; that is, the input is classified into one of a finite set of values. Regression algorithms can be used when the output can have any numerical value (within a certain range). Similarity learning algorithms can be similar to classification and regression algorithms, but are based on learning from examples using a similarity function that measures the similarity or relevance of two objects. Besides supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, input data can be provided (only) and unsupervised learning algorithms can be used to find structure within the input data, for example, by grouping or clustering the input data to find commonalities. Clustering involves assigning input data, which includes multiple input values, into subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, but are dissimilar to input values ​​contained in other clusters.

[0149] Reinforcement learning is the third group of machine learning algorithms. In other words, reinforcement learning can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Rewards are calculated based on the actions taken. Reinforcement learning is based on training one or more software agents to select actions in order to increase cumulative rewards, thereby making the software agents better at a given task (as demonstrated by the increasing rewards).

[0150] Furthermore, certain techniques can be applied to some machine learning algorithms. For example, feature learning can be used. In other words, a machine learning model can be trained, at least in part, using feature learning, and / or a machine learning algorithm can include a feature learning component. Feature learning algorithms (also known as representation learning algorithms) can retain information from their inputs but are also able to transform that information in a way that makes it useful; this transformation is often used as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0151] In some examples, anomaly detection (i.e., outlier detection) can be used to identify input values ​​that are significantly different from the majority of the input or training data and thus raise suspicion. In other words, machine learning models can be trained using anomaly detection at least in part, and / or machine learning algorithms can include anomaly detection components.

[0152] In some examples, machine learning algorithms can use decision trees as predictive models. In other words, machine learning models can be based on decision trees. In a decision tree, observations about a particular item (e.g., a set of input values) can be represented by branches of the decision tree, while the output value corresponding to that item can be represented by leaves. Decision trees can support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree can be represented as a classification tree; if continuous values ​​are used, the decision tree can be represented as a regression tree.

[0153] Association rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or utilize one or more relationship rules, which represent knowledge derived from the data. These rules can be used, for example, to store, manipulate, or apply that knowledge.

[0154] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" can refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" can refer to, for example, a set of data structures and / or rules representing the learned knowledge based on training performed by a machine learning algorithm. In embodiments, the use of a machine learning algorithm can mean the use of a base machine learning model (or multiple base machine learning models). The use of a machine learning model can mean that the machine learning model and / or the set of data structures / rules serving as the machine learning model were trained by a machine learning algorithm.

[0155] For example, a machine learning model can be an artificial neural network (ANN). An ANN is a system inspired by biological neural networks (such as those found in the retina or brain). An ANN consists of multiple interconnected nodes and multiple connections between nodes, known as edges. There are typically three types of nodes: input nodes that receive input values, hidden nodes that are connected to other nodes (only), and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can transfer information from one node to another. The output of a node can be defined as a (non-linear) function of the sum of its inputs. The inputs to a node can be used in this function based on the "weights" of the edges or the "weights" of the nodes providing the input. The weights of nodes and / or edges can be adjusted during the learning process. In other words, training an artificial neural network can involve adjusting the weights of the nodes and / or edges of the artificial neural network to achieve the desired output for a given input.

[0156] Alternatively, the machine learning model can be a Support Vector Machine (SVM), a Random Forest model, or a Gradient Boosting model. A Support Vector Machine (SVM) is a supervised learning model with associated learning algorithms that can be used to analyze data, for example, in classification or regression analysis. An SVM can be trained by providing multiple training input values ​​belonging to one of two categories. The SVM can then be trained to assign new input values ​​to one of the two categories. Alternatively, the machine learning model can be a Bayesian network, a type of probabilistic directed acyclic graphical model. A Bayesian network can represent a set of random variables and their conditional dependencies using a directed acyclic graph. Alternatively, the machine learning model can be based on a genetic algorithm, a retrieval algorithm and heuristic technique that mimics the process of natural selection.

[0157] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated to “ / ”.

[0158] Although some aspects have been described in the context of apparatus, it is apparent that these aspects also represent a description of the corresponding method, where blocks or devices correspond to method steps or features of method steps. Similarly, some aspects described in the context of method steps also represent a description of corresponding blocks or items or features of the corresponding apparatus. Some or all of the steps in the method steps may be performed by (or using) hardware devices (e.g., processors, microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices.

[0159] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. The above implementations can be executed using non-transitory storage media such as digital storage media (e.g., floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or FLASH memories) that store electronically readable control signals thereon, which cooperate (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0160] Some embodiments of the invention include a data carrier having electronically readable control signals that are capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0161] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. For example, the program code may be stored on a machine-readable medium. For example, the computer program may be stored on a non-transitory storage medium. Some embodiments relate to a non-transitory storage medium containing machine-readable instructions that, when executed, implement a method according to the proposed concept or one or more of the examples above.

[0162] Other embodiments include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0163] In other words, embodiments of the present invention are therefore computer programs having program code that, when run on a computer, performs one of the methods described herein.

[0164] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon, which, when executed by a processor, is used to perform one of the methods described herein. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0165] Therefore, another embodiment of the invention is a data stream or signal sequence representing a computer program for performing one of the methods described herein. This data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0166] Another embodiment includes a processing device, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0167] Another embodiment includes a computer on which a computer program is installed for performing one of the methods described herein.

[0168] Another embodiment of the invention includes an apparatus or system configured to transmit (e.g., electronically or optically) to a receiver a computer program for performing one of the methods described herein. The receiver may be, for example, a computer, mobile device, storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0169] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0170] List of reference numerals

[0171] 100 Systems for processing biologically relevant data

[0172] 103 Retrieval data based on biologically relevant images

[0173] 105 Second-highest dimension representation

[0174] 110 One or more processors

[0175] 120 One or more storage devices

[0176] 200 Systems for Processing Biological Data

[0177] 201 Query, Retrieval Query, Retrieval Data Based on Biologically Related Images

[0178] 210 Visual models, classifiers

[0179] 220 trained visual recognition machine learning algorithms and visual models

[0180] 230 Trained visual recognition machine learning algorithms and visual models

[0181] 240 Database

[0182] 250 embeddings, multiple second-highest-dimensional representations

[0183] 255 Databases and intermediate storage devices

[0184] 257 Bypass

[0185] 260 Embedding, First High-Dimensional Representation

[0186] 270 Comparisons in Embedded Space

[0187] 280 closest embeddings

[0188] 290 Corresponding image

[0189] 300 Systems for processing biologically relevant data

[0190] 315 Skipped Pre-classification

[0191] 381 Returns the image corresponding to the closest embedding.

[0192] 383 feeds data to the image source

[0193] 385 users

[0194] 387 Database

[0195] 389 Public Database

[0196] 390 scientific publications, social media entries, or blog posts

[0197] 393 Images of biomolecules

[0198] 395 biological sequences

[0199] 400 Systems for Controlling Microscopes

[0200] 401 Image-based retrieval data

[0201] 405 Second higher dimension representation

[0202] 411 Control Signal

[0203] 500 Systems for Controlling Microscopes

[0204] 501 Microscope

[0205] 510 images

[0206] 550 queries, retrieval queries, image-based data retrieval

[0207] 580 Find the corresponding coordinates

[0208] 590. Transmit the corresponding coordinates back to the microscope.

[0209] 595 Corresponding coordinates, new coordinates

[0210] 600 Systems for Controlling Microscopes

[0211] 700 System for Controlling Microscopes

[0212] 740 Clustering Algorithm

[0213] 750 Determining the cluster centers

[0214] Latent vectors of 760 cluster centers

[0215] 770 Using distance metrics

[0216] 790 Systems for processing biologically relevant data using clustering algorithms

[0217] 791 Change image modality

[0218] 792 users

[0219] 793 Repositories

[0220] 794 Public Database

[0221] 795 scientific publications, social media entries, or blog posts

[0222] Images of 796 biomolecules

[0223] 797 biological sequences

[0224] 800 Systems for Training Machine Learning Algorithms

[0225] 810 Microscope

[0226] 820 Computer Equipment

[0227] 900 Methods for processing retrieval data based on biologically relevant images

[0228] 910 Receives retrieval data based on biologically relevant images

[0229] 920 Generate the first high-dimensional representation

[0230] 930 obtained multiple second-highest-dimensional representations

[0231] 940. Compare the first high-dimensional representation with each second high-dimensional representation.

[0232] 1000 Methods for controlling microscopes

[0233] 1010 Receives image-based retrieval data

[0234] 1020 Generate the first high-dimensional representation

[0235] 1030 yields multiple second-highest-dimensional representations.

[0236] 1040 Choosing the second higher dimension representation

[0237] 1050 Controlling the operation of the microscope

[0238] 1100 Methods for controlling microscopes

[0239] 1110 Identify multiple clusters

[0240] 1120 Determine the first high-dimensional representation

[0241] 1130 Choosing the second higher dimension representation

[0242] 1140 provides control signals.

Claims

1. A system comprising one or more processors (110) and one or more storage devices (120), wherein, The system is configured to: receive biology-related image-based search data (103); generate, by a trained visual recognition machine learning algorithm executed by the one or more processors (110), a first high-dimensional representation (260) of the biology-related image-based search data (103), wherein the first high-dimensional representation (260) includes at least 3 entries each having a different value, wherein more than 50% of the values of the entries of the first high-dimensional representation are not equal to 0, wherein the values of one or more entries of the first high-dimensional representation are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; obtain a plurality of second high-dimensional representations (105, 250) of a plurality of biology-related image-based input data sets or a plurality of biology-related language-based input data sets, wherein more than 50% of the values of the entries of the second high-dimensional representations are not equal to 0, wherein the values of one or more entries of the second high-dimensional representations (105, 250) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; compare the first high-dimensional representation (260) to each of the plurality of second high-dimensional representations (105, 250); select, based on the comparison, a second high-dimensional representation of the plurality of second high-dimensional representations (105, 250) that is closest to the first high-dimensional representation (260); and output at least one of: the closest second high-dimensional representation, a biology-related image-based input data set of the plurality of biology-related image-based input data sets that corresponds to the closest second high-dimensional representation, or a biology-related language-based input data set of the plurality of biology-related language-based input data sets that corresponds to the closest second high-dimensional representation.

2. The system of claim 1, wherein, The biology-related image-based search data (103) is image data of an image of at least one of: a biological structure that includes a nucleotide sequence; a biological structure that includes a protein sequence; a biological molecule; a biological tissue; a biological structure that has a particular behavior; or a biological structure that has a particular biological function or a particular biological activity.

3. The system of claim 1 or 2, wherein, The comparison of the first high-dimensional representation (260) to each of the plurality of second high-dimensional representations (105, 250) is based on a Euclidean distance function or a bulldozer distance function.

4. The system of claim 1 or 2, wherein, The first high-dimensional representation (260) and the second high-dimensional representations (105, 250) are numerical representations.

5. The system of claim 1 or 2, wherein, The first high-dimensional representation (260) and the second high-dimensional representations (105, 250) each include more than 100 dimensions.

6. The system of claim 1 or 2, wherein, The first high-dimensional representation (260) is a first vector and the second high-dimensional representations (105, 250) are second vectors.

7. The system of claim 1 or 2, wherein, a value of 5 or more entries of the first high-dimensional representation (260) is greater than 10% of a maximum absolute value of the entries of the first high-dimensional representation (260), and a value of 5 or more entries of each second high-dimensional representation of the plurality of second high-dimensional representations (105, 250) is greater than 10% of a respective maximum absolute value of the entries of the second high-dimensional representation (105, 250).

8. The system of claim 1 or 2, wherein, The trained visual recognition machine learning algorithm comprises a trained visual recognition neural network.

9. The system of claim 8, wherein, The trained visual recognition neural network comprises 30 or more layers.

10. The system of claim 8, wherein, The trained visual recognition neural network is a convolutional neural network or a capsule network.

11. The system of claim 8, wherein, The trained visual recognition neural network comprises a plurality of convolutional layers and a plurality of pooling layers.

12. The system of claim 8, wherein, The trained visual recognition neural network uses a rectified linear unit activation function.

13. The system of claim 1 or 2, wherein, The system is configured to obtain the second high-dimensional representation (105, 250) by generating, by the trained visual recognition machine learning algorithm executed by the one or more processors, a second high-dimensional representation (105, 250) of the plurality of second high-dimensional representations (105, 250) of the plurality of biology-related image-based input data sets or the plurality of biology-related language-based input data sets, wherein each second high-dimensional representation of the plurality of second high-dimensional representations (105, 250) comprises at least 3 entries each having a different value.

14. The system of claim 1 or 2, further comprising a microscope (501, 810) configured to obtain the plurality of biology-related image-based input data sets by taking images of biological samples.

15. The system of claim 1 or 2, wherein, The system is configured to select the trained visual recognition machine learning algorithm from a plurality of trained visual recognition machine learning algorithms based on the biology-related image-based search data (103).

16. The system of claim 1 or 2, wherein, The system is configured to: receive second biology-related image-based search data and information about a logical operator; generate, by the trained visual recognition machine learning algorithm executed by the one or more processors (110), a first high-dimensional representation of the second biology-related image-based search data; determine, according to the logical operator, a combined high-dimensional representation based on a combination of the first high-dimensional representation (260) of the first biology-related image-based search data (103) and the first high-dimensional representation of the second biology-related image-based search data; and compare the combined high-dimensional representation to each second high-dimensional representation of the plurality of second high-dimensional representations (105, 250). The logical operator is an AND operator and the combined high-dimensional representation is determined by adding the first high-dimensional representation (260) of the first biology-related image-based search data (103) and the first high-dimensional representation of the second biology-related image-based search data.

17. The system of claim 16, wherein, The system is configured to control operation of a microscope (501, 810).

18. The system of claim 1 or 2, wherein, The system is configured to:

19. A system comprising one or more processors (110) and one or more storage devices (120), wherein, receive image-based search data (401); ​ generating, by a trained visual recognition machine learning algorithm executed by the one or more processors (110), a first high-dimensional representation of the image-based search data (401), wherein the first high-dimensional representation comprises at least 3 entries each having a different value, wherein more than 50% of the values of the entries of the first high-dimensional representation are not equal to 0, wherein the values of one or more entries of the first high-dimensional representation (260) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; obtaining a plurality of second high-dimensional representations (405) of a plurality of image-based input data sets, wherein more than 50% of the values of the entries of the second high-dimensional representations are not equal to 0, wherein the values of one or more entries of the second high-dimensional representations (105, 250) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; selecting, based on a comparison of the first high-dimensional representation with each of the plurality of second high-dimensional representations (405), a second high-dimensional representation (405) of the plurality of second high-dimensional representations that is closest to the first high-dimensional representation (260); providing a control signal (411) for controlling an operation of a microscope (501, 810) based on the selected second high-dimensional representation; and outputting at least one of: the closest second high-dimensional representation, an image-based input data set of the plurality of image-based input data sets corresponding to the closest second high-dimensional representation, or a biology-related language-based input data set of the plurality of biology-related language-based input data sets corresponding to the closest second high-dimensional representation.

20. A system comprising one or more processors (110) and one or more storage devices (120), wherein, the system is configured to: determining, by a clustering algorithm executed by the one or more processors (110), a plurality of clusters of the plurality of second high-dimensional representations (405) of a plurality of image-based input data sets, wherein more than 50% of the values of the entries of the second high-dimensional representations are not equal to 0, wherein the values of one or more entries of the second high-dimensional representations (105, 250) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; determining a first high-dimensional representation of a cluster center of a cluster of the plurality of clusters, wherein more than 50% of the values of the entries of the first high-dimensional representation are not equal to 0, wherein the values of one or more entries of the first high-dimensional representation (260) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; selecting, based on a comparison of the first high-dimensional representation with each of the plurality of second high-dimensional representations (405) or a subset of the second high-dimensional representations (405), a second high-dimensional representation (405) of the plurality of second high-dimensional representations that is closest to the first high-dimensional representation (260); providing a control signal (411) for controlling an operation of a microscope based on the selected second high-dimensional representation; and outputting at least one of: the closest second high-dimensional representation, an image-based input data set of the plurality of image-based input data sets corresponding to the closest second high-dimensional representation, or a biology-related language-based input data set of the plurality of biology-related language-based input data sets corresponding to the closest second high-dimensional representation. of the plurality of biologically relevant language-based input data sets corresponds to the closest second high-dimensional representation.

21. The system of claim 20, wherein, The clustering algorithm comprises a k-means clustering algorithm or a mean shift clustering algorithm.

22. The system of any one of claims 19-21, wherein, The system is configured to determine a microscope target position based on the selected second high-dimensional representation, wherein the microscope target position is a position at which an image is to be taken, the image being represented by an image-based input data, and the position corresponding to the selected second high-dimensional representation, wherein the control signal is configured to trigger the microscope to drive to the microscope target position.

23. The system of any one of claims 19-21, wherein, The system is configured to generate the plurality of second high-dimensional representations of the plurality of image-based input data sets by a visual recognition machine learning algorithm executed by the one or more processors (110).

24. The system of any one of claims 19-21, wherein, The system is configured to select a second high-dimensional representation of the plurality of second high-dimensional representations that is closest to the first high-dimensional representation based on the comparison.

25. The system of any one of claims 19-21, further comprising the microscope configured to take a plurality of images of the sample, wherein, The plurality of image-based input data sets represent the plurality of images of the sample.

26. A microscope comprising the system of any one of the preceding claims 1 to 25.

27. A method (900) for processing biologically relevant image-based search data, the method comprising: receiving (910) biologically relevant image-based search data; generating (920), by a trained visual recognition machine learning algorithm, a first high-dimensional representation of the biologically relevant image-based search data, wherein the first high-dimensional representation comprises at least 3 entries each having a different value, wherein more than 50% of the values of the entries of the first high-dimensional representation are not equal to 0, wherein a value of one or more entries of the first high-dimensional representation (260) is proportional to a likelihood of a presence of a specific biological function or a specific biological activity; obtaining (930) a plurality of second high-dimensional representations of a plurality of biologically relevant image-based input data sets or a plurality of biologically relevant language-based input data sets, wherein more than 50% of the values of the entries of the second high-dimensional representation are not equal to 0, wherein a value of one or more entries of the second high-dimensional representation (105, 250) is proportional to a likelihood of a presence of a specific biological function or a specific biological activity; comparing (940) the first high-dimensional representation to each of the plurality of second high-dimensional representations; selecting a second high-dimensional representation of the plurality of second high-dimensional representations (105, 250) that is closest to the first high-dimensional representation (260) based on the comparison; and outputting at least one of: the closest second high-dimensional representation, a biologically relevant image-based input data set of the plurality of biologically relevant image-based input data sets that corresponds to the closest second high-dimensional representation, or a biologically relevant language-based input data set of the plurality of biologically relevant language-based input data sets that corresponds to the closest second high-dimensional representation.

28. A method (1000) for controlling a microscope, the method comprising: receiving (1010) image-based search data; generating (1020), by a trained visual recognition machine learning algorithm, a first high-dimensional representation of the image-based search data, wherein the first high-dimensional representation includes at least 3 entries each having a different value, wherein more than 50% of the values of the entries of the first high-dimensional representation are not equal to 0, wherein the values of one or more entries of the first high-dimensional representation (260) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; obtaining (1030) a plurality of second high-dimensional representations of a plurality of image-based input data sets, wherein more than 50% of the values of the entries of the second high-dimensional representation are not equal to 0, wherein the values of one or more entries of the second high-dimensional representation (105, 250) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; selecting (1040), from the plurality of second high-dimensional representations, a second high-dimensional representation that is closest to the first high-dimensional representation (260) based on a comparison of the first high-dimensional representation to each of the plurality of second high-dimensional representations; controlling (1050) an operation of a microscope based on the selected second high-dimensional representation; and outputting at least one of: the closest second high-dimensional representation, an image-based input data set of the plurality of image-based input data sets that corresponds to the closest second high-dimensional representation, or a language-based input data set of the plurality of language-based input data sets that corresponds to the closest second high-dimensional representation.

29. A method (1100) for controlling a microscope, the method comprising: determining (1110), by a clustering algorithm, a plurality of clusters of a plurality of second high-dimensional representations of a plurality of image-based input data sets, wherein more than 50% of the values of the entries of the second high-dimensional representation are not equal to 0, wherein the values of one or more entries of the second high-dimensional representation (105, 250) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; determining (1120) a first high-dimensional representation of a cluster center of a cluster of the plurality of clusters, wherein more than 50% of the values of the entries of the first high-dimensional representation are not equal to 0, wherein the values of one or more entries of the first high-dimensional representation (260) are proportional to a likelihood of a presence of a particular biological function or a particular biological activity; selecting (1130), from the plurality of second high-dimensional representations, a second high-dimensional representation that is closest to the first high-dimensional representation (260) based on a comparison of the first high-dimensional representation to each of the plurality of second high-dimensional representations or a subset of the second high-dimensional representations; providing (1140) a control signal for controlling an operation of a microscope based on the selected second high-dimensional representation; and outputting at least one of: the closest second high-dimensional representation, an image-based input data set of the plurality of image-based input data sets that corresponds to the closest second high-dimensional representation, or a language-based input data set of the plurality of language-based input data sets that corresponds to the closest second high-dimensional representation. the plurality of biology-related language-based input data sets corresponding to the closest second high-dimensional representation.

30. A computer program product having program code configured to perform the method of any one of claims 27 to 29 when the program code is executed by a processor.

Images