Apparatus for extracting information about images, laboratory automation and image analysis workflows from natural language to automate and improve reproducibility of life science experiments

Abstract

A method of generating an instrument setting includes providing a scientific publication as input, using a pre-trained language model on the input to extract an embedding of a key and a corresponding value associated with a setting of an instrument. The method also includes generating an instrument setting based on the embedding and the corresponding value, the instrument setting causing the instrument to perform a task.

Description

Background Technology

[0001] Reproducibility is crucial in scientific research, ensuring the reliability of data acquired and analyzed using publicly available acquisition parameters and analytical methods. For example, in the field of biomedical imaging, experiments are currently conducted based on manual user input. Due to the complexity of operating biomedical imaging instruments and data analysis methods, setting up and even reproducing experiments requires background knowledge and experience from the user. Furthermore, reproduction is particularly difficult because not all scientific publications adhere to standardized forms of described data acquisition and analysis methods. In fact, the use of synonyms, jargon, or vendor-specific terminology adds further challenges to experimental reproduction. Therefore, improved experimental automation and image analysis workflows may be necessary to enhance experimental reproducibility. Attached Figure Description

[0002] The following description will use only examples of devices and / or methods, with reference to the accompanying drawings, in which...

[0003] Figure 1 A flowchart illustrating an example of a method for generating instrument settings is shown;

[0004] Figure 2 An example of the default mode is shown;

[0005] Figure 3 shows another example of a flowchart of a method for generating instrument settings;

[0006] Figure 4 A flowchart illustrating an example of a training method for a language model used in an instrument is shown;

[0007] Figure 5 A schematic diagram of a system for performing biomedical imaging experiments is shown;

[0008] Figure 6 A schematic diagram of a system for running biomedical imaging experiments is shown. Detailed Implementation

[0009] Some examples will now be described in more detail with reference to the accompanying drawings. However, other possible examples are not limited to the features of these embodiments described in detail. Other examples may include modifications to features, as well as equivalents and alternatives to features. Furthermore, the terminology used herein to describe certain examples should not limit other possible examples.

[0010] Throughout the description of the accompanying drawings, the same or similar reference numerals refer to the same or similar elements and / or features, which may be implemented in the same or modified form while providing the same or similar function. For clarity, the thickness of lines, layers, and / or areas in the drawings may also be exaggerated.

[0011] When the word "or" is used to combine two elements A and B, it should be understood to disclose all possible combinations, namely, A only, B only, and A and B, unless otherwise explicitly defined in individual cases. As alternative wording for the same combination, "at least one of A and B" or "A and / or B" can be used. This is equivalent to combinations of more than two elements.

[0012] If the singular forms such as “a,” “an,” and “the” are used, and the use of a single element is not explicitly or implicitly defined as mandatory, other examples may use several elements to implement the same function. If the function is described below as being implemented using multiple elements, other examples may use a single element or a single processing entity to implement the same function. It should also be understood that the terms “include,” “including,” “comprise,” and / or “comprising”, when used, describe the presence of a specified feature, integer, step, operation, process, element, component, and / or group thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components, and / or groups thereof.

[0013] Figure 1 A flowchart illustrating an example of a method 100 for generating instrument settings is shown. Method 100 includes providing scientific publications as input 110, using a pre-trained language model 120 to extract embeddings of keys and corresponding values ​​associated with the instrument settings, and generating the instrument settings 130 based on the embeddings and corresponding values. The instrument settings enable the instrument to perform tasks when applied to it.

[0014] Optionally, the key describes the instrument's parameters, and the value represents the setting of the instrument's parameters.

[0015] Embeddings in large language models can be understood as representations of words, phrases, or sentences in a continuous vector space. Embeddings capture the semantic meaning and contextual relationships between words. Typically, each word or symbol in the input text is transformed into a high-dimensional vector, where similar words are represented by vectors that are closer together in the vector space. Embeddings are learned during the training process of the language model, where the model adjusts the vectors to minimize the differences between similar words and maximize the differences between different words based on the context in which they appear. This process enables the model to understand the meaning of words and their relationships within sentences, allowing it to generate coherent and context-sensitive responses. As described above, the proposed method 100 is based on a pre-trained language model for learning the semantics of biomedical terms, instrument parameters and associated values, as well as information about specimens and sample preparation. Figure 4 The training of the language model is described in the text.

[0016] For example, the pre-trained language model may optionally be a large language model, but advantageously it is an encoder-decoder type with a transformer architecture. In this context, the encoder-decoder type can be understood as a framework designed to handle various types of input-output mappings. The encoder processes the input data and transforms it into a fixed-size representation or context vector. The decoder takes the context vector produced by the encoder and generates the output sequence from it. The transformer architecture follows the encoder-decoder type by processing sequential data (such as text) to map the global dependencies between the input and output.

[0017] Modern biomedical instruments may require extensive background knowledge across multiple fields to operate the instruments and associated systems and analyze the data acquired using them. For example, imaging with a microscope requires understanding the principles of the microscope, such as magnification, resolution, contrast, light source, sample type, sample preparation, and post-acquisition data processing. Therefore, setting all parameters to properly use the microscope for experiments can be a time-consuming process, and even for experts in the field, the possibility of human error exists when operating the instrument.

[0018] In scientific research, experimental results are communicated through journal articles and conference papers. Depending on the methods used, these results need to be reproduced by different research groups. Currently, automated systems in the research field simplify and optimize all aspects of data acquisition, analysis, and control processes. They leverage artificial intelligence (AI) and robotics to improve efficiency, accuracy, and reproducibility in experiments by reducing the need for human intervention and minimizing human error. For example, the automation of microscopy enables high-throughput screening, real-time monitoring, and precise control of experimental conditions, thereby accelerating research discovery, improving diagnostic workflows, and enhancing productivity in scientific and clinical settings.

[0019] However, verifying the reproducibility of published results is typically done manually, relying on free-text descriptions of the methods. Unfortunately, these descriptions often lack standardized formats, leading to the use of synonyms, jargon, or vendor-specific terminology to describe experimental procedures. Jargon may be understood as proprietary terminology within a specific profession or group, but can be challenging for those outside the field to comprehend. Understanding the entire acquisition workflow and data processing steps may require the involvement of experienced experts in the relevant field to reproduce the experiment accordingly. Therefore, automated methods or systems capable of generating instrument setups to reproduce experiments based on scientific publications may be needed.

[0020] Method 100 can facilitate the reproduction of experiments using publicly available scientific documents without user intervention to understand the methods in the provided scientific publications by providing scientific publications as input 110 and using a pre-trained language model 120 to extract embeddings of keys and corresponding values ​​associated with the settings of instrument 120.

[0021] When extracting key-value embeddings, key-value pairs may still contain synonyms or meronyms that need to be parsed. Entity parsing in the context of a language model can be understood as the process of identifying keys, link keys, and disambiguating keys across different parts of a text or across different texts to ensure consistency and accuracy in understanding and generating language.

[0022] Creating word embeddings involves mapping words or phrases to vectors of real numbers, known as embeddings. These embeddings capture the semantic meaning, relationships, and various linguistic features of the words. In this context, entity parsing can be understood as the process of solving a problem by semantically mapping a number of possible synonyms to a key based on vectors.

[0023] The latent space can be understood as an abstract high-dimensional space where a pre-trained language model learns to encode patterns from input data during training. Semantics is captured by creating vectors of key embeddings in the latent space. As an example, appropriate metrics such as cosine similarity are used to compare, for example, the resulting embeddings of synonyms to measure the angle between the vectors. The closest items are matched with corresponding items in the data patterns, and key-value pairs from a corrected list set are used to generate instrumentation setting 130 in the final instrumentation setting.

[0024] There are different practical possibilities for performing entity resolution. One example is using a size of... The sparse and long vectors are represented by a length equal to the vocabulary of the training dataset for the pre-trained language model. It can utilize embeddings based on term-frequency-inverse-document-frequency (TF-IDF). TF-IDF is a numerical statistic reflecting the importance of words to documents in a set or corpus. It is commonly used in text mining and information retrieval to evaluate the relevance of terms to documents within the context of a corpus, i.e., the frequency with which a term appears in a document. Inverse document frequency measures the importance of a term. When calculating TF, all terms are considered equally important. However, some terms, such as “is,” “of,” and “as stated,” may appear frequently but are not important. Therefore, these frequently occurring terms are weighted less, while rare terms are weighted more. In this example, each word is represented as a sparse and long vector of size [missing information]. Multiplying by D in terms of rows in the document matrix, where D is the number of input documents.

[0025] Alternatively, positive pointwise mutual information (PPMI) can be used. PPMI can be understood in natural language processing as a statistical measure that quantifies the association between two words, going beyond simple co-occurrence frequency. In PPMI, in this example, the interaction (context) of each word is expressed as a value of 1. Multiply Sparse term-term matrix representation. PPMI is widely used in natural language processing, especially in tasks involving semantic similarity for synonym detection and word embedding.

[0026] Another example is using arbitrary size. Short and dense vectors, where d can be, for example, less than 300, less than 400, less than 500, less than 700, or less than 1000. As an example, it uses static embeddings based on training a skip-gram classifier using negative sampling. Skip-gram models aim to predict context words given a target world, thereby learning representations that reflect language patterns and semantic similarities. Negative sampling is an optimization method that makes this training process more efficient and scalable. Secondly, dynamic embeddings can be used, such as, as an example, a bidirectional encoder representation from transformer (BERT) transformer with a masking language model objective. BERT can be understood as a method in natural language processing that is designed to understand the context of a word based on all its surroundings. In other words, the left and right sides of the word. Existing models typically read text unidirectionally (from left to right or from right to left), which can limit understanding. In contrast, BERT uses the transformer's attention mechanism to consider the complete context of a word by looking at words that appear before and after it, resulting in richer text representations.

[0027] Finally, entity resolution can be performed via retrieval augmented generation (RAG). An alternative to resolving entities in the latent space is to use database lookups. For this, specific tokens are inserted into cues given to a pre-trained language model; these tokens are picked up by an orchestration layer built into the application. This will use the entities in question and perform queries on available databases, such as Wikipedia, Wikidata, or domain-specific knowledge databases describing entities specific to imaging devices, laboratory automation systems, or image analysis software. Preferably, Wikidata is preferable to Wikipedia because it is structured for automated retrieval and has shorter descriptions and synonyms that can be looked up.

[0028] Examples of the method enable researchers or users to more easily reproduce published experiments by using scientific publications describing the experiment as input and by using a pre-trained language model on the input to extract embeddings of keys and corresponding values ​​associated with the instrument settings used in the experiment. Instrument settings for reproducing the experiment can then be generated based on the embeddings and corresponding values.

[0029] The generation and use of key embeddings can also enable instrument parameters from one microscope model or brand to be automatically mapped to different instrument settings when reproducing experiments. Because embeddings capture the semantic meaning and context of each parameter value (rather than being limited to rigid or model-specific naming), the system can transform or adapt extracted parameter data from the original publication (where the experiment may have used instruments of different brands or models, such as microscopes) to the corresponding set of values ​​for another instrument (such as a microscope) used to reproduce the experiment. This can be particularly beneficial when other instruments use different naming conventions, parameter ranges, or functional modules, as embeddings provide a flexible layer of abstraction that bridges these differences.

[0030] Method 100 may optionally also include using a default mode 111. The default mode includes at least a complete set of keys and associated values ​​for the instrument settings required for the instrument to perform the task. Therefore, the default mode will enable the instrument to perform the task without user input.

[0031] The choice of method usually depends on the specific requirements of the application, the availability of labeled data, and the available computing resources.

[0032] Method 100 may optionally also include deriving missing embeddings 121 of keys and values ​​required for instrument settings rather than from user input, wherein missing keys and values ​​are derived using a default mode. There is a possibility that some parameters may not be extracted in step 120. Not all scientific publications provide a comprehensive description of all technical parameters of instruments and laboratory automation systems. This may be due to unintentional omissions, and because users of such devices may not have full visibility into all parts of the device or workflow in use, some parts of the device may be automatically configured, or the user may lack the technical expertise required to know all parts. In such cases, a pre-trained language model may not be able to simply guess the missing keys and values, as incorrect hardware parameter settings could lead to instrument damage or harm to the operating user.

[0033] For example, when searching for unknown keys or symbols in the input, special symbols such as UNK can be used to maintain a controlled word size. Words including the UNK symbol can be potential candidates for missing values. The encoder part of the pre-trained language model generates latent vectors from the unknown symbols and then compares them with latent vectors created from all other symbols in the input (e.g., keys). A detailed explanation of the methods used to identify missing keys and values ​​will follow.

[0034] An exemplary method for deriving missing keys and values ​​is to extract them from the default data schema by taking the average between the upper and lower bounds of the respective parameter range. Alternatively, as an example, the p-th percentile between the upper and lower bounds of the missing keys in the default schema can be used to derive the missing keys and values.

[0035] Method 100 may optionally also include iteratively optimizing the keys and values ​​122 by comparing the extracted embeddings with the estimated key-value pairs and the embeddings of the publicly available data using forward inference. Even if the missing keys and values ​​are derived from the default mode, it may not be the optimal setting depending on the sample and instrument used. Forward inference can be understood as the reasoning process used in AI, where the reasoning process starts from known conditions to infer new conditions. In the context of optimizing keys and values, forward inference starts from a starting point derived from the default mode to infer optimized key-value pairs.

[0036] For example, a quantized variational autoencoder (qVAE) can be used to optimize estimated key-value pairs. A VAE can be understood as a generative model used in unsupervised learning. A VAE is designed to learn a latent representation of the input data, which can then be used to generate new data points similar to the training data. In a VAE, the latent space is typically continuous, indicating that each latent variable can take any value within a certain range. However, in a qVAE, the latent space is discretized into a finite number of values. This discretization can be understood as dividing the continuous latent space into a grid of discrete points. The discretization step in the training process can help improve interpretability, stability, or computational efficiency. Using the compressed latent space of the trained qVAE, images acquired using the extracted key-value embeddings and estimated key-value pairs are compared to images of the input data. After forward inference on both, the comparison is iteratively performed using a numerical metric of how closely the output matches the published data to obtain vector embeddings of the published and reproduced data, respectively. As an example, the metric could involve L1 norm, L2 norm, p-norm, or cosine similarity.

[0037] As described above, instrument settings 130 are generated based on embedded and corresponding values, wherein the instrument settings enable the instrument to perform tasks.

[0038] Method 100 may optionally include implementing instrument integrity constraints 131 by adjusting values ​​corresponding to one or more keys. As mentioned above, incorrect parameter settings may harm instrument hardware or even the instrument operator when iteratively optimizing keys and values ​​122. Therefore, for safety reasons, it may be necessary to generate instrument settings 130 after implementing instrument integrity constraints 131.

[0039] Method 100 may optionally also include using a pre-trained language model to extract further embeddings of keys and corresponding values ​​associated with the workflow of the data analysis system. Publications provided as input include the data analysis program and the instrumentation setup used for data acquisition. The extraction of key and corresponding value embeddings associated with the workflow of the data analysis system can further reduce human error in experiments.

[0040] As an example, missing key-value embeddings can also be derived from the default pattern. To optimize key-value pair estimations for workflows used in data analysis systems, such as in perceptual loss, pixel-based metrics (such as mean squared error, mean absolute error, or structural similarity index or PSNR), intermediate activations of convolutional neural networks (CNNs) can be used. CNNs are widely used for pattern recognition in gridded data such as images.

[0041] Method 100 optionally also includes operating the microscope 132 using instrument settings, which enables automated instrument operation for reproducibility experiments. This reduces the time spent manually setting up the instrument and human error.

[0042] Among other things, Figure 2 An exemplary default mode 210 of the instrument is shown. Figure 2An exemplary schematic diagram 200 of modes is shown, which includes not only instrument keys and values ​​(instrument mode 210) but also keys and values ​​for other aspects ultimately required to redo the experiment. As an example of this setup, Figure 200 shows instrument mode 210, analysis mode 220, experiment mode 230, and user mode 240. As an example, instrument mode 210 may include parameter names, their lower and upper limits, and units (if applicable), such as {"LED 488": {"Type": int, "Lower Limit": 0, "Upper Limit": 100, "Unit": "%"}210a, {"storage-x": {"Type": float32, "Lower Limit": 0, "Upper Limit": 65983.7, "Unit": "µm"}210b, ..., {"HyD": {...} 210x}. Analysis mode 220 may include parameters related to image analysis, such as {"Detection": {...} 220a, "Background": {"Type": int, ...} 220b, ..., "Contrast Threshold": {...} 220x}. Experimental mode 230 may include information about the species, model organism, model system (e.g., cell culture, organoids, 3D cell culture, tissue transplantation, tissue sections, and organs), staining and cleanup procedures (immunofluorescence, tagged fluorescent proteins, low molecular weight protein tags for fluorescence, HE staining). Optionally, user mode 240 may define further customized documentation by providing free text descriptions, such as {“Summary”: “Provides a concise summary of the methods section”, “Imaging Workflow”: “Returns a diagram of all microscopic imaging steps as a dictionary with keys corresponding to known instrument parameters and values ​​in the test”}. It is worth noting that this is an exemplary, non-exhaustive list of parameter names that can be used in a particular microscopy experiment, along with associated values ​​with upper and lower limits and units.

[0043] Optionally, hints can be partially automated using a RAG as described above. Default and user modes can be provided as nested dictionaries. Concatenations of such keys, such as {"photomultiplier tube": {"type": int, "lower limit": 0, "upper limit": 900, "gain": 700, "unit": "volts"}} concatenated as "photomultiplier tube volts", can be suitable for querying a database to retrieve a description of an element that can be used for inference. It is worth noting that it is not obvious which combinations produce a reasonable meaning. Therefore, all possible key combinations can be generated as search terms. Optionally, using prior knowledge, some keys can be omitted to reduce search time.

[0044] Figure 3 again illustrates an exemplary flowchart 300 for the method of generating instrument settings. In this case, instrument settings are generated for microscope 391 and data analysis system 392.

[0045] Default mode 301 defines the fields corresponding to instrument parameters, image analysis parameters, and experimental parameters, as well as the data type for each field, and optionally, it can be similar to... Figure 2 Provide user mode 302.

[0046] User input 310 provides the publication 303 to be reproduced. The publication 311 can be found through literature retrieval using experimental metadata or image features. In this context, metadata refers to the keys and associated values ​​used to acquire the image.

[0047] Use or prompt a pre-trained language model 320 to extract key-value pairs as information and output them as a nested dictionary 340, as described above. Figure 1 As described above. These extracted key-value pairs are placed on a graph. In this context, a graph can be understood as a data structure used to represent a set of objects, nodes, and the relationships or connections between these objects. Each node corresponds to an extracted key-value pair. In the simplest case, the graph is a linear list, which is a graph with a depth of 1. Therefore, a unique list of extracted key-value pairs exists. However, in other cases, alternatives may exist at multiple steps, where the graph has a depth > 1. For example, the publication of interest 303 may include multiple image acquisition parameter values ​​or multiple data analysis parameter values ​​for the same parameter for comparison or reference. In this case, it may be necessary to represent the graph as an adjacency list, and a set of lists representing different extracted key-value pairs 351 and 352 can be created.

[0048] Several data issues may exist, including missing keys and values, synonyms, use of jargon or vendor-specific terms, and duplication, which need to be addressed before generating instrument setups or image analysis workflows. Entity resolution and missing key estimation can be performed as described above. As an example, the synonyms “hybrid detector” 371 and “HyD” 372 can be included in the key-value pairs that need to be resolved. In entity resolution, semantics are captured by creating vector embeddings 360 of the keys to be compared using an appropriate similarity metric 370. In Figure 3, cosine similarity is presented as an example. The cosine metric calculates the cosine of the angle between two embedded vectors, reflecting how closely related the two embeddings are in their orientation, regardless of their magnitude.

[0049] Before applying instrument settings or data analysis settings to microscope 391 or data analysis system 392, implement integrity constraints 380 to avoid damage to systems 391 and 392 or injury to the operating user. Finally, the settings can be applied to imaging device 391, laboratory automation system, or data analysis system 392.

[0050] As an example, in Figure 3, prompt 320, or using a pre-trained language model 330 with user input 303, can be described as “find the experimental conditions related to the microscope and output a dictionary of key-value pairs. The values ​​can be nested dictionaries.” As a very simple example, it can be assumed that the user input includes the following description: “The gain on the photomultiplier tube (PMT) is set to 700 V. To achieve good time resolution, the highest available scan speed is used. Resonant scan mode is used with a line frequency of 12000 Hz and zoom 6, and a 63x HCS Plan APO objective is used.” After the steps described above (e.g., 340-380 in Figure 3), the final instrument setup could be “Output: PMT”: {“Gain”: 700}, “Resonant Scanner”: {“Speed”: 12000, “Zoom”: 6}, “Objective”: {“Magnification: 63x, “Planarity”: “Plan APO”}}”. This structured instrument setup can be sent to microscope 391 or any optical instrument to reproduce the experiment described in user input 303.

[0051] Figure 4 A flowchart of a training method 400 for a language model of an instrument is shown. Method 400 includes generating a training dataset 410 from the instrument's technical documentation and training a language-based model 420 using the training dataset. The language-based model is configured to generate embeddings of keys and associated values, where the keys describe the instrument's parameters and the values ​​represent the settings of those parameters.

[0052] As an example, refer to Figure 1 The key is a description of the parameters of biological or medical related terms, experimental instruments, laboratory automation systems and data analysis systems, or components of experimental instruments, laboratory automation systems and data analysis systems, including lighting sources, photodetectors, scanning devices, spectroscopic instruments, actuators and motors.

[0053] As an example, a language-based model can optionally be a medium-sized language model. A language model is a computational model trained to predict the probabilities of words or sequences of symbols in a language. Large language models involve hundreds of billions or even trillions of parameters to be trained on vast datasets, requiring enormous computational power for both training and inference. Medium-sized data models involve significantly fewer parameters, ranging from tens of millions to tens of billions. They are more manageable in terms of computational requirements while efficiently handling a wide range of natural language processing tasks.

[0054] As an example, the medium-sized language model can optionally be fine-tuned in terms of cues including thought chain strategies to improve prediction performance.

[0055] Fine-tuning a medium-sized language model involves slightly adjusting the parameters of a pre-trained model to specialize it for a specific task or domain. The chain-of-thought strategy can be understood as a technique used in natural language processing to solve complex problems by breaking them down into intermediate steps or reasoning paths. In the context of language models, chain-of-thought cues involve providing the model with a hint that not only poses the question but also suggests a way to solve it, typically by outlining a series of logical steps or considerations. The model then generates text that follows this suggested reasoning path, ideally leading to more accurate and interpretable conclusions.

[0056] Using thought chain hints to fine-tune medium-sized language models offers several advantages. The model generates intermediate steps or reasoning leading to the final answer, making the problem-solving process more transparent and understandable. Furthermore, by breaking down complex problems into simpler components, the model can more effectively handle tasks requiring multi-step reasoning. Users can create hints to guide the model in specific directions, customizing the reasoning process to suit the problem at hand.

[0057] For example, a pre-instruction could be given to a medium-sized language model, such as "Find the experimental conditions related to the microscope and output a list of key-value pairs." Then, two exemplary sub-hints involving thought chain reasoning could be provided as follows:

[0058] Method: The gain on the photomultiplier tube (PMT) is set to 700 V. Output: [("PMT", 700)]. Reasoning: A photomultiplier tube, abbreviated as "PMT", is a photoelectric sensor with amplified gain. Gain is measured in volts and abbreviated as "V". Therefore, the key is PMT, and the value is 700.

[0059] Method: To achieve good temporal resolution, the highest available scan speed was used. Resonant scanning mode was used with a line frequency of 12000 Hz and a zoom of 6. Output: [(“Resonant Scanner”, 12000), (“Zoom”, 6)]. Inference: A resonant scanner is a specific implementation of a scanner. A scanner is a device used to move a beam of light across a sample for measurement. The movement speed is measured as frequency, abbreviated as “Hz”. Zoom is achieved using a dimensionless numerical factor relative to the amplitude without zoom, depending on the scanner's amplitude.

[0060] In the fine-tuning step, based on prior instructions, keys (e.g., PMT, resonant scanner, zoom) and their associated values ​​(e.g., 700, 12000, 6) are extracted as output, and reasoning is presented in conjunction with the thought chain described above. The thought chain strategy reflects human cognitive processes, where complex problems are typically solved by breaking them down into simpler components and reasoning about them sequentially. Furthermore, by evaluating each step in the reasoning process, it helps minimize the propagation of errors.

[0061] As an example, method 400 may optionally also include querying a database to retrieve descriptions as an element of reasoning in the CoT strategy. The training dataset may not include all the terminology used in scientific experiments and research. In this case, databases describing entities specific to imaging devices, laboratory automation systems, or image analysis software, such as Wikipedia, Wikidata, or domain-specific knowledge databases, can be retrieved to improve reasoning in the CoT strategy.

[0062] Figure 5 A schematic diagram of an example of a system 500 for performing biomedical imaging experiments using publicly available scientific articles, laboratory notes, or technical documents is shown. The system includes one or more processors 510 and one or more storage devices 520. System 500 can be used to perform a variety of tasks. For example, system 500 can be configured to perform a combination of... Figure 1 Method 100 is shown. Alternatively or additionally, system 500 may be configured to perform a combination Figure 4 Method 400 is shown.

[0063] In conjunction with the proposed concept or one or more of the examples above (e.g., Figures 1 to 4 The document mentions further details and aspects of system 500. System 500 for performing biomedical imaging experiments includes one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above.

[0064] Figure 6 A schematic diagram of an example system 600 for running biomedical imaging experiments is shown. The system includes one or more processors 610 and one or more storage devices 620. The system is configured to acquire experimental information in the form of scientific publications, laboratory notes, or technical documents, and to use a language-based model to combine... Figure 1 The method 100 shown extracts key-value pair embeddings from the experimental information obtained. Based on the combination... Figure 4 The method 400 shown is used to train a language-based model. The system is configured to use key-value pairs to operate a biomedical imaging device, analysis system, or laboratory automation system.

[0065] In conjunction with the proposed concept or one or more of the examples above (e.g., Figures 1 to 4 Further details and aspects of system 600 are mentioned. System 600 for performing biomedical imaging experiments includes one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more examples above.

[0066] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0067] Other implementations include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0068] In other words, embodiments of the present invention are therefore computer programs having program code that, when run on a computer, performs one of the methods described herein.

[0069] The examples described in this article can be summarized as follows:

[0070] Examples (e.g., Example 1) relate to a method for generating instrument settings, comprising: providing a scientific publication as input; using a pre-trained language model on the input to extract embeddings of keys and corresponding values ​​associated with instrument settings; and generating instrument settings based on the embeddings and the corresponding values, the instrument settings enabling the instrument to perform a task.

[0071] Another example (e.g., Example 2) relates to the previously described example (e.g., Example 1) and further includes: providing a default mode that includes a complete set of keys and associated values ​​for instrument settings required for the instrument to perform a task.

[0072] Another example (e.g., Example 3) relates to the previously described examples (e.g., Example 1 or 2), where the key is a description of the parameters of the instrument, and the value represents the setting of the parameters of the instrument.

[0073] Another example (e.g., Example 4) relates to the previously described examples (e.g., Examples 1, 2, or 3) and also includes: further including: exporting missing embeddings of keys and values ​​not extracted from the user input but required by the instrument settings.

[0074] Another example (e.g., Example 5) relates to the previously described example (e.g., Example 4), where the missing keys and values ​​were exported using the default schema.

[0075] Another example (e.g., Example 6) relates to the previously described examples (e.g., Example 4 or 5) and further includes iteratively optimizing keys and values ​​by comparing the extracted embeddings with estimated key-value pairs and the embeddings of the publicly available data using forward inference.

[0076] Another example (e.g., Example 7) relates to the previously described example (e.g., Example 1) and also includes: implementing the instrument’s integrity constraints by adjusting the values ​​corresponding to one or more keys.

[0077] Another example (e.g., Example 8) relates to the previously described examples (e.g., Examples 1 to 7) and further includes: using the pre-trained language model to extract further embeddings of keys and corresponding values ​​associated with the workflow of the data analysis system.

[0078] Another example (e.g., Example 9) relates to the previously described examples (e.g., Examples 1 to 8), where the instrument settings are generated for a microscope.

[0079] Another example (e.g., Example 10) relates to the previously described examples (e.g., Example 9) and also includes: operating the microscope using the instrument settings (123).

[0080] Another example (e.g., Example 11) relates to a method for training a language model for an instrument, comprising: generating a training dataset based on technical documentation of the instrument; and training a language-based model using the training dataset, the language-based model being configured to generate embeddings of keys and associated values, the keys being descriptions of parameters of the instrument and the values ​​representing settings of the parameters of the instrument.

[0081] Another example (e.g., Example 12) relates to the previously described example (e.g., Example 11), wherein the key is a description of the parameters of a biological or medical related term, laboratory instrument, laboratory automation system and data analysis system, or a component of the laboratory instrument, laboratory automation system and data analysis system, the component including a lighting source, photodetector, scanning device, spectroscopic instrument, actuator and motor.

[0082] Another example (e.g., Example 13) relates to the previously described examples (e.g., Example 11 or 12), where the language-based model is a medium-sized language model.

[0083] Another example (e.g., Example 14) relates to the previously described examples (e.g., Examples 11 to 13), where the medium-sized language model is fine-tuned in terms of cues including thought chain strategies to improve predictive performance.

[0084] Another example (e.g., Example 15) relates to the previously described examples (e.g., Examples 11 to 14) and also includes: querying a database to retrieve a description as an element of reasoning in the said thought chain strategy.

[0085] Another example (e.g., Example 16) relates to a method for reproducing an experiment, comprising: providing a scientific publication describing the experiment as input; using a pre-trained language model on the input to extract embeddings of keys and corresponding values ​​associated with the setup of the instrument used in the experiment; and generating an instrument setup for reproducing the experiment based on the embeddings and the corresponding values.

[0086] Another example (e.g., Example 17) relates to the previously described example (e.g., Example 16) and further includes: using the generated instrument setup to control the instruments for reproducing the experiment, based on the experiment described in the scientific publication.

[0087] Another example (e.g., Example 18) relates to the previously described examples (e.g., Example 16 or 17), where the experiment is a biological experiment.

[0088] Another example (e.g., Example 19) relates to the previously described examples (e.g., Examples 16 to 18), where the instrument is a microscope.

[0089] Another example (e.g., Example 20) relates to a system for performing biomedical imaging experiments using publicly available scientific articles, laboratory notes, or technical documents, the system including one or more processors and one or more storage devices, wherein the system is configured to perform at least one of the methods according to any one of Examples 1 to 9 and any one of Examples 11 to 16.

[0090] Another example (e.g., Example 21) relates to a system for running biomedical imaging experiments, the system including one or more processors and one or more storage devices, wherein the system is configured to: acquire experimental information in the form of scientific publications, laboratory notes, or technical documents; extract key-value pair embeddings from the acquired experimental information using a language-based model trained according to any one of Examples 10 to 16, according to any one of Examples 1 to 9; and use the key-value pairs to operate a biomedical imaging device, analysis system, or laboratory automation system.

[0091] Another example (e.g., Example 22) relates to a computer program having program code that, when run on a processor, performs the method according to any one of Examples 1 to 16.

[0092] Aspects and features related to a specific example in the previous examples can also be combined with one or more other examples to replace the same or similar features of the other examples, or to introduce features additionally into the other examples.

[0093] Examples may also include (computer) programs comprising program code that, when executed on a computer, processor, or other programmable hardware component, performs one or more of the methods described above. Therefore, the steps, operations, or processes of the different methods described above may also be executed by a programmed computer, processor, or other programmable hardware component. Examples may also encompass program storage devices, such as digital data storage media, which are machine-readable, processor-readable, or computer-readable and are encoded and / or contain machine-executable, processor-executable, or computer-executable programs and instructions. For example, a program storage device may include or be a digital storage device, a magnetic storage medium such as a disk and tape, a hard disk drive, or an optically readable digital data storage medium. Other examples may include computers, processors, control units, (field-programmable arrays) ((F)PLAs), (field-programmable gate arrays) ((F)PGAs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), integrated circuits (ICs), or system-on-a-chip (SoC) systems programmed to perform the steps of the methods described above.

[0094] It should also be understood that, unless explicitly stated in individual cases or necessary for technical reasons, the disclosure of steps, processes, operations, or functions in the specification or claims should not be construed as implying that these operations necessarily depend on the described order. Therefore, the preceding description does not limit the execution of steps or functions to a particular order. Furthermore, in further examples, a single step, function, process, or operation may include and / or be decomposed into several sub-steps, sub-functions, sub-processes, or sub-operations.

[0095] If aspects of an apparatus or system have been described, these aspects should also be understood as descriptions of the corresponding method. For example, a block, device, or functional aspect of an apparatus or system may correspond to a feature of the corresponding method, such as method steps. Therefore, aspects of the method description should also be understood as descriptions of corresponding blocks, elements, characteristics, or functional features of the corresponding apparatus or system.

[0096] The appended claims are incorporated herein by reference in the detailed description, wherein each claim may be taken independently as a separate example. It should also be noted that while in the claims a dependent claim refers to a specific combination with one or more other claims, other examples may also include combinations of dependent claims with the subject matter of any other dependent or independent claim. Such combinations are explicitly stated herein unless it is stated in individual cases that a particular combination is not intended to be used. Furthermore, for any other independent claim, the features of the claim should also be included, even if the claim is not directly defined as dependent on the other independent claim.

Claims

1. A method (100) for generating instrument settings, comprising: Provide scientific publications as input (110); A pre-trained language model (120) is used on the input to extract the embeddings of keys and corresponding values ​​associated with the instrument settings; as well as An instrument setting (130) is generated based on the embedding and the corresponding value, which enables the instrument to perform a task.

2. The method according to claim 1, further comprising: A default mode (111) is provided, which includes a complete set of keys and associated values ​​for the instrument settings required for the instrument to perform a task.

3. The method according to claim 1 or 2, wherein the key is a description of the parameters of the instrument, and the value represents the setting of the parameters of the instrument.

4. The method according to any one of claims 1 to 3, further comprising: The export is not extracted from the user input but rather from the missing embeddings of the keys and values ​​required by the instrument settings (121).

5. The method of claim 4, wherein the missing keys and values ​​are derived using the default mode.

6. The method according to claim 4 or 5, further comprising: The keys and values ​​are iteratively optimized by using forward inference to compare the extracted embeddings with the estimated key-value pairs and the embeddings of the published data (122).

7. The method according to claim 1, further comprising: The integrity constraints of the instrument are implemented by adjusting the values ​​corresponding to one or more keys (131).

8. The method according to any one of claims 1 to 7, further comprising: The pre-trained language model is used to extract further embeddings of keys and corresponding values ​​associated with the workflow of the data analysis system (132).

9. The method according to any one of claims 1 to 8, wherein the instrument settings are generated for a microscope.

10. The method of claim 9, further comprising: Use the instrument settings (123) to operate the microscope.

11. A method (400) for training a language model for an instrument, comprising: A training dataset (410) is generated based on the instrument's technical documentation. as well as A language-based model (420) is trained using the training dataset, the language-based model being configured to generate embeddings of keys and associated values, the keys being descriptions of the instrument's parameters and the values ​​representing settings of the instrument's parameters.

12. The method of claim 10, wherein the key represents a description of parameters of biological or medical related terms, experimental instruments, laboratory automation systems, and data analysis systems, the experimental instruments, laboratory automation systems, and data analysis systems including lighting sources, photodetectors, scanning devices, spectroscopic instruments, actuators, and motors.

13. The method according to any one of claims 11 to 12, wherein the language-based model is a medium-sized language model.

14. The method according to any one of claims 11 to 13, wherein the medium-sized language model is fine-tuned in terms of cues including thought chain strategies to improve predictive performance.

15. The method according to any one of claims 11 to 14, further comprising: The database is queried to retrieve the description (430) as an element of reasoning in the thought chain strategy.

16. A method for reproducing an experiment, comprising: Provide scientific publications describing the experiment as input; A pre-trained language model is used on the input to extract the embeddings of keys and corresponding values ​​associated with the settings of the instruments used in the experiment; as well as Based on the embedding and the corresponding value, an instrument setting is generated for the instrument used to reproduce the experiment.

17. The method of claim 16, further comprising: Based on the experiment described in the scientific publication, the generated instrument settings are used to control the instruments used to reproduce the experiment.

18. The method according to claim 16 or 17, wherein the experiment is a biological experiment.

19. The method according to any one of claims 16 to 18, wherein the instrument is a microscope.

20. A system (500) for performing biomedical imaging experiments using publicly available scientific articles, laboratory notes, or technical documents, the system comprising one or more processors (510) and one or more storage devices (520), wherein the system is configured to perform at least one of the methods according to any one of claims 1 to 9 and any one of claims 11 to 16.

21. A system (600) for running biomedical imaging experiments, the system comprising one or more processors (610) and one or more storage devices (620), wherein the system is configured to: Obtain experimental information in the form of scientific publications, laboratory notes, or technical documents; A language-based model trained using the method according to any one of claims 11 to 15; and the method according to any one of claims 1 to 9 for extracting key-value pair embeddings from the obtained experimental information; and The key-value pairs are used to operate biomedical imaging devices, analysis systems, or laboratory automation systems.

22. A computer program having program code, wherein when the computer program is run on a processor, the program code is used to perform the method according to any one of claims 1 to 16.

Images