Microscope system and related program

Abstract

In a computer-implemented program for controlling a microscope, a text input (T) describing a desired microscope image and a sample (10) employed is received. The text input (T) of the employed sample and the overview image (20) are input into a large language model (LLM) that is trained that can process the text input (T) and the overview image (20) together to calculate (P4) a microscope setting (40) for capturing a microscope image (50) corresponding to the desired microscope image. Microscope images (50) are then captured with these calculated microscope settings (40).

Description

Technical Field

[0001] The present disclosure relates to a microscopy system and a program for providing desired microscopy images. Background Art

[0002] Modern microscopes are highly complex systems with a wide range of setup options. The microscope setup and components used must be selected based on the desired sample analysis. The ideal setup depends not only on the sample and sample preparation itself, but also on the analytical goals. For example, a microscopist may wish to analyze a population of biological cells within a sample, or analyze individual cells or specific organelles requiring illumination settings suitable for fluorochromes. Users typically begin by describing their planned imaging application from an application perspective—that is, what they want to image. However, setting the optimal imaging technique parameters to obtain the desired image data requires considerable expertise and experience. While microscopes increasingly offer automated processes to increase user comfort, translating the application description into imaging technique parameter settings remains a major obstacle for many microscopists. Due to a lack of expertise, after spending considerable time and effort setting the necessary settings, only marginal results are achieved.

[0003] Voice control systems for microscopes are known, for example voice-controlled surgical microscopes using machine learning language models, as described in CN112149606A. Voice commands are compared with stored relevant commands such as zoom or white balance. DE102020 / 108796A1 describes another voice-controlled surgical microscope, in which the surgeon can use voice commands to switch from one stage with predetermined microscope settings to the next stage with other predetermined microscope settings. Thus, in known (surgical) microscopes, the voice commands indicate specific microscope settings or a set of pre-saved microscope settings, so that the voice commands replace the entry of similar commands via a mouse, touch screen or keyboard, see also paragraph

[0030] in US2018 / 0348500A1. This situation also arises in McDermott, S. et al. in "Controlling and scripting laboratory hardware with open-source, intuitive interfaces: open elastic voice commands and open elastic Blockly library".

[0004] Intuitive interfaces: OpenFlexure Voice Control and OpenFlexureBlockly)" arXiv:2209.14947v2[physics.ins-det], February 2, 2023, a system described in the article: or also in Holburn, D. et al. "Voice Control of the Scanning Electron Microscope Using a Low-Cost Virtual Assistant", Microsc.Microanal.27(Suppl 1) 1), 2021, doi:10.1017 / S1431927621009685. Users can issue voice commands such as "autofocus," "capture image," or "move 100 steps on the X-axis," and the microscope will execute these commands accordingly. Therefore, existing voice control systems do not help solve the aforementioned problem of translating application experimental descriptions into imaging technical parameter settings. Instead, voice commands must specify the imaging technical parameter settings. Experimental descriptions using natural (complex) language are impossible.

[0005] Modern microscope imaging programs offer virtual assistants (wizards) that simplify operation but do not overcome the aforementioned issues. Wizards are often unable to implement complex requests, react to results, or ask the user follow-up questions. They also only operate in a predetermined sequence, even if that sequence is inappropriate for a specific experiment. Extending wizards to new conditions is possible only through specially programmed modules, which requires a significant investment of time and money.

[0006] Likewise, ideal processing of captured images or raw data requires significant expertise and additional time investment. While standard settings for image processing algorithms or learning models generally achieve good results, processing parameters must be individually set to achieve optimal image quality, particularly based on the sample size and image properties such as signal-to-noise ratio.

[0007] As background of the present invention, reference is also made to the following prior art:

[0008] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, L., Kaiser, L., Polosukhin, I.: “Attention Is All You Need” rXiv:1706.03762v5[cs.CL]6 November 2017.

[0009] The neural network described in this article is called a transformer. While traditional recurrent models process input sequences sequentially, transformers include attention blocks, which process the input sequence as a whole rather than sequentially. If attention blocks are used in the encoder, it is called a transformer encoder with self-attention blocks. First, an embedding is computed for each input element (token) in the sequence. For simplicity, token embeddings are also referred to as tokens. A token can correspond to a word or a word part, and the sequence can be a sequence of words or multiple sentences. Tokens in the sequence are positionally encoded and fed into a self-attention block, which computes three representations for each token through matrix multiplication: the query (Q, the query vector), the key (K, the key vector), and the value (V, the value vector). Furthermore, for each token (hereafter referred to as token A), the associated Q vector is multiplied by the K vector of the other token, then a softmax function is applied, and then multiplied by the V vector of the other token. This process is repeated for all the A tokens in the sequence. The result of this computation is the output of the self-attention block for token A. The output for each other token in the sequence is computed in the same manner. These outputs form the input to the next self-attention block. A sequence of tokens is often fed into multiple self-attention blocks in parallel, a process known as multi-head attention. The output computed by the encoder's last self-attention block for each token in the sequence is also known as the final hidden state embedding for that token.

[0010] Similar to the transformer encoder, the transformer decoder is a network with self-attention blocks that use elements of the input sequence together in their computation to estimate the most likely next token of the sequence. This next token then becomes part of the input sequence used to generate the next token. The self-attention blocks perform the same computations as described for the transformer encoder.

[0011] In the transformer encoder-decoder, multiple consecutive attention blocks are used, where the Q vector comes from the decoder’s tokens, and the K and V vectors come from the encoder’s tokens (the final hidden state embeddings).

[0012] For transformer encoders, you can also refer to the following articles:

[0013] Devlin, J., Chang, M. W., Lee, K., Toutanova, K.: “BERT: Pre-training of DeepBidirectional Transformers for Language Understanding”. arXiv:1810.4805v2 [cs.CL] 24 May 2019.

[0014] This paper describes a transformer encoder in which a starting token, called a classification-specific token ([CLS] token), is placed before the input sequence. Given the input sequence, the transformer encoder computes a final hidden state embedding for the [CLS] token, which contains information about the entire sequence as it is applied along with the other tokens to the computations performed in the self-attention block. Training of the transformer encoder is first performed in a pre-trained fashion using a collection of text, for example, by masking out a word or token in the training data that the transformer encoder will predict. After pre-training, another network layer is attached to the transformer encoder, and further training (fine-tuning) is performed using additional training data, such as training data for a classification task. In this process, the final hidden state embedding for the [CLS] token is mapped to a category from the last one or more network layers.

[0015] The input sequence to the Transformer encoder does not necessarily contain text, or does not necessarily contain only text, but can also contain image data, such as the network described in "Vision Transformer":

[0016] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Deghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: “An image is worth 16x16 words: Transformers for image recognition atscale”, arXiv:2010.11929v2 [cs.CV], 3 June 2021.

[0017] Wu, H., et al., “CvT: Introducing Convolutions into Visual Transformers,” arXiv:2103.15808v1 [cs.CV], March 29, 2021.

[0018] The Transformer decoder has been popularized by GPT networks, for example, as described:

[0019] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). “Improving Language Understanding by Generative Pre-training,” in:

[0020] https: / / web.archive.org / web / 20200609115647 / https: / / cdn.openai.com / research-covers / language-unsupervised / language_understanding_paper.pdf.

[0021] After the model has been first trained (pre-trained), it can be further trained (fine-tuned) using reinforcement learning from human feedback to improve the probability of the desired (text) generation, as described, for example:

[0022] Lambert, N. et al., "Illustrating Reinforcement Learning from Human Feedback (RLHF)", December 9, 2022

[0023] https: / / web.archive.org / web / 20230102122855 / https: / / huggingface.co / blog / rlhf

[0024] OpenAI (2023), "GPT-4 Technical Report",

[0025] https: / / web.archive.org / web / 20230314180351 / https: / / cdn.openai.com / papers / gpt-4.pdf

[0026] OpenAI (2020), “Language Models are Few-Shot Learners”, arXiv:2005.14165v4 [cs.CL] 22 July 2020.

[0027] Text input can also be processed along with the input image data, as described in the “GPT-4 Technical Report” cited above.

[0028] Although the output quality of recurrent neural networks such as LSTM is often not as good as that of transformers, they are alternatives to transformers:

[0029] Hochreiter, S., and Schmidhuber, J.: “Long Short-term Memory” in Neural Computation 9(8):1735-1780, December 1997.

[0030] DOI:10.1162 / neco.1997.9.8.1735

[0031] For further background, see US 2020 / 0371333 A1, which describes a machine learning validation model for checking image processing results. Summary of the Invention

[0032] It can be considered that the object of the present invention is to provide a microscope system and program that can provide microscope images with desired characteristics without requiring the user to have expert knowledge in setting microscope parameters or providing desired processing parameters for the images.

[0033] This object is achieved by a microscopy system and a procedure having the features of the independent claims.

[0034] According to one embodiment of the present invention, a microscope is controlled by a computer-implemented program. A textual input describing at least one desired microscope image and a sample is received. Additionally, at least one overview image of the sample is received. The textual input and the overview image are input into a large language model, which is a machine learning neural network trained to process the textual input and the overview image together to calculate microscope settings for capturing a microscope image corresponding to the desired microscope image based on the sample. The microscope then captures the microscope image based on the microscope settings calculated by the large language model.

[0035] Through the present invention, users can, specifically through voice input, describe the experiment from the perspective of the application and name the properties of the microscope image to be captured, without having to name or specify the microscope settings themselves. For example, the user can tell the large language model whether a specific type of single cell or a cluster of cells in the sample should be imaged. The large language model uses this information to identify the appropriate magnification for capturing single cells or cell clusters, and the overview image is used to navigate to the appropriate location where the desired cells are located. Imaging parameters such as illumination intensity or fluorescence settings can be determined by the large language model based on text input and the overview image, without the user having to specify illumination intensity or fluorescence excitation or detection channels. In this way, users can obtain high-quality imaging without having to master a lot of professional knowledge or the laborious operation of manual settings.

[0036] According to another embodiment of the present invention, a computer-implemented program processes a microscope image. At least one microscope image of a sample is received. A textual input describing a desired image processing is also received. The textual input and the microscope image are input into a large language model, which is a trained machine learning neural network, to calculate processing parameters for processing the microscope image from the textual input and the microscope image. The microscope image is processed using the calculated processing parameters.

[0037] Advantageously, the user does not need to specify all processing parameters for processing the microscopy image itself; instead, a textual description of the desired image processing is sufficient, and the large language model uses this along with the microscopy image to determine the processing parameters. For example, a denoising algorithm can be selected from a variety of available algorithms based on the sample type. The specific value of the algorithm's regularization parameter affects the quality of the processing result, and the ideal value depends on factors such as the sample type and the signal-to-noise ratio of the relevant image region. Thus, the user can process microscopy images using ideal processing parameters without having to have detailed knowledge of the image processing program being used or having to configure the processing settings themselves.

[0038] Another embodiment of the present invention relates to a computer-implemented program for retrieving a desired microscope image. A textual input describing the desired microscope image is received and fed into a large language model. The large language model is a machine learning neural network trained to compute at least attributes of the desired microscope image from the textual input. The large language model then loads a specific microscope image from a database of microscope images based on the attributes of the desired microscope image.

[0039] Large databases containing tens or hundreds of thousands of microscope images are often available, often with incomplete annotations or no annotations at all. While the database may contain one or more microscope images that meet the user's specifications, the lack of annotations makes finding and retrieving these images very difficult. In various variations of the present invention, a large language model can select appropriate microscope images from the database based on simple text input from the user, without requiring the user to provide precise specifications or fully annotated image data. This allows for more efficient access and utilization of large databases.

[0040] The microscope system of the present invention includes a microscope for imaging and a computing device configured to execute any computer-implemented program.

[0041] The computer program of the present invention comprises instructions stored on a non-volatile data storage medium, which, when executed by a computer, causes the computer to perform any computer-implemented procedure.

[0042] Optional Features

[0043] Variants of the microscope system according to the invention and of the program according to the invention are the subject matter of the dependent claims and are explained in the following description.

[0044] Microscope setup

[0045] The microscope settings or microscope parameters determined by the large language model may be at least one or in particular all of the following settings:

[0046] - magnification, in particular the choice of an objective lens and / or setting the magnification by zoom optics in order to include individual organelles or cell clusters in the microscope image according to user specifications; the choice of the objective lens is based on the calculated magnification to be set, and can also be based on the objective lens numerical aperture and working distance;

[0047] - Imaging speed, in particular pixel dwell time, scanning speed or scanning resolution;

[0048] - the location of the sample area to be analyzed, where one or more images are captured, and the associated sample stage position;

[0049] - contrast procedures, such as phase contrast microscopy, differential interference contrast (DIC), Hoffman modulation contrast, brightfield or darkfield measurements, fluorescence microscopy, confocal microscopy, two-photon or multiphoton microscopy, polarized light microscopy, interference microscopy, digital holographic microscopy (DHM) or Fourier stylography;

[0050] - illumination power or intensity; duration of sample illumination;

[0051] - camera exposure time, pixel size of the combined camera sensor elements or other general detection parameters;

[0052] - General optical path parameters, such as mirrors, filters and pinhole sizes;

[0053] - Focusing, especially the aperture diaphragm which affects depth of field;

[0054] - condenser settings, in particular setting the condenser aperture and / or condenser height according to the selected magnification;

[0055] - The excitation wavelength used in fluorescence measurements, particularly depending on the stain employed and the type of sample structure of interest;

[0056] - the number of detection channels and the characteristics of the detection channels, in particular the characteristics of the respective wavelength ranges;

[0057] -Camera sensitivity.

[0058] Microscope settings can also specify whether to capture a single image, an image stack consisting of multiple images staggered along the optical axis, or a time series, in which the same sample area is analyzed at time intervals using single images or image stacks. Entire experiments with different settings and sequences of imaging events can also be defined using microscope settings determined by a large language model. In addition to the settings described above, it is also possible to specify which sample wells (in the case of multiwell plates / multichamber slides) should be captured with which settings for an experiment, as not all wells are part of the experiment and these wells are often subdivided into specific experimental lines of inquiry. During an experiment, it is also possible to specify the time intervals between measurements, the Z-stack settings, and whether to find or maintain the focal plane, or to restart an experiment to repeat it.

[0059] The microscope settings are set by the large language model so that at least one microscope image captured with these settings corresponds to the desired microscope image. This can be understood as the captured microscope image and the desired microscope image being consistent with respect to the microscope image properties that the large language model derived from the input received.

[0060] Text Input

[0061] The user can create text input via voice input or input devices on a microscope or computer. In the case of voice input, speech-to-text processing can be performed. However, the user can also create input directly in the form of text, for example, by typing in a chat interface, command line, or scripting API. In particular, the input device can be a keyboard, computer mouse, joystick, or touch screen.

[0062] The text input may be written in a natural language or in a technical or domain-specific language, such as a programming language, wherein the text input is provided in a form such as an XML file.

[0063] Furthermore, a graphical interface can be utilized, such as one that allows sliders to be moved. In contrast to conventional sliders (which can also be optionally used) that directly indicate microscope settings, such as laser illumination intensity, these sliders can be used to describe the desired properties of the microscope image or sample being used. For example, a slider can be used to set the width / length of the microscope image to be captured to approximately 100 times, 10 times, 2 times, 1 time, or 0.5 times the diameter of a cell in the biological sample being used. Based on this setting, the microscope settings will be automatically executed so that only a portion of a cell, the entire cell, or multiple entire cells are visible in the subsequently captured microscope image.

[0064] In particular, when the at least one microscope image is to be captured by text input, the text input may indicate at least one of the following:

[0065] Which objects should be visible in the desired microscope image, specifically the object type and number. For example, it is possible to distinguish between individual organelles, individual cells, and cell clusters. Text input can also specify the cell type or organelle type, such as nucleus, Golgi apparatus, endoplasmic reticulum, or mitochondria.

[0066] - The type of sample involved, e.g. from which tissue or living biological cell.

[0067] - What dyes / stains were used during specimen preparation, particularly what histochemical stains or fluorescent dyes were used.

[0068] - Whether bleaching of the specimen (caused by high illumination intensities) is allowed. This can be expressed as a level, e.g., never allow bleaching, only allow bleaching when necessary to achieve adequate image quality, or always allow bleaching for optimal image quality.

[0069] - Whether to prioritize high image quality, i.e., high signal-to-noise ratio, or sample preservation.

[0070] Textual input can also include data from various sources. For example, a user can enter information in at least one of the ways described above, and the textual input (or separate input of contextual information) can also include provided textual information about the sample being used and / or the current experiment. For example, information about previous sample preparation and / or sample type available on the data storage medium of the sample being used. This information can be input as textual input into the large language model along with the user input. The textual information can also appear on the sample carrier in the form of text or a machine-readable one-dimensional / two-dimensional barcode; in this case, the textual information can be read by analyzing the overview image and input into the large language model as part of the textual input along with the user input. The textual input can also include text created by a computer program, such as a virtual assistant, that suggests measurements or experimental descriptions for a planned experiment to the user. For example, a measurement description might state that individual biological cells of a specific cell type are to be localized, followed by a time series of measurements that include all stages of cell division, and that while high illumination intensity is desirable to achieve a good signal-to-noise ratio, under no circumstances should the illumination intensity be excessive enough to damage the cells. If the user confirms the proposed measurement description, it is then, after revision if necessary, fed as text into a large language model.

[0071] If the text input is used to process a microscopy image, the text input may, for example, indicate the type of specimen depicted and / or the dye or stain employed in specimen preparation, as described above. The text input may also indicate the processing goal to be achieved by the image processing, such as denoising, resolution enhancement, deconvolution, and / or SIM calculation (SIM stands for structured illumination microscopy, which computes a higher resolution image from multiple images with different illumination structures).

[0072] Processing parameters for image processing

[0073] A textual input and a captured or loaded microscope image can be used to determine processing parameters for processing the microscope image. In this case, the textual input includes a description of the desired image processing, in addition to or instead of the possible textual input content described above. The textual input, along with the microscope image, is fed into a large language model that is trained to calculate processing parameters based on the textual input and the microscope image. The image processing program then processes the microscope image using the calculated processing parameters.

[0074] The processing parameters may specifically specify one or more of the following:

[0075] - Selecting one of a plurality of deconvolution algorithms (in particular, deconvolution is calculated by estimating the point spread function (PSF) of the microscope system), a plurality of resolution enhancement algorithms, or a plurality of denoising algorithms. The suitability of an algorithm depends, for example, on the sample type and / or sample carrier type used. Thus, different algorithms with the same functionality can be provided, for example, different denoising algorithms for specific sample types. Then, one of the algorithms is selected based on at least one sample type.

[0076] Specifying parameters for the (selected) deconvolution algorithm, resolution enhancement algorithm, or denoising algorithm, or for the SIM calculation (SIM: Structured Illumination Microscopy). These parameters may relate to, for example, the number of iterations to be performed or the regularization term, in particular the regularization function and regularization parameter. The processing parameters may also be used to estimate the image PSF, which is then used for image processing. The parameters of the denoising algorithm can be set differently depending on the noise level of the microscopic image being processed.

[0077] - Virtual staining of specific specimen structures, or some other virtual modification of the contrast type, such as changing between phase contrast, bright field, DIC, or fluorescence images.

[0078] - Variation of the depth of field, where blurred structures become sharper or more blurred respectively by mathematically simulating a higher or lower depth of field, while structures in the focal plane remain essentially unchanged.

[0079] In addition to the algorithms described by way of example, image processing programs may also involve other functionalities. In particular, image processing programs can be used to improve image quality, increase the visibility of specific types of objects or structures (e.g., by virtual staining or virtual changes in contrast), or suppress undesirable types of objects or structures, in particular by background suppression or artifact removal. Segmentation masks can also be calculated. The output of an image processing program can be an image, although other processing results are generally also possible, such as a classification of object types or image properties, or the regression of specific physical values ​​of objects (e.g., their size or geometry), or the calculation of object statistics (e.g., the number of objects of a specific type or the size distribution of objects of a specific type). The image processing program in question can be a machine learning model or a classical algorithm without a learning model. The processing parameters can specify the choice of a learning model or algorithm and, if necessary, the parameters to be set for the algorithm or model.

[0080] To achieve the desired image processing, the image processing model must be retrained, with microscopy images serving as training data. Large language models can be designed to initiate new training, specifically by specifying training parameters or hyperparameters (e.g., training data augmentation, learning rate or learning rate increments), and a suitable model architecture. Large language models can also be designed to assess whether a given microscopy image can be correctly processed by an existing model or whether new training should be performed.

[0081] Structure and training data for large language models

[0082] A large language model is a deep artificial neural network that receives (among other things) text from a user as input and generates an output that specifies parameters for subsequent image generation. Compared to simple machine learning models, large language models include several tens of millions of model parameters that need to be defined during model training, and typically more than a billion parameters.

[0083] For example, a large language model may comprise a transformer encoder (as described in the introduction with reference to the article "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding") and an optional visual transformer for simultaneous image processing. Alternatively or additionally, a transformer decoder may also be provided having the properties as described in the introduction with reference to the prior art. In addition, the design of the transformer encoder-decoder described in the introduction may be adopted. More generally, an encoder or decoder with a self-attention block may also be implemented in which the Q, K and V vectors of the transformer are not used in the calculations, but vectors calculated from the labels are used for different matrix calculations. In principle, large language models may also be designed using models of the LSTM form as described with reference to the prior art. In the variants of the embodiments described below, the transformer may also be replaced by a different structure, in particular in the case of image data, it may be replaced by an LSTM or a CNN.

[0084] Various implementations of large language models include a (transformer) encoder. The text input is converted into a sequence of tokens in a known manner, for example, on a word or word part basis. Furthermore, a special token, hereinafter referred to as the start token or [CLS] token, is added. This token can be placed before the token in the text input, but in principle can also be placed at another position in the sequence or after it. The value of the start token can be predetermined by training the large language model. The encoder computes a mapping from the text input with the added start token to an output, which can include an output vector (hidden state embedding or representation) for each token. In the case of a transformer encoder, the hidden state embedding of the start token contains information about all other tokens in the text input, which is incorporated into the embedding of the start token through calculations performed in a self-attention block using the Q, K, and V vectors. After the encoder processes the input, the position of the start token in the embedding space becomes a feature vector for further processing. The hidden state representation of the start token is fed into a mapping procedure, which can be considered part of the large language model. This procedure computes a mapping between the feature vector (i.e., the hidden state representation of the start token) and target parameters (particularly the value of the microscope settings). The mapping procedure can include, for example, a linear mapping or a further machine learning network, such as a multilayer perceptron. This procedure is similar to the use of the [CLS] tag in the BERT and ViT models cited in the introduction. As with ViT, the encoder processes the input image data (particularly the overview image in this case) to incorporate image information into the feature vector or generate a separate feature vector for the overview image. A pre-trained language model, particularly one from the prior art, can also be used as the transformer encoder. The mapping procedure / further machine learning network can be designed, for example, as a regression model or a classification model. It can be learned after the transformer encoder has been trained or pre-trained, for example, through supervised learning. In supervised training, the text input specifies a desired set of microscope settings. The further machine learning network then learns to compute a mapping from the hidden state embedding of the [CLS] tag to a set of microscope settings, or from the hidden state embedding of the [CLS] tag and the feature vector of the overview image to a set of microscope settings. Training data can be collected actively or passively by observing the user. Training can optionally be performed with the aid of reinforcement learning. As described in more detail below, an additional evaluation model can be used to assess whether microscope images captured using a certain microscope setting represent a successful implementation of text input. In principle, the above training data can also be used for joint training, where the transformer encoder is trained or fine-tuned in addition to training other machine learning networks.

[0085] Training data for large language models can specifically include one or more of the following:

[0086] -Microscope user manual;

[0087] -Chats between microscope users and microscope service personnel, where the chats include questions and answers related to microscope use and troubleshooting. The chats can include text-based conversations or transcribed conversations such as phone calls.

[0088] - Records of training sessions / events related to the microscope, in particular regarding the use of the microscope, or training of service personnel in microscope user support. In the case of oral training, the records may also be transcribed audio recordings.

[0089] The training data described above describes how to choose specific microscope settings, especially the type of sample or the desired properties of the image to be captured. This training data is particularly useful for designing large language models as a transformer decoder.

[0090] Large language models can also be formed by fine-tuning an existing large language model, especially one that has been pre-trained. The aforementioned training data, such as a microscope user manual, can serve as training data for fine-tuning. Fine-tuning can optionally be self-monitored, meaning the model attempts to reproduce the microscope user manual or other training data during training.

[0091] As described in the paper "Reinforcement Learning from Human Feedback (RLHF)" cited in the introduction, reinforcement learning can be used to reinforce desired responses and suppress undesirable responses during fine-tuning. Microscope settings determined by a large language model and / or microscope images captured using the determined microscope settings can be evaluated. This evaluation can be performed manually or by a learning model, particularly a large language model. The captured microscope images can be images captured after the microscope settings were calculated by the large language model; alternatively, microscope images from a set of images can be used, where the microscope settings used to capture the given microscope image are known. Thus, a given microscope image can be selected that was imaged using microscope settings that correspond to the microscope settings determined by the large language model. The user or the model then evaluates whether the desired properties of the microscope image correspond to the text input and whether the selected microscope image meets the highest possible image quality. This eliminates the need to capture new microscope images, or requires only a very small number of them, making training faster and more cost-effective.

[0092] In principle, fine-tuning can also be performed with a few (or one) learning / prompting steps. In this case, the text input is preceded by a given text, optionally with associated image data. The given text involves a mapping of the text input to the output data (microscope settings and / or processing parameters) of a large language model. For example, multiple text request examples can be used with a few prompts to capture microscope images with specific properties and the associated microscope settings used to capture the corresponding microscope images. This will prompt the model to output microscope settings that match the text input. This type of fine-tuning generally does not change the learned model parameter values.

[0093] The large language model can also include a transformer encoder-decoder, as described in the article by Vaswani et al. cited in the introduction. In this case, the training can be a supervised translation from text input to microscope settings. The training data includes natural language or technical user input as text input and the associated desired parameter output (a set of microscope settings). The transformer encoder-decoder is particularly advantageous when the output is in technical or domain-specific language. For example, the microscope settings can be output by generating an XML file written in a specific programming language that contains the commands to set up the microscope. This setting is particularly suitable when the user's text input is more like a formal experimental description of the microscope settings to be translated, and an interactive conversation with the large language model is not necessarily required. The transformer decoder is also suitable for outputting microscope settings in technical or domain-specific language.

[0094] The training process for all model types is subject to change. For example, transformer encoders can in principle also be trained in an unsupervised manner like transformer decoders.

[0095] Large language models can include multiple machine learning models (sub-models), which can be learned in a joint training or can also be learned separately. Different sub-models are activated according to the needs of specific tasks. Among other things, different sub-models can be provided for image analysis, where at least one sub-model is selected based on, for example, the sample type; different sub-models can also be provided for issuing subsequent queries, determining whether there is sufficient information to create microscope settings or processing parameters, and generating microscope settings. At least one sub-model is a language model for processing text input.

[0096] The large language model or at least one sub-model may also include one or more convolutional neural networks (CNNs), particularly U-Net architectures. Other model architectures of deep neural networks may also be used.

[0097] Subsequent query loop

[0098] Simple language models in the prior art can only detect user-specified microscope settings, but cannot infer appropriate microscope settings from more abstract user descriptions. Another difference from the prior art is that the large language model of the present invention can be designed to provide interactive assistance to users during the microscope setup process, by submitting follow-up queries or comprehension questions to the user to infer appropriate microscope settings.

[0099] A large language model can be trained to determine whether all desired microscope settings (i.e., all required microscope settings) can be set based on the received textual input, or whether a follow-up query should be posed to the microscope user regarding one of the microscope settings. In the follow-up query, the user's subsequent response is fed into the large language model as further textual input. In the follow-up query, the large language model is trained not to inquire about the value of a microscope setting, but rather about sample properties, desired microscope image properties, or properties of the experiment to be performed related to that microscope setting. For example, a follow-up query might be whether sample bleaching is acceptable or whether a relatively long measurement time should be chosen to achieve a sufficient signal-to-noise ratio. In contrast, no questions are asked about the specific values ​​of microscope settings, such as the power of the illumination device.

[0100] If the user's response indicates that the calculated microscope settings are unacceptable (for example, if the user does not allow bleaching of the sample, but the microscope settings specify a high illumination intensity and the user does not want the measurement duration to be too long), the large language model can change other parameters to achieve the desired goal, such as a specific signal-to-noise ratio; in the example cited, these effects can be avoided by, for example, changing the aperture cable or increasing the pinhole size for confocal imaging, provided that the user does not express any conflicting specifications (for example, regarding depth of field).

[0101] Other examples of follow-up queries posed to the user could be: "Which dyes are of interest?" "Is the process time-sensitive?" or "Which sample (of multiple samples on a sample carrier) should be analyzed?"

[0102] For example, special encodings in the output of a large language model can be used to assess whether a follow-up query to the user is necessary. Alternatively, specialized models, in particular transformer encoder models of large language models, can be used to classify the necessity of a follow-up query.

[0103] A large language model can include one or more transformer encoders and transformer decoders for different tasks. For example, a transformer encoder can be used to compute microscope settings based on the inputs to the large language model (specifically, text input and responses received from the user to subsequent queries). On the other hand, a large language model can include a specialized model, such as a transformer decoder, for creating subsequent queries for the user. This enables textual interaction. The transformer decoder can be trained using the microscope user manual and other training data mentioned above.

[0104] The input to the large language model includes the context of the conversation with the user so far, so that the model "remembers" the previous conversation process and uses it to calculate subsequent queries or microscope settings. For a specific session, the input to the large language model may specifically include: all user requests; all subsequent queries to the model with associated user replies; the parameter settings / microscope settings generated so far, the associated images and / or their evaluations by the (image-to-image) evaluation model; and any existing contextual knowledge. The contextual knowledge can come from a database related to the provided samples, sample carriers and previous measurement results, and can also distinguish between different users. The knowledge about previous measurement results is particularly helpful to the large language model when the user mentions previous measurement results in the text input or response, for example, the user indicates that the same dye should be rendered visible as in the measurement result of the previous week. In contrast, traditional language models for microscopy are generally unable to take contextual knowledge into account when defining microscope settings.

[0105] In a variant of the present invention related to microscopy image processing, when the current textual input does not accurately describe the desired image processing, a large language model can be configured to issue subsequent queries to allow the necessary processing parameters to be defined with a predetermined minimum level of reliability. The subsequent queries can relate to, for example, the sample type or imaging technique employed.

[0106] Overview image processing

[0107] The large language model can include a (particularly machine-learned) image analysis model. This model computes analysis results from the overview image, and the large language model uses these results together with the text input in the calculation to determine the microscope settings. The image analysis model can optionally be learned separately from another part of the large language model, such as separately from the transformer encoder that computes the feature vector from the text input. Alternatively, the large language model can also be designed directly for image processing without the need to train a separate image analysis model, for example as described in the article cited in the introduction.

[0108] For large language models, especially image analysis models of the type language model, the analysis results computed based on the overview image can specify, for example, one or more of the following:

[0109] - The type of sample carrier, such as whether it is a slide, culture dish, chamber slide, or multiwell plate;

[0110] - Specimen type, such as adherent / fixed cells, stained / unstained tissue sections, rock specimens, or electronic components;

[0111] - Whether chemical dyes were used during sample preparation, and if so, what chemical dyes were used;

[0112] - Experiment type, such as single image, long-term experiment, or high temporal resolution;

[0113] - one or more focus positions, i.e. one or more height planes of the sample to be analyzed, and the associated microscope settings for focusing on the elevation plane;

[0114] - the contamination state or degree of contamination of the sample carrier or sample, such as the fill level of a liquid sample, discoloration or bacterial infestation of a liquid biological sample, or droplets of sample liquid due to condensation;

[0115] - the location of an area of ​​interest, such as a sample or sample area, a cover slip, a sample chamber or a text-containing area; and / or

[0116] - The information content of the text or barcode on the sample carrier.

[0117] These analysis results can be processed together with the text input through a large language model to calculate the microscope settings. This joint processing contrasts with separate processing, in which one result is calculated separately for the text input and the other is calculated separately for the image data.

[0118] When computing the analysis results for the overview image, you can optionally take the text input (or a piece of information extracted from it by a large language model) into account. If the text input has a name such as sample type, it can be taken into account when locating and identifying sample regions.

[0119] In addition to the analysis results of the image analysis model, other contextual knowledge about the respective samples can usually also be considered and fed into the large language model together with the text input.

[0120] Image evaluation and imaging adjustments

[0121] By evaluating the captured microscope images, it is possible to determine whether the implementation of text entry was successful or whether new imaging should be performed after adjusting the microscope settings.

[0122] Large language models can be trained to derive microscopy image properties from a textual input, which properties a captured microscopy image should satisfy in order to conform to the textual input. Microscopy image properties can represent, for example, that a specific object type (e.g. a specific organelle) should be visible or that a minimum / maximum pixel size of a microscopy image should conform. Therefore, which microscopy image properties should be satisfied is not predetermined but rather inferred at least from the textual input. Other examples of microscopy image properties include image quality (e.g. signal-to-noise ratio, image noise, contrast, image sharpness, depth of field, overexposure, shaded / shadowed sample areas, resolution), contrast type, sample staining, sample state, or the presence of artifacts (e.g. contaminants; image processing errors caused by processing the captured raw data or imaging artifacts (e.g. light source reflections)).

[0123] At least one machine-learned image assessment model calculates relevant microscopic image attributes based on a microscopic image captured using the microscope settings determined using the large language model. Multiple image assessment models can be provided, each designed to determine different microscopic image attributes. Alternatively, a single image assessment model can be designed to calculate different microscopic image attributes. At least one corresponding image assessment model is selected based on the microscopic image attributes to be satisfied, derived from the textual input.

[0124] If the microscope image's microscope image properties match those derived from the textual input, the microscope image is further utilized, such as being displayed to the user, saved, and / or used in the provided workflow. After displaying the microscope image, the large language model may ask the user if any modifications are desired. The user is then given the opportunity to add or modify information to the current textual input, at which point the process is run again. In this new process, the large language model utilizes the user's previous textual input, the user's new specifications, and the microscope image or images that led to the user providing the new information. For example, the user may indicate that they were disturbed by the blurred structures displayed in the microscope image, superimposed on sharp specimen objects. The large language model processes this statement and determines that the depth of field should be improved compared to the previous image as a desired microscope image property. The large language model then alters the previous microscope settings, such as adjusting the aperture stop or reducing the pinhole size for confocal measurements. As a result of these changes, other microscope settings are automatically adjusted to maintain the desired properties. For example, the illumination intensity may be increased to compensate for the smaller pinhole size to maintain a specific signal-to-noise ratio.

[0125] If the image assessment model determines that the microscope image properties of a captured microscope image do not conform to the microscope image properties derived from the text input, the image assessment model provides feedback to the large language model. Specifically, the microscope image properties of the microscope image can be input into the large language model. The large language model then calculates adjusted microscope settings for capturing new microscope images so that the microscope image properties of the new microscope image are closer to the microscope image properties derived from the text input. For example, if the captured microscope image does not display the organelle to be analyzed at the minimum number of pixels defined by the text input, the adjusted microscope settings can specify a higher magnification. If the captured organelle is of an adequate size but the contrast / image quality is insufficient, the adjusted microscope settings can specify an increase in illumination intensity.

[0126] In the event that it is determined that microscope settings were adjusted to capture a new microscope image, the large language model can optionally be designed to perform an evaluation to determine whether a subsequent query should be submitted to the user to authorize the capture of the new microscope image, given the impact of the adjusted microscope settings on the sample or measurement duration, for example, if the illumination intensity is increased, which may risk sample bleaching.

[0127] Furthermore, captured microscopy images can be evaluated using learned quality metrics, i.e., by machine learning models trained specifically to compute image quality.

[0128] Furthermore, the captured microscope image can be evaluated by evaluating its similarity to at least one reference image. For example, the similarity can be calculated by computing a mapping of the microscope image and the reference image in a feature space, and determining the distance between the representations in the feature space. In particular, known techniques from the field of image retrieval can be used to estimate the similarity. For example, when a user provides text input in the form of an instruction to "capture an image similar to yesterday's", a comparison can be made with a reference image. Before imaging, a large language model sets the parameters / microscope settings using contextual knowledge (yesterday's parameters). The image evaluation model then compares the captured microscope image with yesterday's image to ensure that the request was correctly executed. In principle, the image evaluation model can also be considered as part of a large language model, which outputs the evaluation results to another part of the large language model.

[0129] Evaluations performed by the image evaluation model and / or supplements of previous text input performed by users after viewing microscope images can be collected for new / supplemental training of the large language model, e.g., as reinforcement signals, thereby further improving the large language model.

[0130] Image evaluation for image processing

[0131] When image processing is performed on captured microscope images, the processed microscope images can be evaluated to assess whether the implementation of text entry was successful or whether the image processing needs to be repeated using adjusted processing parameters.

[0132] The features described above with respect to the evaluation of captured microscopy images may also be applied to the evaluation of processed microscopy images. In particular, different image evaluation models may be provided, one or more of which may be selected based on textual input or determined processing parameters to evaluate the processed microscopy images.

[0133] Image evaluation can assess whether the intended processing goals were achieved and whether any processing errors occurred. Processing goals can include, for example, resolution enhancement, denoising, background or imaging artifact suppression, deconvolution, SIM calculation, or segmentation. Processing errors can be reflected by added image artifacts, such as bubble-like structures, phantom objects (not present in the original data), and bumping or ringing artifacts. Image evaluation criteria can be selected based on the processing goal, for example, whether the image noise is lower or higher than the image noise in the original microscope image. Optionally, image evaluation only involves image regions classified as relevant by image analysis. For example, the signal-to-noise ratio can be enhanced by comparing the signal-to-noise ratio of the sample's respective image regions in the original microscope image and the processed microscope image, while the environment / background is not considered or only to a lesser extent in determining the signal-to-noise ratio. Specially trained models can detect added image artifacts. Image segmentation can be evaluated using a trained validation model that is trained to detect such segmentation masks or detections, or using correct and / or incorrect segmentation masks to detect deviations from the segmentation masks of the training data. If a negative evaluation result occurs, especially if the evaluation criteria are not met, the image evaluation model can be designed to output a corresponding error description.

[0134] If the evaluation result is positive, the processed microscope image is used further, for example displayed to a user, saved and / or used in a provided workflow.

[0135] However, if a negative evaluation result occurs, feedback is sent to the large language model. Specifically, the error descriptions generated by the image evaluation model can be input into the large language model. The large language model uses the error descriptions / evaluations to calculate adjusted processing parameters for new image processing. For example, if an object is hallucinated, the elephant language model will choose modified processing parameters that are known to have a lower probability of hallucination (even if this means accepting a loss in the desired processing goal, such as increased resolution).

[0136] As described for the evaluation of captured microscopy images, the evaluation of processed microscopy images can also be performed using a learned quality metric or by estimating a similarity to at least one reference image.

[0137] Loading microscope images (image retrieval)

[0138] In various variations of the present invention, the desired microscope image is retrieved from the database using a large language model. The large language model may provide, in addition to the aforementioned functionality for determining microscope settings and / or processing parameters, functionality as described below.

[0139] A text input describing a desired microscope image is received to load or retrieve the microscope image. As described above, the text input can come from a user, such as through voice input, or from a computer program, such as in an automated workflow.

[0140] The text input is fed into a large language model that is trained to compute the expected microscopy image properties from the text input. Microscopy images are then loaded from a database containing microscopy images based on the expected microscopy image properties.

[0141] In simple cases, microscopy image properties stored separately for microscopy images in a database can be used. However, extensive databases often exist only for microscopy images with no or only partial annotation of their properties.

[0142] In particular, the feature space can be used to automatically find microscopic images with desired properties in a database containing microscopic images that are unannotated or annotated only for certain properties. A large language model is trained to compute a mapping of the desired microscopic image properties to the feature space. A corresponding feature space representation is also provided for each microscopic image in the database; in other words, each microscopic image is represented as a point in the feature space by mapping / embedding the microscopic image in the feature space. The large language model can select and load the microscopic image from the database whose feature space representation is closest to the representation of the desired microscopic image property. The selected microscopic image is then displayed to the user or used in an automated workflow. The large language model can also select multiple images, rather than a single image, whose feature space representations have the smallest distance to the representation of the desired microscopic image property.

[0143] Specifically, the large language model may include a text encoder for mapping microscope image features to a point in the feature space; and an image encoder for mapping the microscope image into the feature space. Both encoders can be transformer-based. Using (partially) annotated microscope images, the relationship between the microscope image attributes described in the text and the microscope image is established during training. After the training is completed, the remaining, unannotated or incompletely annotated microscope images can be mapped to the feature space through the image encoder. Therefore, for microscope images without text annotations, their correspondence or similarity with the text input can also be calculated. The distance between two representations in the feature space can measure their similarity. The desired microscope image attributes can be calculated from the text input, for example, by a transformer decoder trained for this purpose.

[0144] A loaded microscope image can be evaluated for its conformance to a textual input. This can be done using a machine learning classifier that checks whether the microscope image matches a specific category or domain. Multiple binary classifiers can be provided for a larger number of classes, with one or more binary classifiers selected based on the desired microscope image properties calculated by a large language model. For example, if the desired microscope image property specifies that a cell nucleus should be visible, a binary classifier trained to assess the presence of a cell nucleus in the image will be selected.

[0145] In another example, a user may request in text input a microscope image of cells at a specific cellular stage, such as the mitotic phase of cell division. The large language model computes a mapping to feature space / latent space, but there is significant uncertainty about the location in the feature space. Therefore, the large language model submits a subsequent query to the user, further restricting the microscope image—for example, asking whether dead cells (or cells at different cellular stages) are also visible in the microscope image. The user responds in the negative, and this response is considered to compute a new mapping to the feature space. Because the specification is more precise, uncertainty is reduced, and the large language model outputs at least one microscope image whose representation in the feature space is closest to the request. This microscope image can optionally be evaluated by a classifier before being displayed to the user. In this example, multiple classifiers trained to detect cells at different cellular stages are selected. One or more microscope images evaluated as correct are displayed to the user. The user can then refine / modify the text request, for example, specifying that an image with a different contrast type than the displayed image should be selected. The large language model processes the user's refined specification and the displayed image. In this example, based on the refined specification, the large language model uses a classifier to determine the contrast type of the selected image, for example, determining the presence of a phase contrast image. The large language model uses the refined text request to calculate a new mapping to the feature space and selects at least one closest microscopy image. The classifier is used to evaluate whether the microscopy image is a phase contrast image. If the evaluation determines that the microscopy image is not a phase contrast image, the image is presented to the user. This creates an interactive loop, where user feedback is used to refine the large language model's calculations, bringing the results closer to user needs.

[0146] The textual input and the resulting microscopy image properties are specific to sample type (e.g., a specific cell line), imaging type (e.g., a specific contrast type or stain), and image content type (e.g., a specific event such as cell division).

[0147] Contextual information

[0148] Large language models can be designed to receive and process contextual information along with text input. Contextual information can come from, for example, a data store, metadata about a given image processed along with the text input, or information about the microscope currently being used, such as its component specifications or settings. User profiles can also be used as contextual information.

[0149] In particular, the contextual information can also be used by an image analysis model of a large language model or another machine learning network of a large language model.

[0150] The context information may specify, among other things, the sample type employed, the sample status, the sample phase, the sample carrier employed, a description of the experiment and / or a description of previous measurements.

[0151] By using contextual information, large language models can avoid having to request the appropriate information from the user. Large language models can also use contextual information to check the reasonableness of user specifications in text input. For example, contextual information can specify a sample type; if the text input specifies that an organelle to be displayed does not exist in this sample type, the large language model can point out this inconsistency to the user without having to calculate microscope settings and capture potentially inappropriate images.

[0152] Contextual information can also come from image analysis of images fed into a large language model, for example by detecting fonts or other objects in the image.

[0153] Common Features

[0154] Both microscope images and overview images can be understood as images captured by a microscope or calculated using measurement data from a microscope. In particular, a microscope image or overview image can be formed from one or more raw or processed microscope images and can include 2D or 3D image data stacks or volume data, or a time series of 2D or 3D image data of the same object captured at different points in time. Overview images have a larger field of view than microscope images.

[0155] In principle, the structures or objects depicted in the microscope images and overview images can be any structures or objects. In addition to the sample itself (e.g., biological structures, electronic components, or rock fragments), they can also depict sample containers, sample carriers, microscope components (e.g., the sample stage), or regions thereof.

[0156] A "received" image may include the possibility of an image captured by the microscope, or it may be an existing image received, such as an image loaded from a data memory.

[0157] The microscope can be an optical microscope, including a system camera and optionally a separate overview camera. Other types of microscopes, such as electron microscopes, X-ray microscopes, or atomic force microscopes, can also be used. A microscope system is a device consisting of at least one computing device and a microscope.

[0158] The computing device can be designed in a decentralized manner and can be a physical part of the microscope, or it can be separately located near the microscope or at a remote location at any distance from the microscope. In general, it can be composed of any combination of electronic devices and software, and specifically can include a computer, a server, a cloud-based computing system, or one or more microprocessors or graphics processors. The computing device can also be configured to control components of the microscope. The decentralized or cloud-based design of the computing device is very advantageous, especially when using specially designed hardware to execute large language models or to collect user feedback for subsequent training of large language models.

[0159] Singular descriptions are intended to encompass both “exactly one” and “at least one” variations. For example, “a microscope image was captured using microscope settings computed by a large language model” is intended to encompass whether exactly one microscope image was captured using the computed microscope settings or more than one microscope image was captured.

[0160] The features described herein as additional device features, when implemented as intended, also result in variants of the program according to the present invention. Conversely, a microscope system, or in particular a computing device, can be configured to execute the described program variants. The described training process for a machine learning network defines the properties of the trained network, and by executing the described training process as a program step of the program, additional program variants according to the present invention can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0161] The further effects and features of the present invention will be described below with reference to the accompanying drawings:

[0162] Figure 1 Schematically illustrates the process of one embodiment of a program according to the present invention;

[0163] Figure 2 Schematically illustrates the process of one embodiment of a program according to the present invention;

[0164] Figure 3 Schematically illustrates the process of one embodiment of a program according to the present invention;

[0165] Figure 4 Schematically illustrates the process of one embodiment of a program according to the present invention;

[0166] Figure 5 Schematically illustrates the process of one embodiment of a program according to the present invention;

[0167] Figure 6 Schematically illustrates the process of one embodiment of a program according to the present invention;

[0168] Figure 7Schematically illustrates the process of one embodiment of a program according to the present invention;

[0169] Figure 8 schematically illustrates the process of one embodiment of a program according to the present invention; and

[0170] Figure 9 The process of one embodiment of a program according to the invention is schematically shown. DETAILED DESCRIPTION

[0171] In the following, different embodiments will be described with reference to the accompanying drawings. Generally, similar elements and elements that operate in a similar manner are denoted by the same reference symbols.

[0172] Figure 1 : A program with a large language model according to the present invention

[0173] Figure 1 In a schematic manner an embodiment of a computer implemented program according to the invention for controlling a microscope 1 is illustrated. The program uses a special Large Language Model (LLM) to capture microscope images 50 according to the user's wishes, without the user having to indicate specific microscope settings.

[0174] In process P1, at least one overview image 20 of a sample 10 is received. In the example shown, the overview image 20 is captured by an (overview) camera of the microscope 1, but it is generally also possible to load an existing overview image 20 from a memory. The overview image 20 shows the sample carrier 9 with the sample 10, which in this example comprises a plurality of spatially separate sample areas, for example a plurality of tissue sections.

[0175] In process P2, a text input T is received from a user U, for example, by capturing a voice command from the user U via a microphone and converting it into text using voice recognition. The text input T describes the desired microscope image and / or the sample to be used. For example, the user may issue a voice command to capture an image of a specific biological cell. Thus, the text input T describes the cell type of interest (as distinct from other cell types) and the magnification, because a single cell is to be displayed, rather than a cluster of multiple cells (requiring a lower magnification or a larger field of view) or an organelle (showing only a slice within the cell).

[0176] In process P3, the text input T and the overview image 20 are input to the Large Language Model (LLM). This is a machine learning neural network that has been trained to process the text input T and the overview image 20 together to calculate microscope settings 40 for capturing microscope images that match the desired microscope image description. The calculated microscope settings 40 are output by the Large Language Model (LLM) in process P4.

[0177] In process P6, the microscope 1 is controlled according to the microscope settings 40. In process P7, at least one microscope image 50 is captured using the microscope settings 40 calculated by the large language model LLM. The calculated microscope settings 40 may also include a sequence of consecutive microscope settings to be set, and multiple microscope images may be captured using these settings.

[0178] The microscope image 50 is input to a machine learning image assessment model M2, process P8, which calculates microscope image attributes 46, such as attributes related to image quality (particularly contrast, image sharpness, or signal-to-noise ratio), and attributes related to the depicted structures or objects (particularly object detection, to determine whether cells of a particular cell type are depicted in the microscope image 50, and if so, how many). The image assessment model M2 uses the microscope image attributes 45 that the microscope image 50 should satisfy, as calculated by the large language model LLM in process P5. The microscope image attributes 45 specified by the large language model LLM can be derived from the text input T and the overview image 20 and can indicate, in particular, the type and number of objects to be displayed. The image assessment model M2 can include multiple object detection models for detecting different types of objects, wherein one or more object detection models are selected for processing the microscope image 50 based on the microscope image attributes 45 calculated by the large language model LLM.

[0179] In process P9, an evaluation result 55 is calculated, which indicates whether the microscope image properties 46 calculated by the image evaluation model M2 for the microscope image 50 correspond to the microscope image properties 45 determined by the large language model. Process P9 can in principle be performed by the image evaluation model M2 or the large language model LLM. If the evaluation result 55 is positive, the microscope image 50 will be further used in process P10, for example, displayed to the user U. In process P10, it is also possible to continue the provided workflow that is performed after successfully capturing the microscope image 50 with the microscope image properties 45 calculated by the large language model LLM, for example, navigating to and analyzing more sample locations, capturing more microscope images of the same sample area with different contrast types, or performing a time series measurement, in which images of the same sample area are captured at different points in time.

[0180] After viewing the microscope image 50 in process P10, the user U can make additions to the text input T in process P11. In particular, the user U can replace and / or improve the previous specifications. The supplement to the text input is input into the large language model LLM, which calculates modified microscope settings based on the previous information and the newly input information in order to capture a new microscope image. In this way, a cycle can be repeated many times, in which imaging is performed using the settings calculated by the large language model LLM and feedback from the user is taken into account. Thus, the user U can use the text input T to interactively control the imaging. For example, as a supplement to the text input T, the user U can say "highlight the cell nucleus better." The previous microscope settings 40 are then modified, for example, the intensity of the illumination wavelength for fluorescence excitation of the dye used for cell nucleus staining is increased.

[0181] If the evaluation result 55 indicates that the microscopic image attributes calculated by the image evaluation model M2 are inconsistent with the microscopic image attributes 45 calculated by the large language model LLM, feedback is provided to the large language model LLM in a flow process P12, which refers to Figure 2 Describe in more detail.

[0182] Figure 2 : A loop with a large language model and / or feedback image evaluation for subsequent user queries

[0183] Reference Figure 2 , first describe Figure 1 According to an additional function of an embodiment of the present invention, the captured microscope image 50 does not meet the microscope image properties 45 required by the large language model LLM according to the evaluation result 55. In process P12, the evaluation result 55 or the microscope image properties 46 calculated by the image evaluation model M2 (by means of a special mark) is input to the large language model LLM to calculate the adjusted microscope settings 40B. To calculate the adjusted microscope settings 40B, the large language model LLM also takes into account the previously calculated microscope settings 40, the text input T and the overview image 20 or the analysis results determined therefrom. The purpose of the adjusted microscope settings 40B is to make the microscope image properties of the newly captured microscope image 50B closer to the microscope image properties 45 derived from the text input T. The newly captured microscope image 50B will then be used in accordance with the microscope image 50 described above. If necessary, the cycle of adjusting the microscope settings and capturing new microscope images can be repeated until the captured microscope image meets the desired microscope image properties 45.

[0184] If adjusted microscope settings 40B are determined for capturing a new microscope image 50B, an evaluation 47 may optionally be performed. Evaluation 47 indicates whether a subsequent query Q should be submitted to user U to authorize capturing a new microscope image 50B, given the effect of the adjusted microscope settings 40B on sample 10 or the measurement duration. For example, desired microscope image properties 45 may indicate that a particular cell type should be visible, while evaluation result 55 indicates that the cell type is barely visible in microscope image 50; in this case, the large language model LLM calculates adjusted microscope settings 40B with increased illumination intensity and / or a modified illumination wavelength. If, in evaluation 47, it is determined that the modified illumination results in bleaching of sample 10, a subsequent query Q should be submitted to the user. Evaluation 47 may be calculated by the large language model LLM, which also uses all previous inputs and outputs of the large language model LLM relevant to the current experiment. The large language model LLM processes the response A of the user U to the subsequent query Q in order to capture a new microscope image 50B using the adjusted microscope settings 40B, or to modify the adjusted microscope settings 40B again based on the response A of the user U. For example, if the user responds that the photodamage of the sample 10 is unacceptable, the large language model LLM can (possibly after further subsequent queries Q and associated responses A) increase the illumination duration and the measurement duration in order to obtain better visibility of a specific cell type without increasing the illumination intensity. Alternatively, the large language model LLM can switch to an objective lens with a higher magnification and capture multiple laterally offset microscope images, stitching these images together to form a single image (image stitching), which can also achieve better visibility of a specific cell type without increasing the illumination intensity.

[0185] If the feedback loop has run through process P12 several times and still has not reached an evaluation result 55 of sufficiently good quality, the large language model LLM can provide feedback to the user U. Specifically, the large language model LLM can use a subsequent query Q to request a change to the imaging conditions specified by the user U, such as asking whether the laser power limit can be increased.

[0186] The large language model LLM is also designed to determine, after receiving the overview image 20 and the text input T, whether the available information is sufficient to calculate all necessary microscope settings 40 for capturing the desired microscope image, or whether the user U needs to provide further information for this purpose. If more information is required, the user U first submits a subsequent query Q. The large language model LLM then calculates the microscope settings 40 using the further text input (response A) received from the user U and the previously provided information. This cycle of subsequent queries Q and responses A can be repeated until the large language model M has determined all necessary microscope settings 40. Only then can the microscope settings 40 be used for imaging.

[0187] Whenever possible, the large language model M formulates subsequent queries so as to ask about properties of the sample or allowed / forbidden influences on the sample, rather than asking about the specific values ​​of the microscope settings 40. For example, the large language model M is designed not to ask about the fluorescence excitation wavelength used as any of the microscope settings 40, but rather about the relevant fluorophores / fluorochromes used. This requires less expertise from the user U and reduces the possibility of error. If the large language model LLM knows which dyes are being used (e.g., by automatically reading labels on the sample carriers identified in the overview image 20), the large language model LLM can also ask about the sample structure of interest in subsequent queries Q and set the fluorescence excitation and detection accordingly; however, the user U does not need to ask about the specific settings for the fluorescence measurement.

[0188] Figure 3 : Large language model with independent decoder and encoder

[0189] The large language model LLM may include a transformer decoder Dec and a transformer encoder Enc, which respectively receive input of the large language model LLM or information derived therefrom, in particular a text input T and an overview image 20 or information determined from the overview image 20.

[0190] The transformer encoder Enc computes an embedding Emb based on the input data. The embedding Emb represents the extracted features of the data fed into the large language model (LLM). A trained machine learning classifier M4 can calculate from the embedding Emb whether (or what) information is needed to determine all necessary microscope settings 40. The transformer encoder Enc and classifier M4 are collectively referred to as a transformer encoder model.

[0191] The transformer decoder Dec is a language model that is used to create subsequent queries Q for the user U. To this end, the decoder Dec receives the classification result of the classifier M4 and the input of the large language model LLM, in particular the text input T, optional context information and the overview image 20 or information obtained from the overview image 20. Using the response A to the subsequent query Q, the transformer encoder Enc once again calculates an embedding Emb. If the classifier M4 indicates that there is now enough information to determine all desired microscope settings 40, the mapping program M3 calculates a mapping from the embedding Emb to the microscope settings 40. The mapping program M3 can be a machine learning network, wherein the mapping program M3 and the classifier M4 can optionally be formed by a common neural network. In this case, the classification result can depend on the respective confidence level of each determined microscope setting 40.

[0192] In a variation of the illustrated embodiment, the transformer decoder Dec can be designed to submit the subsequent query Q to the user U without the classification result of the classifier M4.

[0193] It can also be provided that the transformer decoder Dec first communicates with the user U to obtain all necessary information and only then calculates the translation or regression to the microscope setting 40 using the mapping program M3 (or the transformer encoder Enc and the mapping program M3).

[0194] Figure 4 : Image Analysis Using Large Language Models

[0195] Reference Figure 4 Describes variants of image analysis and image processing using Large Language Models (LLMs). Additionally, Figure 4 The process of a computer-implemented program according to the invention for controlling the microscope 1 is also schematically shown, wherein in particular the process in connection with the previous figures can be implemented.

[0196] The large language model LLM may include an image analysis model M1 that receives and processes an input overview image 20. The language model / machine learning network LLM1 further processes the analysis result 21 calculated by the image analysis model M1 together with the text input T (or information derived therefrom). The output of the machine learning network LLM1 corresponds to the output of the large language model LLM in the previous figures, and the machine learning network LLM1 may include a transformer encoder and / or a transformer decoder as described in the previous figures.

[0197] The analysis result 21 can be an image, at least one segmentation mask or information about the image content. For example, different object types can be identified and the relevant image coordinates can be output optionally. Object properties such as the degree of contamination, the uniformity of the staining process, possible bacterial infestation or the filling level in the sample container can also be evaluated. The image analysis model M1 can be based on a convolutional network (CNN), such as a U-Net architecture, or on a transformer, such as a visual transformer, as described in the introduction to the referenced prior art. Optionally, the image analysis model M1 can be designed to use information Inf from the text input T, context information K and information A from the user U's response to a subsequent query Q from the machine learning network LLM1. The information Inf is calculated by the machine learning network LLM1 and can represent the sample type, sample carrier type or sample preparation (such as the staining process), etc. The image analysis model M1 uses this information Inf in the image analysis. For example, different segmentation models or detection models can be provided and any of these models can be used depending on the sample type.

[0198] The analysis result 21 output by the image analysis model M1 can be generated in text form, which allows the machine learning network LLM1 to process it together with the text information T. In this case, the machine learning network LLM1 can specifically use a pre-trained language model without having to make extensive adjustments to the analysis result 21. If necessary, the pre-trained language model can be fine-tuned to determine the microscope settings. For example, in order to display the analysis result 21 in text form, for example, an equivalent text specification of the segmented object type can be used together with the relevant image coordinate specifications instead of the segmentation mask.

[0199] Alternatively, the overview image 20 may be encoded / mapped in a feature space, for example, using a visual transformer. The encoding of the overview image 20 is then combined with the text input T (or the result of further processing of the text input T by a language model) and jointly processed by a machine learning model. In this case, the encoding of the overview image 20 can substantially preserve the entire information content of the overview image 20, thereby obtaining a more comprehensive image analysis result.

[0200] Figure 5 : Large language models with transformer encoders

[0201] Figure 5 The process of one embodiment of the program according to the present invention is schematically shown, wherein a large language model LLM includes a transformer encoder En for processing a text input T at least once. Other text information - such as user responses, contextual information or text image analysis results - can be processed together with the text input T in the form of combined text.

[0202] The text input T is converted into a sequence of tokens S, where each token is a sequence of the text input T "Show a hydatid cyst with some larvae" larvae)". The tokens can also be divided in other ways. The sequence S is preceded by a start token [CLS], whose value is defined by the training of the transformer encoder Enc. The sequence S with the start token [CLS] is input to the transformer encoder Enc. The transformer encoder Enc computes an output from it, which contains the representation (hidden state representation) [CLS]-Emb of the start token [CLS]. In particular, the associated hidden state representation can be computed for each token. To this end, the transformer encoder Enc can contain multiple self-attention blocks in a known manner, which compute the Q, K and V vectors of the tokens respectively. The block output for the start token [CLS] is calculated by computing the Q vector of the start token [CLS] with the K vector and V vector of each other token, as described in the introduction with reference to the prior art. This output forms the input of the next self-attention block. The self-attention mechanism can be executed multiple times in parallel to provide multi-head attention.

[0203] The hidden state representation [CLS]-Emb of the starting token [CLS] contains the information of the entire sequence S, which is why only the hidden state representation [CLS]-Emb is used as the aggregated sequence representation for further processing by the large language model LLM, without using the hidden state representations of the remaining other tokens.

[0204] The hidden state representation [CLS]-Emb is fed into a mapping procedure M3, which is designed to compute a mapping to the values ​​of the microscope settings 40. The mapping procedure M3 can be learned through supervised training, where the relevant microscope settings 40 are specified as ground truth data for different values ​​(= positions in feature space) of the hidden state representation [CLS]-Emb. The values ​​of the hidden state representation [CLS]-Emb for the training data can be computed from a given sequence S by a trained transformer encoder Enc.

[0205] Depending on the design of the large language model (LLM), overview image 20 is considered at various points, specifically in the transformer encoder (Enc) and / or in the mapping procedure (M3). In the example shown, the hidden state representation [CLS]-Emb of the text input T is fed into mapping procedure M3 along with overview image 20. More generally, overview image 20 can also be preprocessed so that the result of the processing of overview image 20 is fed into mapping procedure M3 instead of overview image 20 itself. For example, a machine-learned feature extractor can be used for preprocessing.

[0206] The text input T contains information about the desired microscopy image, in this case the sample used, i.e. the specification of the sample type (hydatid cyst), and information about the desired magnification of the microscopy image, which is derived from the specification that a hydatid cyst with larvae is to be shown, rather than multiple separate cysts, a single larva or just a specific cell type within the larva.

[0207] Using the text input T, the large language model LLM can deduce information about the size of the object (hydatid cyst) and thus determine the corresponding optical magnification to be set, such as the objective lens selection and / or zoom setting, so that just one hydatid cyst can be seen in the captured microscope image. In this example, the overview image 20 shows about six hydatid cysts, i.e., fluid-filled bubbles, each containing several tapeworm larvae. Through the hidden state representation [CLS]-Emb, the mapping program M3 knows which object type should appear in the overview image 20, and can thus perform segmentation or determine the object position specifically for this object type (or objects of a given size that are known to have characteristics of this object type). Therefore, the large language model LLM uses the overview image 20 to deduce the coordinates of the object to be analyzed based on the text input T.

[0208] In order for the segmentation or determination of the object position to work in a manner that is as error-free as possible, it is usually necessary that the pixel size of the depicted objects corresponds to the pixel size of the objects used in training. Depending on the resolution of the overview image, this is often not the case or would place too high demands on the scope of the training data. However, the large language model LLM is able to estimate how many pixels an object of this object type (here the object type is hydatid cysts and larvae) should usually occupy in the overview image 20 based on the knowledge of the object type (the large language model LLM knows the typical actual object size of this object type) and based on the knowledge of the imaging conditions of the overview image 20. The overview image 20 can then be enlarged or reduced so that the estimated pixel size corresponds to the average pixel size of the objects used in training. The rescaled overview image can then be further processed to determine, in particular, the coordinates of the desired object, etc.

[0209] The object coordinates, objective lens selection and / or zoom setting, and other settings are output by the mapping program M3 as microscope settings 40. In process P6, the microscope 1 is controlled using these microscope settings 40 so that a microscope image 50 is captured in process P7. The microscope image 50 corresponds in terms of microscope image properties to the requirements derived from the text input T. The microscope image 50 can be further used as previously illustrated.

[0210] and Figure 5The description is not intended to be exhaustive. For example, the sequence S may be supplemented with more content, including specific context information, and may also include an "end of sentence" marker and a position code for the marker.

[0211] Figure 6 :Image processing process

[0212] Figure 6 The flowchart of one embodiment of a program for processing microscope images according to the present invention is schematically shown. The illustrated process can be optionally used in combination with the processes of one or more other embodiments.

[0213] In process P1 ', at least one microscope image 50 of a sample is received, for example captured by microscope 1 or loaded from a memory. Furthermore, in flow process P2, a text input T is received from a user U, describing the desired image processing.

[0214] In process P3', text input T and a microscope image 50 are input to the Large Language Model (LLM). This is a machine learning neural network, which, in process P4', calculates processing parameters 41 for processing the microscope image 50 based on the text input T and the microscope image 50. In process P6', these processing parameters 41 are used to select at least one image processing program 60 from a plurality of provided image processing programs. The parameters of the selected image processing program 60 are also determined by the processing parameters 41. Then, in process P7', the microscope image 50 is processed using the calculated processing parameters 41. Specifically, the selected image processing program 60 performs processing using the set parameters. In the example shown, the image processing program 60 calculates a processed microscope image 50A.

[0215] For example, the image processing programs 60 may include, for example, different image sharpening programs, different denoising programs, resolution enhancement programs, artifact suppression programs, and / or programs for virtual coloring different object types. Different programs for the same task may be used to process different object types.

[0216] In the example shown, text input T indicates that microscope image 50 should be sharpened. The large language model (LLM) learns the depicted biological cell types from contextual information K associated with microscope image 50. The large language model (LLM) evaluates microscope image 50 based on the contrast type, in this example, determining that multiple fluorescence channels were used in the imaging (as opposed to, for example, phase contrast or DIC images). The large language model (LLM) also determines which object types (e.g., cell types or organelle types) are located in which image regions. To this end, a segmentation or location determination model sensitive to the present cell types is selected based on the contextual information K. The large language model (LLM) indicates which of multiple provided image sharpening programs 61 to use as processing parameters 41. The properties of microscope image 50 (e.g., contrast type, determined SNR) and the present object types are mapped to one or more provided image sharpening programs 61. Different image processing programs can be selected for different objects located in microscope image 50. In this example, different image regions are defined based on the different cell types located; these different image regions are then processed by different image sharpening programs 61 to form processed microscope image 50A.

[0217] The processed microscope image 50A may be evaluated by the image evaluation model M2 similar to the evaluation of microscope images described with reference to the other figures.

[0218] The large language model LLM uses the text input, the optionally provided contextual information K and the microscope image 50 to evaluate whether the most suitable image processing procedure can be selected and all necessary parameters defined, or whether there is still uncertainty about which of multiple possible image processing procedures or parameter settings are necessary for optimal image processing. In the case of uncertainty, the large language model LLM checks what information about the sample used, the imaging performed or the image processing performed can be used to eliminate the uncertainty. The large language model LLM then submits a corresponding subsequent query Q to the user U. For example, it is possible that the cell type is known, but this cell type varies greatly from one living organism to another, and different organisms have different image sharpening procedures 61. Via the subsequent query Q, the large language model LLM obtains a response A from the user U, which response A, for example, is about the sample and whether the depicted cell type is from a human or a reptile, which is incorporated into the determination of the processing parameters 41.

[0219] Figure 7 : Training of large language models

[0220] Figure 7The training process of a large language model (LLM) is schematically shown. The large language model (LLM) is or includes a pre-trained language model, which is, for example, based on a transformer and has been trained (pre-trained) using a wide range of text databases.

[0221] In subsequent training (fine-tuning), the language model will be adjusted according to the specific task. The figure shows three different fine-tuning processes, which can be performed alternately or cumulatively. The specific process is carried out in the order shown in the figure.

[0222] In the subsequent training of the large language model LLM (fine-tuning FT1), the training data 30 used includes the microscope user manual, the chat logs between the microscope user and the microscope service personnel, and the microscope setting files based on the programming language (for example in XML format). Therefore, the large language model LLM is trained to be able to calculate the microscope settings 40 for the text input T and / or to ask subsequent queries. The specific use of the training data 30 in fine-tuning FT1 depends on the structure of the large language model LLM. For example, using a structure in the form of a transformer decoder, the chat logs of the training data can be copied. Using a transformer encoder-decoder, it is possible to use, for example, a translation of the provided experimental description and the associated (specifically programming language-based) microscope setting specification pairs, where the experimental description is processed in the encoder and the microscope settings are predicted by the decoder.

[0223] The figure also schematically shows the process of fine-tuning FT2 of the large language model LLM, which can be optionally performed after fine-tuning FT1. In fine-tuning FT2, text input T is input into the large language model LLM. The text input T can be loaded from a database or generated by the large language model LLM. The large language model LLM uses the text input T to calculate the microscope settings 40 and the desired microscope image properties 45 that the microscope image to be captured should meet. The microscope settings 40 are used to control the microscope 1 to capture the microscope image 50. The microscope image 50 and the desired microscope image properties 45 are then input into the large language model LLM (or a dedicated image evaluation model) to evaluate whether the microscope image 50 meets the desired microscope image properties 45. The corresponding evaluation results are used as feedback FB to adjust the model parameter values ​​of the large language model. For example, the text input T can indicate that an image with a specific type of cell nucleus clearly visible should be captured. Any desired microscope image property 45 is a clearly visible cell nucleus. If the evaluation result of the microscope image 50 shows that there is no cell nucleus, only a part of the cell nucleus or only a cell nucleus with low visibility in the microscope image 50, corresponding feedback FB can be returned, by which, for example, the illumination wavelength and / or detection wavelength or illumination intensity are adjusted, or in the case where only the cell nucleus is captured in the slice, a modified microscope setting for lateral translation of the sample is performed.

[0224] Similarly, a rule-based reward model can be used where the microscope settings are evaluated by a large language model without capturing microscope images. For example, the calculated microscope settings can be evaluated based on predefined rules that define the minimum illumination intensity and / or exposure time required to achieve a sufficient signal-to-noise ratio, optionally based on different sample types and other microscope settings. For example, the microscope settings 40 can be calculated by a large language model LLM in response to a text input T, for which a sample-saving analysis will result in an insufficient signal-to-noise ratio, in which case the relevant parameter decision will not be reinforced.

[0225] The figure also shows the process of fine-tuning FT3 of the large language model (LLM), which can be performed instead of or in addition to (particularly after fine-tuning FT2) the process of fine-tuning FT3. Fine-tuning FT3 uses RLHF (reinforcement learning from human feedback). For a text input T, the large language model (LLM) computes multiple candidate microscope settings 40A, each for capturing a microscope image 50. A person provides evaluations 56 of the microscope images 50, e.g., which microscope image 50 best implements the text input T. Feedback FB1 can reinforce the parameter decision based on which candidate microscope setting 40A obtains the microscope image 50 with the best evaluation. User feedback can also be collected during normal operation of the microscope and then used for fine-tuning.

[0226] Figure 8 : Image Retrieval

[0227] exist Figure 8 , schematically illustrates the process of a program for retrieving a desired microscope image according to the present invention. Based on a user request, a suitable microscope image 50 is selected from a database D using a large language model (LLM). In addition to the processes of other embodiments, the process described below may also be optionally performed.

[0228] In process P2 , a text input T is received from a user U describing a desired microscope image. The description of the desired microscope image in the text input T can be as described in the previous figures. Specifically, it can be described which sample type and which sample stage should be visible in the microscope image.

[0229] The text input T is input to the large language model LLM, which calculates the desired microscope image attributes 45 based on the text input T in process P13. In terms of content, the microscope image attributes 45 can correspond to the specifications of the text input T, but can also include specifications derived from the text input T. For example, based on the desired object type, a sample preparation procedure or a suitable measurement procedure suitable for the object type can be inferred, so the microscope image to be selected should be captured using such a suitable measurement procedure. For example, if, according to the text input T, cell nuclei should be clearly visible, then as the microscope image attributes 45, it can be determined that a sample image with DAPI stain (or other DNA-binding stain) should be selected, or an image using a typical excitation wavelength of the stain in the imaging should be selected.

[0230] In process P14 , the desired microscope image attributes 45 are input to the transformer encoder Enc2 , which calculates a mapping 45 ′ of the microscope image attributes 45 to the feature space Z / latent space.

[0231] The database D comprises a plurality of microscope images 50, for example tens of thousands or hundreds of thousands of images, for which respective mappings to a feature space Z are obtained (feature space representations 50'). In process P16, the feature space representations 50' are calculated in advance using an encoder Enc1. The encoder Enc1 may be, for example, a transformer encoder or other learned model. Alternatively, the encoder Enc1 may also calculate the embedding of the microscope image 50 by stochastic neighbor embedding (SNE), in particular by t-distributed stochastic neighbor embedding (t-SNE), as described in more detail in the German patent application DE102022121545 of the applicant on August 25, 2022. This embedding creates a feature space in which the distance between the embeddings of two microscope images reflects their similarity. The embeddings of the microscope images 50 can be used together with the relevant microscope image attributes as training data for the encoder Enc2, so that the encoder Enc2 learns a mapping from a set of (textual) microscope image attributes to a point in the feature space Z.

[0232] In process P17, it is determined which feature space representation 50' in the feature space Z is closest to the representation 45' of the microscope image feature 45. In process P18, the associated microscope image 50 is loaded from the database D.

[0233] In the example shown, the loaded microscope image 50 undergoes an optional evaluation or verification, for which purpose it is input into the image evaluation model M2 in process P19. The image evaluation model M2 evaluates whether the selected microscope image 50 satisfies one or more desired microscope image properties 45. To this end, the image evaluation model M2 calculates the corresponding microscope image properties 46 for the selected microscope image 50. The determination of which microscope image properties are to be determined is not fixed but rather determined by the desired microscope image properties 45. For example, if the desired microscope image properties 45 specify a particular contrast type, then the presence or absence of the desired contrast type is determined as the microscope image property 46. If the desired microscope image properties 45 specify a sample type, a classification is performed, alternatively or also, on the sample type depicted in the selected microscope image 50.

[0234] If the evaluation result 55 of the image evaluation model M2 indicates that the microscopic image attributes 46 do not fully match the desired microscopic image attributes 45, corresponding feedback is provided to the large language model LLM in process P20. The large language model LLM, specifically the transformer encoder Enc2, calculates a new mapping 45' of the desired microscopic image attributes 45 to the feature space Z taking into account the feedback. For example, the undesired microscopic image attributes 46 in the previously selected microscopic image 50 are added as desired microscopic image attributes through the feedback. A new microscopic image is selected and evaluated using the new representation.

[0235] If the evaluation result 55 of the image evaluation model M2 concludes that the selected microscope image 50 meets the desired microscope image attributes, it is displayed to the user U in process P10. If the user U is not satisfied with the selected microscope image 50, he or she can supplement the previous text input T in process P21, for example, describing the expected differences in the selected microscope image 50. The above cycle is repeated using the supplemented text input T to select a new microscope image and display it to the user U.

[0236] The large language model LLM can also submit subsequent queries Q to user U, similarly utilizing the elicited user response A to determine desired microscope image attributes 45. For example, when the large language model LLM classifies text input T as ambiguous or potentially misleading, the subsequent query Q can be issued. The large language model LLM can determine the distance between different feature space representations 50' whose associated microscope images 50 satisfy the microscope image attribute 45. If the distance exceeds a predetermined value, the large language model LLM can infer that the text input T is too vague, and therefore a wide range of microscope images 50 meet this requirement. For the distribution of feature space representations 50' whose associated microscope images 50 do meet the requirement, one or more principal axes in feature space Z can be determined, i.e., directions of maximum distance. The large language model LLM can also determine which microscope image attributes contribute most to these principal axes. Subsequent queries Q can then be issued based on these determined microscope image attributes. In this way, through several subsequent queries Q, the desired microscope image attributes 45 can be determined, allowing selection of potentially suitable microscope images.

[0237] Figure 9 :microscope

[0238] Figure 9An embodiment of a microscope system 100 according to the present invention is shown. Microscope system 100 comprises a microscope 1 and a computing device 12, which can be part of or separate from microscope 1. Microscope 1 includes a stand 2, which supports the other components of the microscope. Specifically, these components include: an illumination device 3; a condenser 5 for directing illumination light onto a sample area; a nosepiece / turret, on which an objective 6 is mounted in the illustrated example; a sample stage 13 with a holder 8 for holding a sample carrier 9; and a microscope camera 7. When objective 6 is pivoted into the microscope's optical path, microscope camera 7 receives detection light from a sample area, where a sample may be located, in order to capture an image of the sample. In principle, sample 10 can be or include any object, fluid, or structure. To capture an overview image 20 of the sample environment, microscope 1 can use a low-magnification objective and / or an optional additional overview camera (not shown) to observe the sample area at a perpendicular or oblique angle. To capture the overview image, an illumination device can be used, such as a plurality of light-emitting diodes 4, whose illumination direction, for dark-field measurements, is oriented at an oblique angle relative to the detection axis. In addition to the microscope camera 7, the sample can also be observed through the eyepiece 14. The microscope system 100 also includes a computer program 11, which is stored on a non-volatile storage medium and is illustrated only schematically. The computer program 11 or the computing device 12 is configured to execute the program variants described with reference to the other figures. In the example shown, the microscope 1 is an optical microscope, but other types of microscopes are also possible.

[0239] The variants described with reference to the different figures can be combined with each other. The described embodiments are purely illustrative and modifications are possible within the scope of the appended claims.

[0240] Reference Symbol List

[0241] 1 Microscope

[0242] 2 brackets

[0243] 3 Lighting equipment

[0244] 4 LEDs

[0245] 5 Condenser

[0246] 6 Objective lens

[0247] 7 Microscope / System Camera

[0248] 8 Fixing bracket for fixing sample carrier

[0249] 9 Sample carrier

[0250] 10 samples

[0251] 11 Computer Program

[0252] 12 Computing Devices

[0253] 13 Sample table

[0254] 14 Eyepiece

[0255] 20 Overview Images

[0256] 21 Analysis Results

[0257] 30 training data

[0258] 40 Microscope Setup

[0259] 40A Candidate Microscope Setup in Large Language Model Training

[0260] 40B Adjusted microscope settings

[0261] 41 Processing Parameters

[0262] 45 Microscope image properties that need to be satisfied from text input

[0263] 45' Mapping of desired microscope image attributes to feature space Z

[0264] 46 Microscope image properties of the captured microscope image 50

[0265] 50, 50B Microscope image

[0266] Microscope image after 50A processing

[0267] 50' Feature Space Representation of Microscope Images

[0268] 55 Evaluation results of the image evaluation model

[0269] 56 Evaluation of Microscope Images

[0270] 60 Image Processing Programs

[0271] 61 Image Sharpening Program

[0272] 100 Microscope System

[0273] A. User response

[0274] [CLS] Start tag

[0275] [CLS]-Hidden state representation of Emb start token

[0276] D database

[0277] Dec Transformer decoder for large language models

[0278] Enc, Enc1, Enc2 (Transformer) encoders

[0279] Emb is the embedding of the received input data calculated by the encoder Enc

[0280] Feedback in large language model training for FB, FB1

[0281] FT1-FT3 Subsequent training (fine-tuning) of large language models

[0282] Information to consider in Inf image analysis

[0283] K context information

[0284] LLM Large Language Model

[0285] LLM1 LLM Machine Learning Network

[0286] M1 Image Analysis Model

[0287] M2 Image Evaluation Model

[0288] M3 Mapper

[0289] M4 classifier

[0290] M5 submodel

[0291] P1-P21 Process of the Invention

[0292] P1'-P7' Process of the present invention

[0293] Q is a follow-up query for the user's large language model

[0294] S-labeled sequence

[0295] T Text Input

[0296] U microscope users

[0297] Z feature space

Claims

1. A computer-implemented program for controlling a microscope, comprising: receiving (P2) text input (T) describing a desired microscope image and a sample (10) to be employed; receiving (P1) an overview image (20) of a sample (10) taken; feeding (P3) the text input (T) and the overview image (20) into a large language model (LLM), which is a trained machine learning neural network, for processing the text input (T) and the overview image (20) together to compute (P4) microscope settings (40) for capturing a microscope image (50) corresponding to a desired microscope image; and A microscope image (50) is captured (P7) using microscope settings (40) computed using a large language model (LLM).

2. A procedure according to any preceding claim, in, The microscope settings (40) determined the following: - magnification; - Imaging speed; - The location of the sample area to be analyzed and the position of the sample stage; - lighting power or intensity; - Duration of lighting; - Focus; - Condenser settings; -The excitation wavelength used for fluorescence excitation; - the number of detection channels and the characteristics of the detection channels; - Comparison procedures; - Single shot or time series imaging.

3. A program according to any preceding claim, Wherein the text input (T) indicates at least one of the following: - Which objects should be visible in the desired microscope image; - the type of sample involved; - Which dyes were used in sample preparation; - whether bleaching of the sample (10) is permitted; and -Whether to prioritize image quality or sample preservation.

4. A program according to any preceding claim, in, A large language model (LLM) is trained to determine whether all microscope settings to be set can be set via the received text input (T), or whether a subsequent query (Q) is necessary to the microscope user for any of the microscope settings to be set; Therein, in case of a subsequent query (Q), the user response (A) is fed into a large language model (LLM) as further textual input (T).

5. The program according to the preceding claim, in, In the case of subsequent queries (Q), a large language model (LLM) is trained not to ask for the values ​​of microscope settings, but rather to query for properties of samples (10), desired microscope images, or experiments to be performed that are associated with these microscope settings.

6. A program according to claim 4 or 5, in, Evaluate whether it is necessary to submit a follow-up query (Q) to the user Through special encodings in the output of large language models (LLMs) or The necessity of the subsequent query (Q) is classified through a transformer encoder model of a large language model (LLM).

7. The program according to any one of claims 4 to 6, in, The large language model (LLM) includes a transformer decoder (Dec) and a transformer encoder (Enc), wherein the transformer decoder (Dec) is trained to issue a subsequent query (Q) and the transformer encoder (Enc) is used to calculate microscope settings (40) based on text input (T) and a user response (A) received in response to the subsequent query (Q).

8. A program according to any preceding claim, in, The text input (T) is converted into a sequence of tokens (S); A start marker ([CLS]) is placed before the sequence (S), whose value is learned during the training of the transformer encoder (Enc); The start token ([CLS]) and the sequence (S) are input into the transformer encoder (Enc), and the output of the transformer encoder (Enc) calculated thereby contains the hidden state representation ([CLS]-Emb) of the start token ([CLS]); Therein, the hidden state representation ([CLS]-Emb) of the start token ([CLS]) is input into a mapping program (M3) designed to calculate a mapping to the value of the microscope setting (40).

9. A program according to any preceding claim, in, training a large language model (LLM) to derive (P5) from a text input (T) the microscopic image properties (45) that a microscopic image (50) needs to satisfy in order to conform to the text input (T); wherein the machine learning image assessment model (M2) calculates relevant microscope image attributes (46) based on a microscope image (50) captured using a microscope setting (40) determined using a large language model (LLM); wherein the microscope image (50) is used (P10) if the microscope image attribute (46) of the microscope image (50) meets the required microscope image attribute (45) derived from the text input (T); and wherein, in the event that the microscope image properties (46) of the microscope image (50) do not meet the microscope image properties (45) to be met, feedback (P12) is provided to a large language model (LLM), and the large language model (LLM) uses the feedback to determine adjusted microscope settings (40B) to capture a new microscope image (50B); Therein, a large language model (LLM), upon determining adjusted microscope settings (40B) for capturing a new microscope image (50B), performs an evaluation (47) to determine whether a subsequent query (Q) should be issued to a user (U) to authorize capturing a new microscope image (50B), due to the impact that the adjusted microscope settings (40B) may have on the sample (10) or the measurement duration.

10. The program according to any one of the preceding claims, in, The large language model (LLM) comprises an image analysis model (M1) which computes an analysis result (21) from an overview image (20), the analysis result (21) being computed by the large language model (LLM) together with a text input (T) to determine a microscope setting (40); The analysis result (21) determines one or more of the following: - sample carrier type, - sample type, - Whether chemical dyes were used in sample preparation, and if so, what chemical dyes were used, - Experiment type, - one or more focus positions, - the state or degree of contamination of the sample carrier or the sample (10), - the location of the area in question, and - The information content of the text or barcode on the sample carrier.

11. A procedure according to any preceding claim, in, When calculating the analysis result (21) based on the overview image (20), the text input (T) is also taken into account.

12. A program according to any preceding claim, The text input (T) also describes the desired image processing; The program also includes Inputting (P3') the text input (T) and the microscope image (50) to a large language model (LLM) which is trained to calculate (P4') processing parameters (41) for processing the microscope image (50) based on the text input (T) and the microscope image (50); and The microscope image (50) is processed (P7') using the calculated processing parameters (41).

13. A computer-implemented program for processing microscope images, comprising: receiving (P1') a microscope image (50) of a sample (10); receiving (P2) a text input (T) describing a desired image processing; Inputting (P3') the text input (T) and the microscope image (50) into a large language model (LLM), the large language model being a machine learning neural network trained to calculate (P4') processing parameters (41) for processing the microscope image (50) based on the text input (T) and the microscope image (50); and The microscope image (50) is processed (P7') using the calculated processing parameters (41).

14. A program according to claim 12 or 13, in, Text input (T) indicates at least one of the following: - the type of sample in question; - What dyes or stains were used during sample preparation; - Whether the processing of the microscope image (50) requires denoising, resolution enhancement, deconvolution, changing the depth of field, changing the type of virtual contrast, background suppression or artifact removal.

15. A program according to any one of claims 12 to 14, in, The processing parameters (41) determine one or more of the following: - Selection of any of a variety of deconvolution algorithms, any of a variety of resolution enhancement algorithms, or any of a variety of denoising algorithms; - The number of iterations to perform, or the regularization term for the deconvolution, resolution enhancement, or denoising algorithms.

16. A program according to any preceding claim, in, Text input (T) is created by the user through voice input and subsequent speech-to-text processing, or through an input device or graphical interface.

17. A program according to any one of the preceding claims, in, The large language model (LLM) is formed by fine-tuning (FT1-FT3) an existing large language model, wherein the training data (30) of the fine-tuning (FT1-FT3) includes one or more of the following: - Microscope user manual; -Chat logs between microscope users and microscope service personnel; - Notes from microscope training sessions.

18. A computer-implemented program for providing a desired microscope image, comprising: receiving (P2) a text input (T) describing a desired microscope image; feeding the text input (T) into a large language model (LLM) (P3), the large language model being a machine learning neural network trained to infer (P13) desired microscopy image attributes (45) from the text input (T); and A microscope image (50) is loaded (P18) from a database (D) containing microscope images according to desired microscope image properties (45).

19. A program according to any preceding claim, in, training a large language model (LLM) to compute (P15) a mapping (45') of desired microscope image features (45) to a feature space (Z); wherein each microscope image (50) in the database (D) is provided with a respective feature space representation (50'); and Therein, a large language model (LLM) loads from a database (D) the microscope image (50) whose feature space representation (50') is closest to the representation (45') of the desired microscope image features (45).

20. A microscope system having A microscope (1) for imaging; and Computer device (12) configured to execute a computer-implemented program according to any one of the preceding claims.

21. A computer program comprising instructions stored on a non-volatile data storage medium, which, when executed by a computer, causes the computer to execute a program according to any one of claims 1 to 19.

Images