Microscope system and related procedures

Abstract

In a computer-implemented program for controlling a microscope, a textual input (T) is received that describes a desired microscope image and an employed sample (10). The textual input (T) and an overview image (20) of the employed sample are input into a large language model (LLM) that is trained to process the textual input (T) and the overview image (20) together to compute (P4) microscope settings (40) for capturing a microscope image (50) that corresponds to the desired microscope image. The microscope image (50) is then captured with these computed microscope settings (40).

Description

Technical Field

[0001] This disclosure relates to a microscope system and a procedure for providing desired microscope images. Background Technology

[0002] Modern microscopes are highly complex systems with a wide range of setup options. Microscope settings and the microscope components used must be selected based on the desired sample analysis. The ideal setup depends not only on the sample and its preparation but also on the objectives of the analysis. For example, a microscope user might want to analyze a group of biological cells in a sample, or a single cell or specific organelle requiring an illumination setting suitable for fluorescent dyes. Users typically begin by describing the planned imaging from an application perspective—what needs to be imaged. However, setting the optimal imaging technique parameters to obtain the desired image data requires a high level of expertise and experience. While microscopes increasingly offer automated processes to improve user comfort, translating application descriptions into imaging technique parameter settings remains a major obstacle for many microscope users. Due to a lack of sufficient expertise, after spending considerable time and effort on the necessary setup, only negligible results are often obtained.

[0003] Voice-controlled systems for microscopes are known, such as voice-controlled surgical microscopes using machine learning language models, as described in CN112149606A. Voice commands are compared with stored related commands such as zoom or white balance. DE102020 / 108796A1 describes another voice-controlled surgical microscope where the surgeon can use voice commands to switch from one stage with predetermined microscope settings to the next stage with other predetermined microscope settings. Thus, in known (surgical) microscopes, voice commands indicate specific microscope settings or a set of pre-saved microscope settings, thereby replacing similar commands entered via a mouse, touchscreen, or keyboard, see also paragraph

[0030] of US2018 / 0348500A1. This also appears in McDermott, S. et al.'s paper, "Controlling and scripting laboratory hardware with open-source, intuitive interface: open-source flexible voice commands and open-source flexible Blockly library."

[0004] Intuitive interfaces: OpenFlexure Voice Control and Open FlexureBlockly”) arXiv:2209.14947v2 [physics.ins-det], 2 February 2023, a paper describing a system in which: or also in Holburn, D. et al. “Voice Control of the Scanning Electron Microscope Using a Low-Cost Virtual Assistant”, Microsc. Microanal. 27 (Suppl 1), 2021, doi:10.1017 / S1431927621009685.

[0005] The user can issue voice commands here, such as “autofocus”, “capture image”, “move 100 steps in X axis”, and the microscope will execute these commands accordingly. Thus, the known voice control systems do not help to solve the problem of converting an application experiment description into imaging technology parameter settings for imaging. On the contrary, the voice commands must specify the imaging technology parameter settings. It is not possible to use natural (complex) language for the experiment description.

[0006] The imaging procedures of modern microscopes provide a virtual assistant (wizard) that can simplify the operation but does not overcome the above problems. The wizard usually cannot implement complex requirements, react to results, ask the user follow-up questions, and can only operate in a predetermined order, even if these orders are not suitable for a particular experiment. Only through specially programmed modules can the wizard be extended to new conditions, which represents a significant investment of time and money.

[0007] Likewise, the ideal processing of the captured images or raw data requires a lot of specialist knowledge and additional time investment. While standard settings of image processing algorithms or learning models generally achieve good results, the processing parameters must be set individually to achieve the best image quality, in particular depending on the sampled sample and image properties, such as signal-to-noise ratio.

[0008] As background to the invention, the following prior art is also referred to:

[0009] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, Gomez, L., Kaiser, L., Polosukhin, I.: "Attention Is All You Need" rXiv: 1706.03762v5 [cs.CL] 6 Nov 2017.

[0010] The neural network presented herein is called a transformer. Traditional recurrent models process input sequences sequentially, while a transformer includes attention blocks that process the input sequence as a whole, not sequentially. If attention blocks are used in an encoder, it is called a transformer encoder with self-attention blocks. First, the embeddings of each input element (token) in a sequence are computed, where a token embedding is also simply called a token for brevity of language. A token can correspond to a word or a word piece, where the sequence can be a sequence of words or multiple sentences. Tokens in a sequence have position encodings and are input into a self-attention block, which computes three representations for each token by matrix multiplication, called query (Q), key (K), and value (V) vectors, respectively. In addition, for a token (hereafter called token A), the relevant Q vector is multiplied with the K vector of another token, then a softmax function is applied, and then multiplied with the V vector of the other token. This process is done for token A with all other tokens in the sequence. The result of the computation is the output of the self-attention block for token A. The outputs are computed in the same way for every other token in the sequence. These outputs form the input to the next self-attention block. A chain of tokens is usually input into multiple self-attention blocks used in parallel, which is called multi-head attention. The output computed by the last self-attention block of the encoder for a token in the sequence is also called the final hidden state embedding of the token.

[0011] Similar to a transformer encoder, a transformer decoder is a network with self-attention blocks that are used together in the computation of elements in an input sequence to estimate the most likely next token of the sequence. The next token will then become part of the input sequence for generating the next token. The self-attention blocks perform the same computation as described for the transformer encoder.

[0012] In a transformer encoder-decoder, multiple consecutive attention blocks are used, where the Q vectors come from the tokens of the decoder, and the K and V vectors come from the tokens of the encoder (final hidden state embeddings).

[0013] With respect to the transformer encoder, also refer to the following article:

[0014] Devlin, J., Chang, M. W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv: 1810.4805v2 [cs.CL] 24 May 2019.

[0015] Described herein is a transformer encoder, in which one start token, called special classification token ([CLS] token), is placed before the input sequence. From the input sequence, the transformer encoder computes a final hidden state embedding for the [CLS] token, which contains information of the whole sequence, since it is applied together with the other tokens to the computations made in the self-attention blocks. The training of the transformer encoder is first performed in a pre-training fashion using a collection of texts, e.g. by masking one word or token in the training data for which the transformer encoder is to make a prediction. After the pre-training is finished, another network layer is attached on top of the transformer encoder, and further training (fine-tuning) is performed using other training data, like training data for a classification task. In this process, the final hidden state embedding of the [CLS] token is mapped to a class from the last network layer or layers.

[0016] The input sequence of the transformer encoder does not necessarily have to contain text, or not necessarily only contain text at all, but can contain image data, like the network described in Vision Transformer:

[0017] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Deghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929v2 [cs.CV] 3 June 2021;

[0018] Wu, H., et al. CvT: How to integrate convolution into vision transformers. arXiv:2103.15808v1 [cs.CV] 29 March 2021.

[0019] Transformer decoders have become popular through GPT networks and the like, as described, for example:

[0020] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. (2018). Improving language understanding by generative pre-training. arXiv: 1312.0439v2 [cs.CL] 2018.

[0021] https: / / web.archive.org / web / 20200609115647 / https: / / cdn.openai.com / research-covers / language-unsupervised / language_understanding_paper.pdf.

[0022] After the first training (pre-training) of the model, further training (fine-tuning) can be performed using reinforcement learning obtained from human feedback to improve the probability of desired (text) generation, as described, for example:

[0023] Lambert (N.), et al., "Illustrating Reinforcement Learning from Human Feedback (RLHF)," December 09, 2022

[0024] https: / / web.archive.org / web / 20230102122855 / https: / / huggingface.co / blog / rlhf

[0025] OpenAI (2023), "GPT-4 Technical Report," https: / / web.archive.org / web / 20230314180351 / https: / / cdn.openai.com / papers / gpt-4.pdf

[0026] https: / / web.archive.org / web / 20230314180351 / https: / / cdn.openai.com / papers / gpt-4.pdf

[0027] OpenAI (2020), "Language Models are Few-Shot Learners," arXiv:2005.14165v4 [cs.CL] 22 Jul 2020.

[0028] Text input can also be processed with inputted image data, as described in the "GPT-4 Technical Report" cited above.

[0029] Although the output quality of recurrent neural networks such as LSTMs is often inferior to that of transformers, they are an alternative to transformers:

[0030] Hochreiter, S., and Schmidhuber, J.: "Long Short-term Memory," Neural Computation 9(8): 1735-1780, December 1997,

[0031] DOI: 10.1162 / neco.1997.9.8.1735

[0032] As further background, reference is made to US 2020 / 0371333 Al, which describes a machine learning validation model for checking image processing results. SUMMARY

[0033] It can be considered that the object of the present application is to provide a microscope system and a program that can provide a microscope image with desired properties without the user having to have expert knowledge in setting microscope parameters or providing desired processing parameters for the image.

[0034] This object can be achieved by a microscope system and a program having the features of the independent claims.

[0035] According to one embodiment of the present application, a microscope is controlled by a computer-implemented program. A textual input is received that describes at least one desired microscope image and a sample used. In addition, an overview image of at least one sample used is received. The textual input and the overview image are input into a large language model that is a machine-learned neural network that, after training, can jointly process the textual input and the overview image to compute microscope settings for capturing a microscope image corresponding to the desired microscope image from the sample used. Then, the microscope captures the microscope image according to the microscope settings computed by the large language model.

[0036] With the present application, a user can, in particular by voice input, describe an experiment from an application perspective and name properties of a microscope image to be captured without having to name or specify the microscope settings themselves. For example, the user can tell the large language model that a certain type of single cell or a cell cluster of a sample shall be imaged. The large language model can use this information to identify an appropriate magnification for capturing the single cell or the cell cluster, while the overview image is used to navigate to the appropriate location where the desired cell is located. Imaging parameters like illumination intensity or fluorescence settings can be determined by the large language model from the textual input and the overview image without the user having to specify the illumination intensity or the fluorescence excitation or detection channels. In this way, the user can obtain high-quality imaging without having to master a large amount of expert knowledge and without having to perform laborious manual settings.

[0037] According to another embodiment of the present application, a microscope image is processed by a computer-implemented program. At least one microscope image of a sample is received. A textual input is also received that describes a desired image processing. The textual input and the microscope image are input into a large language model that is a machine-learned neural network that, after training, can compute processing parameters for processing the microscope image from the textual input and the microscope image. The microscope image is processed using the computed processing parameters.

[0038] Advantageously, the user does not need to specify all processing parameters for processing the microscope image itself; instead, a text description of the desired image processing is sufficient, which the large language model uses together with the microscope image to determine the processing parameters. For example, from a number of de-noising algorithms provided, one can be selected for de-noising depending on the sample type, wherein the specific value of the algorithm regularization parameter influences the quality of the processing result, and the ideal value depends on, e.g., the sample type and the signal-to-noise ratio of the relevant image region. Thus, the user does not need to have detailed knowledge of the image processing program used, nor does he / she need to make the processing settings himself / herself, in order to process the microscope image with ideal processing parameters.

[0039] The computer-implemented program of another embodiment of the present application is directed to retrieving a desired microscope image. A text input describing the desired microscope image is received and input into a large language model. The large language model is a machine-learned neural network trained to at least compute properties of the desired microscope image from the text input and to load a specific microscope image from a database containing microscope images depending on the properties of the desired microscope image.

[0040] Typically, a large database containing hundreds of thousands or millions of microscope images can be used, wherein the microscope images are either not fully annotated or not annotated at all. While one or more microscope images in the database can comply with the user’s specifications, finding and retrieving these microscope images is very difficult due to the lack of annotations. In different variants of the present application, the large language model can select a suitable microscope image from the database based on the user’s simple text input without the user providing precise specifications or full annotated image data. This way, the large database can be accessed and utilized in a more efficient way.

[0041] The microscope system of the present application comprises a microscope for imaging and a computing device configured to execute any of the computer-implemented programs.

[0042] The computer program of the present application comprises instructions stored on a non-transitory data storage medium that, when executed by a computer, cause the computer to execute any of the computer-implemented programs.

[0043] Optional features

[0044] Variants of the microscope system according to the present application and of the program according to the present application are the subject of dependent claims and are explained in the following description.

[0045] Microscope settings

[0046] The microscope settings or microscope parameters determined by the large language model can at least be at least one or in particular all of the following settings:

[0047] - magnification, in particular selection of the objective and / or setting of the magnification by a zoom optics in order to include single organelles or cell clusters in the microscope image according to user specifications; the selection of the objective is according to the set calculated magnification, but also according to the numerical aperture and working distance of the objective;

[0048] - imaging speed, in particular pixel dwell time, scan speed or scan resolution;

[0049] - position of the sample area to be analyzed, where one or more images are captured, and the associated sample stage position;

[0050] - contrast procedure, for example phase contrast microscopy, differential interference contrast (DIC), Hoffman modulation contrast, bright field or dark field measurements, fluorescence microscopy, confocal microscopy, two- or multiphoton microscopy, polarized light microscopy, interferometric microscopy, digital holographic microscopy (DHM) or Fourier ptychographic microscopy;

[0051] - illumination power or intensity; duration of the sample illumination;

[0052] - camera exposure time, pixel size of the combined camera sensor elements or generally other detection parameters;

[0053] - general optical path parameters like mirror, filter and pinhole size;

[0054] - focusing, in particular aperture stop influencing the depth of field;

[0055] - condenser settings, according to the selected magnification, in particular setting of the condenser aperture and / or condenser height;

[0056] - excitation wavelength used in fluorescence measurements, in particular according to the employed stain and the type of sample structure of interest;

[0057] - number and characteristics of the detection channels, in particular the respective wavelength ranges;

[0058] - camera sensitivity.

[0059] The microscope settings can also specify whether a single image is to be captured, whether an image stack consisting of multiple images staggered along the optical axis is to be captured, or whether a time series is to be captured in which the same sample region is analyzed at time intervals by a single image or an image stack. An entire experimental procedure with different settings and sequences of imaging events can also be defined by the microscope settings determined by the large language model. In addition to the settings described above, it can also be specified for an experimental procedure which of the sample wells (in the case of multi-well plates / multi-chamber slides) are to be captured using which settings, since not all wells are part of the experiment, and these sample wells are often subdivided into specific experimental directions of inquiry. During the experimental procedure, it can also be specified that there is a time interval between measurements, Z-stack settings and search or holding or restarting of the focal plane, or that the experiment is to be repeated.

[0060] The microscope settings are set by the large language model such that at least one microscope image captured with these settings corresponds to a desired microscope image. This can be understood as meaning that the captured microscope image and the desired microscope image are consistent in terms of microscope image properties, which are derived by the large language model from the received input.

[0061] Text input

[0062] The user can create the text input by voice input or an input device on the microscope or computer. In the case of voice input, a voice-to-text process can be carried out. However, the user can also create the input directly in the form of text, for example by typing via a chat interface, command line or script API. In particular, the input device can use a keyboard, computer mouse, joystick or touchscreen.

[0063] The text input can be written in natural language or in a technical or domain-specific language, such as a programming language, wherein the text input is provided in the form of an XML file, for example.

[0064] In addition, a graphical interface can also be used, for example via which a slider can be moved. In contrast to a conventional slider which directly indicates a microscope setting, such as the intensity of the laser illumination (which can also be used), the slider described above can be used to describe properties of the desired microscope image or of the sample used. For example, a slider can be used to set the width / length of the microscope image to be captured to be approximately 100 times, 10 times, 2 times, 1 time or 0.5 times the diameter of the cells of the biological sample used. Depending on this setting, the microscope settings will be automatically performed such that only part of a cell, the entire cell or multiple entire cells can be seen in the subsequently captured microscope image.

[0065] In particular when at least one microscope image is to be captured by the text input, the text input can indicate at least one of the following:

[0066] - Which objects, in particular object types and object numbers, should be visible in the desired microscope image. For example, it can be distinguished between single organelles, single cells and cell clusters. The text input can also indicate cell types or organelle types, for example, nuclei, Golgi apparatus, endoplasmic reticulum, mitochondria.

[0067] - Which sample type is involved, for example, from which tissue or living organism cells.

[0068] - Which dyes / stains were used during sample preparation, in particular which histochemical stains or fluorescent dyes were used.

[0069] - Whether bleaching of the sample (caused by high illumination light intensity) is allowed. This can be indicated by a scale, for example, bleaching is not allowed in any case, bleaching is only allowed if it is necessary to achieve sufficient image quality, or bleaching is always allowed in order to achieve the best image quality.

[0070] - Whether high image quality, i.e. high signal-to-noise ratio, is prioritized or the preservation of the sample.

[0071] The text input can also include data from various sources. For example, the user can enter the information in at least one of the ways described above, while the text input (or a separate input of contextual information) can also include provided text information about the sample used and / or the current experiment. For example, information available in a data storage medium of the sample used about previous sample preparation and / or sample type. This information can be entered as text input together with the user input into the large language model. The text information can also be present on the sample carrier in the form of text or machine-readable one-dimensional / two-dimensional barcodes; in this case, the text information can be read by analyzing the overview image and entered as part of the text input together with the user input into the large language model. The text input can also include text created by a computer program, for example, by a virtual assistant, which suggests a measurement or experiment description to the user for the planned experiment. For example, the measurement description can state that a single biological cell of a specific cell type is to be localized and then measured in a time series, where all stages of cell division should be included, although the illumination intensity should be very high in order to achieve a good signal-to-noise ratio, but the cell should not be damaged by the high illumination intensity in any case. If the user confirms the suggested measurement description, if necessary after making revisions, it is entered as text input into the large language model.

[0072] If the text input is used for processing the microscope image, the text input can, as indicated, depict the type of sample and / or the dye or stain employed in the sample preparation process, as described above. The text input can also indicate the processing goal to be achieved by the image processing, for example de-noising, resolution enhancement, de-convolution and / or SIM calculation (SIM is an abbreviation for structured illumination microscopy, i.e. the calculation of a higher resolution image from a plurality of images with different illumination structures).

[0073] Processing parameters for image processing

[0074] The text input and the captured or loaded microscope image can be used to determine processing parameters for processing the microscope image. In this case, the text input comprises a description of the desired image processing, in addition or alternatively to the above-mentioned possible text input contents. The text input is input into the large language model together with the microscope image, which model is trained to calculate processing parameters from the text input and the microscope image. The image processing program then processes the microscope image using the calculated processing parameters.

[0075] The processing parameters can in particular specify one or more of the following:

[0076] - a selection from a plurality of de-convolution algorithms (in particular calculated from an estimate of the point spread function PSF of the microscope system), a plurality of resolution enhancement algorithms or a plurality of de-noising algorithms. The suitability of the algorithms depends primarily on, for example, the type of sample and / or the type of sample carrier used. Different algorithms with the same functionality can therefore be provided, for example different de-noising algorithms for specific different sample types. One of the algorithms is then selected depending on at least one sample type.

[0077] - specification of parameters for the (selected) de-convolution algorithm, resolution enhancement algorithm or de-noising algorithm, or for the SIM calculation (SIM: structured illumination microscopy). These parameters can relate to, for example, the number of iterations to be performed, or to the regularization term, in particular the regularization function and the regularization parameter. The processing parameters can also be used to estimate the PSF of the imaging, wherein the PSF is then used for the image processing. The parameters of the de-noising algorithm can be set differently depending on the noise level of the microscope image to be processed.

[0078] - virtual staining of specific sample structures, or some other virtual modification of the contrast type, for example between phase contrast, bright field, DIC or fluorescence images.

[0079] - a change in the depth of field, wherein a higher or lower depth of field is mathematically simulated, so that blurred structures become correspondingly sharper or more blurred, while the structures in the focal plane remain essentially unchanged.

[0080] In addition to the exemplarily illustrated algorithms, the image processing program can also involve other functions. In particular, the image processing program can be used to improve the image quality, increase the visibility of specific object or structure types (for example, by virtual staining or virtual changes of the contrast type), or suppress undesired object or structure types, in particular by background suppression or artifact removal. Segmentation masks can also be calculated. The output of the image processing program can be an image, although it can also be other processing results in general, for example, a classification with respect to object types or image properties, or a regression with respect to physical values of specific objects (such as their size or geometry), or a calculation of statistical values of objects (such as the number of objects of a specific type or the size distribution of objects of a specific type). The cited image processing programs can be machine learning models or classic algorithms without a learning model. The processing parameters can specify the selection of a learning model or an algorithm and, if necessary, also parameters to be set for the algorithm or model.

[0081] In order to achieve the desired image processing, new training of the image processing model is also required, in which, in particular, microscope images are used as training data. The large language model can be designed to initiate new training, in particular to propose training parameters or hyperparameters (for example, scaling of the training data, learning rate or learning rate increment) and suitable model architectures. The large language model can also be designed to evaluate whether a given microscope image can be processed correctly by an existing model or whether new training should be carried out.

[0082] Structure and training data of the large language model

[0083] The large language model is a deep artificial neural network which, among other things, receives text from a user as input and generates an output which specifies parameters for the subsequent image generation. In contrast to simple machine learning models, the large language model comprises several million model parameters which need to be defined in the model training, in particular more than one billion parameters.

[0084] For example, a large language model can comprise a transformer encoder (as described in the introduction with reference to the article "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding") and an optional visual transformer for synchronous image processing. Alternatively or additionally, a transformer decoder can also be provided, which has the properties described in the introduction with reference to the prior art. Furthermore, the liquid can employ the design of a transformer encoder-decoder as described in the introduction. More generally, an encoder or decoder with self-attention blocks can also be implemented in which, instead of the transformer's Q, K and V vectors, different matrix calculations are carried out using vectors calculated from the tokens when calculating. In principle, a large language model can also be designed using models in the form of LSTMs as described with reference to the prior art. In variants of the embodiments described below, the transformer can also be replaced by a different structure, in particular in the case of image data, an LSTM or a CNN can be used instead.

[0085] The different implementations of the large language model comprise one (transformer) encoder. The text input is converted into a sequence of tokens in a manner known per se, e.g. on the basis of words or word parts. In addition, a special token, in the following referred to as the start token or [CLS] token, is added. It can be located before the tokens of the text input, but in principle also at another position in the sequence or after the sequence. The value of the start token can be predetermined by the training of the large language model. The encoder computes a mapping of the text input with the added start token to an output, which can comprise an output vector (hidden state embedding or representation) for each token. In the case of a transformer encoder, the hidden state embedding of the start token contains information of all other tokens in the text input, which is incorporated into the embedding of the start token by the computation in the self-attention block using the Q, K and V vectors. After the encoder has processed the input, the position of the start token in the embedding space will be the feature vector for further processing. The hidden state representation of the start token is input into a mapper, which can be regarded as part of the large language model. It is intended to compute a mapping of the feature vector (i.e. the hidden state representation of the start token) to a target parameter, in particular a value of a microscope setting. The mapper can comprise a linear mapping or a further machine learning network, e.g. a multi-layer perceptron. This procedure has similarities to the use of the [CLS] token in the BERT and ViT models cited in the introduction. As with ViT, the encoder can also process input image data (in particular the overview image here) in order to incorporate image information into the feature vector or to generate a separate feature vector for the overview image. A pre-trained language model, in particular a language model of the prior art, can also be used as transformer encoder. The mapping procedure / further machine learning network can be designed as a regression model or a classification model. It can be learned after the transformer encoder has completed training or pre-training, e.g. by supervised learning. In supervised training, a set of desired microscope settings is specified for a text input. The further machine learning network learns from this to compute a mapping of the hidden state embedding of the [CLS] token to a set of microscope settings, or to compute a mapping of the hidden state embedding of the [CLS] token and a feature vector of the overview image to a set of microscope settings. The training data can be actively collected or obtained by passive observation of users. The training can optionally be implemented with the aid of reinforcement learning. As described in more detail below, an additional evaluation model can be used to evaluate whether a microscope image captured using a determined microscope setting represents a successful implementation of the text input. In principle, the training data described above can also be used for joint training, in which the transformer encoder is trained or fine-tuned in addition to training the other machine learning network.

[0086] The training data for the large language model can in particular comprise one or more of the following:

[0087] - a microscope user manual;

[0088] - a chat between a microscope user and a microscope service person, wherein the chat comprises questions and answers related to the use and troubleshooting of the microscope. The chat can comprise a conversation in text form or a transcribed conversation like a telephone call.

[0089] - a recording of a training session / activity related to the microscope, in particular related to the use of the microscope or the training of the service person in the support of the microscope user. If the training is oral, the recording can also be a transcribed recording.

[0090] The above training data describes in particular how to choose microscope settings, in particular as a sample type or desired properties of the image to be captured. Such training data is particularly useful for the case that the large language model is designed as a transformer decoder.

[0091] The large language model can also be formed by fine-tuning an existing large language model, in particular that has already been pre-trained. The above training data, e.g. the microscope user manual, can be used as training data for the fine-tuning. The fine-tuning can optionally be self-monitored, i.e. the model tries to reproduce the microscope user manual or other training data during training.

[0092] As described in the paper “Reinforcement Learning from Human Feedback” cited in the introduction, with reinforcement learning, desired reactions can be strengthened and undesired reactions can be suppressed during the fine-tuning. The microscope settings determined by the large language model and / or the microscope images captured using the determined microscope settings can be evaluated. The evaluation can be done by a human or a learning model, in particular a large language model. The captured microscope images can be images captured after the large language model has calculated the microscope settings; or, given microscope images from an image set can also be used, wherein the microscope settings used to capture the given microscope images are known. Thus, given microscope images can be selected for which the microscope settings used for imaging correspond to the microscope settings determined by the large language model. Then, the user or the model evaluates whether the desired properties of the microscope images correspond to the text input and whether the selected microscope images meet the highest possible image quality. In this way, reinforcement learning does not require capturing new microscope images or only a very small number of microscope images, thus making the training faster and more cost-effective.

[0093] In principle, fine-tuning can also be done by a few (or one) learning / prompting. In this case, there is a given text before the text input, optionally with associated image data. The given text relates to the mapping of the text input to the large language model output data (microscope settings and / or processing parameters). For example, multiple text request examples can be used in a few prompts to capture microscope images with specific properties and the associated microscope settings for capturing the respective microscope images. This would prompt the model to output microscope settings that match the text input. This type of fine-tuning typically does not change the learned model parameter values.

[0094] The large language model can also comprise a transformer encoder-decoder as described in the article by Vaswani et al. cited in the introduction. In this case, the training can be supervised translation from text input to microscope settings. The training data comprises natural language or technical user input as text input and the associated desired parameter output (set of microscope settings). The transformer encoder-decoder is particularly advantageous when the output is in technical or domain-specific language. For example, the microscope settings can be output by generating an XML file that is written in a specific programming language containing commands to set up the microscope. This is particularly suitable when the user's text input is more like a formal experiment description to be translated into microscope settings and an interactive dialogue with the large language model does not necessarily take place. The transformer decoder is also suitable for outputting microscope settings in technical or domain-specific language.

[0095] It is possible that the training process varies for all model types. For example, the transformer encoder can in principle also be trained in an unsupervised manner like the transformer decoder.

[0096] The large language model can comprise multiple machine learning models (submodels) that can be learned in joint training or also individually. Different submodels are activated depending on the requirements of the specific task. Different submodels can be provided for image analysis, for example, in which at least one submodel is selected depending on the sample type; different submodels can also be provided for issuing follow-up queries, determining whether there is enough information for creating microscope settings or processing parameters, and generating microscope settings. At least one submodel is a language model for processing the text input.

[0097] The large language model or at least one submodel can also comprise one or more convolutional neural networks (CNNs), in particular a U-Net architecture. Other model architectures of deep neural networks can also be used.

[0098] Follow-up query loop

[0099] The simple language model of the prior art can only detect microscope settings specified by the user and cannot infer suitable microscope settings from more abstract user descriptions. A further difference to the prior art is that the large language model of the present invention can be designed to provide interactive help to the user during execution of the microscope settings by submitting follow-up questions or comprehension questions to the user to infer suitable microscope settings.

[0100] The large language model can be trained to determine whether all microscope settings to be set, i.e. all required microscope settings, can be set according to the received text input or whether a follow-up question to the microscope user is required for one of the microscope settings. Upon follow-up query, the subsequently received user reply is input into the large language model as further text input. Upon follow-up query, the large language model is trained not to ask for a value of a microscope setting, but to query a sample property related to this microscope setting, a property of the desired microscope image or a property of the experiment to be carried out. For example, the follow-up question can be whether bleaching of the sample can be accepted or whether a relatively long measurement time should be chosen to obtain a sufficient signal-to-noise ratio. In contrast, no question is asked for a specific numerical value of a microscope setting, e.g. the power of the illumination device.

[0101] If the user's reaction indicates that the calculated microscope settings are not acceptable (e.g. if the user does not allow bleaching of the sample, but the microscope settings prescribe a higher illumination intensity and the user also does not want the measurement duration to be too long), the large language model can change other parameters to achieve the intended goal, e.g. a certain signal-to-noise ratio; in the example cited, these effects can be avoided by, for example, changing the aperture cable or increasing the pinhole size of the confocal imaging, provided that the user has not specified any conflicting specifications (e.g. regarding the depth of field).

[0102] Further examples of follow-up questions submitted to the user can be: "Which dyes are of interest?" "Is the procedure time-sensitive?" or "Which sample (of the plurality of samples on the sample carrier) should be analyzed?"

[0103] Whether it is necessary to submit a follow-up query to the user can be assessed, for example, by a special coding in the output of the large language model. Alternatively, a special model, in particular a transformer encoder model of the large language model, can also be used for this purpose, which classifies the necessity of a follow-up query.

[0104] The large language model can comprise one or more transformer encoders and transformer decoders for different tasks. For example, a transformer encoder can be used to compute the microscope settings from the input of the large language model, in particular the text input and the received user responses to the subsequent queries. On the other hand, the large language model can comprise a dedicated model, e.g. a transformer decoder, for creating the subsequent queries for the user. This enables the text interaction. The transformer decoder can be trained using the microscope user manual and the other training data mentioned above.

[0105] The input to the large language model comprises the context of the user conversation up to this point in order for the model to "remember" the previous conversation process and utilize it for computing the subsequent query or the microscope settings. For a particular session, the input in the large language model can in particular comprise: all user requests; all subsequent queries of the model with the associated replies of the user; the parameter settings / microscope settings generated up to this point, the associated images and / or (image-to-image) evaluation model's evaluation of them; and any existing contextual knowledge. The contextual knowledge can come from a database, relating to the provided samples, sample carriers and previous measurement results, while also distinguishing between different users. The knowledge about previous measurement results is particularly helpful for the large language model when the user refers to previous measurement results in the text input or the response, e.g. the user indicates that the same dye should be rendered visible as in the previous week's measurement. In contrast, traditional language models for microscopes typically cannot take into account contextual knowledge when defining the microscope settings.

[0106] In a variant of the invention relating to microscope image processing, the large language model can also be configured to issue a subsequent query when the description of the desired image processing by the current text input is not precise enough, in order to allow defining the necessary processing parameters with a predetermined minimum level of reliability. The subsequent query can relate to e.g. the type of sample taken or the imaging technique taken.

[0107] Processing of the overview image

[0108] The large language model can contain a (in particular machine-learned) image analysis model. This model computes analysis results from the overview image, which the large language model uses in the computation together with the text input to determine the microscope settings. The image analysis model can optionally be learned separately from another part of the large language model, e.g. separately from the transformer encoder that computes the feature vector from the text input. Alternatively, the large language model can also be designed directly for image processing without training a separate image analysis model, e.g. as described in the article cited in the introduction.

[0109] Large language models, in particular image analysis models, can specify, based on the analysis results computed for the overview image, for example, one or more of the following:

[0110] - the sample carrier type, such as whether it is a glass slide, a petri dish, a chamber slide, or a multi-well plate;

[0111] - the sample type, such as adherent / fixed cells, stained / unstained tissue sections, rock samples, or electronic components;

[0112] - whether chemical dyes were used in the sample preparation process, and if so, which chemical dyes were used;

[0113] - the experiment type, such as single image, long-term experiment, or high temporal resolution;

[0114] - one or more focal positions, i.e. one or more height planes of the sample to be analyzed, and the related microscope settings for focusing on the respective height plane;

[0115] - the contamination status or contamination level of the sample carrier or the sample, such as the fill level of a liquid sample, discoloration or bacterial infestation of a liquid biological sample, or droplets of the sample liquid due to coagulation;

[0116] - the location of the relevant area, such as the sample or sample area, the cover glass, the sample chamber, or the location data of a text area; and / or

[0117] - the information content of a text or barcode on the sample carrier.

[0118] These analysis results can be processed by the large language model together with the text input to compute microscope settings. This joint processing is in contrast to separate processing, in which one result is computed separately for the text input independent of the image data, and another result is computed separately for the image data independent of the text input.

[0119] When computing the analysis results for the overview image, the text input (or an information extracted from it by the large language model) can be selected to be taken into account. If the text input has a name such as sample type, it can be taken into account when localizing and identifying the sample area.

[0120] In addition to the analysis results of the image analysis model, other context knowledge about the respective sample can also be taken into account, which is typically entered together with the text input into the large language model.

[0121] Image evaluation and imaging adjustment

[0122] By evaluating the captured microscope images, it can be determined whether the implementation of the text input was successful or whether new imaging should be carried out after adjusting the microscope settings.

[0123] A large language model can be trained to derive microscope image properties from a text input, which a captured microscope image should fulfill in order to comply with the text input. Microscope image properties can be expressed, for example, that a certain object type (e.g. a certain organelle) should be visible or that a minimum / maximum pixel size of a microscope image should be complied with. Thus, which microscope image properties should be fulfilled are not predetermined but at least derived from the text input. Other examples of microscope image properties include image quality (e.g. signal-to-noise ratio, image noise, contrast, image sharpness, depth of field, overexposure, shadow / shading sample area, resolution), contrast type, sample staining, sample state or presence of artifacts (e.g. contaminations; image processing errors caused by processing captured raw data or imaging artifacts (e.g. light source reflections)).

[0124] At least one machine-learned image evaluation model calculates relevant microscope image properties from a microscope image captured using the microscope settings determined using the large language model. Multiple image evaluation models can be provided, which are designed to determine different microscope image properties. Also, one image evaluation model can be designed to calculate different microscope image properties. At least one respective image evaluation model is selected depending on the microscope image properties to be fulfilled derived from the text input.

[0125] In case the microscope image properties of the microscope image comply with the microscope image properties derived from the text input, the microscope image will be further used, e.g. displayed to a user, saved and / or used in a provided workflow. After displaying the microscope image, the large language model can ask the user whether a modification is required. The user then has the opportunity to add information or change information to the current text input, at which point the described process will be run again. In the new process, the large language model uses the previous text input of the user, the new specification of the user and the microscope image or images that led to the user providing the new information. For example, the user can state that they were disturbed by blurry structures displayed superimposed with sharp sample objects in the microscope image. The large language model processes this statement and determines that the depth of field should be improved compared to the previous image as a desired microscope image property; changes the previous microscope settings, e.g. adjusts the aperture diaphragm or reduces the pinhole of a confocal measurement. Due to these changes, other microscope settings are also automatically adjusted to maintain desired properties, e.g. to maintain a certain signal-to-noise ratio, the illumination intensity can be increased to compensate for the smaller pinhole.

[0126] If the image evaluation model determines that the microscopy image properties of the captured microscopy image do not comply with the microscopy image properties derived from the text input, the image evaluation model provides feedback to the large language model. Specifically, the microscopy image properties of the microscopy image can be input into the large language model. The large language model then calculates adjusted microscopy settings for capturing a new microscopy image, such that the microscopy image properties of the new microscopy image are closer to the microscopy image properties derived from the text input. For example, if the captured microscopy image does not show the organelle to be analyzed with the minimum number of pixels as defined by the text input, the adjusted microscopy settings can specify a higher magnification. If the captured organelle has an adequate size, but insufficient contrast / image quality, the adjusted microscopy settings can specify an increased illumination intensity.

[0127] In the case of determining adjusted microscopy settings for capturing a new microscopy image, the large language model can optionally be designed to perform an evaluation to determine whether a follow-up query should be submitted to the user for authorizing the captured new microscopy image, since the adjusted microscopy settings have an impact on the sample or the measurement duration. For example, if the illumination intensity is increased, there can be a risk of sample bleaching as a result.

[0128] Furthermore, the captured microscopy image can also be evaluated using a learned quality metric, i.e. by a machine learning model that is specifically trained with the goal of calculating image quality.

[0129] Furthermore, the captured microscopy image can also be evaluated by comparing it to at least one reference image. For example, the similarity can be calculated by calculating the mapping of the microscopy image and the reference image in a feature space and determining the distance between the representations in the feature space. In particular, the similarity can be estimated using known techniques from the field of image retrieval. For example, when the user provides the text input in the form of an instruction to "capture an image similar to yesterday's", the reference image can be compared. Before imaging, the large language model sets the parameters / microscopy settings using the contextual knowledge (parameters of yesterday). The image evaluation model then compares the captured microscopy image to the image of yesterday to ensure that the request was executed correctly. In principle, the image evaluation model can also be considered a part of the large language model, which outputs the evaluation result to another part of the large language model.

[0130] The evaluations performed by the image evaluation model and / or the supplementation of the previous text input performed by the user upon viewing the microscopy image can be collected in order to newly / supplementally train the large language model, for example as a reinforcement signal, in order to further improve the large language model.

[0131] Image evaluation for image processing

[0132] When the captured microscope image is image-processed, the processed microscope image can be evaluated. The implementation of the text input is evaluated whether it was successful or whether the image processing needs to be repeated using adjusted processing parameters.

[0133] The features described above in connection with the evaluation of the captured microscope image can also be applied analogously to the evaluation of the processed microscope image. In particular, different image evaluation models can be provided from which one or more image evaluation models are selected depending on the text input or the determined processing parameters for evaluating the processed microscope image.

[0134] The image evaluation can assess whether the intended processing goal was achieved and whether any processing errors occurred. The processing goal can be, for example, resolution enhancement, de-noising, background or imaging artifact suppression, deconvolution, SIM calculation or segmentation. Processing errors can be reflected by added image artifacts, such as bubble-like structures, hallucinated objects (not contained in the original data) and ringing or ringing artifacts. The image evaluation criteria can be selected depending on the processing goal, for example, whether the image noise is lower or higher than in the original microscope image. Alternatively, the image evaluation only relates to image regions that are classified as relevant by image analysis. For example, the signal-to-noise ratio can be enhanced by comparing the signal-to-noise ratio of the respective image regions of the sample in the original microscope image and the processed microscope image, while the environment / background is not taken into account or only to a lesser extent when determining the signal-to-noise ratio. A specially trained model can detect added image artifacts. Image segmentation can be evaluated using a trained validation model that is trained to detect such segmentation masks or deviations from the training data or using correct and / or incorrect segmentation masks. If a negative evaluation result occurs, in particular if the evaluation criteria are not met, the image evaluation model can be designed to output a corresponding error description.

[0135] If the evaluation result is positive, the processed microscope image is further used, for example, displayed to the user, saved and / or used in the provided workflow.

[0136] If, however, a negative evaluation result occurs, feedback is sent to the large language model. In particular, the error description generated by the image evaluation model can be input into the large language model. The large language model uses the error description / evaluation result to calculate adjusted processing parameters for a new image processing. For example, if an object hallucination has occurred, the large language model selects modified processing parameters with a lower known probability of hallucination (even if this entails accepting a loss in terms of the desired processing goal, for example, resolution enhancement).

[0137] As described for the evaluation of the captured microscope images, the evaluation of the processed microscope images can also be performed using a learned quality measure or by estimating a similarity to at least one reference image.

[0138] Loading microscope images (image retrieval)

[0139] In different variants of the invention, the desired microscope image is retrieved from a database by a large language model. The large language model can provide the functionality described below in addition to the functionality described above for determining microscope settings and / or processing parameters.

[0140] A text input describing the desired microscope image is received to load or retrieve the microscope image. As described above, the text input can come from a user, e.g. by voice input, or from a computer program, e.g. in an automated workflow.

[0141] The text input is input into a large language model that is trained to compute the desired microscope image properties from the text input. Subsequently, a microscope image is loaded from a database containing microscope images according to the desired microscope image properties.

[0142] In simple cases, microscope image properties stored for the microscope images in the database can be used. However, extensive databases often only exist in a state where the microscope images are not annotated or only partially annotated with their properties.

[0143] In particular, a feature space can be used to automatically find a microscope image with the desired properties in a database containing microscope images that are not annotated or only annotated with certain properties. The large language model is trained to compute a mapping of the desired microscope image properties to the feature space. A respective feature space representation is also provided for each microscope image in the database; in other words, by mapping / embedding the microscope images in the feature space, each microscope image is represented as a point in the feature space. The large language model can select and load a microscope image from the database whose feature space representation is closest to the representation of the desired microscope image properties. The selected microscope image is then shown to the user or used for an automated workflow. The large language model can also select multiple images whose feature space representations have the smallest distance to the representation of the desired microscope image properties instead of a single image.

[0144] In particular, the large language model can comprise a text encoder for mapping microscope image features to a point in a feature space and an image encoder for mapping microscope images to the feature space. Both encoders can be transformer-based. Using (partially) annotated microscope images, the relation between microscope image properties described by text and the microscope images is established in training. After training, remaining, unannotated or not fully annotated microscope images can be mapped into the feature space by the image encoder. Thus, for microscope images without text annotation, a correspondence or similarity to a text input can also be computed. The distance between two representations in the feature space can measure their similarity. Desired microscope image properties can be computed from a text input, e.g. by a transformer decoder trained for this purpose.

[0145] It can be assessed whether a loaded microscope image matches a text input. For this a machine learning classifier can be used which checks whether the microscope image matches a certain class or domain. Multiple binary classifiers can be provided for more classes, wherein one or more binary classifiers are selected depending on the desired microscope image properties computed from the large language model. For example, if the desired microscope image properties specify that a cell nucleus should be visible, a binary classifier trained to assess whether a cell nucleus is present in the image is selected.

[0146] In another example, the user can request in a text input a microscope image with cells in a specific cell phase, e.g. in the mitotic phase of cell division. The large language model computes a mapping to a feature space / latent space, but there is a large uncertainty in the location in the feature space. Therefore, the large language model submits a follow-up query to the user to refine the request for the microscope image, e.g. it asks whether dead cells (or cells in a different cell phase) can also be seen in the microscope image. The user responds negatively and the user’s response is taken into account to compute a new mapping to the feature space. Since the specification is more precise, the uncertainty is reduced and the large language model outputs at least one microscope image whose representation in the feature space is closest to the request. The microscope image can optionally be evaluated by a classifier before being shown to the user. In this example, multiple classifiers are selected that are trained to detect cells in different cell phases. One or more of the evaluated microscope images that are correct are shown to the user. The user can then refine / modify the text request, e.g. by specifying that images of a different contrast type than the shown image should be selected. The large language model processes the refined specification of the user and the shown image. Based on the refined specification, the large language model uses the classifiers to determine the contrast type of the selected image, e.g. it determines that there is a phase contrast image. The large language model uses the refined text request to compute a new mapping to the feature space and selects at least one closest microscope image. The classifier is used to evaluate whether the microscope image is a phase contrast image - if the evaluation determines that the microscope image is not a phase contrast image, the image is presented to the user. This forms a loop of interaction, where the feedback of the user is used to refine the computation of the large language model to make the result closer to the user’s needs.

[0147] The text input and the determined properties of the microscope image are specific to the sample type (e.g. a specific cell line), the imaging type (e.g. a specific contrast type or staining) and the image content type (e.g. a specific event like cell division).

[0148] Context information

[0149] The large language model can be designed to receive and process context information together with the text input. The context information can come from a data store, be metadata of the given image processed together with the text input, come from the microscope currently in use and relate to the component specifications or settings of the microscope, or be a user profile.

[0150] In particular, the context information can also be used by an image analysis model of the large language model or another machine learning network of the large language model.

[0151] The context information can specify, among other things, the sample type employed, the sample state, the sample stage, the sample carrier employed, the experiment description, and / or the description of previous measurements.

[0152] By using the context information, the large language model can be saved from having to ask the user for the respective information. The large language model can also use the context information to check the plausibility of the user specifications in the text input. For example, the context information can specify a sample type; if the text input specifies an organelle to be displayed that does not exist in this sample type, the large language model can point out this inconsistency to the user without having to calculate microscope settings and capture possibly unsuitable images.

[0153] The context information can also come from an image analysis of an image input into the large language model, for example by detecting fonts or other objects in the image.

[0154] General features

[0155] Both the microscope images and the overview images can be understood as images captured by a microscope or calculated from measurement data with a microscope. In particular, the microscope images or overview images can be formed from one or more raw images or processed microscope images and can comprise two-dimensional image data or three-dimensional image stacks or volume data, or also time series data of two- or three-dimensional image data of the same object captured at different points in time. The field of view of the overview images is larger than that of the microscope images.

[0156] In principle, the structures or objects described in the microscope images and the overview images can be any structures or objects. In addition to the sample itself, such as a biological structure, an electronic component or a rock fragment, it is also possible to describe the sample container, the sample carrier, a microscope component, such as a sample stage, or a region thereof.

[0157] The "received" images can comprise the possibility of images captured by a microscope or also existing images received, for example, loaded from a data store.

[0158] The microscope can be an optical microscope comprising one system camera and optionally one separate overview camera. Other types of microscopes can also be used, such as electron microscopes, X-ray microscopes or atomic force microscopes. The microscope system is understood to mean a device consisting of at least one computing device and one microscope.

[0159] The computing device can be designed in a decentralized manner, can be a physical part of the microscope, or can also be arranged separately in the vicinity of the microscope or at a remote location at any distance from the microscope. In general, it can be composed of any combination of electronic devices and software, and in particular can include a computer, a server, a cloud-based computing system, or one or more microprocessors or graphic processors. The computing device can also be configured to control the microscope assembly. The decentralized or cloud-based design of the computing device is very advantageous, especially when a large language model is executed using a specially designed hardware or user feedback is collected for subsequent training of the large language model.

[0160] The singular form of the description is intended to cover both the "exactly 1" and "at least 1" variants. For example, the microscope setting calculated using the large language model to capture the microscope image is intended to understand that exactly one microscope image is captured using the calculated microscope setting or more than one microscope image is captured.

[0161] The features described in the present invention as features of the additional device also result in variants of the program based on the invention when implemented as intended. Conversely, the microscope system or in particular the computing device can be configured to execute variants of the described program. The training process of the described machine learning network defines the properties of the trained network, and by executing the described training process as a program step of the program, additional program variants according to the invention can be formed. BRIEF DESCRIPTION OF DRAWINGS

[0162] Further effects and features of the present invention will be described below with reference to the accompanying drawings:

[0163] Figure 1 The process of one embodiment of the program according to the invention is schematically illustrated;

[0164] Figure 2 The process of one embodiment of the program according to the invention is schematically illustrated;

[0165] Figure 3 The process of one embodiment of the program according to the invention is schematically illustrated;

[0166] Figure 4 The process of one embodiment of the program according to the invention is schematically illustrated;

[0167] Figure 5 The process of one embodiment of the program according to the invention is schematically illustrated;

[0168] Figure 6 The process of one embodiment of the program according to the invention is schematically illustrated;

[0169] Figure 7The process of one embodiment of a program according to the application is schematically illustrated;

[0170] Figure 8 The process of one embodiment of a program according to the application is schematically illustrated; and

[0171] Figure 9 The process of one embodiment of a program according to the application is schematically illustrated. DETAILED DESCRIPTION

[0172] Different embodiments will be described below with reference to the drawings. In general, like elements and elements working in a similar way are denoted by the same reference signs.

[0173] Figure 1 Program with large language model according to the application

[0174] Figure 1 One embodiment of a computer implemented program according to the application for controlling a microscope 1 is schematically illustrated. The program uses a special large language model LLM to capture a microscope image 50 according to the user's wishes, without the user having to indicate specific microscope settings.

[0175] In process P1, at least one overview image 20 of the sample 10 is received. In the illustrated example, the overview image 20 is captured by a (overview) camera of the microscope 1, although it is also generally possible to load an existing overview image 20 from a memory. The overview image 20 shows the sample carrier 9 with the sample 10, which in the present example comprises a plurality of spatially independent sample regions, for example a plurality of tissue sections.

[0176] In process P2, a text input T from the user U is received, for example by capturing a voice instruction of the user U by a microphone and converting it into text using voice recognition. The text input T describes the desired microscope image and / or the employed sample. For example, the user can issue a voice instruction to capture an image of a specific biological cell. The text input T thus describes the cell type of interest (as opposed to other cell types) as well as the magnification, since a (single) cell is to be shown, not a cluster of cells (requiring a lower magnification or a larger field of view) or an organelle (showing only one slice within a cell).

[0177] In process P3, the text input T and the overview image 20 are input into the large language model LLM. This is a neural network trained to process the text input T and the overview image 20 together in order to compute therefrom microscope settings 40 for capturing a microscope image that complies with the description of the desired microscope image. The computed microscope settings 40 are output by the large language model LLM in process P4.

[0178] In process P6, the microscope 1 is controlled according to the microscope settings 40. In process P7, at least one microscope image 50 is captured using the microscope settings 40 computed by the large language model LLM. The computed microscope settings 40 can also comprise a sequence of consecutive microscope settings to be set, using which a plurality of microscope images can be captured.

[0179] The microscope image 50 is input into a machine learning image evaluation model M2, process P8, which computes microscope image properties 46, for example properties related to the image quality, in particular contrast, image sharpness or signal-to-noise ratio, and properties related to the depicted structures or objects, in particular object detection to determine whether cells of a specific cell type are depicted in the microscope image 50 and, if so, how many. The image evaluation model M2 uses the microscope image properties 45 that the microscope image 50 should satisfy, which are computed by the large language model LLM in process P5. The microscope image properties 45 specified by the large language model LLM can be derived from the text input T and the overview image 20 and can indicate, in particular, the type and number of objects to be displayed. The image evaluation model M2 can comprise a plurality of object detection models for detecting different types of objects, wherein one or more object detection models for processing the microscope image 50 are selected according to the microscope image properties 45 computed by the large language model LLM.

[0180] In process P9, an evaluation result 55 is computed which indicates whether the microscope image properties 46 computed by the image evaluation model M2 for the microscope image 50 correspond to the microscope image properties 45 determined by the large language model. Process P9 can in principle be performed by the image evaluation model M2 or the large language model LLM. If the evaluation result 55 is positive, the microscope image 50 will be further used in process P10, for example displayed to the user U. In process P10, it can also be continued to execute a provided workflow, which is executed after successful capture of a microscope image 50 having the microscope image properties 45 computed by the large language model LLM, for example, navigating to and analyzing more sample locations, capturing more microscope images of the same sample region with different contrast types, or performing a time series measurement in which images of the same sample region are captured at different points in time.

[0181] After viewing the microscope image 50 in process P10, the user U can add a text input T in process P11. In particular, the user U can replace and / or improve the previous specification. The supplement to the text input is input into the large language model LLM, which computes modified microscope settings from the previous information and the newly input information in order to capture a new microscope image. In this way, a cycle can be repeated in which imaging is performed using settings computed by the large language model LLM and feedback from the user is taken into account. Thus, the user U can interactively control the imaging using the text input T. For example, as a supplement to the text input T, the user U can say "better highlight the cell nuclei". The previous microscope settings 40 are then modified, for example, the intensity of the illumination wavelength for the fluorescence excitation of the dye used for the cell nuclei staining is increased.

[0182] If the evaluation result 55 indicates that the microscope image properties computed by the image evaluation model M2 do not agree with the microscope image properties 45 computed by the large language model LLM, feedback is provided to the large language model LLM in a flow process P12, which will be described with reference to Figure 2 in more detail.

[0183] Figure 2 : Cycle with large language model for user follow-up queries and / or feedback image evaluation

[0184] With reference to Figure 2 , first the additional functionality of an embodiment of Figure 1 will be described, according to which the captured microscope image 50 does not comply with the microscope image properties 45 required by the large language model LLM, according to the evaluation result 55. In process P12, the evaluation result 55 or the microscope image properties 46 computed by the image evaluation model M2 are input (by special marking) to the large language model LLM in order to compute adjusted microscope settings 40B. For computing the adjusted microscope settings 40B, the large language model LLM also takes into account the previously computed microscope settings 40, the text input T and the overview image 20 or the analysis results determined therefrom. The purpose of the adjusted microscope settings 40B is to bring the microscope image properties of the newly captured microscope image 50B closer to the microscope image properties 45 derived from the text input T. The newly captured microscope image 50B will then be used instead of the microscope image 50 as described above. If necessary, the cycle of adjusting the microscope settings and capturing a new microscope image can be repeated until the captured microscope image meets the desired microscope image properties 45.

[0185] If adjusted microscope settings 40B are determined for capturing a new microscope image 50B, an evaluation 47 can optionally be performed. The evaluation 47 indicates whether a follow-up query Q should be submitted to the user U to authorize capturing the new microscope image 50B in view of the impact of the adjusted microscope settings 40B on the sample 10 or the measurement duration. For example, the desired microscope image properties 45 can indicate that a particular cell type should be visible, while the evaluation result 55 indicates that the cell type is hardly visible in the microscope image 50; in this case, the large language model LLM computes adjusted microscope settings 40B with increased illumination intensity and / or modified illumination wavelength. In the evaluation 47 it is determined that the modified illumination causes bleaching of the sample 10, so a follow-up query Q should be submitted to the user. The evaluation 47 can be computed by the large language model LLM, which also uses all previous inputs and outputs of the large language model LLM related to the present experiment in order to compute the evaluation 47. The large language model LLM processes a response A of the user U to the follow-up query Q in order to capture a new microscope image 50B using the adjusted microscope settings 40B or to modify the adjusted microscope settings 40B again according to the response A of the user U. For example, if the user responds that the light damage of the sample 10 is not acceptable, the large language model LLM can (possibly after further follow-up queries Q and related responses A) increase the illumination duration and the measurement duration in order to achieve better visibility of the particular cell type without increasing the illumination intensity. Or, the large language model LLM can switch to an objective with a higher magnification and capture multiple laterally offset microscope images, which are stitched together to form a single image (image stitching), which can also achieve better visibility of the particular cell type without increasing the illumination intensity.

[0186] If the feedback loop has been run several times through the process P12 and still no evaluation result 55 of sufficient quality has been achieved, the large language model LLM can provide feedback to the user U. In particular, the large language model LLM can use a follow-up query Q to request a change of the imaging conditions specified by the user U, e.g. asking whether the laser power limit can be increased.

[0187] The large language model LLM is also designed to determine, after receiving the overview image 20 and the text input T, whether the available information is sufficient to calculate all necessary microscope settings 40 for capturing the desired microscope image, or whether the user U needs to provide further information for this purpose. If more information is needed, the user U first submits a follow-up query Q. Then, the large language model LLM uses the further text input received from the user U (response A) and the previously provided information to calculate the microscope settings 40. The loop of follow-up queries Q and responses A can be repeated until the large language model M has determined all necessary microscope settings 40. Only then can the microscope settings 40 be used for imaging.

[0188] As far as possible, the large language model M formulates follow-up queries so as to ask about properties of the sample or about allowed / disallowed influences on the sample, without asking for specific values of the microscope settings 40. For example, the large language model M is designed so as not to ask for the fluorescence excitation wavelength used as any of the microscope settings 40, but to ask for the relevant fluorophore / flourescent dye used instead. This places lower demands on the user U's expertise and reduces the likelihood of errors. If the large language model LLM knows which dyes are being used (for example, by automatically reading a label on the sample carrier identified in the overview image 20), the large language model LLM can also ask about the sample structure of interest in the follow-up query Q and set the fluorescence excitation and detection accordingly; however, the user U does not need to ask for specific settings of the fluorescence measurement.

[0189] Figure 3 Large language model with separate decoder and encoder

[0190] The large language model LLM can comprise a transformer decoder Dec and a transformer encoder Enc, which receive the input to the large language model LLM or information derived therefrom, in particular the text input T and the overview image 20 or information determined from the overview image 20, respectively.

[0191] The transformer encoder Enc calculates an embedding Emb from the input data. The embedding Emb represents a feature extraction of the data input to the large language model LLM. A trained machine learning classifier M4 can calculate from the embedding Emb whether (and which) information is needed to determine all necessary microscope settings 40. The transformer encoder Enc and the classifier M4 can also be referred to collectively as a transformer encoder model.

[0192] The transformer decoder Dec is a language model that creates a follow-up query Q for the user U. For this, the decoder Dec receives the classification result of the classifier M4 and the input of the large language model LLM, in particular the text input T, optionally the context information and the overview image 20 or information obtained from the overview image 20. With the response A to the follow-up query Q, the transformer encoder Enc again calculates an embedding Emb. If the classifier M4 indicates that now there is enough information to determine all desired microscope settings 40, the mapper M3 calculates a mapping of the embedding Emb to the microscope settings 40. The mapper M3 can be a machine learning network, wherein the mapper M3 and the classifier M4 can optionally be formed by a common neural network. In this case, the classification result can depend on the respective confidence of each determined microscope setting 40.

[0193] In a variant of the illustrated embodiment, the transformer decoder Dec can be designed to submit the follow-up query Q to the user U without the classification result of the classifier M4.

[0194] It can also be provided that the transformer decoder Dec first communicates with the user U to obtain all necessary information and then calculates the translation or the regression to the microscope settings 40 using the mapper M3 (or the transformer encoder Enc and the mapper M3).

[0195] Figure 4 Image analysis using a large language model

[0196] Reference is made to Figure 4 Variants of image analysis and image processing using a large language model LLM are described. Furthermore, Figure 4 It is also schematically shown a process of a computer-implemented program for controlling a microscope 1 according to the present application, wherein in particular the processes related to the previous figures can be implemented.

[0197] The large language model LLM can comprise an image analysis model M1 that receives and processes the input overview image 20. The language model / machine learning network LLM1 further processes the analysis result 21 calculated by the image analysis model M1 together with the text input T (or information derived therefrom). The output of the machine learning network LLM1 corresponds to the output of the large language model LLM in the previous figures, and the machine learning network LLM1 can comprise a transformer encoder and / or a transformer decoder as described in the previous figures.

[0198] The analysis result 21 can be an image, at least one segmentation mask or information about the content of the image. For example, different object types can be identified and optionally the associated image coordinates can be output. It is also possible to assess object properties such as the degree of contamination, the uniformity of the staining process, possible bacterial infestation or the filling level in the sample container. The image analysis model M1 can be based on a convolutional network (CNN), for example a U-Net architecture, or on a transformer, for example a visual transformer as described in the introduction to the prior art. Optionally, the image analysis model M1 can be designed to use information Inf from the text input T, context information K and information from the response A of the user U to a subsequent query Q of the machine learning network LLM1. The information Inf is calculated by the machine learning network LLM1 and can represent the sample type, the sample carrier type or the sample preparation, for example the staining process. The image analysis model M1 uses this information Inf in the image analysis. For example, different segmentation models or detection models can be provided and one of these models can be used depending on the sample type.

[0199] The analysis result 21 output by the image analysis model M1 can be generated in text form, which makes it possible for the machine learning network LLM1 to process it together with the text information T. In this case, the machine learning network LLM1 can in particular use a pre-trained language model without having to make major adjustments to take the analysis result 21 into account. If necessary, the pre-trained language model can be fine-tuned to determine the settings of the microscope. For example, in order to display the analysis result 21 in text form, for example, an equivalent text specification of the segmented object types can be used together with a specification of the associated image coordinates instead of a segmentation mask.

[0200] Alternatively, the overview image 20 can also be encoded / mapped in a feature space by, for example, a visual transformer; the encoding of the overview image 20 is then combined with the text input T (or the result of a further processing of the text input T by a language model) and jointly processed by the machine learning model. In this case, the encoding of the overview image 20 can essentially retain the full information content of the overview image 20, so that a more comprehensive image analysis result can be obtained.

[0201] Figure 5 Large language model with transformer encoder

[0202] Figure 5 The process of one embodiment of a program according to the application is schematically illustrated, in which the large language model LLM comprises a transformer encoder En for processing the text input T at least once. Further text information, such as a user response, context information or a text image analysis result, can be processed together with the text input T in the form of a combined text.

[0203] The text input T is converted into a sequence of tokens S, in the present example each token being a word of the text input T "Show a hydatid cyst with some larvae". The tokens can also have other divisions. The sequence S is preceded by a start token [CLS] whose value is defined by the training of the transformer encoder Enc. The sequence S with the start token [CLS] is input into the transformer encoder Enc. The transformer encoder Enc computes therefrom an output which contains a representation (hidden state representation) [CLS]-Emb of the start token [CLS]. In particular, a related hidden state representation can be computed for each token. To this end, the transformer encoder Enc can comprise a plurality of self-attention blocks in a known manner, which compute the Q, K and V vectors for the tokens, respectively. The output for the start token [CLS] is computed by computing the Q vector of the start token [CLS] with the K and V vectors of each of the other tokens, as described in the introduction with reference to the prior art. This output forms the input for the next self-attention block. The self-attention mechanism can be executed in parallel several times to provide multi-headed attention.

[0204] The hidden state representation [CLS]-Emb of the start token [CLS] contains information about the entire sequence S, which is why only the hidden state representation [CLS]-Emb is used as the aggregated sequence representation for further processing by the large language model LLM, and not the hidden state representations of the remaining tokens.

[0205] The hidden state representation [CLS]-Emb is input into a mapping program M3 which is designed to compute a mapping to values of the microscope settings 40. The mapping program M3 can be learned by supervised training, in which the relevant microscope settings 40 are specified as ground truth data for different values (= positions in the feature space) of the hidden state representation [CLS]-Emb. The values of the hidden state representation [CLS]-Emb of the training data can be computed from a given sequence S by the trained transformer encoder Enc.

[0206] Depending on the design of the large language model LLM, the overview image 20 can be considered at different points, in particular at the transformer encoder Enc and / or the mapping program M3. In the illustrated example, the hidden state representation [CLS]-Emb of the text input T is input into the mapping program M3 together with the overview image 20. More generally, the overview image 20 can also be pre-processed so that the result of the processing of the overview image 20 is input into the mapping program M3 instead of the overview image 20. For example, a machine-learned feature extractor can be used for the pre-processing.

[0207] The text input T contains information about the desired microscope image, in this example relating to the used sample, i.e. the specification of the sample type (hydatid cysts), and information about the magnification of the desired microscope image, which is derived from the specification that a hydatid cyst with a larva is to be shown, not multiple independent cysts, a single larva, or just a specific cell type within the larva, etc.

[0208] Using the text input T, the large language model LLM can thus derive information about the size of the object (hydatid cysts) and thus determine the respective optical magnification to be set, e.g. the objective selection and / or zoom setting, in order to see exactly one hydatid cyst in the captured microscope image. In this example, the overview image 20 shows approximately six hydatid cysts, i.e. gas bubbles filled with liquid, each of which contains several tapeworm larvae. By means of the hidden state representation [CLS]-Emb, the mapper M3 knows which object type should occur in the overview image 20 and can thus segment or determine the object position specifically for this object type (or objects of a given size known to have this object type characteristic). The large language model LLM thus uses the overview image 20 to derive the coordinates of the object to be analyzed according to the text input T.

[0209] In order for the segmentation or determination of the object position to work as error-free as possible, it is usually necessary to correspond the pixel size of the depicted object to the object pixel size used in training. Depending on the resolution of the overview image, this is often not the case or puts too high a demand on the range of the training data. However, the large language model LLM is able to estimate how many pixels an object of this object type (here the object type is hydatid cysts and larvae) should typically occupy in the overview image 20 based on the knowledge of the object type (the large language model LLM knows the typical actual object size of this object type) and based on the knowledge of the imaging conditions of the overview image 20. The overview image 20 can then be scaled up or down so that the estimated pixel size corresponds to the average object pixel size used in training. The rescaled overview image can then be further processed to determine, in particular, the coordinates of the desired object, etc.

[0210] The object coordinates, the objective selection and / or zoom setting, and other settings are output by the mapper M3 as microscope settings 40. In a process P6, the microscope 1 is controlled with these microscope settings 40 in order to capture a microscope image 50 in a process P7. The microscope image 50 corresponds to the requirements derived from the text input T on the microscope image properties. The microscope image 50 can be further used as previously illustrated.

[0211] With Figure 5The related description is not intended to be exhaustive. For example, the sequence S can also be supplemented with more content, including specific contextual information, also with a "end of sentence" marker, and the position coding of the markers.

[0212] Figure 6 : image processing procedure

[0213] Figure 6 The flow of one embodiment of a procedure for processing microscope images according to the application is schematically shown. The illustrated procedure can optionally be used in combination with the procedure of one or more other embodiments.

[0214] In a procedure P1 ', at least one microscope image 50 of a sample is received, for example captured by a microscope 1 or loaded from a memory. Furthermore, in a flow procedure P2, a text input T of a user U is received, in which a desired image processing is described.

[0215] In a procedure P3', the text input T and the microscope image 50 are input into a large language model LLM. This is a neural network trained in machine learning, which calculates in a procedure P4' processing parameters 41 for processing the microscope image 50 from the text input T and the microscope image 50. The processing parameters 41 are used in a procedure P6' to select at least one image processing program 60 from a plurality of provided image processing programs. The parameters of the image processing program 60 employed are also determined by the processing parameters 41. Then, in a procedure P7', the microscope image 50 is processed using the calculated processing parameters 41, i.e. by the selected image processing program 60 using the set parameters. In the example shown, the image processing program 60 calculates a processed microscope image 50A.

[0216] For example, the image processing program 60 can include, for example, different image sharpening programs, different denoising programs, resolution enhancement programs, artifact suppression programs and / or programs for virtually staining different object types. Different programs for the same task can be used for different object types.

[0217] In the illustrated example, the text input T indicates that an image sharpening process is to be performed on the microscope image 50. The large language model LLM can learn from the context information K related to the microscope image 50 which biological cell types are depicted. The large language model LLM evaluates the microscope image 50 according to the contrast type, in this example determining that multiple fluorescence channels were used in the imaging (as opposed to e.g. phase contrast or DIC images). The large language model LLM can also determine which object types (e.g. which cell types or organelle types) are located in which image regions. For this purpose, a segmentation or location determination model sensitive to the cell types present can be selected according to the context information K. The large language model LLM indicates which of the provided image sharpening procedures 61 to use as processing parameters 41. The properties of the microscope image 50 (e.g. contrast type, determined SNR) and the object types present are mapped to one or more of the provided image sharpening procedures 61 to use. For different objects located in the microscope image 50, different image processing procedures can be selected for use. In this example, different image regions are defined according to different cell types located; different image regions are then processed by different image sharpening procedures 61, resulting in a processed microscope image 50A.

[0218] The processed microscope image 50A can be evaluated by an image evaluation model M2, similar to the evaluation of microscope images described with reference to other figures.

[0219] The large language model LLM uses the text input, optionally provided context information K, and the microscope image 50 to evaluate whether the most suitable image processing procedure can be selected and all necessary parameters defined, or whether there is still relevant uncertainty, i.e. which of multiple possible image processing procedures or parameter settings are necessary for optimal image processing. In case of uncertainty, the large language model LLM checks which information related to the sample used, the imaging performed, or the image processing performed is available to resolve the uncertainty. The large language model LLM then submits a corresponding follow-up query Q to the user U. For example, it can be known which cell type, but the cell type varies greatly depending on the living being, and different image sharpening procedures 61 are used for different organisms. Through the follow-up query Q, the large language model LLM obtains a response A from the user U, which for example relates to the sample and whether the depicted cell type is from a human or a reptile, which is incorporated into the determination of the processing parameters 41.

[0220] Figure 7 Training of the large language model

[0221] Figure 7The training process of a large language model LLM is schematically shown. The large language model LLM is or comprises a pre-trained language model, which is e.g. transformer-based and has been trained (pre-trained) using a broad database of text.

[0222] In the subsequent training (fine-tuning), the language model is adapted to a specific task, in the figure three different fine-tuning processes are shown, which can be performed alternatively or cumulatively, in particular in the order as shown.

[0223] In the subsequent training (fine-tuning FT1) of the large language model LLM, the training data 30 used comprises a microscope user manual, chat logs between microscope users and microscope service personnel and programming language based microscope setup files (e.g. in XML format). Thus, the large language model LLM is trained to compute microscope settings 40 and / or to formulate follow-up questions for a text input T. The specific use of the training data 30 in the fine-tuning FT1 depends on the structure of the large language model LLM. For example, in case of a structure in the form of a transformer decoder, the chat logs of the training data can be copied. In case of a transformer encoder-decoder, a translation of e.g. provided experiment descriptions and related (in particular programming language based) microscope setup specifications pairs can be used, wherein the experiment descriptions are processed in the encoder and the microscope settings are predicted by the decoder.

[0224] The process of fine-tuning FT2 of the large language model LLM is also shown schematically in the figure, which can optionally be performed after fine-tuning FT1. In fine-tuning FT2, a text input T is input into the large language model LLM, which can be loaded from a database or generated by the large language model LLM. The large language model LLM uses the text input T to compute microscope settings 40 and desired microscope image properties 45 that should be fulfilled by the microscope images to be captured. The microscope settings 40 are used to control the microscope 1 to capture microscope images 50. The microscope images 50 and the desired microscope image properties 45 are then input into the large language model LLM (or a dedicated image evaluation model) to evaluate whether the microscope images 50 comply with the microscope image properties 45. The respective evaluation result is used as feedback FB to adjust the model parameter values of the large language model. For example, the text input T can indicate that images with clearly visible cell nuclei of a certain type should be captured. Any of the desired microscope image properties 45 is a clearly visible cell nucleus. If the evaluation of the microscope images 50 results in that there are no cell nuclei, only parts of cell nuclei or only cell nuclei with low visibility in the microscope images 50, the respective feedback FB can be returned, by which, for example, the illumination wavelength and / or the detection wavelength or the illumination intensity are adjusted, or in the case of capturing cell nuclei only in sections, modified microscope settings for sample lateral translation are executed.

[0225] Likewise, a rule-based reward model can also be used, in which the microscope settings are evaluated by the large language model without the need to capture microscope images. For example, the computed microscope settings can be evaluated according to predefined rules, which define a minimum illumination intensity and / or exposure time to obtain a sufficient signal-to-noise ratio, which can optionally be dependent on different sample types and other microscope settings. For example, the microscope settings 40 can be computed by the large language model LLM in response to a text input T, which performs a sample preservation analysis, which would result in an insufficient signal-to-noise ratio, in which case the related parameter decision is not reinforced.

[0226] The process of fine-tuning FT3 of the large language model LLM is also shown in the figure, which can be performed instead of or in addition to (in particular after fine-tuning FT2) the process of fine-tuning FT3. Fine-tuning FT3 uses RLHF (reinforcement learning from human feedback). For a text input T, the large language model LLM computes a plurality of candidate microscope settings 40A, each for capturing a microscope image 50. A person provides an evaluation 56 of the microscope images 50, for example, which of the microscope images 50 best implements the text input T. The feedback FB1 can reinforce the parameter decision according to which the candidate microscope settings 40A can obtain the microscope image 50 with the best evaluation. User feedback can also be collected during normal operation of the microscope and then used for fine-tuning.

[0227] Figure 8 : image retrieval

[0228] In Figure 8 , the process of the procedure for retrieving a desired microscope image according to the present application is schematically illustrated. Upon a user request, a suitable microscope image 50 is selected from the database D by the large language model LLM. In addition to the process of other embodiments, the process described below can optionally be performed.

[0229] In the process P2, a text input T is received by the user U describing a desired microscope image. The desired microscope image can be described in the text input T as described in the previous figures. In particular, it can be described which sample type and which sample stage should be visible in the microscope image.

[0230] The text input T is input to the large language model LLM, which computes the desired microscope image properties 45 from the text input T in the process P13. In terms of content, the microscope image properties 45 can correspond to the specification of the text input T, but can also include specifications derived from the text input T. For example, from the desired object type, it can be inferred that a sample preparation procedure suitable for this object type or a suitable measurement procedure is appropriate, so the microscope image to be selected should be captured with such a suitable measurement procedure. For example, if from the text input T the cell nucleus should be clearly visible, then as microscope image properties 45 it can be determined that a sample image with DAPI stain (or other stain binding to DNA) or an image using the typical excitation wavelength of this stain in imaging should be selected.

[0231] In the process P14, the desired microscope image properties 45 are input to a transformer encoder Enc2, which will calculate a mapping 45' of the microscope image properties 45 into the feature space Z / latent space.

[0232] The database D comprises a plurality of microscope images 50, for example several ten- or hundred-thousands of images for which a respective mapping into a feature space Z (feature space representation 50’) is available. In a process P16, the feature space representations 50’ are computed in advance using an encoder Enc1. The encoder Enc1 can be, for example, a transformer encoder or other learned model. Alternatively, the encoder Enc1 can also compute an embedding of the microscope images 50 by a random neighborhood embedding (SNE), in particular by a t-distributed stochastic neighborhood embedding (t-SNE), as described in more detail by the applicant in German patent application DE 10 2022 121 545 of 25 August 2022. Such an embedding creates a feature space in which the distance between the embeddings of two microscope images reflects their similarity. The embeddings of the microscope images 50 can be used together with the associated microscope image properties as training data for an encoder Enc2 in order for the encoder Enc2 to learn a mapping from a set of (textual) microscope image properties to a point in the feature space Z.

[0233] In a process P17, it is determined which feature space representation 50’ in the feature space Z is closest to the representation 45’ of the microscope image feature 45. In a process P18, the associated microscope image 50 is loaded from the database D.

[0234] In the example shown, the loaded microscope image 50 is optionally evaluated or verified for which it is input into an image evaluation model M2 in a process P19. The image evaluation model M2 evaluates whether the selected microscope image 50 fulfills one or more of the desired microscope image properties 45. For this purpose, the image evaluation model M2 computes the respective microscope image properties 46 of the selected microscope image 50. Which microscope image properties are determined is not fixed, but is determined by the desired microscope image properties 45. For example, if the desired microscope image properties 45 specify a particular contrast type, it is determined which contrast type or whether the desired contrast type is present as a microscope image property 46. If the desired microscope image properties 45 specify a sample type, a classification is performed instead or also of the sample type depicted in the selected microscope image 50.

[0235] If the evaluation result 55 of the image evaluation model M2 indicates that the microscope image properties 46 do not fully match the desired microscope image properties 45, corresponding feedback is provided to the large language model LLM in a process P20. The large language model LLM, in particular the transformer encoder Enc2, calculates a new mapping 45' of the desired microscope image properties 45 into the feature space Z taking into account the feedback, for example by adding microscope image properties that were previously selected in the microscope image 50 not to appear as desired microscope image properties by means of the feedback. A new microscope image is selected and evaluated using the new representation.

[0236] If the evaluation result 55 of the image evaluation model M2 leads to the conclusion that the selected microscope image 50 meets the desired microscope image properties, this is displayed to the user U in a process P10. If the user U is not satisfied with the selected microscope image 50, he or she can supplement the previous text input T in a process P21, for example to describe the expected differences in the selected microscope image 50. The above-described loop is repeated using the supplemented text input T to select a new microscope image and display it to the user U.

[0237] The large language model LLM can also submit follow-up queries Q to the user U in order to likewise determine the desired microscope image properties 45 using the elicited user responses A. For example, a follow-up query Q can be issued when the large language model LLM classifies the text input T as ambiguous or possibly misleading. The large language model LLM can determine how large the distances are between the different feature space representations 50' of the microscope images 50 that meet the microscope image properties 45. If the distances exceed a predetermined value, the large language model LLM can infer that the text input T is too unspecific and therefore that very different microscope images 50 all meet this requirement. For the distribution of the feature space representations 50' for which the associated microscope images 50 actually meet the requirement, one or more principal axes in the feature space Z can be determined, i.e. directions of maximum distance. The large language model LLM can also determine which microscope image properties contribute most to these principal axes. A follow-up query Q is then issued on the basis of these determined microscope image properties. In this way, the desired microscope image properties 45 can be determined in a few follow-up queries Q in order to select a possibly suitable microscope image.

[0238] Figure 9 : microscope

[0239] Figure 9An embodiment of a microscope system 100 according to the application is shown. The microscopy system 100 comprises a microscope 1 and a computing device 12, which can be part of the microscope 1 or separate from the microscope 1. The microscope 1 comprises a stand 2 by which other components of the microscope are supported. The latter specifically comprise an illumination device 3, a condenser 5 for directing illumination light to a sample region, an objective changer / turret on which in the illustrated example an objective 6 is mounted, a sample stage 13 with a holder 8 for fixing a sample carrier 9, and a microscope camera 7. The microscope camera 7 receives detection light from the sample region in which a sample can be located when the objective 6 is pivoted into the optical path of the microscope, in order to capture a sample image. In principle, the sample 10 can be or comprise any object, fluid or structure. For capturing an overview image 20 of the sample environment, the microscope 1 can use an objective with a lower magnification and / or an optional additional overview camera (not shown) for observing the sample region perpendicularly or at an oblique angle. For capturing the overview image, an illumination arrangement can be used, including e.g. a plurality of light emitting diodes 4, which are oriented at an oblique angle to the detection axis for illumination in dark field measurements. In addition to the microscope camera 7, the sample can also be observed through an eyepiece 14. The microscope system 100 further comprises a computer program 11, which is stored in a non-volatile storage medium, which is only schematically illustrated. The computer program 11 or the computing device 12 is configured to perform the program variants described with reference to the other figures. In the illustrated example, the microscope 1 is an optical microscope, but in principle also other types of microscopes.

[0240] The variants described with reference to the different figures can be combined with each other. The described embodiments are purely illustrative and can be varied within the scope of the appended claims.

[0241] List of reference symbols

[0242] 1 microscope

[0243] 2 stand

[0244] 3 illumination device

[0245] 4 light emitting diode

[0246] 5 condenser

[0247] 6 objective

[0248] 7 microscope / system camera

[0249] 8 holder for fixing a sample carrier

[0250] 9 sample carrier

[0251] 10 sample

[0252] 11 computer program

[0253] 12 computing device

[0254] 13 sample stage

[0255] 14 eyepiece

[0256] 20 overview image

[0257] 21 analysis result

[0258] 30 training data

[0259] 40 microscope settings

[0260] 40A candidate microscope settings in large language model training

[0261] 40B adjusted microscope settings

[0262] 41 processing parameters

[0263] 45 microscope image properties to be fulfilled derived from text input

[0264] 45' mapping of desired microscope image properties to feature space Z

[0265] 46 microscope image properties of captured microscope image 50

[0266] 50, 50B microscope image

[0267] 50A processed microscope image

[0268] 50' feature space representation of microscope image

[0269] 55 evaluation result of image evaluation model

[0270] 56 evaluation of microscope image

[0271] 60 image processing program

[0272] 61 image sharpening program

[0273] 100 microscope system

[0274] A user response

[0275] [CLS] start token

[0276] [CLS]-Emb hidden state representation of start token

[0277] D database

[0278] Dec Transformer decoder of large language model

[0279] Enc, Enc1, Enc2 (Transformer) encoder

[0280] Emb embedding of the received input data calculated by the encoder Enc

[0281] FB, FB1 feedback in the training of large language models

[0282] FT1-FT3 follow-up training (fine-tuning) of large language models

[0283] Inf information to be taken into account in the image analysis

[0284] K context information

[0285] LLM large language model

[0286] LLM1 machine learning network of the LLM

[0287] M1 image analysis model

[0288] M2 image evaluation model

[0289] M3 mapping program

[0290] M4 classifier

[0291] M5 submodel

[0292] P1-P21 processes of the inventive procedure

[0293] P1'-P7' processes of the inventive procedure

[0294] Q follow-up query for the user of the large language model

[0295] S sequence of labels

[0296] T text input

[0297] U user of the microscope

[0298] Z feature space

Claims

1. A method for controlling a microscope, comprising: Receive (P2) a text input (T) describing the desired microscope image and the sample (10) used; The overview image (20) of the sample (10) used by the receiver (P1) is received. The text input (T) and overview image (20) are input (P3) into a large language model (LLM), which is a trained machine learning neural network used to process the text input (T) and overview image (20) together to compute (P4) a microscope setup (40) for capturing a microscope image (50) corresponding to the desired microscope image; and Microscope settings (40) calculated using a large language model (LLM) are used to capture (P7) microscope images (50); Among them, a large language model (LLM) is trained to determine whether all microscope settings to be set can be derived from the received text input (T), or whether it is necessary to issue a follow-up query (Q) to the microscope user for any microscope setting to be derived. In the case of a subsequent query (Q), the user response (A) is fed into a large language model (LLM) as further text input (T) to derive the microscope parameters.

2. The method according to claim 1, in, The microscope settings (40) determined the following: - Magnification; -Imaging speed; -The location of the sample area to be analyzed and the location of the sample stage; - Lighting power or intensity; - Illumination duration; -Focus; - Concentrator settings; - The excitation wavelength used for fluorescence excitation; - The number of detection channels and the characteristics of the detection channels; - Comparison procedure; - Single-shot imaging or time-series imaging.

3. The method according to claim 1, The text input (T) indicates at least one of the following: - Which objects should be visible in the desired microscope image; - The types of samples involved; -Which dyes were used in sample preparation? - Whether bleaching of sample (10) is permitted; and - Should priority be given to image quality or sample preservation? 4. The method according to claim 1, in, In the case of a subsequent query (Q), the training of a large language model (LLM) is not to request values ​​for the microscope settings, but to query the properties of samples (10) associated with those microscope settings, the desired microscope images, or the experiments to be performed.

5. The method according to claim 1, in, The necessity of submitting a follow-up query (Q) to the user is assessed using the following methods: Through special encoding in the output of large language models (LLM) or The necessity of subsequent queries (Q) is classified using a transformer encoder model based on a large language model (LLM).

6. The method according to claim 1, in, Large language models (LLMs) include a transformer decoder (Dec) and a transformer encoder (Enc), the transformer decoder (Dec) being trained to emit subsequent queries (Q), and the transformer encoder (Enc) being used to compute microscope settings (40) based on text input (T) and the received user response (A) to the subsequent query (Q).

7. The method according to claim 6, in, The text input (T) is converted into a token sequence (S); A start marker ([CLS]) is placed before the sequence (S), the value of which is learned during the training of the transformer encoder (Enc); The start marker ([CLS]) and sequence (S) are input into the transformer encoder (Enc), and the output of the transformer encoder (Enc) is calculated to include the hidden state representation ([CLS]-Emb) of the start marker ([CLS]). The hidden state representation ([CLS]-Emb) of the starting marker ([CLS]) is input into the mapping procedure (M3), which is designed to calculate the mapping to the value of the microscope setting (40).

8. The method according to claim 1, in, Train a large language model (LLM) to derive (P5) microscope image (50) from text input (T) as microscope image properties (45) that need to be satisfied to conform to text input (T); Among them, the machine learning image evaluation model (M2) calculates relevant microscope image attributes (46) based on microscope images (50) captured using microscope settings (40) determined using a large language model (LLM). Wherein, if the microscope image attribute (46) of the microscope image (50) meets the microscope image attribute (45) that needs to be satisfied derived from the text input (T), then the microscope image (50) (P10) is used; and In cases where the microscope image attribute (46) of the microscope image (50) does not meet the microscope image attribute (45) to be satisfied, feedback (P12) is provided to the large language model (LLM), which uses the feedback to determine the adjusted microscope settings (40B) to capture a new microscope image (50B). The Large Language Model (LLM) evaluates (47) the adjusted microscope settings (40B) for capturing new microscope images (50B) to determine whether a follow-up inquiry (Q) should be made to the user (U) to authorize the capture of new microscope images (50B), since the adjusted microscope settings (40B) may affect the sample (10) or measurement duration.

9. The method according to claim 1, in, The large language model (LLM) includes an image analysis model (M1) that calculates analysis results (21) from an overview image (20), which are calculated by the large language model (LLM) together with text input (T) to determine the microscope settings (40). Among them, the analysis results (21) determine one or more of the following: -Sample carrier type, -Sample type, - Were chemical dyes used in sample preparation? If so, what chemical dyes were used? -Experiment type, -One or more focus positions, - The state or degree of contamination of the sample carrier or sample (10). -The location of the relevant area, and - The information content of the text or barcode on the sample carrier.

10. The method according to claim 9, in, When calculating the analysis results (21) based on the overview image (20), the text input (T) is also taken into account.

11. The method according to claim 1, The text input (T) also describes the desired image processing; The method further includes The text input (T) and the microscope image (50) are input (P3') into a large language model (LLM), which is trained to calculate (P4') processing parameters (41) for processing the microscope image (50) based on the text input (T) and the microscope image (50); and The microscope image (50) was processed using the calculated processing parameters (41).

12. The method according to claim 11, in, Text input (T) indicates at least one of the following: - Whether the processing of the microscope image (50) requires denoising, resolution enhancement, deconvolution, changing depth of field, changing virtual contrast type, suppressing background or removing artifacts.

13. The method according to claim 11 or 12, in, Processing parameter (41) determines one or more of the following: -A choice of any one of multiple deconvolution algorithms, any one of multiple resolution enhancement algorithms, or any one of multiple denoising algorithms; - The number of iterations to be performed, or the regularization term for the deconvolution algorithm, resolution enhancement algorithm, or denoising algorithm.

14. The method according to any one of claims 1-10, in, Text input (T) is created by the user through voice input and subsequent voice-to-text processing, or through an input device or graphical interface.

15. The method according to any one of claims 1-10, in, Large language models (LLMs) are formed by fine-tuning (FT1-FT3) an existing large language model, wherein the training data (30) for the fine-tuning (FT1-FT3) includes one or more of the following: -Microscope User Manual; - Chat logs between microscope users and microscope service personnel; - Records of a microscope training meeting.

16. A microscope system having Microscope for imaging (1); and Computer device (12), configured to perform the method according to claim 1.

17. A non-volatile data storage medium storing a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to claim 1.

Images