Synthesized fluorescence images with sample-specific references

Abstract

The prediction algorithm determines the synthesized fluorescence image based on the measured image. The prediction algorithm can be trained based on reference images automatically acquired during the measurement. The synthesized fluorescence image can also be validated based on the reference images.

Description

Technical Field

[0001] Various examples of the present disclosure generally relate to techniques for digitally post-processing measurement images acquired using a microscopic imaging modality. In particular, a synthetic fluorescence image can be determined. Background Art

[0002] Fluorescence imaging is used in various fields of "green science" (Life Sciences). Fluorescence imaging can be used to identify regions and structures in biological samples with high sensitivity and specificity. Specificity is achieved by constructing markers that specifically bind to specific molecules or cell structures. Specificity can also be achieved with markers that are only activated in a specific chemical environment. Specificity can also be achieved with markers produced within the cells themselves. This may involve, for example, autofluorescence or markers after appropriate genetic manipulation procedures. Sensitivity is achieved through the fluorescence of the marker at a specific wavelength. This allows for filtering signals within a spectral range.

[0003] Labels typically include or consist of fluorophores. A fluorophore is a molecule whose electron shell can be excited by light of a specific wavelength and then emits light of a longer wavelength. The use of fluorophores allows for spectral separation of excitation and emission. This allows for the measurement of signals that contain only a small fraction of background light. This allows for a high signal-to-noise ratio (SNR).

[0004] The two advantages of fluorescence imaging, namely specificity and sensitivity, make it possible to identify specific types of cell structures, such as cell nuclei, within a corresponding population. Another application involves, for example, the identification of tumors.

[0005] Fluorescence microscopy can present several drawbacks. For example, exposure to light to stimulate fluorescence can lead to photobleaching, where fluorophores lose their ability to fluoresce. The use of light to stimulate fluorescence can also cause phototoxicity, potentially damaging the cell structure of the sample. This can distort experiments.

[0006] Live imaging of ensembles of cellular structures can be particularly negatively impacted by photobleaching and phototoxicity. This is because live imaging typically requires the acquisition of numerous fluorescence images at a high temporal density. Consequently, exposure to light is extremely high. Consequently, the available observation period is limited, for example to a few hours, because after this time, the cellular structures can become damaged, thereby distorting the experiment.

[0007] The following publications are known: EP3553165A1; Wagner, Nils et al., “Deep learning-enhanced light-field imaging with continuous validation”, published in the journal Nature Methods (18.5 (2021): 557-563). Summary of the Invention

[0008] Therefore, better techniques are needed to obtain fluorescence images.

[0009] This object is achieved by the features of the independent claim. The features of the dependent claims define embodiments.

[0010] Some of the examples described here are generally based on the recognition that classic hardware-based fluorescence imaging can significantly influence the properties of the sample being examined, particularly cells, by using light to stimulate fluorescence. On the other hand, the methods described here are based on the recognition that other imaging modalities are less invasive because they require less sample exposure to acquire the corresponding image data. The examples described here are also based on the recognition that, even with these less invasive imaging modalities, there is a correlation between the contrast of the measurement image of interest and the expected contrast of the corresponding fluorescence image. Therefore, measurement images can be acquired using less invasive imaging modalities, such as conventional transmitted light microscopy in brightfield, without compromising the integrity of the sample. In particular, a very large number of measurement images can be acquired, for example with high temporal resolution.

[0011] Some of the examples described herein are also based on the recognition that, when using a prediction algorithm to determine a synthesized fluorescence image, it is often inherently impossible or difficult to estimate how reliable the prediction algorithm is in determining the synthesized fluorescence image. For example, it may be unclear whether the contrast contained in the synthesized fluorescence image accurately characterizes the sample or is an artifact. This means that the information content contained in the synthesized fluorescence image may be unclear. The quality of the synthesized fluorescence image may also be unclear.

[0012] Some of the examples described in this article are also based on the recognition that situations can sometimes arise in which, after the conclusion of a biological experiment, it turns out that a prediction algorithm does not provide good results, meaning that the contrast of the synthesized fluorescence image does not represent the sample correctly or only partially.

[0013] Next, some techniques for validating synthesized fluorescence images or prediction algorithms are described. The quality of the synthesized fluorescence images can be ascertained or evaluated. Furthermore, according to some examples described herein, the prediction algorithm can be adjusted through training, for example, when the corresponding validation results indicate inaccuracies. Sample-specific training can be performed.

[0014] In various examples described herein, a series of measurement images can be acquired during an observation period. The measurement images can be acquired using an imaging modality that results in a smaller exposure of the sample than fluorescence imaging. During the observation period, a smaller number of reference images can also be acquired using fluorescence imaging.

[0015] The measured images can then be digitally post-processed using a prediction algorithm. This prediction algorithm can determine a synthetic fluorescence image based on the measured images. The prediction algorithm can be trained and / or validated using reference images.

[0016] Each example relates to a computer-implemented method. The method includes controlling at least one imaging device to acquire a measurement image of a sample using microscopic imaging. The measurement image is acquired during an observation period. The method also includes controlling at least one imaging device to acquire a plurality of reference images of at least a portion of the sample using microfluorescence imaging during the observation period. The method also includes training parameters of a prediction algorithm based on at least a portion of the reference images as ground truth and also based on a portion of the measurement images. The method also includes determining a synthesized fluorescence image based on one or more of the measurement images and using the prediction algorithm after the training is complete.

[0017] The prediction algorithm can therefore be machine-learned.

[0018] With respect to the observation period, the reference images may be acquired with a smaller spatial density and / or a smaller temporal density than the measurement images.

[0019] The sample may be a biological sample. The sample may, for example, comprise a collection of cell structures. The cell structures may, for example, undergo a biological change process that occurs distributed over the collection and the observation period.

[0020] The computer-implemented method may further comprise synchronizing the determination of the synthesized fluorescence image with the acquisition of the measurement image after the training is completed. Synchronization may refer to the temporal coordination of the two processes. The synthesized fluorescence image may be determined separately in response to the acquisition of the measurement image. This enables real-time digital contrast post-processing of the measurement image.

[0021] Training can be performed based on a loss contribution to a loss function. The loss contribution can be based on a comparison of semantic content of at least a portion of a reference image and semantic content of at least a portion of a measurement image. The semantic content can quantify biological structure.

[0022] The computer-implemented method includes controlling at least one imaging device to acquire a measurement image of the sample during an observation period using microscopic imaging. Furthermore, the computer-implemented method includes controlling at least one imaging device to acquire a plurality of reference images of at least a portion of the sample during the observation period using microfluorescence imaging. The computer-implemented method also includes determining a synthesized fluorescence image based on the measurement image and based on a prediction algorithm. The computer-implemented method also includes performing validation of the synthesized fluorescence image based on a comparison between at least a portion of the reference image and at least a portion of the synthesized fluorescence image.

[0023] The computer-implemented method may also include, for example, adjusting one or more parameter values ​​of the prediction algorithm based on the validation results. For a machine-learned prediction algorithm, the prediction algorithm may, for example, be selectively retrained or retrained (i.e., retrained) based on the validation results.

[0024] It is also possible that the computer-implemented method includes adjusting the acquisition of reference images based on the results of the verification. In the event of a verification failure, for example, additional reference images may be acquired selectively or on demand so that the prediction algorithm can be retrained based on these additional reference images.

[0025] The verification can be based on a comparison of the semantic content of at least a portion of the reference image with the semantic content of at least a portion of the measured image. This may obviate the need for a registration of the respective measured image with the respective reference image. Pixel-based correspondence need not be present.

[0026] However, a pixel-based comparison can also be performed for verification.

[0027] In principle, reference images can be acquired during the observation period, interleaved with the measurement images.

[0028] Reference images may also be acquired in response to a trigger event.

[0029] In principle, various triggering events are conceivable, such as user commands or changes in the semantic content of the measurement image or the output of an object recognition algorithm (which can operate on the measurement image, for example).

[0030] At least one imaging device can be controlled to acquire reference images more frequently during at least one calibration phase (also referred to as calibration period), which is arranged at the beginning and / or end of the observation period or in relation to expected points in time of biological processes of the sample. In particular, reference images can be acquired more frequently during the calibration phase than outside of the calibration phase.

[0031] At least one imaging device can be controlled to capture a reference image in a first area of ​​the sample. At least one imaging device can also be controlled to capture a measurement image at least partially in a second area of ​​the sample that is different from the first area of ​​the sample. In this case, the reference image cannot be captured in the second area of ​​the sample. This protects the second area of ​​the sample.

[0032] At least one imaging device can be controlled to acquire a reference image using microfluorescence imaging with a first value of an exposure parameter for microfluorescence imaging. At least one imaging device can be controlled to acquire a measurement image using microfluorescence imaging with a second value of an exposure parameter for microfluorescence imaging. The first value can cause a first illumination of the sample and each reference image, and the second value can cause a second illumination of the sample and each measurement image. The second illumination is smaller than the first illumination.

[0033] In principle, much fewer reference images than measurement images can be acquired in an observation period. For example, the number of reference images in an observation period cannot be greater than 50% (or 5% or one percent) of the number of measurement images in the same observation period.

[0034] A device includes a processor. The processor is configured to control at least one imaging device to acquire a measurement image of a sample using microscopic imaging during an observation period. Furthermore, the processor is configured to control the at least one imaging device to acquire a plurality of reference images of at least a portion of the sample using microfluorescence imaging during the observation period. Furthermore, the processor is configured to train parameters of a prediction algorithm, specifically based on at least a portion of the reference images as ground truth and also based on at least a portion of the measurement images. The processor is also configured to determine a synthesized fluorescence image after the training is complete based on one or more of the measurement images and using the prediction algorithm.

[0035] An apparatus includes a processor configured to: control at least one imaging device to acquire a measurement image of a sample during an observation period using microscopic imaging; control at least one imaging device to acquire a plurality of reference images of at least a portion of the sample during the observation period using microfluorescence imaging; determine a synthesized fluorescence image based on the measurement image and based on a prediction algorithm; and perform verification of the synthesized fluorescence image based on a comparison between at least a portion of the reference image and at least a portion of the synthesized fluorescence image.

[0036] A computer program or computer program product or a computer-readable storage medium includes program code. This program code can be loaded and executed by a processor. This causes the processor to implement a method. The method includes controlling at least one imaging device to acquire a measurement image of a sample using microscopic imaging. The measurement image is acquired during an observation period. The method also includes controlling at least one imaging device to acquire multiple reference images of at least a portion of the sample using microfluorescence imaging during the observation period. The method also includes training parameters of a prediction algorithm based on at least a portion of the reference images as ground truth and also based on a portion of the measurement images. The method also includes determining a synthesized fluorescence image based on one or more of the measurement images and using the prediction algorithm after the training is complete.

[0037] A computer program, a computer program product, or a computer-readable storage medium includes program code. This program code can be loaded and executed by a processor. This causes the processor to implement a method. The method includes controlling at least one imaging device to acquire a measurement image of a sample using microscopic imaging during an observation period. Furthermore, the computer-implemented method includes controlling at least one imaging device to acquire multiple reference images of at least a portion of the sample using microfluorescence imaging during the observation period. The computer-implemented method also includes determining a synthesized fluorescence image based on the measurement image and based on a prediction algorithm. The computer-implemented method also includes verifying the synthesized fluorescence image based on a comparison between at least a portion of the reference image and at least a portion of the synthesized fluorescence image.

[0038] The features mentioned above and those explained below can be used not only in the respectively explicitly stated combination, but also in other combinations or alone, without departing from the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A system is shown with at least one imaging device and means for digitally post-processing images acquired by the at least one imaging device;

[0040] Figure 2Schematically illustrates digital contrast post-processing of measurement images acquired with at least one imaging device;

[0041] Figure 3 is a flow chart of an exemplary method;

[0042] Figure 4 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0043] Figure 5 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0044] Figure 6 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0045] Figure 7 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0046] Figure 8 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0047] Figure 9 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0048] Figure 10 The timing of acquiring the measurement image and acquiring the reference image is shown exemplarily and schematically;

[0049] Figure 11 The locations and timings for acquiring measurement images and acquiring reference images are shown exemplarily and schematically. DETAILED DESCRIPTION

[0050] The above-mentioned characteristics, features and advantages of the present invention and the manner in which these characteristics, features and advantages are achieved will become more clearly and distinctly understood in conjunction with the following description of an embodiment which is explained in more detail with reference to the accompanying drawings.

[0051] The present invention will now be explained in more detail with reference to the accompanying drawings using preferred embodiments. In the drawings, identical reference numerals designate identical or similar elements. The drawings are schematic representations of various embodiments of the present invention. The elements shown in the drawings are not necessarily shown true to scale. Rather, the various elements shown in the drawings are reproduced so that their functions and general application can be understood by a person skilled in the art. The connections and couplings between the functional units and elements shown in the drawings can also be implemented as indirect connections or couplings. The connections or couplings can be implemented wired or wirelessly. The functional units can be implemented as hardware, software, or a combination of hardware and software.

[0052] The following describes some techniques for digitally post-processing the contrast of measured images of biological specimens. In particular, the techniques described herein can be used to generate computer-generated images that replicate the contrast of a specific imaging modality without actually using the corresponding imaging modality.

[0053] According to the various examples described herein, synthetic images can be generated that mimic the imaging of stained samples (called "virtual staining" in English). In this type of imaging of stained samples, the biological sample is usually stained with a corresponding dye. This is done during the staining process. The dye is brought into contact with the sample. The dye contains one or more markers. These markers produce a special contrast in the imaging. For example, the marker can specifically bind to a specific cell part, such as the cell nucleus or mitochondria. With the help of the marker, tumor tissue can be made visible. For example, some markers that are activated by illumination with a specific wavelength and then emit light of other wavelengths, that is, fluorescent markers, can be envisioned.

[0054] As a further general rule, the most different types of samples can be tested in the examples described herein. Biological samples can be tested. For example, living cells or dead cells, for example living cells or dead cells in tissue samples, can be tested. Cell cultures can be tested, for example. Biological samples can therefore be referred to as biological model systems, such as cells or spheroids (that is to say, common cell clusters, such as tumors or neural cell clusters) or organoids (that is to say, organ-specific cell clusters). Therefore, biological samples can especially include cells or tissues. Cell cultures can be tested. For example, primary cell cultures can be tested, that is to say, cells that are obtained from an organism and remain active. Dead cells can also be tested. Such samples therefore include, for example, a collection of cell structures.

[0055] The following describes some techniques, particularly in conjunction with fluorescence imaging. These techniques allow for the generation of synthetic fluorescence images. Fluorescence images have a contrast that characterizes a specific cell structure from a larger number of cell structures with high specificity and sensitivity. This allows for the differentiation of a specific cell type from other cell structures of other types.

[0056] In the following, the various techniques are described in particular with reference to sample staining for fluorescence imaging; however, in other examples, the staining can also be used for other fields of application, for example, histopathology.

[0057] In the following, techniques are described which combine measurement techniques for fluorescence imaging, ie, for wavelength-selective excitation and wavelength-band-selective detection of fluorescence, with the aid of suitable measurement hardware, with techniques for digital image post-processing.

[0058] This combination of classical hardware-based measurement technology and digital image post-processing allows, on the one hand, high-quality fluorescence images to be generated, while, on the other hand, effects such as photobleaching and phototoxicity caused by fluorescence excitation due to light exposure can be reduced. In particular, image quality can be optimized, meaning a particularly high degree of reliability, ensuring that fluorescence images have the necessary sensitivity and specificity for specific cell structures. Furthermore, the reduction in photobleaching and phototoxicity allows, for example, longer observation periods compared to classical hardware-based fluorescence imaging techniques. Higher temporal resolution, meaning more images per unit time, can also be achieved.

[0059] The techniques described herein can also be used to reliably check whether a synthesized fluorescence image has the desired accuracy. In particular, it can be checked, for example, whether a prediction algorithm used to determine the synthesized fluorescence image has produced the intended prediction. This means that the synthesized fluorescence image or the prediction algorithm can be verified.

[0060] According to various examples, a series of measurement images of a sample are acquired during an observation period using microscopic imaging. This involves a biological sample. A biological sample can therefore be referred to as a biological model system, such as cells or spheroids (i.e., general cell clusters, such as tumors or neural cell clusters) or organoids (i.e., organ-specific cell clusters).

[0061] Biological samples can therefore include, in particular, cells or tissues. Cell cultures can be examined. For example, primary cell cultures can be examined, that is, cells obtained from an organism and kept alive. Dead cells can also be examined. Such samples can therefore include, for example, a collection of cellular structures. During the same observation period, reference images are also acquired using microfluorescence imaging. These reference images are acquired for at least a portion of the sample.

[0062] A composite fluorescence image is then generated for the plurality of measurement images using a prediction algorithm and taking into account the reference image. This means that the contrast of the measurement images (that is, the pixel values, which encode, for example, color and / or brightness) is digitally post-processed using the prediction algorithm.

[0063] In principle, the prediction algorithm can be parameterized manually, i.e., appropriately set using appropriate user prior knowledge. However, it is also conceivable that the prediction algorithm is machine-learned. The corresponding training can then be performed in a computer-implemented manner. For example, an artificial neural network (KNN) can be used in this context. KNNs consist of a large number of layers that perform various operations. An example of a KNN is a convolutional neural network (CNN), which uses convolutional layers that perform convolutions on kernel input values. The various layers can be interconnected via suitable weights. Nonlinear activations are conceivable. Pooling operations can be performed, where information is discarded. An example is max pooling, where only the strongest values ​​in a region (e.g., 2 x 2 neurons) are retained. KNNs can have a feedforward architecture. Here, the results of one layer are always communicated only to another layer. If so-called skip connections are present, the output of one layer can be communicated to multiple subsequent layers. In principle, different types of KNNs can be used, such as generative adversarial networks (GANs) or autoencoder networks, such as variational autoencoder networks.

[0064] The measurement images are acquired using microscopy such that effects such as photobleaching and phototoxicity are relatively low in each measurement image compared to each reference image. This means that the imaging modality of microscopy for acquiring the measurement images is relatively less invasive.

[0065] As a general rule, various imaging modalities can be used for microscopic imaging in the various examples described herein to acquire measurement images during the observation period. 2D or 3D imaging modalities can be used. In principle, it is conceivable to acquire measurement images in each sample layer in conjunction with the acquisition of measurement images. This can be achieved by using a suitable objective lens and arranging the layer to be acquired in the focal point of the objective lens. A so-called Z-stack of multiple sample layers can also be acquired. This can be achieved, for example, by moving the sample stage in the Z direction between the acquisition of two measurement images.

[0066] For example, the measurement images can be acquired with the aid of a conventional transmitted light microscope. Brightfield imaging can be used. Darkfield imaging or phase contrast, such as differential phase contrast, can also be used. Further examples relate to holographic imaging, holographic tomography, quantitative phase tomography, optical coherence tomography, dynamic optical coherence tomography, angle-selective illumination imaging, phase gradient imaging, structured illumination imaging. Another example relates to the acquisition of measurement images with a high dynamic range (HDR imaging) using burst mode (that is to say, short time intervals between the raw images of an image series, which are in particular smaller than the time scale of the sample dynamics) with different exposure times. This means that multiple raw images can be acquired per measurement image, which have different exposure times and can then be combined to form a single measurement image.

[0067] Microscopic imaging can therefore use imaging modalities other than fluorescence imaging. Here, the prediction algorithm implements a picture-to-picture (B2B) prediction algorithm that converts a first contrast (e.g., brightfield imaging in transmitted light) into a second contrast (e.g., fluorescence, e.g., from a sample stained with a fluorescent marker or autofluorescence).

[0068] In some examples, an autofluorescence imaging modality can be used to acquire the measurement images. Here, fluorescence can be emitted without staining the biological sample with a marker. This can avoid physiological changes in the sample caused by temporal changes in the marker or the marker itself.

[0069] However, in some examples, it is also conceivable to use fluorescence imaging as an imaging modality for acquiring measurement images. However, the exposure parameters of fluorescence imaging (e.g., light intensity and / or exposure time) can be set to the values ​​used for acquiring measurement images so that the illumination or light exposure of each measurement image and each sample surface is less than the imaging parameters of fluorescence imaging used to acquire reference images. This means, in other words, that the photobleaching and phototoxic effects of each measurement image may be smaller than those of each reference image, even though, in principle, the same imaging modality of fluorescence imaging is used for acquiring both measurement images and reference images. For example, it is also conceivable to activate other markers to acquire measurement images instead of activating the measurement image. For example, a marker that can be activated with a smaller amount of light or with minimal invasiveness can be used to acquire reference images. However, in principle, it is also possible to predict another second fluorescence contrast based on the first fluorescence contrast, thereby reducing light exposure overall. For example, the tdTomato marker and the corresponding fluorescence contrast can be predicted based on the "green fluorescent protein" (GFP) marker and the corresponding fluorescence contrast.

[0070] If fluorescence imaging is also used to acquire the measurement image, but fluorescence imaging results in a lower light exposure, this typically means that the resulting fluorescence measurement image has a lower signal-to-noise ratio (SNR). In this scenario, the synthesized fluorescence image has a higher SNR than the measurement image. Therefore, the prediction algorithm in this scenario is sometimes also called a denoising algorithm.

[0071] As can be seen above, both the measurement image and the reference image are acquired during the observation period. The measurement image can be acquired last. The observation period defines the duration of the measurement. The observation period can be coordinated, in particular, with the underlying biological properties of the sample. This means that during the observation period, one or more biological processes may occur in the cellular structures of the cellular structure collection of the sample, and the measurement is intended to detect these biological processes. Therefore, the reference image is acquired during the actual measurement (rather than, for example, in a separate calibration measurement).

[0072] Then, using an algorithm, synthetic fluorescence images can be determined based on the measurement images. Compared to the measurement images, these synthetic fluorescence images have (enhanced) fluorescence contrast. In this relationship, the reference image can acquire one or more of the functions described below in conjunction with Table 1.

[0073]

[0074]

[0075]

[0076] Table 1: Various possible approaches for using reference images acquired during an observation period. The various possible approaches can also be combined with one another. For example, it is conceivable to use a portion of the reference images for training a machine-learned prediction algorithm (hereinafter: training-reference images) according to example I, and to use another portion of the reference images for validation (hereinafter: validation-reference images) according to example II.

[0077] The prediction algorithm may utilize the correlation between the image contrast of the measurement image and the image contrast of the fluorescence image.

[0078] This technique, also known as "virtual staining," is based on the fact that the contrast of the measured image is altered or enhanced through digital post-processing using a prediction algorithm, resulting in a synthetic fluorescence image. This virtual staining offers various advantages, including reduced exposure and scanning time, improved signal-to-noise ratio (SNR), reduced phototoxicity, and reduced experimental costs.

[0079] Various techniques are based on the recognition that reference techniques for virtually staining cell structures have certain drawbacks. These reference techniques are described, for example, in Christiansen, Eric M. et al., “In silico labeling: predicting fluorescent labels in unlabeled images” (Cell, 173.3 (2018): 792-803) and in Ounkomol, Chawin et al., “Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy” (Nature Methods, 15.11 (2018): 917-920). In these reference techniques, a parameterization of the prediction algorithm is performed based on reference measurements on a separate sample or before the actual measurement. The parameterized prediction algorithm is then used to determine a synthetic fluorescence image based on measurement images acquired during the observation period of the actual sample measurement. However, during the observation period, no additional reference images are collected in reference techniques. These reference techniques have the disadvantage of requiring a separate reference measurement. This can be time-consuming. The user must implement a separate workflow to perform the reference measurement, which requires time, user skills, and additional implementation effort. These reference techniques also have the disadvantage that differences can often arise between the reference measurement and the actual measurement of the sample. When using different samples to parameterize the prediction algorithm on the one hand and perform the measurement on the other, it may happen that the parameterization of the sample does not match the measurement. This may occur, for example, due to different sample preparations. It may also occur due to different imaging modalities used to collect measurement images in the reference measurement and in the actual measurement. If, for example, supervised learning techniques are used for the prediction algorithm through machine learning, this means that a large amount of training data must be collected and annotated. This can be particularly expensive.

[0080] Another disadvantage that can arise with reference techniques is that the prediction quality of the prediction algorithm—that is, the quality of the synthesized fluorescence images—can be verified only to a limited extent or not. This means that the user cannot verify the information content of the synthesized reference images. This can, for example, lead to incorrect diagnoses or incorrectly derived measurement variables based on the synthesized fluorescence images. This can also have an impact on the safety of the corresponding method, for example, if it impairs subsequent data evaluation.

[0081] The techniques described herein enable these disadvantages to be eliminated or mitigated.

[0082] According to various examples, reference images can be acquired during an observation period with a smaller temporal density and / or a smaller spatial density than the measurement images. This means, for example, that fewer reference images are acquired during an observation period than measurement images. Fewer reference images can be acquired for each sample volume of the sample than measurement images. This allows the sample load to be limited by acquiring reference images. Imaging is performed (at least primarily) based on the measurement images. Reference images that are sparsely sampled in time and / or spatial space can, in turn, have an auxiliary function (training and / or validation).

[0083] Figure 1 A system 100 according to various examples is schematically illustrated. System 100 includes a device 101. Device 101 can be, for example, a computer or server. Device 101 includes a processor 102 and a memory 103. Device 101 also includes an interface 104. Device 101 can receive image data, such as measurement images and / or reference images, from one or more imaging devices 111, 112 via interface 104. Processor 102 can also send control data to the one or more imaging devices 111, 112 via interface 104 to control the imaging devices to acquire image data. Processor 102 can also use the control data to set values ​​for one or more imaging parameters, such as illumination parameters.

[0084] Generally speaking, processor 102 may be configured to load control instructions from memory 103 and execute the control instructions. When processor 102 loads and executes the control instructions, this causes processor 102 to execute the techniques described herein. These techniques include, for example, controlling imaging device 111 and optional imaging device 112 to acquire image data. For example, processor 102 may be configured to control imaging device 111 to acquire multiple measurement images of a sample using microscopic imaging during an observation period. Processor 102 may also be configured to control imaging device 111 to acquire multiple reference images of at least a portion of the sample during the same observation period. Reference images can, in principle, be acquired using the same imaging modality as the measurement images. Depending on the capabilities of imaging device 111 with respect to the imaging modality used to acquire the reference images, processor 102 may also control imaging device 112 to acquire the reference images.

[0085] Furthermore, the processor 102 can be configured to determine a synthesized fluorescence image based on the measured image and based on the prediction algorithm based on control instructions from the memory 103. This means that the processor 102 can perform virtual staining or other image-to-image transformations, such as denoising.

[0086] Furthermore, the processor 102 may be configured to perform one or more of the example techniques according to Table 1, that is, for example, training a prediction algorithm based on at least a portion of a reference image and / or validating a synthesized fluorescence image as an output of the prediction algorithm based on at least a portion of a reference image.

[0087] Also there Figure 2 The corresponding technology is schematically shown in FIG.

[0088] Figure 2 Some aspects are schematically illustrated with a virtual staining of a measurement image 201 depicting a sample comprising a collection of cellular structures. The measurement image 201 serves as input to a prediction algorithm 205, which provides a synthesized fluorescence image 202 as output. The prediction algorithm 205 is trained 6001 using a training-reference image 251 (see Table 1: Example I). The synthesized fluorescence image 202 is validated 6002 using a validation-reference image 261 (see Table 1: Example II).

[0089] In principle, training reference images 251 and validation reference images 261 can be acquired using the same imaging modality. This means, in particular, that it is not necessary to determine at the time of acquisition whether the corresponding reference image will be used as a training reference image or a validation reference image. In principle, a specific reference image can also be used as both a training reference image and a validation reference image.

[0090] The prediction algorithm can be implemented, for example, by KNN. The KNN architecture can be implemented, for example, according to Christiansen, Eric M. et al., "In silico labeling: predicting fluorescent labels in unlabeled images" (Cell, 173.3 (2018): 792-803) and Ounkomol, Chawin et al., "Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy" (Nature Methods, 15.11 (2018): 917-920).

[0091] Another architecture can be implemented as a U-net with convolutional layers and skip connections across certain layers. See, for example, Ronneberger, O., Fischer, P., & Brox, T. (October 2015), "U-net: Convolutional networks for biomedical image segmentation" (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241, Springer-Verlag, Champlain). However, in this case, grayscale regression is used as the output instead of a classification layer (to obtain image-to-image predictions). U-nets can also be trained with an additional adversarial loss contribution. This is also known as a "conditional survivable adversarial network."

[0092] Another example is for example an autoencoder network, such as a variational autoencoder network.

[0093] In principle, it may be desirable to use different reference images for training 6001 and validation 6002 in order to perform these processes independently of each other. The training-reference image 251 and the validation-reference image 261 can therefore be acquired at different points in time during the observation period and / or at different areas of the sample. Figure 3 Some aspects are discussed in detail accordingly.

[0094] Figure 3 is a flow chart of an exemplary method. When the processor loads the control instructions from the memory and executes the control instructions, the method can be implemented by a device with a processor and a memory, for example. Figure 3 , the method can be performed by Figure 1 101 is implemented, and in particular is implemented by the processor 102 when it loads the program code from the memory 103. Figure 3 Indicated by dotted lines.

[0095] In block 3005, a plurality of measurement images (e.g., time series) are collected during the observation period in the corresponding measurement. The measurement images depict, for example, a biological sample including a collection of cellular structures. The cellular structures may be undergoing a biological process. Such biological processes may occur distributed across the collection and the observation period. The purpose of the measurement may therefore be to depict or measure such biological processes. More generally, the purpose of the measurement may be to characterize the biological sample.

[0096] Because biological processes occur distributed in time and space, that is, they are observed at different points in time at different locations, it can be helpful to acquire measurement images with a high repetition rate. This means acquiring many measurement images per unit time. Alternatively or additionally, measurement images can also be acquired with a high spatial density, that is, for example, Z-stacks of measurement images can be acquired (repeatedly). This allows for the depiction of biological processes. This can be particularly helpful when the duration of each individual occurrence of a biological process is much smaller than the temporal dispersion of the occurrences of the biological process in the set.

[0097] To acquire the measurement image 3005, the apparatus may appropriately control the corresponding imaging device. The processor may, for example, send a control command to the imaging device. The control command may indicate a time point for triggering exposure and / or a value of one or more lighting parameters.

[0098] A reference image is acquired in block 3010. The reference image is acquired during the same observation period during which the measurement image was acquired and during which the cell structure of the collection of samples was, for example, subjected to a biological change process.

[0099] The typically independent distribution of the processes in time and location makes it possible, in particular, to also record reference images in a distributed manner along the temporal and / or spatial dimensions. It is conceivable to record the reference images as a plurality of individual images of different regions, where the samples are substantially similar in these regions. Alternatively or additionally, the reference images can be acquired in a distributed manner.

[0100] The reference image is acquired using fluorescence microscopy as the imaging modality. This means that, to acquire the reference image, the cell structure is excited with suitable light, specifically one that is tuned to the electronic excitation state of the fluorophore. The wavelength of the light is tuned to the corresponding excitation energy of the fluorophore. Therefore, the sample can be stained with a marker before the observation period. However, autofluorescence can also be stimulated (e.g., by hemoglobin naturally present in the sample).

[0101] Therefore, in some examples, autofluorescence imaging modalities can also be used to acquire reference images. This allows fluorescence to be emitted without staining the biological sample with markers. This can prevent physiological changes in the sample caused by temporal variations in the marker or the marker itself. However, due to recording limitations, acquiring autofluorescence images is invasive to the sample. Therefore, replicating the autofluorescence signal using a less invasive imaging modality significantly protects the sample.

[0102] In principle, measurement images can also be acquired in block 3005 using imaging modalities other than fluorescence imaging. However, fluorescence imaging can also be used to acquire measurement images, where other values ​​can be used for one or more exposure parameters, resulting in a lower illumination level for each measurement image than for each reference image. The prediction algorithm can then be trained in block 3015 based on at least a portion of the reference images, specifically the training reference images. This means that specific parameter values ​​for the prediction algorithm are appropriately set. The training reference images serve as ground truth. This means that the output of the prediction algorithm in the respective training state is compared with the respective training reference images, whereby the output of the prediction algorithm can be determined based on the associated measurement images.

[0103] Machine learning techniques can be used to train prediction algorithms. For example, KNN can be trained. This means, for example, that gradient descent methods can be used to adjust the parameter values ​​of the prediction algorithm based on a value of a loss function based on a comparison (e.g., pixel comparison) between the corresponding training reference image and the associated measurement image. Different loss functions can be used, such as the sum of squared deviations between pixel values. Parameter values ​​can also be adjusted using so-called backpropagation.

[0104] The training-reference image serves as the ground truth for a specific measurement pattern. To determine the pairing between the training-reference image and the measurement image for determining the value of the loss function based on the corresponding comparison, the spacing between the training-reference image and the measurement image in time and / or space can be taken into account. For example, the training-reference image can be used as the ground truth for the measurement image when the training-reference image and the measurement image depict the same region of the sample and are acquired at very short intervals. Short intervals can be time intervals so short that biological processes cannot occur, or only with a sufficiently low probability. For example, it may be desirable for the time interval between acquiring a measurement image and acquiring reference images 251, 261 of a pair 601, 602 to be no longer than 1 second, optionally no longer than 500 ms, and further preferably no longer than 10 ms. This is based on the understanding that such time scales are short enough to avoid significant changes in fluorescence contrast between the measurement image and the associated reference image.

[0105] Sometimes it is possible that the reference image and the measurement image may inherently depict substantially the same area of ​​the sample in the same pose. This may be the case when the same imaging optics are used, such as when a specific objective of the imaging device 111 is used both for acquiring the reference image and for acquiring the measurement image, e.g. Figure 1When using different imaging optics, it may be necessary to register the reference image pairs to one another, that is to say to assign corresponding sample positions in the images.

[0106] However, in all examples, it is not necessary to inherently depict the same sample region or to perform a registration between the measurement image and the training-reference image to identify corresponding sample positions. In some examples, regardless of the specific microstructure depicted, basic morphological properties of the contrast in the synthetic fluorescence image reconstructed based on the measurement image and in the training-reference image can be compared with one another by appropriately selecting a corresponding loss function considered during training. This can especially be the case with loss functions that use loss contributions without a pixel-by-pixel comparison between the synthetic fluorescence image and the training-reference image.

[0107] Unregistered images can be compared with one another, for example, by using a recurrent generative adversarial network, where a non-pixel-dependent adversarial loss contribution or recurrent loss is taken into account in the loss function during training.

[0108] As another example of non-pixel-based loss contributions, the semantic information content of a synthesized fluorescence image can be compared with a reference image. This semantic information content can be determined by evaluating the corresponding images. For example, this semantic information content can describe biological properties of the sample. This semantic information content can be selected from the following group: the number or surface density of specific biological structures; the average size or size distribution of specific biological structures; and the average spacing between specific biological structures. Biological structures can be, for example, cellular components such as nuclei or mitochondria. Such biological properties can be determined, for example, using an object recognition algorithm. This can again be achieved, for example, using a kNN algorithm. The association between the measurement image and the reference image can be learned using training in block 3015. However, training in block 3015 is generally optional. In various examples, it is conceivable that the prediction algorithm is parameterized already in preparation for acquiring the measurement image 3005. This can be performed during the corresponding reference measurement. Furthermore, it is not necessary to use a machine-learned prediction algorithm in all examples. In this case, a classic prediction algorithm with fixed settings can be used.

[0109] In block 3020, a composite fluorescence image is then determined based on the measurement images and using a prediction algorithm. If the prediction algorithm was previously trained in block 3015, the corresponding trained prediction algorithm is used. The prediction algorithm thus receives one of the measurement images as input and determines the associated fluorescence image.

[0110] Optionally, it is conceivable to verify the synthesized fluorescence image from block 3020 in block 3025. This can be based on a comparison between at least a portion of the reference image from block 3019 (the verification reference image) and an initial portion of the synthesized fluorescence image. For example, individual image points can be compared with one another and deviations in corresponding color or brightness values ​​determined. It may also be appropriate to pair the synthesized fluorescence image and the verification reference image based on their spacing in time and / or spatial space, as already explained above in conjunction with block 3015 with the training reference image. These verification images can be compared to one another, provided they have a particularly close spacing in time and / or spatial space. If different sample areas and / or different positions of the imaging optics relative to the sample areas are used, it may be helpful (but not necessary) to perform a registration between the paired measurement image and the verification reference image.

[0111] As a further example of a non-pixel-based comparison for verification, the semantic information content of a synthetic fluorescence image can be compared with a reference image. The semantic information content can be determined by evaluating the corresponding images. This semantic information content can, for example, describe biological properties of the sample. This semantic information content can be selected, for example, from the following group: the number or surface density of specific biological structures; the average size or size distribution of specific biological structures; and the average spacing between specific biological structures. Biological structures can, for example, be cellular components such as nuclei or mitochondria. These biological properties can be determined, for example, using an object recognition algorithm. This can, in turn, be achieved, for example, using a KNN.

[0112] In optional block 3030, a check can be performed to determine whether the verification in block 3025 was successful. Based on this check, blocks 3010, 3015, 3020, and 3025 can be re-executed in the corresponding loop 3099. This means that the acquisition of reference images in block 3010 can be adjusted and, in particular, extended, based on the result of the verification in block 3025. In other words, the training in block 3015 and the verification in block 3025 can be performed iteratively according to loop 3099. If the verification does not indicate a high quality of the synthesized fluorescence image, additional reference images can be acquired in block 3010, for example. A successful verification can be defined, for example, if the degree of agreement between the synthesized fluorescence image and the verification reference image (according to a predetermined metric) exceeds a predetermined threshold. The training can then be further refined in block 3015 based on the additional detected reference images during the subsequent execution of block 3010.

[0113] With the help of Figure 3 The proposed method can eliminate the shortcomings previously mentioned in conjunction with classic hardware-only fluorescence imaging. In addition, it can also eliminate the shortcomings previously mentioned in conjunction with the reference implementation for virtual staining.

[0114] The disadvantages of reference techniques can also be mitigated by combining digital contrast post-processing for denoising, see for example Weigert, Martin et al., “Content-aware image restoration: pushing the limits of fluorescence microscopy” (Nature Methods, 15.12 (2018): 1090-1097) or Prakash, Mangal et al., “Removing Pixel Noises and Spatial Artifacts with Generative Diversity Denoising Methods” (arXiv preprint arXiv: 2104.01374 (2021)). In particular, the sample or at least parts of the sample can be virtually stained, that is, the result corresponds to the result of a reference technique for virtual staining accompanied by a separate reference measurement. However, a separate reference measurement is not necessary because the reference image is acquired during the actual observation period.

[0115] The described technique allows for imaging samples with high image resolution, while using a less invasive imaging modality to acquire the measurement images. Furthermore, the sample can be imaged with high temporal resolution, since the light exposure for each measurement image is lower in the less invasive imaging modality.

[0116] The reliability of the prediction of the synthetic fluorescence image can also be verified. In particular, independent reference images can be used to assess the quality of the prediction.

[0117] Then it was given Figure 3A specific example of the application of the method is provided. This example relates to time-lapse experiments in the observation of living cell cultures. Here, a "video" is to be created from a series of synthetic fluorescence images acquired over a long observation period, for example, several hours or even days. Cell cultures undergo a number of biological processes, such as the individual steps of the cell cycle, particularly mitosis. In particular, organelles, the cell nucleus, and DNA can be observed here. Corresponding specific markers are known. However, the duration of the individual steps of the cell cycle or other intracellular processes is extremely short compared to the observation period. Therefore, the mitotic process typically lasts only a few minutes, and vesicle movement may require a time resolution of several minutes. Since, in some scenarios, different cells of a cell culture are evenly distributed in different steps of the cell cycle, the time points of mitosis can again appear evenly distributed across the cell collection. Therefore, it is generally desirable to acquire measurement images with a high repetition rate, that is, a series of measurement images should have a spacing between adjacent measurement images that is of the same order of magnitude as the duration of the individual steps of the cell cycle.

[0118] Classic hardware-based fluorescence imaging involves very high light exposure, which can damage the cell culture and prevent the various steps of the cell cycle to be observed from occurring unaffected by the measurement. Phototoxicity and photobleaching are no longer negligible. Therefore, in classic hardware-based fluorescence imaging techniques, a trade-off may need to be made between opposing target parameters: a high signal-to-noise ratio of the fluorescence image; a long observation period; and low phototoxicity and photobleaching.

[0119] It should be noted that specific biological processes do not necessarily occur evenly distributed over time during the observation period for each cell in the collection. Sometimes, specific biological processes can be triggered in a targeted manner by changing external conditions, such as adding a chemical substance or changing process conditions, for example by using so-called cell synchronization. In other words, specific biological processes may occur frequently during specific, shorter time intervals within the observation period. For example, cell cultures can be prepared in such a way that the majority of cells in the sample cell culture enter the mitotic phase after a short time interval from the start of the experiment, for example, after 1 hour. In the preceding period, only few or no interesting processes occurred. However, in these scenarios, it may be difficult to achieve sufficiently high quality synthesized fluorescence images using classic hardware-based fluorescence imaging.

[0120] Therefore, by using the previously described techniques to acquire measurement images using a suitable imaging modality, it is possible to achieve lower light exposure and thus reduce phototoxicity and photobleaching. This in turn achieves higher temporal resolution. Overall, fluorescence imaging can be used to acquire significantly fewer reference images (training reference images and / or validation reference images) than measurement images. The number of reference images within an observation period may, for example, be no more than 50%, optionally no more than 5%, and further optionally no more than 1% of the number of measurement images within the same observation period. This means that the temporal density of the reference images can be less than that of the measurement images. This allows for targeted training of the prediction algorithm based on the reference images and / or validation of the synthesized fluorescence image determined by the prediction algorithm, while also achieving high temporal resolution for the synthesized fluorescence image.

[0121] As an alternative to or in addition to this sparse sampling over time via the reference image, the samples can also be sparsely sampled in the spatial space via the reference image and the measurement image.

[0122] Here, as a general rule, there are various possibilities for configuring the acquisition of reference images. Reference images can be acquired at different time positions and / or in different areas of the sample. These different implementations can differ in terms of the illumination of the sample, the efficiency of the workflow, the validity of the results, and the time required to perform the measurement. Figures 8 to 11 Let’s discuss some different examples in detail. Figures 4 to 11 Various variations can also be combined with each other to form new variations.

[0123] Figure 4 Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 for training and the acquisition of validation-reference images 261 for validating a prediction algorithm are shown. Figure 4 In particular, some aspects of the corresponding timing of the observation period 301 are shown.

[0124] Figure 4 It is shown that a series 200 of measurement images 201 are acquired at approximately the same sampling rate during the observation period 301. At the same time, the training-reference images 251 are acquired only during the calibration phase 311, which is at the beginning of the observation period 301. The measurement images 201 are also acquired during the calibration phase 311, so that a pair 601 of corresponding measurement images 201 and training-reference images 251 can be formed, which can be combined as before. Figure 3 The method is described in block 3015 for training the prediction algorithm. The measurement image 201 and the training reference image 251 of the pair 601 are acquired adjacently within a certain time.

[0125] After the prediction algorithm 205 has been trained, a synthesized fluorescence image 202 can be determined based on the measurement image 201 .

[0126] These synthesized fluorescence images 202 can then be verified based on the verification-reference images 261. The verification-reference images 261 are acquired at a low sampling rate during the entire observation period 301. That is, the light exposure is limited.

[0127] The reliability of the synthesized fluorescence image can be checked with the aid of the verification reference image 261. For example, a comparison between the nearest neighbors (time intervals) of the corresponding verification reference image 261 and the measurement image 201 can be determined. Figure 3 The description is in conjunction with box 3025.

[0128] exist Figure 4 In the example of FIG, it is generally conceivable to also train the prediction algorithm during observation period 301 after the calibration phase 311 has concluded. The training can be completed, for example, at a point in time 651. Then, immediately following point in time 651, during the observation period, after the training, real-time post-processing of the measurement image 201 can be performed, and the synthesized fluorescence image 202 can be output to the user via a corresponding screen. This means that, in general, using the subsequently trained prediction algorithm 205, the synthesized fluorescence image can be determined synchronously or in a time-coupled manner with the acquisition of the measurement image 201.

[0129] exist Figure 4 As can be seen in the example of FIG. 3 , the temporal density of the reference images 261 , 262 , when measured over the observation period 301 , is smaller than the temporal density of the measurement image 201 .

[0130] Figure 5 Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 and validation-reference images 261 are shown. Figure 5 In particular, some aspects regarding the respective timing in the observation period 301 are shown.

[0131] Figure 5 The example basically corresponds to Figure 4 But in Figure 5 In the example of FIG. 3 , the calibration phase 311 is not arranged at the beginning of the observation period 301 (see FIG. Figure 4 ), but is arranged at the end. In this way, after the corresponding measurement is completed, when the prediction algorithm is trained, the measurement image 201 is digitally post-processed based on this prediction algorithm 205.

[0132] It can also be based on Figure 4The example of FIG. 2 performs validation of the prediction algorithm 205 based on the validation-reference image 261 , which is acquired at a very low sampling rate during the entire observation period 301 .

[0133] The calibration phase 311 is provided at the end of the observation period 301 and causes the light exposure during most of the observation period 301 to be equal to, for example, Figure 4 Compared with the Figure 4 The light exposure for acquiring the training-reference image 251 occurs at the beginning of the observation period 301 .

[0134] Figure 6 Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 and validation-reference images 261 are shown. Figure 6 In particular, some aspects regarding the corresponding timing in the observation period 301 are shown.

[0135] exist Figure 6 In the example of , a calibration phase 311 is again used, during which a training reference image 601 is acquired. Figure 6 In the example of FIG. 3 , the calibration phase 311 is set with reference to an expected time point during the biological change process.

[0136] For example, as already explained above, instead of a statistically uniform distribution of the occurrence of biological processes during the entire observation period 301, a corresponding local distribution 661 can be present. This distribution 661 of the frequency of occurrence of biological processes defines the expected time point. Therefore, in general, the expected time point can be determined based on prior knowledge. In this example, the process conditions can be modified by adding chemical substances according to the measurement protocol to define the expected time point.

[0137] Since the calibration phase 311 and the acquisition of the training reference image 251 are arranged with reference to an expected point in time, the prediction algorithm 205 can be trained with particularly informative information.

[0138] exist Figure 4 、 Figure 5 and Figure 6 In the example of FIG. 301 , a calibration phase 311 is described with reference to a training-reference image 251. As an alternative or in addition to this calibration phase 311, a calibration phase can also be used for acquiring training-reference images 251, which calibration phase results in gating in the time period for acquiring training-reference images 251. For example, it is conceivable to acquire training-reference images 251 in a calibration phase 311 provided at the beginning of observation period 301 (see FIG. 302 ). Figure 4 ), a verification-reference image 251 is acquired in a corresponding calibration phase arranged at the end of the observation period 301.

[0139] Combine Figure 4 、 Figure 5 and Figure 6 The example of illustrates the concentration of the acquired training-reference images 251 in the calibration phase 311. Generally speaking, this is optional. Figure 7 Another example is shown in .

[0140] Figure 7 Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 and the acquisition of validation-reference images 261 are shown. Figure 7 In particular, some aspects regarding the corresponding timing in the observation period 301 are shown.

[0141] exist Figure 7 In the example shown, the training reference images 251 and the measurement images 201 are acquired in a time-staggered manner, ie, approximately evenly distributed over the entire observation period 301 .

[0142] For example, a fixed repetition rate can be used (eg, a training reference image 251 for every Nth measured image 201 , where, for example, N≧10 applies). The same statements also apply in principle to the validation reference image 261 .

[0143] This distribution of the training reference images 251 over the entire measurement period 301 allows for targeted training of the prediction algorithm 205 for different phases of the respective experiment, which is particularly helpful when the respective biological processes do not occur uniformly distributed over the observation period 301 .

[0144] The above describes some techniques for acquiring reference images 251, 261 in a targeted manner within a certain time. As an alternative to or in addition to the arrangement for acquiring reference images 251, 261 within a certain time, the acquisition of training-reference images 251 and / or validation-reference images can also be triggered by one or more events. Corresponding examples are shown in Figure 8 , here for a validation reference image 261 (wherein a corresponding example can also be used for the training reference image 251 ).

[0145] Figure 8 Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 and validation-reference images 261 are shown.

[0146] exist Figure 8 In the example of , a verification reference image is acquired in response to a trigger event. However, a training reference image 251 may alternatively or additionally be acquired in response to a trigger event 681 .

[0147] In principle, various triggering events are conceivable. For example, a triggering event can be implemented by a user command. Alternatively or additionally, monitoring can be performed to determine whether a change in the semantic content of the measurement image has been detected. Alternatively or additionally, object recognition algorithms can also be used. This will be explained in detail below.

[0148] For example, an object recognition algorithm can be used to identify the occurrence of specific biological processes. As input, the object recognition algorithm can receive, for example, a measurement image 201 or a synthesized fluorescence image 202. If the object recognition algorithm then identifies a certain number of biological processes, a trigger event may occur, and corresponding verification reference images 261 can be acquired. Other semantic content can also be monitored, such as cell division, enlargement or reduction of cellular elements, etc. This ensures that verification reference images 261 are available even for relatively rare biological processes.

[0149] In addition to event-specific object recognition algorithms, simpler implementations can also be carried out which, for example, check whether the image contrast in the measurement image 201 changes suddenly and significantly.

[0150] It is also possible to receive corresponding control instructions relating to the triggering criteria via the user interface.

[0151] Figure 9 Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 and validation-reference images 261 are shown.

[0152] Figure 9 The example basically corresponds to Figure 4 But in Figure 9 In the example shown, at the end of observation period 301, another calibration phase 311 is performed. Additional training reference images 251 are acquired there. For example, another calibration phase 311 can be selectively performed based on the results of a verification based on verification reference images 261 acquired between two calibration phases 311. If, for example, the verification based on verification reference images 261 reveals poor quality in synthesized fluorescence image 202, prediction algorithm 205 can be newly (re)trained at the end of observation period 301 based on additional training reference images 251 acquired during a subsequent calibration phase 311 at the end of observation period 301. Generally speaking, the acquisition of training reference images 251 and the parameter values ​​of prediction algorithm 205 can be adjusted within the scope of (re)training based on the results of the verification. This allows for more than just simple verification; appropriate countermeasures can also be implemented. The synthesized fluorescence image can then be re-determined based on measurement images 201 after the retraining is complete. Additional training data can be acquired as needed.

[0153] These techniques are based on the understanding that the cell culture of a given sample can often change during observation period 301, for example, fading or discoloring. Such changes in the sample may often not be foreseeable from the outset. Despite this, a previously trained prediction algorithm 205 can still lose accuracy. Accuracy decreases further, in particular, with longer intervals from the corresponding calibration phase 311. Therefore, it may be helpful to optionally acquire additional training-reference images 251 at the end of observation period 301.

[0154] With the aid of an optional further calibration phase 311 , the light exposure of the sample can also be set to be as low as possible, but also as much as required.

[0155] Figure 10 Some aspects of acquiring measurement images 201 and acquiring training-reference images 251 and validation-reference images 261 are shown. Figure 10 In the example of , the calibration phase 311 is used to acquire the training-reference images 251 and the calibration phase 312 is also used to acquire the validation-reference images 261. Figure 10 In the example shown, both calibration phases 311, 312 are arranged at the end of the observation period 301. However, other solutions are also conceivable, where the calibration phase 311 is arranged, for example, at the beginning of the observation period 301 or vice versa. Figure 6 Different combinations of timings are contemplated.

[0156] exist Figure 10 Loopback 3099 is also shown in FIG. Figure 3 ). Alternatively, it is possible to acquire and re-validate further training-reference images 251 depending on the result of the validation until the validation is successful. The acquisition of training-reference images 251 can thus depend on the result of the validation.

[0157] Figure 10 The example of has a particularly low light exposure of the sample until the start of the calibration phases 311 , 312 , so that the sample quality is rarely affected by the measurement.

[0158] Previous references Figures 4 to 10 The scenes of FIG. 1 show some examples in which the time arrangement of acquiring reference images 251, 261 is changed. As an alternative to or in addition to the change in the time arrangement of acquiring reference images 251, 261, there is also a change bandwidth involving the spatial arrangement of the acquired reference images 251, 261. The corresponding technical combination Figure 11 Shown.

[0159] Figure 11Some aspects regarding the acquisition of measurement images 201 and the acquisition of training-reference images 251 and regarding the acquisition of validation-reference images 261 are shown. Figure 11 In particular, some aspects are also shown regarding the arrangement in position space (vertical axis) and the arrangement in time (horizontal axis). Figure 11 As shown in FIG, the measurement images 201 can be acquired layer by layer (sometimes also called Z-Stack). It is also possible to form pairs 601, 602 based on the smallest possible spacing in position space and time, such as Figure 11 As shown.

[0160] In addition to this 3D imaging based on a Z-stack of 2D measurement images, true 3D imaging using corresponding imaging methods is also conceivable. Examples are optical coherence tomography or holographic tomography. 3D image data are obtained in this manner. In combination with other specific imaging methods, such as fluorescence imaging, the measurement image 201 and reference images 251, 261 can be derived. For example, a cut plane in the 3D volume of 3D image data can be determined accordingly. This may require, for example, registration between the measurement image 201 and the reference images 251, 261 in order to verify 6002 their trained relationship.

[0161] according to Figure 11 The example can also be used in a generalized manner in other scenarios. For example, it is conceivable that multiple 2D measurement images are recorded laterally for different fields of view. It is then conceivable that reference images 251, 261 are acquired for only one of the multiple laterally offset fields of view. The same description also applies to multi-well plates (MWPs). There, for example, it is conceivable that reference images are acquired for only one sample area of ​​the MWP and measurement images are acquired for multiple sample areas of the MWP. Nevertheless, a composite fluorescence image can still be determined for all sample areas of the MWP.

[0162] In summary, the techniques described above enable the acquisition of measurement images with high temporal and / or spatial density. On the other hand, reference images are acquired with less temporal and / or spatial density. These reference images can be used to validate synthetic images digitally post-processed based on the measurement images and / or to train corresponding prediction algorithms. Because acquiring reference images is typically relatively invasive, meaning potentially damaging the specimen, these techniques enable gentler measurements. However, they can also improve or verify the reliability and quality of the prediction algorithms.

[0163] In summary, some techniques for determining synthetic fluorescence images by digital post-processing of measurement images have been described. Here, a sample or a portion of a sample is stained with a marker, wherein a virtual fluorescent surrogate of this marker is to be obtained. A series of measurement images of this sample are then acquired along one or more dimensions (time, Z position, and / or lateral offset). The measurement images are acquired using a less invasive imaging modality, that is, an imaging modality that causes less phototoxicity and / or photobleaching than fluorescence imaging, which provides the desired fluorescence contrast. However, reference images can also be acquired using this very fluorescence imaging.

[0164] Pairs of measurement and reference images can then be formed with the smallest possible separation in time and positional space. Registration can be performed if necessary, for example, if there is an offset in positional space. These pairs can then be used to train and / or validate the prediction algorithm that determines the synthesized fluorescence image. This means that the prediction algorithm uses the correlation between the fluorescence image and the measurement image to determine the synthesized fluorescence image. The prediction algorithm can then be used to determine the associated synthesized fluorescence image for all acquired measurement images.

[0165] Of course, the features of the previously described embodiments and aspects of the present invention can be combined with one another. In particular, the features can be used not only in the described combinations, but also in other combinations or alone without departing from the scope of the present invention.

[0166] For example, previously described techniques involve digital post-processing of 2D measurement images to obtain synthetic fluorescence images. For this purpose, prediction algorithms based on 2D measurement image operations are used. As a general rule, it is also conceivable to use prediction algorithms based on 3D measurement image operations.

[0167] In addition, some techniques are described in conjunction with digital contrast post-processing for simulating fluorescence contrast. These techniques can also be used to simulate other imaging modalities or application scenarios.

[0168] Furthermore, some techniques have been described above in which synthetic fluorescence images are determined that simulate the fluorescence contrast produced by the markers used to stain the sample. It is also conceivable, however, to simulate autofluorescence contrast without markers.

Claims

1. A computer-implemented method comprising: - actuating (3005) at least one imaging device (111, 112) in order to acquire a measurement image (201) of the sample during an observation period (301) by means of microscopic imaging, - controlling (3010) at least one imaging device (111, 112) in order to acquire a plurality of reference images (251) of at least a portion of the sample during an observation period (301) by means of microfluorescence imaging, wherein the reference images (251) are acquired with a smaller spatial density and / or a smaller temporal density than the measurement images (201) with respect to the observation period (301), - performing (3015) training (6001) of parameters of the prediction algorithm (205) based on at least a portion of the reference image (251, 261) as ground truth and also based on at least a portion of the measurement image (201), and After the training ( 6001 ) has been completed, a synthetic fluorescence image ( 202 ) is determined ( 3020 ) based on one or more of the measurement images ( 201 ) and using the prediction algorithm ( 205 ).

2. The computer-implemented method of claim 1 , wherein: The computer-implemented method further comprises: After the training is completed, the determination of the synthesized fluorescence image (202) is synchronized with the acquisition of the measurement image (201).

3. The computer-implemented method of claim 1 or 2, wherein: The training is performed based on a loss contribution to a loss function, the loss contribution being based on a comparison of semantic content of at least a portion of the reference image (251, 261) and semantic content of at least a portion of the measurement image (201).

4. A computer-implemented method comprising: - actuating (3005) at least one imaging device (111, 112) in order to acquire a measurement image (201) of the sample during an observation period (301) by means of microscopic imaging, - controlling (3010) at least one imaging device (111, 112) in order to acquire a plurality of reference images (261) of at least a portion of the sample during an observation period (301) by means of microfluorescence imaging, wherein the reference images (261) are acquired with a smaller spatial density and / or a smaller temporal density than the measurement images (201) with respect to the observation period (301), - determining (3020) a synthesized fluorescence image (202) based on the measured image (201) and based on the prediction algorithm (205), and - performing (3025) a verification (6002) of the synthesized fluorescence image (202) based on a comparison between at least a portion of the reference image (261) and at least a portion of the synthesized fluorescence image (202).

5. The computer-implemented method of claim 4, wherein: The computer-implemented method further comprises: - Based on the result of said validation (6002): adjusting one or more parameter values ​​of said prediction algorithm (205).

6. The computer-implemented method of claim 4 or 5, wherein: The computer-implemented method further comprises: - According to the result of the verification (6002): adjusting the acquisition of the reference images (251, 261).

7. A computer-implemented method according to any one of claims 4 to 6, wherein: The verification is based on a comparison of the semantic content of at least a portion of the reference image (261) with the semantic content of at least a portion of the measurement image (201).

8. A computer-implemented method according to any one of the preceding claims, wherein: The at least one imaging device (111, 112) is controlled to acquire the reference image (251, 261) during the observation period (301) in an alternating manner with the measurement image (201).

9. A computer-implemented method according to any one of the preceding claims, wherein: In response to a trigger event (681), the at least one imaging device (111, 112) is respectively controlled to acquire at least one of the reference images (251, 261).

10. The computer-implemented method of claim 9, wherein: The triggering event (681) is selected from the following group: a user command; a change in the semantic content of the measurement image; an output of an object recognition algorithm.

11. A computer-implemented method according to any one of the preceding claims, wherein: The at least one imaging device (111, 112) is controlled so as to acquire the reference images (251, 261) more frequently during at least one calibration phase (311, 312) than outside of at least one calibration phase (311, 312), wherein the calibration phase (311, 312) is set at the beginning and / or end of the observation period (301) or in relation to an expected time point of a biological change process of the sample.

12. A computer-implemented method according to any one of the preceding claims, in, controlling the at least one imaging device (111, 112) to acquire the reference image (251, 261) in a first region of the sample, The at least one imaging device (111, 112) is controlled so as to acquire the measurement image (201) at least partially in a second area of ​​the sample that is different from the first area of ​​the sample.

13. A computer-implemented method according to any one of the preceding claims, in, controlling the at least one imaging device (111, 112) to acquire the reference image (251, 261) by means of microfluorescence imaging using a first value of an exposure parameter for microfluorescence imaging, wherein the at least one imaging device (111, 112) is controlled to acquire the reference image (201) by means of microfluorescence imaging with a second value of an exposure parameter for microfluorescence imaging, where the first value contributes to the first illumination of the sample and each reference image, wherein the second value contributes to a second illumination of the sample and of each measurement image, Among them, the second illumination is smaller than the first illumination.

14. A computer-implemented method according to any one of the preceding claims, in, The number of the reference images (251, 261) in the observation period (301) is no more than 50%, optionally no more than 5%, and further optionally no more than 1% of the number of the measurement images (201) in the observation period (301).

15. An apparatus (101) comprising a processor (104) configured to perform the following steps: - actuating (3005) at least one imaging device (111, 112) in order to acquire a measurement image (201) of the sample during an observation period (301) by means of microscopic imaging, wherein: Acquiring reference images (251, 261) with a smaller spatial density and / or a smaller temporal density than the measurement images (201) with respect to an observation period (301), - driving (3010) at least one imaging device (111, 112) in order to acquire a plurality of reference images (251) of at least a portion of the sample during an observation period (301) by means of microfluorescence imaging, - performing (3015) training of parameters of the prediction algorithm (205) based on at least a portion of the reference image (251, 261) as ground truth and also based on at least a portion of the measurement image (201), and After the training is completed, a synthetic fluorescence image (202) is determined (3020) based on one or more of the measurement images (201) and using the prediction algorithm (205).

16. The device according to claim 15, wherein The processor is configured to perform the computer-implemented method of claim 1 .

17. An apparatus (101) comprising a processor (104) configured to carry out the following steps: - actuating (3005) at least one imaging device (111, 112) in order to acquire a measurement image (201) of the sample during an observation period (301) by means of microscopic imaging, - driving (3010) at least one imaging device (111, 112) to acquire a plurality of reference images (251) of at least a portion of the sample during an observation period (301) by means of microfluorescence imaging, wherein Acquiring reference images (251, 261) with a smaller spatial density and / or a smaller temporal density than the measurement images (201) with respect to an observation period (301), - determining (3020) a synthesized fluorescence image (202) based on the measured image (201) and based on the prediction algorithm (205), and - performing (3025) verification of the synthesized fluorescence image (202) based on a comparison between at least a portion of the reference image (261) and at least a portion of the synthesized fluorescence image (202).

18. The device according to claim 17, wherein The processor is configured to perform the computer-implemented method of claim 2 .

Images