Method and apparatus for producing an image of a sample in a predetermined target contrast type

Abstract

Computer-implemented method (10) for generating a second image (2) of a sample in a predetermined target contrast type, comprising: recognizing (15) at least one structure (3) contained in a first image (1) of the sample in a predetermined first input contrast type, which is to be visible in the second image (2) of the sample in the target contrast type, by means of a correspondingly designed segmentation algorithm;and generating (17) the second image (2) of the sample in the target contrast type, wherein generating (17) the second image (2) in the target contrast type comprises: determining (172) first image intensity values ​​of the first image (1) of the sample in the first input contrast type and / or second image intensity values ​​of a third image of the sample in a second input contrast type, in order to generate the second image (2) of the sample in the target contrast type based on the determined first and / or second image intensity values ​​and the at least one structure (3) detected in the first image (1), wherein the predetermined target contrast type corresponds to a microscopy contrast type.;

Description

[0001] A computer-implemented method for generating a second image of a sample in a predetermined target contrast is provided. Additionally, or alternatively, a training method for training a segmentation algorithm that can be used in the method is provided. Additionally, or alternatively, a data processing device is provided, configured to execute the method and / or the training method for the segmentation algorithm. Additionally, or alternatively, a computer program and a computer-readable medium are provided.

[0002] Fluorescence images are used in various applications, such as fluorescence microscopy.

[0003] Fluorescence microscopy is a special form of light microscopy. When fluorescent substances are excited with light of specific wavelengths, they emit light of other, longer wavelengths (known as the Stokes shift). Using this physical effect of fluorescence, fluorescence microscopy produces images of a sample, which are referred to here as fluorescence images.

[0004] In fluorescence microscopy, the magnified image of the object under investigation is generated solely by emitted light. Color filters prevent excitation light from reaching the image. Fluorescence microscopy images are informative when not the entire microscopic specimen fluoresces uniformly, but rather when only certain structures glow. These structures produce bright signals against a dark background.

[0005] Fluorescence-based color channels, which depict DNA dyes (such as 4',6-diamidin-2-phenylindole, or DAPI for short, or Hoechst 33342), often serve as a guide for users during microscopy, as this allows for the rapid acquisition of positional and state information of cells in the image area.

[0006] Even though other contrast types can provide similar information regarding the cells, users are often accustomed to fluorescence imaging (e.g., DAPI-like images) and prefer it to other contrast types.

[0007] However, conventional fluorescence imaging has several disadvantages. For example, sample staining is required, which in turn incurs additional costs and effort. Furthermore, fluorescence imaging occupies a fluorescence channel (usually 461 nm emission in DAPI) on the imaging system, which cannot be used for other imaging. Additionally, the sample can be damaged by fluorescence imaging (e.g., through phototoxicity or bleaching), potentially negatively impacting or extending the duration of the entire experiment. Moreover, inaccuracies can occur, for example, during DAPI staining if compounds do not bind perfectly to the sample (so-called "missed transfection"). This can lead to faulty DAPI images, as parts of the sample appear inactive.

[0008] In Ounkomol et al. (Ounkomol, C., Seshamani, S., Maleckar, MM, Collman, F., & Johnson, GR (2018). Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nature methods, 15(11), 917-920), a technique called virtual staining is described. Virtual staining refers to the determination of a "virtual" image of a fluorescence stain; that is, for a given image of a sample, the corresponding fluorescence image is determined / calculated algorithmically. This calculation is performed using data-driven artificial intelligence methods, such as trained deep neural networks, which are trained with images of the fluorescence stain as target values ​​as well as inputs from other imaging modalities. This calculated fluorescence image usually has the same intensity distribution as a real fluorescence image, i.e., with the same intensity gradations (e.g.,(int16 remains int16 and does not switch to binary segmentation). However, the presence of these fluorescence gradations is often not sufficiently robust or reliable. For example, realistic images can be calculated, but these do not correspond to real biology or chemistry. Furthermore, the models used sometimes lack sufficient generalization or extrapolation properties, especially for samples that have not previously been used in model training. Additionally, virtual staining sometimes fails to adequately preserve the structural information of subcellular details (e.g., nucleoli).

[0009] DE 10 2021 114 287 A1 relates to a computer-implemented method for generating an image processing model that produces output data from a microscopy image, defining a stylized contrast image. For this purpose, model parameters of the image processing model are adjusted by optimizing at least one objective function based on training data. The training data comprises microscopy images as input data and contrast images, where the microscopy images and the contrast images are generated using different microscopy techniques. To ensure that the output data defines a stylized contrast image, a reduction in detail is enforced by the objective function, or the contrast images are reduced-detail contrast images with a level of detail lower than that found in the microscopy images and higher than that found in binary images.

[0010] WO 2021 / 198243 A1 relates to a method for the virtual staining of a tissue sample, comprising selecting a virtual stain, obtaining digital imaging data of the tissue sample, wherein the digital imaging data of the tissue sample were acquired using one or more imaging modalities, determining a region of interest (ROI) of the tissue sample, and providing an output image that shows the tissue sample with the virtual stain only in the ROI.

[0011] In light of this prior art, one objective of the present disclosure is to specify a method and a device, each of which is suitable to enrich the prior art.

[0012] One possible specific purpose of the present disclosure can be seen as avoiding hallucinations when generating a (second) image of a sample in a predetermined target contrast type based on a (first) image of the sample in a predetermined (first) input contrast type. In other words, the image in the target contrast type should only show those structures in the target contrast that are actually present in the sample according to the image in the input contrast type.

[0013] The problem is solved by the features of the independent claims. The dependent and subordinate claims each contain optional further developments of the disclosure.

[0014] The task is then solved by a computer-implemented method for generating a second image of a sample in a predetermined target contrast. The method involves identifying at least one structure present in a first image of the sample in a predetermined first input contrast, which should be visible in the second image of the sample in the target contrast, using a suitably designed segmentation algorithm. The method then includes generating the second image of the sample in the target contrast.Generating the second image in the target contrast type involves determining the first image intensity values ​​of the first image of the sample in the first input contrast type and / or the second image intensity values ​​of a third image of the sample in a second input contrast type. Based on these determined first and / or second image intensity values ​​and the at least one structure detected in the first image, the second image of the sample in the target contrast type is then generated. The predetermined target contrast type corresponds to a microscopy contrast type.

[0015] In other words, as a first step, a (microscopic) recording of the real world can be obtained in a first (input) contrast type.

[0016] In a second step, a segmentation of (relevant) structures included in the obtained image, which would be visible in a second contrast type / the target contrast type (optionally in a fluorescence image, and optionally in a correspondingly stained sample), can be carried out using a (trained) segmentation method.

[0017] In a third step, an image (optionally fluorescence-like) can be generated based on the output of the segmentation process.

[0018] A texture of at least one structure detected by the segmentation algorithm can be transferred to the second image. Furthermore, image intensity values, for example pixel by pixel, can be transferred from the first image to the second image using a predetermined transfer rule in the first input contrast type.

[0019] In microscopy, a contrast type can be understood as a combination of a chosen imaging modality (e.g., fluorescence imaging) and a sample treatment (e.g., staining or genetic modification). This allows the treated sample to be imaged in which (desired) parts of the sample are visually (easily) identifiable, i.e., they contrast with the rest of the image. For example, the same imaging modality (e.g., fluorescence imaging / fluorescence imaging) can result in different contrast types depending on the sample treatment. Each contrast type (e.g., in multichannel fluorescence imaging) can be acquired using a monochrome camera, i.e., imaging to grayscale levels, and optionally with a large bit depth. For visual viewing, the image can be displayed on an 8-bit RGB monitor.For this purpose, the monochromatic image can be displayed in a single color gamut – e.g., (as is the case with DAPI) in the blue channel of an RGB image (although the other two color channels are also conceivable). The grayscale channel can then be converted to 8-bit display by scaling and clipping.

[0020] Insofar as the text refers to the target contrast type corresponding to a (predetermined) microscopy contrast type, this can be understood to mean that the target contrast type is modeled on the microscopy contrast type or comes as close as possible to it, without necessarily being identical. For example, the target contrast type can exhibit the essential properties of the microscopy contrast type, such as sample selectivity. The target contrast type can also have more in common with the microscopy contrast type than the input contrast type.

[0021] According to one possible embodiment, described here as purely optional, the method can generate a DAPI-like microscopy image from a phase-contrast image, which can be used, for example, as a guide during microscopy. For this purpose, a phase-contrast image of a cell-containing sample can be obtained. The cell nuclei within the phase-contrast image can then be segmented. The DAPI-like image, or DAPI-like microscopy image, can then be generated by transferring a texture of the segmented nuclear regions from the phase-contrast image into the image to be generated, and by transforming (inverting, smoothing) and coloring the image intensity values ​​of the phase-contrast image in a DAPI-like manner.

[0022] However, determining and / or transferring the image intensity values ​​does not necessarily require using only the same image of the sample that serves as input for the segmentation algorithm. Instead, an additional or alternative image in a different input contrast can be used, from which the image intensity values ​​are then determined. In other words, determining the image intensity values ​​used to stain the second image of the sample in the target contrast can also (i.e., additionally or alternatively) use the image intensity values ​​of a third image of the sample in a second input contrast. It is therefore conceivable that the image intensity values ​​used to stain the sample could be derived from a different and / or the same image from which the structure is segmented.

[0023] This means that multiple source contrast images can be used. For example, a phase contrast can be used as the first source contrast and a (highly) noisy DAPI channel as the second. The reconstruction, or the generated second image, can then be a DAPI-like image with high-resolution structural information and improved DAPI signal strength. The high-resolution or fine-resolution structure can be derived from the phase contrast image (i.e., the first source contrast), and the fluorescence intensity from the noisy DAPI channel (i.e., the second source contrast). In other words, the structures can be taken from the phase contrast image, for example. The fluorescence intensity from the DAPI channel or the fluorescence image can represent or define the intensity level (optionally low-pass filtered) of the structures in the generated image. At least the background noise, i.e.,In areas of the phase-contrast image where no structure was detected, suppression is possible. Specifically, in a concrete implementation (described here as non-restrictive), a transfer rule for generating the image from the two contrast images can be implemented as follows: l_new(x,y)=l_seg(x,y)*l_PC(x,y)*Lowpass(l_Fluo)(x,y) / Max(l_Fluo) with: I_new(x,y) / / Image / recording to be generated at pixel coordinates x, y I_PC(x,y) / / Image in first source contrast, e.g. phase contrast image / phase contrast recording at pixel coordinates x, y I_Fluo(x,y) / / Image in second source contrast, e.g., fluorescence image / fluorescence recording at pixel coordinates x, y I_seg(x,y) / / Binary mask determined by segmentation of I_PC Lowpass filter (I_Fluo)(x,y) / / Result of the low-pass filtering of the image in the second source contrast at the pixel coordinates x,y Max(I_Fluo) / / Maximum image intensity of the image in the second source contrast

[0024] This method thus enables noise reduction of fluorescence images. As explained in detail later, segmentation can also optionally be performed in both source contrast images.

[0025] It is conceivable that, based on the specific image intensity values, a color scheme for the second image of the sample is determined in the target contrast type.

[0026] This means that spatial information (texture, structure) from the first image can be preserved, but the underlying image intensities can be modified. This can be achieved, for example, through grayscale transformations (e.g.,

[0027] These include inversion, linear transformation, gamma correction, histogram adjustment, ...) and / or filter operations (e.g. smoothing, background addition, ...).

[0028] In other words, the texture of the segmented object regions from the (optionally microscopy) image can be transferred to a new image, and the image intensity values ​​can be processed to create a virtually stained (optionally fluorescence) image (e.g., using DAPI staining or Hoechst 33342). This means that the texture of the segmented structure, e.g., cell nuclei, from the original image, such as one or more phase-contrast images, can be transferred to the new image. The image intensity values ​​of the original image can also be transformed (optionally using DAPI-like or Hoechst 33342-like techniques), optionally inverted, and / or smoothed, and the virtual image can be stained accordingly.

[0029] This offers the advantage that hallucinations are avoided when generating the second image of the sample in the predetermined target contrast type, based on the first image of the sample in the predetermined first input contrast type. In other words, the image in the target contrast type only shows those structures in the target contrast that are actually present in the sample according to the image in the input contrast type.

[0030] A texture can be understood as a spatial arrangement and distribution of the (segmented) structure. "Spatial" can refer to both two-dimensional space, i.e., a two-dimensional representation of the sample's surface, and three-dimensional space, i.e., a three-dimensional representation of the sample.

[0031] The image intensity value can also be referred to as the grayscale value. The grayscale value can indicate the brightness of a single image point or pixel. The image intensity value can therefore indicate how bright or dark the object or pixel in question appears to the human eye in the initial image at the initial input contrast. In other words, the grayscale value can represent the brightness or intensity value of a single image point.

[0032] The image intensity value determined for each image point or pixel can then be used to determine a color of an image point in the second image to be generated in the target contrast type, which corresponds to the image point in the first image of the sample in the first input contrast type.

[0033] For staining the second image in the target contrast type, either DAPI staining or Hoechst 33342 staining can be used. 4',6-Diamidine-2-phenylindole, or DAPI for short, is a fluorescent dye used in fluorescence microscopy to label deoxyribonucleic acid (DNA). The fluorescent dye Hoechst 33342 (bisbenzimide) is also used in fluorescence microscopy to stain DNA. In this case, a color can be assigned to each image intensity value, corresponding to the color that would have appeared in a fluorescence image of the sample using either the fluorescent dye DAPI or Hoechst 33342. Thus, an (artificial) DAPI stain or a Hoechst 33342 stain can be generated using the image intensity values.

[0034] A computer-implemented method can be understood as a method in which one, several or all steps of the method are at least partially carried out by a computer or a data processing device.

[0035] Insofar as the discussion concerns an algorithm, it can be implemented in software and / or hardware. The software can be provided as Software as a Service (SaaS). SaaS can be understood as a cloud-based software model that delivers applications to end users, for example, via a web browser.

[0036] A recording can be understood as a two-dimensional (2D) or three-dimensional (3D) image or representation of a sample, i.e., an object existing in the real world. The recording can also be a time-series recording in 2D (2D+t results in 3D data) or 3D (3D+t results in four-dimensional (4D) data).

[0037] The first image in the first input contrast type can be acquired with a sensor, optionally a microscope. The same applies to the third image in the second input contrast type, described later. An image in an input contrast type can be understood as either the direct result of an optical image being projected onto an (AD-converting) sensor, or an image that has been processed, calculated, or (post-)processed after being captured by the sensor (e.g., using appropriately configured software).

[0038] A structure that should be visible in an image of the sample in the second contrast type can be understood as a predetermined structure or a structure of a predetermined type, for example a cell nucleus.

[0039] A predetermined structure can be understood as a structure that can be recognized by, or is automatically recognized by, a segmentation algorithm specifically designed or configured for this purpose. In other words, the design of the segmentation algorithm determines which structure is recognized, or which objects forming the structure (and thus relevant) are identified. The relevant objects are typically defined by the application. In the life sciences, for example, these might be cell nuclei, cytoskeleton(s), cell organelle(s), and / or tissue. In materials science, they might be components of a printed circuit board. In metrology, rock samples, semiconductors, and / or rough surfaces might be examined.

[0040] Contrast can define or determine the difference between light and dark areas of an image or photograph. When referring to a specific type of contrast, this can refer to the contrast within a particular photograph, especially between predetermined structures and the rest of the image.

[0041] The method described above offers the advantage, among others, that no artifacts or hallucinations arise in the generated images, since their texture originates from real measured images, e.g., microscopy data. Specifically, the structure relevant for the target contrast type can first be extracted from the original image of the first input contrast type using a segmentation method. The structure thus obtained can then be used to generate the subsequent image or image in the target contrast type without requiring a direct transfer of the (image) intensity distribution from the original image, nor does this occur automatically via a machine learning model. This provides a robust method for generating virtual images of an additional or target contrast type, which also requires minimal computational effort. This also eliminates the need for artifacts or hallucinations.Hallucinations in the generated images are prevented. In other words, the process does not initially use a machine learning (ML) algorithm trained to convert the contrast of a given image into another contrast. Instead, a relevant structure is extracted or segmented, and the second image is then generated by coloring the extracted structure and a surrounding background according to the desired (target) contrast.

[0042] The method can include the detection of at least one structure contained in the third image of the sample, which should be visible in the second image of the sample at the target contrast. This can be achieved using a further and / or appropriately designed segmentation algorithm. In this way, the second image of the sample at the target contrast can also be generated based on the structure contained in the third image. This can, for example, allow for a plausibility check of the at least one structure detected or segmented in the first image and / or its supplementation.

[0043] The third image can be in a second input contrast type. This second input contrast type can be the same as the first, or a different one. The second input contrast type can essentially correspond to the target contrast type.

[0044] It is conceivable that, in addition to or alternatively to the second or further structural information from the third recording, second image intensity values ​​can be obtained from the third recording.

[0045] Generating the second image can involve transferring the specified first and / or second image intensity values ​​to the second image using a predetermined transfer rule. Optionally, the transfer of the specified first (and / or second) image intensity values ​​using the predetermined transfer rule can occur only in those areas of the second image where the detected structure, according to the segmentation of the first and / or third image, is located.

[0046] The second image can be stained according to a predetermined (or usual) staining of the corresponding microscopy contrast.

[0047] The coloring can also be referred to as coloring or color representation.

[0048] In other words, the grayscale values ​​contained in the grayscale image can be transferred, for example, to a color channel of a multi-channel fluorescence image, and then a predetermined color can be assigned to the channel (which is usually used for the corresponding microscopy contrast). It is conceivable that this color corresponds to the observable or perceptible color of the sample under the microscope (e.g., blue as fluorescence emission for DAPI staining).

[0049] It is conceivable that the other or remaining areas of the second image, in which the structure is not located, have a brightness according to a predetermined value.

[0050] The value can be a grayscale value. In the case of a single-channel recording, this can be derived or obtained directly from the recording. In the case of a multi-channel recording, it can be determined, among other methods, as described below. Specifically, in the example case of an RGB color value, the following formula can be used: Gray value=0.299×red component+0.587×green component+0.114×blue component The grayscale value is calculated. The result is a value that represents the brightness of the pixel, independent of the colors. This pixel brightness can then be transferred to the second image using a predetermined transfer rule; that is, another grayscale value can be determined, and this additional grayscale value can be used for the same pixel in the second image. It is not necessary to use all RGB channels to determine the grayscale value; it is also conceivable that only the red, green, and / or blue components are used. It is also conceivable that the transfer rule states that the grayscale value determined based on the red, green, and / or blue components is transferred directly or unchanged to the second image.

[0051] It is conceivable that the color of each pixel in the second image is also determined by, or depends on, the gray value. For example, it is possible that each gray value is assigned a color by a transfer rule. This means that the gray value or the image intensity values ​​from the source image(s) can determine not only the contrast of the second image to be generated, but also the color of the individual pixels within it.

[0052] The first and / or third image obtained of the sample can be a microscopy image, optionally two- or three-dimensional, optionally a phase contrast image, a phase gradient image, a differential interference contrast image, a bright field image, an oblique illumination image, a dark field image, an image obtained by quantitative phase imaging, an image obtained by optical coherence microscopy, an image obtained by holography, an image obtained by angular illumination microscopy, an image obtained by transport of intensity equation phase imaging and / or an image obtained by differential phase contrast.

[0053] The target contrast type of the second image can correspond at least partially to a predetermined fluorescence contrast type, optionally at least partially to the DAPI staining or the Hoechst 33342 staining.

[0054] The coloring of the second image can correspond at least partially to the coloring of a fluorescence image, optionally at least partially to the DAPI staining or the Hoechst 33342 staining.

[0055] This means that the colors of the individual pixels in the target contrast can be at least partially similar to the colors that would have been obtained from a fluorescence image of the sample, optionally using DAPI or Höchst 33342 staining. It is conceivable that the target contrast is more similar to the predetermined fluorescence contrast than to the first and / or second input contrast type. It is also conceivable that the target contrast is more similar to the second input contrast than to the first input contrast.

[0056] It is conceivable that the first and / or third image is a fluorescence image, optionally according to DAPI staining or Hoechst 33342 staining.

[0057] The procedure may include preprocessing of the first and / or third image, performed before the detection of at least one structure contained in the first and / or third image. This preprocessing may include shading correction, denoising, deconvolution, brightness correction, normalization of geometric properties, registration of a lateral offset, and / or conversion of the phase-contrast image into a virtual dark-field image.

[0058] Shading correction can be understood as the process of using (software-based) image processing to calculate or remove effects caused by uneven lighting during the acquisition of the first and / or third image. In other words, shading can be understood as a superimposed background signal independent of the sample, which represents a large-scale artifact in the image and which shading correction aims to eliminate.

[0059] Noise can have both sample-dependent and sample-independent components. This noise can be undesirable because it represents a stochastic component of the acquired image or recording that overlays the structure to be segmented. This can, for example, make accurate segmentation difficult at object boundaries. During denoising, this noise can be calculated or removed using (software-based) image processing.

[0060] Due to hardware limitations, individual light points during image capture cannot be mapped onto individual pixels. Instead, they can be "spread out" by the imaging optics – specifically, folded using the system's point-spread function (PSF). Provided the optics' characteristics are known, this process can be computed out (unfolded). This improves the image resolution.

[0061] Over- or underexposure can make it difficult to visually distinguish relevant structures from the background or other structures. Adjusting the brightness (e.g., gamma correction) as part of brightness correction can improve the differences between the captured brightness ranges.

[0062] Microscopy images are often acquired in partial images and combined into a larger image. This can lead to misalignment at the boundaries of the partial images. This misalignment can be corrected by registering a lateral shift.

[0063] Normalizing geometric properties, for example to an average size of the objects, can be advantageous, especially if a magnification factor is known.

[0064] The phase contrast image can be converted into a virtual dark field image, whereby a bright-appearing object is computationally generated on a dark background.

[0065] The procedure can include recording the first and / or the third image. The recording of the first and / or the third image can each be performed at a single point in time and / or as a time series recording.

[0066] In other words, the acquisition, e.g., a microscopy image, can be a 2D or 3D image (e.g., a Z-stack). The acquisition can be captured at a single point in time, i.e., it can be a single image. However, the acquisition can also comprise multiple acquisitions or images. It is conceivable that the multiple acquisitions could be taken at different times (optionally sequentially) as a so-called time-series acquisition, i.e., several images taken over a period of time and optionally combined into a single time-series image. The method is not limited to microscopy images but can be applied to all imaging modalities. For example, the method is also applicable to microsurgical scenarios.

[0067] The phase contrast image can be acquired in such a way that artificial contrast inversions exist around at least one structure contained in the phase contrast image, which is visible in the recording of the sample in the second contrast type.

[0068] This means that image acquisition can be manipulated to facilitate the segmentation of relevant objects or structures. For example, so-called "halos" can be generated around the structure, such as cell nuclei, in phase contrast. These halos represent an artificial contrast inversion, thus facilitating segmentation.

[0069] Generating the virtual image of the sample in the second contrast type may involve generating a background component and / or adding noise to the virtual image of the sample in the second contrast type.

[0070] This means that, in order to obtain a more realistic image impression, a background component can be created and / or noise can be applied to the image and / or the background.

[0071] The procedure may include performing a plausibility check, in which the plausibility of the at least one structure detected by the segmentation algorithm and / or the generated second image of the sample in the target contrast type is checked based on predetermined criteria, optionally using a machine learning model.

[0072] In other words, a plausibility check of the generated output can be performed. This can involve automatically checking the plausibility of the resulting image, the generated second image, or even the intermediate step of segmenting the structure or areas. For example, in the case of nuclear staining, it can be checked whether the shape and internal structure of the cell nuclei are plausible. This can be done, for instance, using a watchdog model (discriminator) trained for this purpose. Additionally, or alternatively, classic image processing can be used to evaluate and validate the shape, ellipticity, size of the segmentation masks, etc. In this context, "plausible" can mean determining whether predetermined values ​​fall within a tolerance range.

[0073] Furthermore, a computer-implemented training procedure for training a segmentation algorithm is provided. The training procedure includes providing a training dataset. This dataset comprises several training examples, each consisting of an initial image of a sample in a first input contrast type and positional information regarding at least one structure contained in the first image, which should be visible in a second image of the sample in a target contrast type. The first input contrast type can differ from the target contrast type.The training procedure involves training the segmentation algorithm with the training dataset, so that after training the segmentation algorithm is designed to recognize a further structure contained in a further image of a further sample in the first input contrast type, which should be visible in a further image of the further sample in the target contrast type.

[0074] Segmentation can be understood as the creation of conceptually related regions by grouping neighboring pixels or voxels according to a predetermined criterion. Segmentation can be implemented as semantic segmentation using machine learning approaches (optionally deep learning), for example, using a convolutional neural network (CNN) and / or a transformer-based model. Additionally, or alternatively, instance segmentation, object detection, and / or point localization with a predefined region of interest (ROI) can be used.

[0075] The training procedure can, for each training example, include determining the positional information regarding at least one structure contained in the first image of the sample. This can be supplemented by further positional information regarding at least one structure obtained from a third image of the sample in a second input contrast type. The training procedure can also, for each training example, include annotating the first images of the sample in the first input contrast type based on the determined positional information regarding the at least one structure contained in the first image.

[0076] Therefore, semi-automated or fully automated generation of annotations based on a further image acquisition is possible. In a more concrete example, this could mean that the annotations of the phase-contrast image are generated automatically or using an algorithm derived from fluorescence images. For example, nuclear masks can be generated from measured DAPI images (e.g., using classical image processing techniques such as smoothing, thresholding, and subsequent detection of the object or structure contour). This is also referred to as chemical annotation. This eliminates (at least partially) the need for manual generation of annotations, particularly the manual annotation of object areas, in the images used to train the segmentation algorithm.

[0077] To annotate the first images of the sample in the first input contrast type, information concerning a spatial relationship between areas of the first image of the sample in the first input contrast type and the third image of the sample in the second input contrast type can be used.

[0078] It can be advantageous if the fluorescence images and the input data have a known spatial relationship to each other, particularly pixel-specific relationships. This can be achieved, for example, by alternately acquiring the images through the same optical path. Alternatively, the images can first be registered to determine their spatial relationship. This then facilitates the automated generation of annotations described above.

[0079] The above description with reference to the computer-implemented procedure for generating the virtual recording of the sample in the second contrast type also applies analogously to the training procedure and vice versa.

[0080] Furthermore, a method for generating a training dataset for a segmentation algorithm can be provided. The training dataset comprises several training examples. For each training example, the generation method involves determining positional information concerning at least one structure contained in a first image of a sample, using further positional information concerning the at least one structure obtained from a third image of the sample in a second input contrast type. The generation method also includes annotating the first images of the sample in the first input contrast type based on the determined positional information concerning the at least one structure contained in the first image of the sample.

[0081] The above descriptions regarding the processes also apply analogously to the manufacturing process and vice versa.

[0082] Furthermore, a data processing device is provided. The device is designed to at least partially execute the computer-implemented method and / or the computer-implemented training method and / or the manufacturing method described above.

[0083] Powerful computing hardware is advantageous for the learning phase and also for the later application phase, for example through the use of graphics cards (GPUs), Tensor Processing Units (TPUs) and / or similar accelerators.

[0084] The data processing device can be a computer that is at least partially integrated into a recording device (e.g., a microscope) that captures the image(s). It is conceivable that the data processing device is configured to control the recording device so that it captures the image(s). It is also conceivable that the data processing device is configured to control the recording device based on a result of the process. The data processing device can, additionally or alternatively, be located remotely from the recording device and be connected to it via a wired and / or wireless connection, e.g., via the internet. Additionally or alternatively, the data processing device can be at least partially integrated into a cluster, e.g., a local area network.Additionally, or alternatively, the data processing device can be at least partially part of a cloud computing instance.

[0085] The above descriptions regarding the procedures also apply analogously to the data processing device and vice versa.

[0086] Furthermore, a computer program is provided. The computer program includes instructions which, when executed by the computer, cause it to at least partially execute the computer-implemented procedure and / or the computer-implemented training procedure and / or the manufacturing procedure described above.

[0087] The computer program or software can include the algorithm or instructions in the form of program code which, when executed on a computing unit or device, performs the above procedure.

[0088] The program code can be in any type of code, in particular code suitable for processing, and optionally control, in a microscope.

[0089] The above descriptions relating to the methods and the device for data processing also apply analogously to the computer program and vice versa.

[0090] Furthermore, a computer-readable medium is provided. The computer-readable medium contains instructions which, when executed by a computer, cause it to at least partially execute the computer-implemented procedure and / or the computer-implemented training procedure and / or the manufacturing procedure described above.

[0091] The computer-readable storage medium, which may include a computer program as defined above, can be any digital data storage device, such as a USB flash drive, a hard drive, a CD-ROM, an SD card and / or an SSD card.

[0092] The computer program can also be obtained additionally or alternatively from other sources, e.g., via the internet. The computer-readable medium can therefore be a data signal containing instructions which, when executed by a computer, cause it to at least partially execute the computer-implemented procedure and / or the computer-implemented training procedure described above.

[0093] The above descriptions relating to the methods, the data processing device and the computer program also apply analogously to the computer-readable medium and vice versa.

[0094] An optional embodiment of the disclosure is described below with reference to Fig. 1, Fig. 2 to Fig. 3 described. Fig. Figure 1 schematically and exemplarily shows a flowchart of a computer-implemented procedure for generating a second image of a sample in a target contrast type based on an image of the sample in a first input contrast type. Fig. Figure 2 shows schematically and exemplarily the acquisition of the sample in the first and second input contrast modes and the second acquisition of the sample in the target contrast mode. Fig. Figure 3 shows schematically and exemplarily a segmentation result that is generated from the first or third recording using the segmentation algorithm, and Fig.Figure 4 schematically and exemplarily shows a flowchart of a computer-implemented training procedure for training a segmentation algorithm, which is used in the procedure whose flowchart is shown in Fig. As shown in 1, the application is found.

[0095] Identical reference symbols refer to identical or similar objects.

[0096] The computer-implemented method 10 for generating a second image 2 of a sample in a target contrast type is described in detail below with reference to Fig. 1 and Fig. 2 described.

[0097] In a first step 11 of the procedure 10, a first image 1 of the sample is acquired using a first input contrast. Additionally, a third image 4 of the sample is acquired using a second input contrast, where the first input contrast differs from the second input contrast. Both images 1 and 4 can be acquired using a microscope (not shown).

[0098] As can be seen from the left side of the Fig. 2 results in the first image 1 being a two-dimensional phase-contrast image 1. The third image 4 is a two-dimensional (real) fluorescence image. On the right side in Fig.Figure 2 shows the second image 2 of the sample in the target contrast type, which is generated using method 10. This predetermined target contrast type corresponds to a microscopy contrast type. More precisely, the second image 2 is a two-dimensional image 2 whose contrast is similar to that of a fluorescence image. The second input contrast is therefore more similar to the target contrast than the first input contrast.

[0099] In all three images 1, 2, 4, the same cell nuclei 31, which in this case form the structure 3 relevant for procedure 10, can be seen in the same spatial distribution.

[0100] The first recording 1 and / or the third recording 4 is taken either at a single point in time or as a time series recording, i.e., both or one of the recordings 1, 4 can be obtained through several recordings taken at different times (optionally one after the other).

[0101] The first recording 1 is carried out in such a way that artificial or artificially generated contrast inversion 32 exists around the at least one structure 3 contained in the first recording 1, which should be visible in the second recording of the sample in the target contrast type.

[0102] In a second step 12 of the procedure 12, the first recording 1, which was recorded in the first step 11, is received by a (not shown) data processing device, which performs the steps of the procedure 10 described below. It is conceivable that this data processing device controls the first step 11.

[0103] In a third step 13 of the procedure 10, the third recording 4, which was also recorded in the first step 11, is also obtained at the device for data processing.

[0104] In a fourth step 14 of the method 10, the images 1, 4 obtained in the second and third steps 12, 13, i.e., the phase contrast image and / or the fluorescence image, are preprocessed using the appropriately configured data processing device. The preprocessing includes shading correction, denoising, deconvolution, brightness correction, normalization of geometric properties, and / or conversion of the phase contrast image, i.e., the first image 1, into a virtual dark-field image. The preprocessing is performed before the structure 3 contained in the phase contrast image 1 is detected.

[0105] In a fifth step 15 of the procedure 10, the structure 3 contained in the first recording 1 is recognized by means of a segmentation algorithm (trained for this purpose). The segmentation algorithm is trained in such a way that it recognizes a structure in the first recording 1 which should be visible in the second recording 2.

[0106] In a sixth step 16 of procedure 10, at least one structure contained in the third image 4 is detected using a segmentation algorithm. The structure 3 detected in the third image 4 is, in this case, the same structure 3 that is also detected in the first image 1. In other words, in the fifth step 15 of procedure 10, the structure 3 in the phase-contrast image 1 is detected, which is also visible in the third image 4 and is detected there in the sixth step 16. It is therefore conceivable that the structure 3, or the structural information, could also be obtained from the third image 4 using the same or another appropriately trained or configured segmentation algorithm. The structure 3 extracted from the third image 4 can be compared with the structure 3 extracted from the first image 1, e.g.,to supplement and / or check the plausibility of the structure 3 extracted from the first recording 1.

[0107] A segmentation result or binary mask5 resulting from the segmentation of the first and / or third recording 1, 4 is exemplified in Fig. Figure 3 shows the image. A binary segmentation has been carried out, i.e., the parts of the first and third images 1 and 4 that can be assigned to structure 3 have been assigned to the foreground, and all other parts of the first and third images 1 and 4 have been assigned to the background.

[0108] In a seventh step 17 of the procedure 10, the second recording 2 is processed based on the structure 3 identified in the first and / or third recording 1, 4 using the segmentation algorithm(s) (see also...). Fig. 3) generated.

[0109] In the first sub-step 171 of the seventh step 17, a texture of the structure 3 detected in the phase contrast image 1 using the segmentation algorithm is transferred to an initial second image. The initial second image corresponds to the one in Fig. The segmentation result shown in Figure 3 is 5 and can be considered the output of the segmentation algorithm.

[0110] In a second sub-step 172 of the seventh step 17, image intensity values ​​of the first and / or third image 1, 4 are determined, optionally exclusively in the areas corresponding to the structure 3 detected therein. Based on the determined image intensity values ​​and the segmented structure 3, the second image 2 is generated; that is, the texture or structure 3 contained in the initial second image or the segmentation result 5 is colored according to the determined image intensity values. The coloring of the second image is carried out according to a predetermined coloring of the corresponding microscopy contrast type, i.e., in this case, corresponding to the DAPI staining. This coloring can be performed using a transfer rule, which is selected depending on the target contrast and which assigns a color and / or brightness to a specific image intensity value.

[0111] In the third substep 173 of the seventh step 17, a background component can be added to the second image 2 obtained in the second substep 2; that is, a background component, such as an (artificial) shading gradient and / or structure(s) representing a sample holder, can be inserted into the second image 2. Additionally, or alternatively, in a fourth substep 174 of the seventh step 17, noise can be added to the second image 2.

[0112] In an eighth step 18 of procedure 10, a plausibility check is performed. This check can verify the plausibility of the structure 3 identified by the respective segmentation algorithm in the first recording 1 and / or third recording 4. Additionally, or alternatively, the plausibility of the generated second recording 2 can be checked. Additionally, or alternatively, the plausibility of the initial second recording 2 (see first sub-step 171 of the seventh step 17) can be checked. Predefined criteria, for example, regarding the form of the identified structure 3 and / or the structure generated in the (initial) second recording 2, can be used for the plausibility check. It is conceivable that the plausibility check uses a (specifically trained) machine learning model.

[0113] The following describes the computer-implemented training procedure 20, which can be used to train the segmentation algorithm used in the fifth step 15 of procedure 10 to recognize the structure 3 contained in the first recording 1. The description of the training procedure 20 with reference to the segmentation algorithm used in the fifth step 15 of procedure 10 applies mutatis mutandis to the segmentation algorithm used in the sixth step 16 of procedure 10, i.e., the segmentation algorithm used to recognize the structure 3 in the third recording 4. A flowchart of the training procedure 20 is shown as an example in Fig. 4 shown.

[0114] In the first step 21 of the training procedure 20, a training dataset is provided, comprising several training examples. Each training example includes a first image 1 in a first input contrast (e.g., a phase contrast image) and positional information regarding at least one structure (3) contained in the first image (1), which should be visible in a second image (2) of the sample in a target contrast. The first input contrast is different from the target contrast.

[0115] Providing the training dataset includes, in a first substep 211 of the first step 21, determining the positional information regarding the at least one structure 3 contained in the first image 1 of the sample. This positional information can specify which pixels and / or image points or image areas of the first image 1 correspond to the structure 3. Determining this positional information can be done using further positional information regarding the at least one structure 3, obtained from a third image 4 in a second input contrast (e.g., a fluorescence image). In other words, the third image 4 may already be annotated and / or, as described above with reference to the sixth step 16, the structure 3 in the third image 4 can be identified using a segmentation algorithm.The position of structure 3 contained in the third image 4 can then be determined and provided as further positional information. For example, if information concerning a spatial relationship between areas of the first image 1 and the corresponding third image 4 is known (e.g., which pixel from the first image 2 corresponds to which pixel from the third image 4), this spatial relationship can be used to convert the further positional information into the positional information to be determined.

[0116] In a second sub-step 212 of the first step 21, an (automated) annotation of the first recording 1 can then be carried out based on the determined location information.

[0117] The first and second sub-steps 211, 212 can also be described as a computer-implemented procedure for producing the training data set for the segmentation algorithm.

[0118] In a second step 22 of the training procedure 20, the segmentation algorithm is trained using the training dataset, specifically, in this case, the annotated fluorescence images. After the training is complete, the segmentation algorithm is configured to detect a structure 3 in a further image (i.e., not included in the training dataset) in the first input contrast type, which should be visible in a further second image in the target contrast type. Reference symbol list 1. First recording of the sample in the first input contrast type 2. Second image of the sample in the target contrast mode 3 Structure 31 cell nuclei 32 Contrast inversion 4. Third recording of the sample in the second input contrast mode 5. Segmentation result or segmented first / third recording 10 computer-implemented methods 11. Recording the sample in the first / second contrast mode 12. Obtaining the sample in the first contrast mode 13 Obtaining the sample image in the second contrast type 14. Preprocessing of sample acquisition in the first contrast type 15. Detection of at least one structure contained in the image of the sample in the first contrast type 16. Detection of at least one structure contained in the image of the sample in the second contrast type 17 Creating the virtual recording 171 Transferring a texture 172 Determining image intensity values 173 Creating a background component 174 Noise in the virtual recording of the sample 18. Performing a plausibility check 20 computer-implemented training methods 21. Providing a training dataset 211 Identifying a structure contained in the image of the sample in the second contrast type 212 Annotating the recordings of the sample in the first contrast type 22 Training the segmentation algorithm with the training dataset QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2021 114 287 A1

[0009] WO 2021 / 198243 A1

[0010] Cited non-patent literature

[0000] Ounkomol et al. (Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F., & Johnson, G. R. (2018). Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nature methods, 15(11), 917-920

[0008]

Claims

A computer-implemented method (10) for generating a second image (2) of a sample in a predetermined target contrast type, characterized in that the method (10) comprises: - detecting (15) at least one structure (3) contained in a first image (1) of the sample in a predetermined first input contrast type, which is to be visible in the second image (2) of the sample in the target contrast type, by means of a suitably designed segmentation algorithm, and - generating (17) the second image (2) of the sample in the target contrast type, wherein the generating (17) of the second image (2) in the target contrast type comprises: - determining (172) first image intensity values ​​of the first image (1) of the sample in the first input contrast type and / or second image intensity values ​​of a third image (4) of the sample in a second input contrast type,to generate the second image (2) of the sample in the target contrast type based on the determined first and / or second image intensity values ​​and the at least one structure (3) detected in the first image (1), wherein the predetermined target contrast type corresponds to a microscopy contrast type. Computer-implemented method (10) according to claim 1, characterized in that the method (10) comprises: - detecting (15) at least one structure (3) contained in the third image (4) of the sample, which is to be visible in the second image (2) of the sample in the target contrast type, by means of a further and / or the appropriately designed segmentation algorithm, in order to additionally generate the second image (2) of the sample in the target contrast type based on the structure (3) contained in the third image (4) of the sample. Computer-implemented method (10) according to claim 1 or 2, characterized in that the generation (17) of the second image (2) comprises: - transferring the determined first and / or second image intensity values ​​to the second image (2) using a predetermined transfer procedure, optionally exclusively in the areas of the second image (2) in which the detected at least one structure (3) is located according to the segmentation of the first and, as far as related to claim 2, the third image (1, 4). Computer-implemented method (10) according to one of claims 1 to 3, characterized in that the second image (2) is colored according to a predetermined coloring of the corresponding microcopy contrast. Computer-implemented method (10) according to one of claims 1 to 4, characterized in that: - the first and / or the third image (1, 4) of the sample is a microscopy image, optionally two- or three-dimensional, optionally a phase contrast image, a phase gradient image, a differential interference contrast image, a brightfield image, an oblique illumination image, a darkfield image, an image obtained by quantitative phase imaging, an image obtained by optical coherence microscopy, an image obtained by holography, an image obtained by angular illumination microscopy, an image obtained by transport of intensity equation phase imaging and / or an image obtained by differential phase contrast, - the target contrast type of the second image (2) corresponds at least partially to a predetermined fluorescence contrast type, optionally at least partially to the DAPI staining or the Hoechst 33342 staining,and / or- the first and / or the third image (1, 4) of the sample is a fluorescence image, optionally according to DAPI staining or Hoechst 33342 staining. A computer-implemented method (10) according to one of claims 1 to 5, characterized in that the method (10) comprises a preprocessing (14) of the first and / or third image (1, 4) which is carried out before the detection (15) of the at least one structure (3) contained in the first and / or, as far as referenced back to claim 2, the third image (1, 4), wherein the preprocessing (14) optionally comprises: - a shading correction, - a denoising, - a deconstruction, - a brightness correction, - a normalization of geometric properties, and / or - as far as referenced back to claim 5, the phase contrast image is converted into a virtual dark field image. Computer-implemented method (10) according to one of claims 1 to 6, characterized in that the method (10) comprises recording (11) the first and / or the third recording (1), wherein the recording (11) of the first and / or the third recording (1) is optionally carried out at a single time point in time and / or as a time series recording. Computer-implemented method (10) according to claim 7, insofar as it relates back to claim 5, characterized in that the acquisition (11) of the phase contrast image is carried out in such a way that at least one artificial contrast inversion exists around the at least one structure (3) contained in the phase contrast image, which is to be visible in the second acquisition (2) of the sample in the target contrast type. Computer-implemented method (10) according to one of claims 1 to 8, characterized in that the generation (17) of the second image (2) of the sample in the target contrast type comprises generating (173) a background component and / or adding noise (174) to the second image (2) of the sample in the target contrast type. Computer-implemented method (10) according to one of claims 1 to 9, characterized in that the method (10) comprises performing (18) a plausibility check in which the plausibility of: - the at least one structure (3) detected by means of the segmentation algorithm, and / or - the generated second image (2) of the sample in the target contrast type is checked, based on predetermined criteria, optionally using a machine learning model. A computer-implemented training method (20) for training (22) a segmentation algorithm, characterized in that the training method (20) comprises: - providing (21) a training dataset, wherein the training dataset comprises several training examples, each comprising a first image (1) of a sample in a first input contrast type and positional information relating to at least one structure (3) contained in the first image (1), which is to be visible in a second image (2) of the sample in a target contrast type, wherein the first input contrast type is optionally different from the target contrast type, and - training (22) the segmentation algorithm with the training dataset, such that the segmentation algorithm after training (22) is designed to detect a further structure contained in a further image of a further sample in the first input contrast type,which should be visible in a further image of the further sample in the target contrast type. Computer-implemented training method (20) according to claim 11, characterized in that the training method (20) comprises, for each training example: - Determining (211) the positional information relating to the at least one structure (3) contained in the first image (1) of the sample using further positional information relating to the at least one structure (3) obtained from a third image (4) of the sample in a second input contrast type, and - Annotating (212) the first images (1) of the sample in the first input contrast type based on the determined positional information relating to the at least one structure (3) contained in the first image (1). Computer-implemented training method (20) according to claim 12, characterized in that information concerning a spatial relationship between areas of the first image (1) of the sample in the first input contrast type and the third image (4) of the sample in the second input contrast type is used for annotating (212) the first images (1) of the sample in the first input contrast type. A computer-implemented method for producing a training dataset for a segmentation algorithm, wherein the training dataset comprises several training examples, characterized in that the training procedure (20) comprises for each training example: - Determining (211) positional information relating to at least one structure (3) contained in a first image (1) of a sample using further positional information relating to the at least one structure (3) obtained from a third image (4) of the sample in a second input contrast type, and - Annotating (212) the first images (1) of the sample in the first input contrast type based on the determined positional information relating to the at least one structure (3) contained in the first image (1) of the sample. Device for data processing, characterized in that the device is configured to execute the computer-implemented method (10) according to one of claims 1 to 10, the computer-implemented training method (20) according to one of claims 11 to 13 and / or the computer-implemented method for producing the training data set according to claim 14. Computer program, characterized in that the computer program includes instructions which, when the program is executed by the computer, cause the computer to execute the computer-implemented method (10) according to one of claims 1 to 10, the computer-implemented training method (20) according to one of claims 11 to 13 and / or the computer-implemented method for producing the training data set according to claim 14. Computer-readable medium, characterized in that the computer-readable medium comprises instructions which, when executed by a computer, cause the computer to execute the computer-implemented method (10) according to one of claims 1 to 10, the computer-implemented training method (20) according to one of claims 11 to 13 and / or the computer-implemented method for producing the training data set according to claim 14.

Images