Registration of microscope images with specific contrast

Abstract

Techniques for registering a first microscope image (81) with a second microscope image (82) are described. The first microscope image images a sample at a first specific contrast, and the second microscope image images the sample at the first specific contrast or at a second specific contrast. For this registration, a reference microscope image (85) is used, which images the sample with a non-specific contrast.

Description

Technical Field

[0001] The various examples in this disclosure relate to techniques for registering two microscope images with specific contrast. Background Technology

[0002] In biology, optical microscopy utilizing fluorescence contrast is crucial because it enables molecularly specific insights into cells and tissue sections via fluorescent labels. Colocalization and quantification of fluorescence concentrations are of particular interest, as they can provide information about the temporary progression of disease and its drug treatment. Corresponding information can also be obtained through other specific contrasts involving selective staining of specific molecules or cellular structures. Another example of a common specific contrast that does not utilize fluorescent labels is hematoxylin and eosin (H&E) staining.

[0003] Today, fluorescent dyes can be selected with virtually unlimited freedom in terms of their spectral properties. For example, primary antibody-based markers are available that can be specifically adapted to tissue types and organelles. Flexible secondary dyes can bind to the markers or to their surrounding environment, allowing users to assign the spectral characteristics of the measured fluorescence signal in a targeted manner.

[0004] In fluorescence microscopy, a selected fluorescent dye can be made to emit light using a characteristic excitation signature or a characteristic excitation wavelength. The corresponding microscopic image thus exhibits fluorescence contrast. A specifically selected color filter can then be used to separate the fluorescence, which typically emits at a longer wavelength, from the excitation light. By appropriately combining different color spectra in the excitation light and by appropriately selecting specific color filters in the microscope's detection arm, multiple fluorescence contrasts and / or bright-field contrasts can be recorded sequentially or in parallel or within a single timeframe. For example, multichannel recording can be obtained where multiple microscopic images are captured sequentially at different contrasts, but without changing optics or with only as few optics as possible between the captures of the various microscopic images in the multichannel recording. Typically, this results in multiple microscopic images of the sample with different fluorescence contrasts. Different fluorescence contrasts specifically label different structures of the sample. Exemplary samples examined are cell samples or tissue section samples (e.g., in histopathology).

[0005] It has been observed that microscope images with different specific contrasts (different microscope images can be stained with different fluorescent labels or different non-fluorescent labels (e.g., H & E)) often have different appearances. Figure 1 An example is shown in the figure. Figure 1Microscopic image 91 is shown, which images a cell sample with a first fluorescence contrast; and microscopic image 92 is shown, which images the same cell sample with a second fluorescence contrast different from the first fluorescence contrast. Both microscopic images 91 and 92 were obtained through corresponding patch scanning. In particular, it is conceivable that microscopic images 91 and 92 were captured through common patch scanning, wherein, for each patch, corresponding multi-channel records with two fluorescence contrasts from different channels are captured, followed by the next scanning step.

[0006] These two fluorescence contrasts specifically label different cell structure types, making structures that are particularly clear in microscope image 91 different from those that are particularly clear in microscope image 92. This becomes particularly clear in the overlay image 95 (corresponding to the overlay of microscope images 91 and 92).

[0007] However, the different appearances of the two microscope images 91 and 92 are not necessarily due solely to the use of different fluorescence contrasts or, more generally, different specific contrasts. This is because the image planes of the two microscope images 91 and 92 may be shifted relative to each other. This means that there is a systematic shift in the lateral direction (xy direction) or the axial direction (z direction). The different appearances can be caused not only by lateral shifts but also by different distortions and / or rotations of the microscope images relative to each other.

[0008] Various reasons can cause this shift and / or distortion and / or rotation of the image plane. Different optical channels are generally used to capture different microscope images with different specific contrasts. Different optical channels have different optical beam paths. For example, a practical problem when recording microscope images with fluorescence contrast is that misalignment of the color filters in the detection arm can cause microscope images with different fluorescence contrasts to be laterally shifted relative to each other in the image plane. This can happen even when different microscope images are part of the same multichannel recording (although this is typically less noticeable in this case). Another reason for the lateral shift of channels with different contrasts may be that each channel records a so-called “patch scan,” and the patch position is not accurately reached again in successive patch scans. These patches are then combined to form a mosaic image, sometimes referred to as “stitching.” Yet another reason may be that the microscope has multiple recording sensors (e.g., cameras) that image slightly different sample areas with slightly different optics. This implies the use of multiple optical channels. In this context, it is also conceivable to use multiple microscopes with different optical properties to record individual fluorescence channels. In addition to lateral offset (xy plane), different fluorescence channels may also be axially offset (z direction), for example, due to longitudinal chromatic aberration of the optical system. Similarly, relative rotation of the images is possible if they are recorded on different microscopes or by different cameras. It is also conceivable to manipulate the sample (i.e., staining or destaining, for example, between capturing microscope images) so as to make specific structures visible one after another without causing interference between fluorescence signals.

[0009] To enable reliable evaluation of the information contained in microscope images 91 and 92, image registration is typically desired between microscope image 91 and microscope image 92, given the possibility of shift, rotation, and / or distortion of the image plane. This registration quantifies or compensates for such shift and / or distortion and / or rotation by identifying common image features in these microscope images, for example. Therefore, such image registration makes it possible, for example, to evaluate only the cellular biological characteristics of a cell sample, which can be obtained from a comparison of the two microscope images 91 and 92.

[0010] However, it has been observed that direct image registration between microscope images 91 and 92, or in general between microscope images with different fluorescence contrasts, does not provide reliable results.

[0011] Therefore, existing technologies disclose pathways for indirectly or using auxiliary methods to perform corresponding registration between microscope images 91, 92. For example, additional microscope images with additional fluorescence contrast for labeling similar cell structure types can be captured. However, registration via additional staining has the following major drawbacks: the reference fluorescence stain bleachs upon prolonged exposure. Ideally, it would be desirable to use a method in which the sample can be fully labeled without bleaching the reference channel, and indeed other fluorescence channels. In cases of intensified bleaching, the labeling of the sample is lost, making repeated measurements and long-term experiments impossible. Additionally, reference staining may lead to a deterioration in the quality of the fluorescence channel of interest. For example, additional fluorescence staining may cause spectral overlap with other channels, resulting in computationally complex spectral separation. Furthermore, reference staining may mask weak emission regions of interest. In terms of hardware, the microscope must be equipped with additional excitation wavelengths (e.g., additional colored LEDs or laser diodes) as well as additional color filters / beam splitters. The sample itself requires additional dyes. Overall, this leads to higher production costs for both the system and the sample. Because of the low fluorescence yield (the ratio of fluorescence emission to excitation radiation), the acquisition time of fluorescence channels is typically longer than that of transmitted light modes (such as bright field). Therefore, the addition of fluorescence channels significantly increases the acquisition time in experiments.

[0012] Another technique is known from US 2012 / 0257811. This document discloses the registration of two microscope images with fluorescence contrast using a bright-field image as a reference. A drawback of this technique is that some structures in the bright-field image are either invisible or only weakly visible.

[0013] Another technique is known from WO 2023 / 044071 A1. This document describes how to obtain a first image of a biological sample on a first substrate and how to obtain a second image of the biological sample on a second substrate. The second substrate has one or more spatial reference points. Registration between the first and second images can then be determined, implemented using patterns in both images. This registration can then be used to overlay the first image onto a spatial dataset including spatial analysis data, the reference frame of which is known relative to the second image, based on these spatial reference points. The disadvantages of this technique are: a specific transparent substrate must be used in conjunction with the spatial reference points (called reference points). This is relatively expensive and complex. Prior knowledge about the appearance of the spatial reference points in the microscopic image must be available. Appropriate image processing algorithms must be used to find the corresponding markers. Summary of the Invention

[0014] Therefore, there is a need for improved techniques to determine the values ​​of registration parameters for registering microscopic images that image the structure of a sample with specific contrast.

[0015] This objective is achieved through the features of the independent claims of the patent. The features of the dependent claims define the embodiments.

[0016] The techniques described herein enable channels with specific contrast (especially channels labeled with completely complementary structures of a sample (i.e., channels with different specific contrast)) to be explicitly aligned with each other, for example, in two dimensions (2D) or three dimensions (3D).

[0017] The corresponding microscope images can be part of a common multichannel recording. For example, a microscope image can be a patch image of a specific patch location from a patch scan. A corresponding multichannel recording can be captured at each patch location.

[0018] However, it is also conceivable that these microscope images are part of different multichannel recordings. For example, these microscope images could be patch images of adjacent patch locations from a common patch scan, where different multichannel recordings are captured at different patch locations.

[0019] It is also conceivable that the sample has been manipulated (i.e., stained and / or destained during a staining cycle) between capturing the corresponding microscope images. In this case, the microscope images could be, for example, mosaic images, each consisting of multiple patches.

[0020] The techniques described in this paper can be used, for example, to recover specific regions of interest (ROIs) in samples captured in different imaging modalities and / or at different microscope magnifications. The techniques described in this paper can also be used to achieve reliable stitching of microscope images: this means that multiple smaller, overlapping patch images of a sample can be merged to form a larger mosaic image.

[0021] The technique described herein does not require reference staining or substrate-based benchmarks / landmarks / reference points. This technique enables particularly accurate registration between microscope images with different fluorescence contrasts / different specific contrasts. In particular, the technique described herein is robust to shifts, distortions, or rotations of the image planes of the two microscope images relative to each other.

[0022] A computer-implemented method is disclosed. This method is used to register a first microscope image with a second microscope image. The first microscope image images a sample with a first specific contrast, and the second microscope image images the same sample with either the first specific contrast or another (i.e., a second) specific contrast.

[0023] For example, a sample can be a cell sample or a tissue slice sample. A sample can contain cellular structures.

[0024] For example, a first specific contrast specifically labels at least one first cell structure type. A second specific contrast specifically labels at least one second cell structure type. This at least one first cell structure type is, for example, at least partially different from the at least one second cell structure type. In other words, this means that the first specific contrast labels at least one cell structure type, while this at least one cell structure type is not exactly labeled by the second specific contrast. There may be overlap in labeling between different specific contrasts, even if the first specific contrast is different from the second specific contrast.

[0025] For example, the first specific contrast can be a specific fluorescence contrast, or it can be obtained through a non-fluorescent label (e.g., H&E).

[0026] The second specific contrast can be a specific fluorescence contrast, or it can be obtained through a non-fluorescent label (e.g., H&E).

[0027] The first and second microscope images can completely overlap, or they can have at least one overlapping area, that is, at least partially image the same area of ​​the sample.

[0028] The first microscope image can be a patch image or a mosaic image.

[0029] The second microscope image can be a patch image or a mosaic image.

[0030] If both the first and second microscopic images are patch images, they can be captured at the same patch location within a common patch scan or at different patch locations within a common patch scan. It is also conceivable that the first microscopic image is part of a first patch scan, and the second patch image is part of another (i.e., a second) patch scan, which may optionally be associated with different staining cycles of the sample. The first and second microscopic images can be captured at corresponding patch locations within the two patch scans.

[0031] The method includes: obtaining a reference microscope image that images the sample with non-specific contrast. Additionally, the method includes: using the reference microscope image to perform registration between a first microscope image and a second microscope image.

[0032] The reference microscope image can be a mosaic image (if both the first and second microscope images are mosaic images). However, the reference microscope image can also be a tiled image (if both the first and second microscope images are tiled images). For example, a tiled scan can be performed that captures one or more microscope images and an associated reference microscope image at multiple tile locations in each case. For example, the first microscope image, the second microscope image, and the reference microscope image can all be captured at the same tile location and are part of a common multichannel recording.

[0033] Non-specific contrast should be distinguished from first-specific and second-specific contrast. Non-specific contrast does not label specific molecules or cellular structures—unlike first-specific and second-specific contrast. Typically, non-specific contrast can be obtained without staining the sample. Non-specific contrast can be label-free contrast.

[0034] For example, nonspecific contrast can be selected from the following group: phase-like contrast; phase contrast; bright-field contrast; contrast due to oblique illumination; dark-field contrast; autofluorescence contrast.

[0035] With a reference microscope image possessing non-specific contrast, for example, different labeled cell structure types from both the first and second microscope images can be visible in the reference microscope image, making registration of the first microscope image with the reference microscope image, and registration of the first microscope image with the second microscope image using the reference microscope image, robust and reliable based on image features. The fact that the reference microscope image lacks fluorescence contrast additionally reduces the light exposure of the cell sample. Cell exposure to fluorescent dyes is reduced, thereby mitigating the impact of measurements on cell biology. For example, registration can include the application of a multimodal registration algorithm. In particular, a multimodal registration algorithm can be applied between microscope images with different contrasts.

[0036] For example, a multimodal registration algorithm can be applied between a microscope image with specific contrast and a (reference) microscope image with non-specific contrast.

[0037] For example, multimodal registration algorithms can be applied between different tile images that overlap in a tile scan. Multimodal registration algorithms can also be implemented between tile images scanned in different (i.e., sequential) sequences.

[0038] For example, the first microscopic image can be associated with a first staining cycle of the sample. For example, the first staining cycle can involve staining the sample with at least one non-fluorescent tag. This non-fluorescent tag can define a first specific contrast in the first microscopic image. For example, H&E can be used as a non-fluorescent tag. However, it is also conceivable that the first staining cycle involves staining the sample with a fluorescent tag.

[0039] The second microscopic image can be associated with a second staining cycle of the sample. For example, the second staining cycle can use at least one non-fluorescent tag to stain the sample. This non-fluorescent tag can define a second specific contrast in the second microscopic image. For example, H&E can be used as a non-fluorescent tag. The second staining cycle can use one or more fluorescent tags to stain the sample. For example, a fluorescent tag can be used exclusively in the second staining cycle, i.e., unlike the use of a non-fluorescent tag in the first staining cycle. One of these one or more fluorescent tags from the second staining cycle can define a second specific contrast in the second microscopic image.

[0040] If different fluorescence contrasts, i.e. multiple different fluorescent tags, are used in the second staining cycle, multi-channel recording can be completed, thereby capturing the corresponding different fluorescence contrasts at different wavelengths.

[0041] For example, the second staining cycle can be performed after the first staining cycle.

[0042] The reference microscope image can then be associated with the first staining cycle of the sample. Therefore, this means that the reference microscope image is captured, for example, as a multi-channel record along with the first microscope image. For instance, it would be conceivable that the first microscope image and the reference microscope image are each mosaic images derived from a common patch scan.

[0043] Reference microscope images can be captured before the second staining cycle is performed.

[0044] In some examples, the method further includes obtaining an additional reference microscope image. This additional reference microscope image can image the sample with the same specific contrast as the reference microscope image used to image the sample. However, it is also conceivable that the additional reference microscope image uses a non-specific contrast different from that of the reference microscope image.

[0045] This additional reference microscope image can be associated with the second staining cycle. For example, this additional reference microscope image can be captured together with the second microscope image as a multi-channel record. For example, it is conceivable that the second microscope image and the additional reference microscope image are each mosaic images from a common patch scan.

[0046] Therefore, this means that in one variant, there are two reference microscope images with non-specific contrast. The first microscope image can then be compared or aligned with the reference microscope image; additionally, the second microscope image can be compared or aligned with another reference microscope image. Furthermore, the comparison or alignment of reference microscope images with other reference microscope images can also be performed.

[0047] However, such comparisons between the first microscope image and the reference microscope image, and between the second microscope image and another reference microscope image, are optional. It is conceivable that the first microscope image is already inherently registered with the reference microscope image; and / or the second microscope image is already inherently registered with another reference microscope image. For example, if the first and reference microscope images are recorded as part of a common multichannel recording or recorded sequentially, the microscope may be able to record the exact same segment again (e.g., if the stage does not move between recordings and changes to the beam splitter / filter do not cause an offset between images). Then, only the reference microscope image is compared with another reference image for the next staining cycle or with the second microscope image (if no other reference microscope image exists).

[0048] The disclosed techniques are based on the insight that, typically, the differences in sample imaging between a first microscopic image and a reference microscopic image, and between a second microscopic image and another reference microscopic image, are relatively small. This is because these images are each associated with the same staining cycle and can be captured, for example, as part of a common multichannel recording and specifically through common patch scanning.

[0049] For example, it is conceivable that the first microscopic image is a first mosaic image composed of first patch images. The second microscopic image could be a second mosaic image composed of second patch images. In particular, the first and second patch images can be captured in different patch scans because these images are associated with different staining rounds. A multimodal registration algorithm can then be implemented between the first patch image and a corresponding reference patch image of a reference microscopic image; and again between the second patch image and a corresponding additional reference patch image of another reference microscopic image. For example, multiple reference patch images can exist for each patch location in a corresponding patch scan. In other words, this means that the multimodal registration algorithm is applied twice, once for the first patch scan (which captures the first patch image and the corresponding reference patch image) and once for the second patch scan (which captures the second patch image and the corresponding additional reference patch image). Therefore, a multimodal registration algorithm can be applied at the patch hierarchy level.

[0050] Furthermore, it is not necessary to perform comparisons between the first tile image and the corresponding reference tile image and / or between the second tile image and the corresponding additional reference tile image in all variations. This is the case if the corresponding image pairs are inherently registered.

[0051] Regardless of whether the multimodal registration algorithm is applied at the tile level or whether the different tile images are already inherently registered at the tile level, it is conceivable that registration will involve applying a single-modal registration algorithm. A single-modal registration algorithm can be applied between a reference microscope image and another reference microscope image (i.e., after mosaic-level stitching).

[0052] However, it is also conceivable that stitching is performed only after the application of a single-modal registration algorithm. In this case, for example, the registration parameter values ​​of the two multimodal registration algorithms can be combined with the registration parameter values ​​of the single-modal registration algorithm to register each corresponding patch of the first patch scan with each corresponding patch of the second patch scan for specific contrast.

[0053] For example, registering a first microscope image and a second microscope image may include determining registration parameter values ​​for the registration based on a comparison between the second microscope image and a reference microscope image. Meanwhile, it is conceivable that the first microscope image and the reference microscope image are already inherently registered—therefore, a comparison between the first microscope image and the reference microscope image is unnecessary. This is useful, for example, if the second microscope image and the reference microscope image were not captured in the same temporal context, i.e., not part of the same multichannel recording or not part of the same patch scan. For example, the first microscope image and the second microscope image may be associated with different staining cycles. Additionally, this is useful if the first microscope image and the second microscope image were captured through significantly different optical channels (e.g., through optical channels with different objectives).

[0054] An optical path generally describes the optical imaging pattern from object space to image space. Therefore, the optical path describes the optical imaging of a sample onto the camera of a microscope. The optical path is determined by the optical components used (such as objectives, filters, etc.).

[0055] Multiple microscope images can be part of a multichannel recording. This means that the multiple microscope images are captured within a single temporal context, and that as many optical components as possible remain unchanged between the captured multichannel microscope images. For example, between different captured multichannel microscope images, only a color filter can be inserted into the beam path or only a color filter can be removed.

[0056] Typically, registration parameter values ​​can indicate, for example, the shift in the x-direction, the shift in the y-direction, rotation, scaling and / or shearing, or deformation, distortion, compression, etc., between two images.

[0057] If the first and reference microscopic images are part of the same multichannel recording, the optical channels used are typically very similar, and no manipulation of the sample is performed between capturing the first and reference microscopic images (i.e., the first and reference microscopic images are captured within a narrow temporal context). In particular, (as explained above) if the first and reference microscopic images are part of the same multichannel recording, these images can be inherently registered: that is, the registration parameter values ​​indicate an identity mapping, or are at least static and previously known. Then, it may not be necessary to determine the corresponding registration parameter values ​​by comparing the first microscopic image with the reference microscopic image. The first and second microscopic images are directly registered to each other by determining the registration parameter values ​​by comparing the second microscopic image with the reference microscopic image.

[0058] However, sometimes it may happen that (even though the first and reference microscope images are part of the same multichannel recording) there is no inherent registration. For example, chromatic aberration in different color filters used to select different wavelengths can cause shifts, distortions, etc., between the two microscope images relative to each other. Therefore, multimodal registration algorithms (e.g., as described above) can be used.

[0059] However, it is also conceivable that the reference microscope image and the first microscope image are not part of the same multichannel recording. For example, the reference microscope image and the first microscope image may have been captured at different times or with respect to completely different optical channels. For example, the reference microscope image and the first microscope image may have been associated with different patch scans. In this case, in particular, additional registration parameter values ​​for the registration can be determined based on a comparison of the first microscope image and the reference microscope image (e.g., by applying a multimodal registration algorithm). In this way, and by determining the registration parameter values ​​by comparing the second microscope image with the reference microscope image, the first microscope image and the second microscope image are registered to each other. For example, the registration parameter values ​​and additional registration parameters can then be combined with each other (without having to consider image features again) to obtain an imaging specification that indicates the final registration between the first microscope image and the second microscope image, i.e., the so-called "resulted registration parameter values".

[0060] The examples described above illustrate the comparison between at least one microscope image with specific contrast and a reference microscope image with non-specific contrast (to determine registration parameter values). In some examples, it is conceivable that such a comparison may also be performed, or exclusively, between two reference microscope images with non-specific contrast. For example, it is conceivable that registration parameter values ​​for registration between a first microscope image and a second microscope image, which also images the sample with non-specific contrast, are determined based on a comparison of the reference microscope image with another reference microscope image.

[0061] For example, it is conceivable that a reference microscope image and a first microscope image are part of the same multichannel recording; and another reference microscope image and a second microscope image are part of the same (another) multichannel recording. However, on the other hand, these two reference microscope images may have been captured at different times (e.g., in the context of a staining cycle, such as after sample processing) and / or with different optical channels (e.g., especially with different microscopes). Therefore, these two reference microscope images may be part of different multichannel recordings. Accordingly, the first microscope image and the second microscope image will be part of different multichannel recordings. In the case of inherent registration of different microscope images in a specific multichannel recording, it is not necessary to perform an additional comparison between the first microscope image and the reference microscope image (because these two microscope images are already inherently registered); additionally, it is not necessary to perform an additional comparison between the second microscope image and another reference microscope image (because these two microscope images are also already inherently registered). In other words, this means that by determining the registration parameter values ​​based on a comparison of the two reference microscope images, registration between the first microscope image and the second microscope image has already been obtained.

[0062] If multiple reference microscope images exist, they can have the same non-specific contrast, such as a specific digital phase contrast.

[0063] In one variation, additional registration parameter values ​​are determined by comparing a reference microscope image with a first microscope image, and / or by comparing another reference microscope image with a second microscope image. This is useful, for example, if the reference microscope image and the first microscope image were not captured using the same optical channels, and / or were not captured in the same temporal context, i.e., particularly not part of the same multichannel recording (this also applies to the other reference microscope image and the second microscope image). Additionally, this further determination of registration parameter values ​​can be useful if chromatic aberration exists in the optical channels of the corresponding pair of reference microscope images and microscope images used to capture the same multichannel recording.

[0064] For example, the non-specific contrast of a reference microscope image can be a phase-like contrast. Phase-like contrast can be, for example, phase contrast. Examples include, for instance, Zernike phase contrast and Nomarsky phase contrast. In this case, specific optical elements are used in the beam path of light, such as a phase ring in the objective lens and an annular stop in the condenser lens. In this way, the interference between the background light and the object light can be made visible. Image contrast can be enhanced by using phase contrast. This means that these cellular structures are particularly clearly visible. Cells are phase objects that do not cause any reduction in the amplitude of light or any significant reduction when light passes through a cell sample, and therefore, phase contrast is preferred for making phase shifts visible.

[0065] However, digital phase contrast can also be used as a type of phase contrast. Here, multiple images are recorded, and then computationally combined to form a single phase contrast image. Therefore, this type of technique can be called digital phase contrast. Phase contrast is obtained by digitally post-processing the recorded intensity images. Examples include the intensity transport equation (TIE) and differential phase contrast (DPC). TIE is described in: Streibl, Norbert. “Phase imaging by the transport equation of intensity.” Optics Communications 49.1 (1984): 6-10. DPC is described in: Mehta, Shalin B. and Colin JR Sheppard. “Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast.” Optics Letters 34.13 (2009): 1924-1926. To record a TIE dataset, the sample is shifted along the optical axis (z-direction), i.e., axially, and a so-called z-stacking of at least two images is recorded. The data is then combined by computation to obtain a phase-contrast image. For this purpose, a diffusion-type partial differential equation is solved. In DPC, the sample is illuminated from at least two different directions (tilted illumination) while it is held at a fixed z-position. All types of segmented sources are possible sources for tilted illumination; examples include segmented diodes, light-emitting diode arrays, digital micromirror devices (DMDs), liquid crystal displays (LCDs or SLMs), or variable condenser apertures. The recorded data is then converted into a phase-contrast image by solving a deconvolution problem. Combinations of TIE and DPC are also conceivable, for example, as described in European Patent Application 24,184,623.7 dated June 26, 2024. The advantage of using digital phase contrast (compared to hardware-based phase contrast) is that when capturing digital phase contrast, it is not necessary to complexly insert or remove objects into or from the beam path of light. Alternatively, the lighting can be modified in a targeted manner, for example, by means of a switchable array of light-emitting diodes arranged in the plane of the lighting pupil. This can be implemented quickly and easily.

[0066] However, it is also conceivable that phase-like contrast is digital phase gradient contrast. For example, DE 10 2015208 084 A1 describes a corresponding technique for various tilted illuminations. Intensity images of cell samples are captured from different illumination directions, and then the differences between the intensity images are calculated. For example, normalized differences can be calculated. This phase gradient contrast is particularly easy to compute. For example, deconvolution is not required.

[0067] For example, a first temporary reference microscope image and a second temporary reference microscope image can be obtained. Both the first and second temporary reference microscope images can image the cell culture with the same intensity contrast but combined with different defocus values ​​and / or illumination geometries. A reference microscope image can then be determined based on the pixel-by-pixel difference formation between the first and second temporary reference microscope images. Optionally, normalization can be performed. Phase gradient contrast can be generated based on such microscope images with intensity contrast captured combined with different defocus values ​​and / or illumination geometries. The pixel-by-pixel difference formation between these two temporary reference microscope images is a simple computational operation that can be performed particularly quickly without requiring large computational resources. Therefore, the corresponding technique is particularly suitable for real-time or near-real-time registration.

[0068] For example, specific registration parameter values ​​for affine transformations between corresponding image pairs can be determined using the techniques described herein. Affine transformations include translation, rotation, scaling, and shearing in matrix form. However, nonlinear deformations using elastic registration algorithms are also conceivable. If elastic registration algorithms are used, it is conceivable, for example, that cell samples undergo changes between capturing different microscopic images, with these changes manifesting as different morphologies of the individual cells. In the context of elastic registration algorithms, such changes in structural geometry can be accounted for through corresponding distortions or deformations.

[0069] As already described, multimodal registration algorithms can be used in various examples. Such multimodal registration algorithms are robust with respect to the inversion of brightness histograms between microscope images being compared to each other. This is particularly useful when comparing a reference microscope image with non-specific contrast (e.g., phase-like contrast) to a microscope image with specific contrast (e.g., fluorescence contrast or H&E contrast).

[0070] For example, registration algorithms selected from the group consisting of: mutual information; normalized gradient fields; structural similarity indices; segmentation-based similarity measures; feature detection; and artificial neural networks can be used. Multimodal registration algorithms using, for example, one of the similarity measures explained above, enable the comparison of a first or second microscope image (each with specific contrast, such as fluorescence contrast) with a reference microscope image (with non-specific contrast, such as lacking fluorescence contrast but possessing phase-like contrast). Typically, in fluorescence contrast, specific portions of cells in the corresponding sample appear particularly bright. However, since cells are phase objects that do not cause significant attenuation of the amplitude of transmitted light, it is possible that the same portions of the cells appear darker relative to the background in the reference microscope image. This inversion of the brightness histogram can be taken into account in multimodal registration algorithms.

[0071] Landmark-based registration algorithms can also be used. In landmark-based registration algorithms, specific characteristic landmarks—that is, characteristic structures or patterns that appear in both images—are sought through a suitable object recognition algorithm. For example, predefined structures can be sought, that is, structures expected to exist in the images and for which there is some prior knowledge about their appearance in the images. However, it is also conceivable to use object recognition algorithms that do not include any specifications regarding the type of structure to be sought. Such landmark-based registration algorithms can also be particularly robust to variations in brightness values. This is due to the fact that landmark-based registration algorithms focus less on the brightness of different pixels and more on specific real-space patterns.

[0072] An electronic data processing apparatus is described. This apparatus is used to register a first microscope image with a second microscope image. The apparatus includes a processor and a memory, wherein the processor is configured to load program code from the memory and execute the program code. When the program code is executed, the processor performs the steps of the method described herein.

[0073] Without departing from the scope of protection of this invention, the features set forth above and described below may be used not only in the explicitly stated corresponding combinations, but also in other combinations or individually. Attached Figure Description

[0074] Figure 1 A first microscope image with a first fluorescence contrast and a second microscope image with a second fluorescence contrast are shown, as well as an overlay of the first microscope image and the second microscope image.

[0075] Figure 2 Showing from Figure 1 Phase-contrast images of cell samples.

[0076] Figure 3 This is a flowchart of an exemplary method.

[0077] Figure 4 This is a flowchart of an exemplary method.

[0078] Figure 5 The distance functionals of the NGF registration algorithm are shown based on various examples.

[0079] Figure 6 The illustration schematically demonstrates the registration between two microscope images with different specific contrasts according to various examples.

[0080] Figure 7 The illustration schematically demonstrates the registration between two microscope images with different specific contrasts according to various examples.

[0081] Figure 8 The illustration schematically demonstrates the registration between two microscope images with different specific contrasts according to various examples.

[0082] Figure 9 The illustrations depict tile-to-tile registration and scan-to-scan registration based on various examples.

[0083] Figure 10 The diagram illustrates a system based on various examples, which has a data processing device and a data source. Detailed Implementation

[0084] The features, characteristics, and advantages of the invention described above, as well as the ways in which they are implemented, will become clearer and more apparent in conjunction with the following description of exemplary embodiments, which are explained in more detail in connection with the accompanying drawings.

[0085] The invention will now be explained in more detail with reference to the accompanying drawings, based on preferred embodiments. In the drawings, identical reference numerals denote identical or similar elements. The drawings are schematic representations of various embodiments of the invention. Elements shown in the drawings are not necessarily shown to scale. Instead, the various elements shown in the drawings are presented in a manner that makes their function and general purpose readily understandable to those skilled in the art. Connections and linkages between functional units and elements shown in the drawings may also be implemented as indirect connections or linkages. Connections or linkages may be implemented in a wired or wireless manner. Functional units may be implemented as hardware, software, or a combination of hardware and software.

[0086] The following discloses a technique for registering two microscope images. These two microscope images may have different specific contrasts; however, it is also conceivable that they may have the same specific contrast. These two microscope images may be part of a common multichannel recording or part of different multichannel recordings. These two microscope images may each be a mosaic image or a patch image. These two microscope images may be associated with different patch scans or with the same patch scan.

[0087] In various examples, instead of obtaining direct registration between two microscope images based on image comparison, one or more “auxiliary registrations” are determined based on one or more reference microscope images (e.g., with non-specific contrast). The two microscope images can then be represented as different fluorescence channels in the microscope images.

[0088] For example, combining Figure 1 The figure shows: a microscope image 91 having a first fluorescence contrast (as an example of a first nonspecific contrast); and a microscope image 92 having a second fluorescence contrast (as an example of a second nonspecific contrast). Figure 2 The image shows an associated reference microscope image 99 with phase contrast. In this image, different cell structure types are all visible together. The two microscope images 91 and 92 may be laterally shifted relative to each other, or even non-linearly deformed and / or rotated. As part of the registration process, such and other registration parameters can be quantified to enable a more comprehensive evaluation.

[0089] In one exemplary variant, microscope images 91 and 92 are first individually aligned with reference microscope image 99 via corresponding multimodal registration. Initially, microscope images 91 and 92 are shifted and / or rotated and / or otherwise distorted relative to reference microscope image 99. If multimodal registration is successful, all microscope images 91, 92, and 99 are aligned both axially and laterally relative to each other. Possible rotational errors, etc., are also corrected accordingly.

[0090] Registration of microscope images 91, 92 relative to reference microscope image 99 can be captured on the sample after multichannel recording (where the two microscope images 91, 92 are captured simultaneously or at least rapidly sequentially; i.e., both microscope images 91, 92 are part of the same multichannel recording), and can also be captured during a re-enactment of multichannel recording after temporary sample processing (e.g., cyclic staining of the sample (staining cycle)). In this case, a repeat recording of the reference microscope image can be completed during the re-enactment of multichannel recording, which can then be used for alignment with both the phase-contrast image of the first staining cycle and the fluorescence channel of that staining cycle and / or other staining cycles. Therefore, this means that multiple multichannel recordings each have a corresponding reference microscope image. For example, each multichannel recording can be captured by patch scanning. If the sample is deformed due to mechanical influences during the imaging step, elastic image registration may also be suitable.

[0091] Figure 3 This is a flowchart illustrating an exemplary method. (Source: [Original Source Name]) Figure 3 This method can be implemented by an electronic data processing device. For example, from... Figure 3 The method can be executed by the processor when the processor loads program code from memory and executes the program code. Therefore, from Figure 3 The method is implemented by computer.

[0092] From Figure 3 The method is used to capture a microscope image and one or more reference microscope images. The reference microscope images have non-specific contrast, such as phase-like contrast.

[0093] Microscopic images exhibit different specific contrasts, such as varying fluorescence contrasts. For example, different dyes and / or different wavelengths can be used for fluorescence excitation. Different labels can also be used.

[0094] Microscopic images with different specific contrasts can be captured in one or more “staining cycles” (i.e., successive iterations 904). In each “staining cycle,” the sample is stained with a corresponding (e.g., fluorescent) dye or label in box 905, and / or the specific dye or label is removed. For example, multiple microscopic images can be captured for each iteration 904 for a multi-channel recording; that is, different iterations correspond to different multi-channel recordings. For each iteration 904, a corresponding patch scan can be performed, wherein, for example, a multi-channel patch image is captured per patch (i.e., at each patch location).

[0095] In box 905, the sample is manipulated, i.e., stained, for example. The sample may be stained with one or more dyes. Different dyes specifically label particular cell structure types, for example, with one or more fluorescent tags and / or with one or more non-fluorescent tags. For example, one dye may label organelles, while another dye may label the cell membrane or nucleus. Different fluorescent dyes may be excited by light of different wavelengths and / or may fluoresce at different wavelengths.

[0096] Manipulations other than staining can also be envisioned. For example, the sample could be destained alternatively or additionally. This means removing the specific fluorescent dye.

[0097] Box 910 involves capturing one or more microscopic images, each with nonspecific contrast. If fluorescent labels are used, for this purpose, light is typically used to excite the fluorescence of a fluorescent dye used to stain the sample in one or more previous iterations in box 905. Autofluorescence may also be used.

[0098] If multiple microscope images are captured in frame 910, they can be captured using different optical channels and therefore may have lateral shifts, distortions, and / or rotations relative to each other. For example, changing a beam splitter or filter may cause image shifts. Optics may also have slightly different magnification for different colors; or other chromatic aberrations may also be possible.

[0099] These multiple microscope images can be part of a multi-channel recording. This means that, in principle, the same optical channels are used in each case; and between capturing each microscope image, only the color filter is changed, for example.

[0100] These microscope images can be what are called mosaic images: performing corresponding patch scans and combining the individual patches.

[0101] Box 915 relates to capturing image data that corresponds to a reference microscope image, or based on which a reference microscope image can subsequently be determined.

[0102] For example, a reference microscope image may be a portion of the same multichannel record used to capture one or more microscope images in box 905 during the same iteration 904. For example, a reference microscope image may be a portion of the same patch scan used to capture one or more microscope images in box 905 during the same iteration 904.

[0103] For example, hardware-based phase contrast can be captured. Then, in box 915, this phase contrast (e.g., Zernike phase contrast or Nomarsky phase contrast) can be used directly to capture a reference microscope image. Other examples include bright-field contrast, dark-field contrast, or autofluorescence contrast. However, multiple intensity images can also be captured in box 915, each captured by combining different defocus values ​​and / or different illumination geometries (contrast resulting from oblique illumination). Based on such a provisional reference microscope image, a reference microscope image can then be later determined using digital phase contrast or digital phase gradient contrast. This means further processing of the provisional reference microscope image to obtain the (final) reference microscope image. Details are further explained below in connection with box 925.

[0104] In some examples (as explained above), it is conceivable that the (reference) microscope image captured in box 910 and / or box 915 consists of multiple patches, i.e., a mosaic image. Here, the microscope used to capture the microscope image may have a motorized sample holder. For example, a motorized microscope stage may be used, which can translate, for example, in two or three dimensions. In this case, a corresponding patch image (e.g., a multi-channel image) is captured at a specific location on the motorized sample holder; then, the motorized sample holder is moved to the next location, and another corresponding patch image (e.g., a multi-channel image) is captured at that next location. This process is repeated until the patch images cover a predefined sample area. The different patch images may each have overlap with each other, making registration between adjacent patch images possible in the overlap. The resulting microscope image (so-called "stitching") can then be derived from the patch images, i.e., the mosaic image. The scenario described above involves capturing multiple microscopic images in a common patch scan; thus, this implies that each patch image is a multi-channel record. In other examples, it is conceivable that multiple patch scans are performed sequentially, even within a single iteration 904 (i.e., for one staining cycle), to capture different microscopic images with specific and / or non-specific contrast. It has been observed that significant offsets and / or rotations can exist between microscopic images from different patch scans, particularly when different patch scans are used to capture different microscopic images. Box 920 relates to determining whether additional staining or destaining or some other manipulation of the sample is intended. This means determining whether an additional staining cycle is intended. If additional staining or destaining is intended, an additional iteration 904 of box 905 is performed. Subsequently, one or more additional microscopic images with one or more corresponding additional specific contrasts can be captured in box 910, and, where appropriate, one or more additional reference microscopic images can be captured in box 915.

[0105] If the microscope images are captured in different iterations 904, they are temporally offset relative to each other (without a common temporal context); this temporal offset can be translated into lateral offsets between the corresponding captured microscope images through various temporal drift mechanisms. Additionally, the sample has been processed simultaneously, and therefore the appearance of the sample may also change.

[0106] In principle, it is not necessary to execute box 915 in every iteration 904. For example, it is conceivable to capture data for the reference microscope image only in the first iteration 904 (i.e., in one staining cycle) and not in subsequent iterations in box 915. It is also conceivable to implement multiple iterations 904 including box 915, thereby obtaining multiple reference microscope images, each of which is used for registration with respect to microscope images with specific contrast in other iterations.

[0107] Figure 3 Techniques related to the capture of microscope images are demonstrated. In principle, it is also conceivable that various microscope images were captured at an earlier time and stored in a database, as illustrated in the various examples described herein. Figure 3 The method is optional in principle.

[0108] Figure 4 This is a flowchart illustrating an exemplary method. (Source: [Original Source Name]) Figure 4 This method can be implemented by an electronic data processing device. For example, from... Figure 4 The method can be executed by the processor when the processor loads program code from memory and executes the program code. Therefore, from Figure 4 The method is implemented by computer.

[0109] From Figure 4 The method is used to register a first microscope image with a second microscope image. The first microscope image has, for example, a first specific contrast, which is different from the second specific contrast of the second microscope image.

[0110] For example, it can be obtained from Figure 3 The method involves capturing a first microscope image, a second microscope image, and one or more reference microscope images. In this regard, [the method] comes from... Figure 4 The method can be found in Figure 3 Following this method, it could also be implemented such that it is partially time-dependent on the method originating from... Figure 3 The methods overlap.

[0111] Box 925 relates to optionally generating one or more reference microscope images with digital phase contrast or digital phase gradient contrast (if such images do not already exist). For example, it would be conceivable that in box 915 (see...) Figure 3In iteration 904, multiple temporary reference microscope images captured are combined with each other. For this purpose, techniques such as those known to be associated with DPC or TIE can be used. Pixel-by-pixel difference formation can also be performed to obtain a corresponding reference microscope image with phase gradient contrast. If a reference microscope image with phase contrast was previously captured directly in box 915, box 925 is unnecessary.

[0112] It is not necessary to obtain one or more reference microscope images with phase-like contrast in all variants. Other types of non-specific contrast can also be used. In particular, unlabeled contrast can be used. Examples (in addition to phase-like contrast, such as phase contrast) include bright-field contrast, contrast due to oblique illumination, dark-field contrast, and autofluorescence contrast.

[0113] Box 930 relates to performing registration between a first microscope image and a second microscope image. Registration is performed using one or more reference microscope images with nonspecific contrast.

[0114] In box 945, the application can then optionally be implemented based on the registration of the first and second microscopic images. For example, the sample can be evaluated based on a comparison between the first and second microscopic images, taking into account the corresponding registration parameter values. For example, the ROI can be labeled in one microscopic image and then displayed in the other microscopic image—thus taking registration into account. Co-localization and quantification of label concentrations can be implemented to quantify disease progression. Multiple microscopic images (e.g., those captured in a common temporal context) can be merged to form a multichannel image.

[0115] However, it is also conceivable to stitch together the first and second microscope images (in this case, these microscope images are patch images). In this scenario, the first and second microscope images typically have the same specific contrast. Stitching expands the field of view, resulting in a mosaic image. For this purpose, the first and second microscope images image different but overlapping regions of the sample. If the first and second microscope images are registered with each other, accurate alignment of the two microscope images can be achieved as part of the stitching process.

[0116] The details relating to the determination of the registration parameter values ​​in box 930 will be discussed next. Different registration algorithms can be used to determine the registration parameter values. For example, an elastic registration algorithm can be used. It is also conceivable to use a landmark-dependent registration algorithm. In particular, it would be useful to use a registration algorithm in box 930 that can handle differences in the appearance of the sample in different microscopic images. For example, it would be conceivable to invert the brightness histograms of those images compared to each other. In other words, this means that specific structures that appear bright in a microscopic image with specific contrast appear dark in a reference microscopic image (and vice versa). The corresponding registration should be robust to both local and global inversion of the brightness histogram. Furthermore, it is useful to use a multimodal registration algorithm to determine the first and / or second registration parameter values. This multimodal registration algorithm differs from a single-modal registration algorithm. Single-modal registration is typically used to align images with a common modality (e.g., two absorption images recorded at wavelengths close together). However, multimodal registration algorithms can yield better results because they compare microscopic images with specific contrast to reference microscopic images with non-specific contrast: while single-modal registration algorithms can use simple metrics (e.g., least squares) as a quality factor for the alignment of channels with each other, multimodal registration algorithms are computationally more complex. This is due to the fact that the similarity metric between the images to be aligned must withstand multimodality. For example, as described above, the brightness histogram can be locally inverted (fluorescence: dark background, bright field / phase: bright / gray background). Table 1 lists typical similarity metrics that can be used for multimodal registration algorithms based on the examples disclosed herein.

[0117]

[0118] Table 1: Various similarity measures for implementing multimodal registration algorithms.

[0119] First, we will explain the "mutual information" in Example 1 according to Table 1. Let's assume that image A is a given image with gray levels, and image B is a second image with gray levels, which, in the most general case, has a non-linear relationship with image A. Without constraining the generality, a functional relationship can be used. If all pixel value pairs are formed (in, If these values ​​are represented relative to each other in a two-dimensional coordinate system (using linear pixel indices), then the nonlinear functional relationship is... This becomes apparent. In particular, this non-linear relationship makes it understandable that, for example, a bright pixel in image A appears dark in image B, and vice versa (this corresponds to a brightness histogram inversion). This means there exists a rule according to which the grayscale value of a pixel in image B can be predicted given the corresponding grayscale value in image A. Now, suppose image A is slightly shifted relative to image B: if image B shifts one pixel to the right, this simple functional relationship is disrupted. The correspondence between image A and the shifted image B becomes more complex. It is no longer possible to simply predict the grayscale value of image B given the grayscale value of image A. The predictability of predicting the second image from the first image can be quantified using the mathematical concept of mutual information. This is achieved by first forming a pixel in image A with a grayscale value 'a' and simultaneously forming a corresponding (identical linear index) pixel in image B. The pixel is determined by the relative frequency of occurrence of grayscale value b. If images A and B are determined by the relative frequencies of occurrence of grayscale values ​​a and b respectively, using bivariate probabilities... Mutual information (MI) is mathematically defined by the following expression:

[0120] Among them, univariate probability and The mutual information of two multimodal images can be directly calculated from the bivariate probabilities through marginalization. Therefore, if the mutual information of two multimodal images is maximized, then the two multimodal images are aligned with each other. This principle makes it possible to register images relative to each other by shifting or rotating the images three-dimensionally relative to each other until their mutual information is maximized.

[0121] The “Normalized Gradient Field” (NGF) in Example 2 of Table 1 will be explained next. Although multimodal images with identical structures can have different brightness values, the corners and edges of these multimodal images should be superimposed in the aligned state. Haber and Modernitzki (Haber, Eldad, and Jan Modersitzki. “Intensity gradient-based registration and fusion of multi-modal images.” Methods of information in medicine 46.03(2007): 292-299.) describe a corresponding metric for aligning images using this observation, and this corresponding metric is referred to here as the Normalized Gradient Field (NGF). The Normalized Gradient Field of Image A is described by the following equation:

[0122] Furthermore, the normalized gradient field of image B is described in a similar manner. Then, the following distance functional can be maximized by shifting one image relative to the next, so that images A and B can be aligned with each other:

[0123] Among them, operations Represents a vector field and The cross product. It is the root of the square of each component of the calculated cross product. For example, Figure 5 The registration parameter values ​​311 and 312 are shown for the translational shifts (in pixels) in the x and y directions of two microscope images with different fluorescence contrasts, each microscope image being correlated with a TIE phase-contrast image. The image shifts can be obtained by deriving the function... The maximum value is used for calculation. In a real-world example on a commercially available laptop, registration based on the "NGF registration algorithm" takes less than 1.7 seconds, while the registration algorithm using mutual information similarity metric takes 14.5 seconds computationally (assuming identical computing hardware). Therefore, for performance reasons, the NGF registration algorithm outperforms the mutual information-based registration algorithm. Specifically, subpixel shifts can be obtained through regression using a suitable quadratic polynomial function or Gaussian function. The maximum value is obtained with subpixel accuracy by estimating the parameters of the polynomial or Gaussian function.

[0124] For segmentation-based similarity according to Example 4 in Table 1, such as masking a specific structure (i.e., cells): the masked image takes a value of 1 in the presence of yeast cells and a value of 0 in the absence of yeast cells. Through segmentation, multimodal images can be converted into unimodal images: both the microscope image with fluorescence contrast and the reference microscope image are segmented into binary images of the same modality. Therefore, after segmentation into binary images, registration can be performed using conventional unimodal similarity metrics (such as least squares distance).

[0125] According to Example 6 in Table 1, an artificial neural network can be used to merge images with different contrast types. In the context of microscopy, this is known as virtual staining (see, for example, WO 2021 / 198 241 A1) or image-to-image learning. For this purpose, the UNET or CycleGAN architecture is primarily used for artificial neural networks. Thus, for example, a phase image (as a reference microscope image) can be converted into a virtual fluorescence image. Other fluorescence channels can then be registered with the virtual fluorescence image in a unimodal manner.

[0126] Figure 6 A first microscope image 81 with a first specific contrast and a second microscope image 82 with a second specific contrast are schematically shown.

[0127] Microscopic images 81 and 82 can be, for example, patch images or mosaic images. Mosaic images 81 and 82 may overlap each other at least partially. Typically, the degree of overlap in mosaic images is greater than that in patch images.

[0128] For example, the two microscope images 81 and 82 may be obtained using different optical channels and / or in separate temporal contexts (e.g., in...). Figure 3 The images were captured in different iterations (904). This means that there may be lateral displacement and / or rotation and / or distortion between the two microscopic images 81 and 82. Figure 6 Two microscope images, 81 and 82, are shown, captured using different optical channels, 61 and 62.

[0129] also, Figure 6 A reference microscope image 85, captured using the third optical channel 63, is also schematically shown. Therefore, lateral shifts, rotations, distortions, etc., may exist between the reference microscope image 85 and microscope image 81, and between the reference microscope image 85 and microscope image 82. For example, the reference microscope image 85 may be in the same iteration 904 as microscope image 81 (see...). Figure 3 Captured in ).

[0130] Figure 6 The diagram shows that a first registration parameter value 71 was determined based on a comparison between microscope image 81 and reference microscope image 85; furthermore, a second registration parameter value 72 was determined based on another comparison (i.e., a comparison between microscope image 82 and reference microscope image 85). Then, to perform registration of microscope images 81 and 82 in the same way, registration parameters 71 and 72 can be explicitly concatenated (or generally combined) to obtain another registration parameter 73 directly between the two microscope images 81 and 82.

[0131] Figure 7 Showing Figure 6A variant of this. Here, reference microscope image 85 and microscope image 81 are part of the same multichannel recording (and are, for example, mosaic images). The optical channels of reference microscope image 85 and microscope image 81 are particularly similar (e.g., the two optical channels may differ only in the color filters used); therefore, reference microscope image 85 and microscope image 81 are inherently registered, and it is not necessary to determine the corresponding registration parameter value 71 separately by comparing microscope image 81 with reference microscope image 85. On the other hand, a comparison can be made between reference microscope image 85 and a second microscope image 82 (microscope image 82 is not part of the multichannel recording) to determine the registration parameter value 72.

[0132] However, in some examples, the registration parameter value 71 can also be determined by comparing the reference microscope image 85 with the first microscope image 81; for example, if there is chromatic aberration due to different color filters used (chromatic aberration causes lateral shift, compression, rotation, etc. of microscope images 81, 85 captured at different wavelengths).

[0133] Figure 8 Showing Figure 6 Another variation involves capturing reference microscope images 85, 86 with nonspecific contrast, each inherently registered with a corresponding microscope image 81, 82. For example, microscope image 81 and reference microscope image 85 may be part of the same multichannel recording; alternatively, microscope image 82 and reference microscope image 86 may be part of the same multichannel recording. The two reference microscope images 85, 86 may be captured, for example, in a corresponding staining cycle or iteration 904 of capturing the correspondingly associated microscope images 81, 82. By comparing the reference microscope images 85, 86 with each other, a corresponding registration parameter value 75 corresponding to the registration of the two microscope images 81, 82 can be determined. Optionally (e.g., in the presence of significant color difference), even additional registration parameter values ​​76, 77 may be determined in each case by corresponding comparisons between microscope images 81, 82 and associated reference microscope images 85, 86. For comparisons between reference microscope images 85 and 86, a single-modal registration algorithm can typically be used; while for comparisons between microscope image 81 and reference microscope image 85, a multimodal registration algorithm should be used, for example.

[0134] In principle, it is conceivable (but not necessary) that microscope images 81, 82, and reference microscope images 85 and / or 86 (as discussed above in conjunction with the various figures) are all part of a common multichannel recording.

[0135] In the various examples described herein, registration can be performed at different stages of image processing. In the first stage, registration can be performed between different patch images from a patch scan. This registration can then be used to combine these patch images to form a mosaic image. In the second stage, registration can then be performed between different mosaic images (e.g., from different staining cycles). Figure 9 The corresponding technology is shown in the figure.

[0136] Figure 9 It demonstrates various aspects of two-stage registration. Figure 9 This is shown in the first staining cycle (see Figure 3 In iteration 904, a multi-channel record 610 is captured for the first patch 680, and another multi-channel record 620 is captured for the other patch 681. In this case, the multi-channel record 610 in the illustrated example includes four channels: three microscopic images 611-613 with specific contrast; and a reference microscopic image 614 with non-specific contrast. The multi-channel record 620 then has the same four channels: corresponding microscopic images 621-624 with the same specific contrast as the corresponding microscopic images 611-613; and another reference microscopic image 624 with the same non-specific contrast as the reference microscopic image 614. For each patch location in each patch scan, a corresponding reference microscopic image is captured.

[0137] Replace the multi-channel record for each tile (e.g.) Figure 9 As shown, individual patch scans can also be performed for each contrast. This means, for example, capturing microscope images 611, 612 in the first patch scan; capturing microscope images 612, 622 in the second patch scan; and capturing microscope images 613, 623 in the third patch scan. The second patch scan begins after the first patch scan is completed, and so on. Then, reference contrasts can be recorded for only one patch scan or even for each patch scan.

[0138] Two multichannel recordings 610 and 620 are captured in a common patch scan. This means that microscope images 611-614 and 621-624 image different but overlapping regions of the sample (where the sample stage moves between these regions). Based on the overlap, for example, microscope image 611 can be registered with the corresponding microscope image 621 (patch-to-patch registration 651). For example, if the intention is to register microscope image 611 with microscope image 621, this can be done as follows: Figure 8 As shown in the diagram.

[0139] In detail, the microscope image 611 can be aligned with the reference microscope image 614, for example, using a multimodal registration algorithm (see...). Figure 8 (Registration parameter value 76); Furthermore, the microscope image 621 can be aligned with the reference microscope image 624, for example, using the same multimodal registration algorithm (see...). Figure 8 (Registration parameter value 77). However, this is only necessary if microscope image 611 and reference microscope image 614 are not inherently registered in any way; or only if microscope image 621 and reference microscope image 624 are not inherently registered in any way. Furthermore, an additional single-modal registration algorithm (registration parameter value 75) can be implemented to register reference microscope image 614 with reference microscope image 624, so as to align reference microscope image 614 and reference microscope image 624 in the overlapping area. Thus, registration of microscope image 611 with microscope image 621 is obtained, making it possible to accurately stitch these images together. This is the first stage of registration.

[0140] The first stage of registration (i.e., additional patch-to-patch registration 652) can also be completed for the microscope images 631 to 633 of patch 685 and the microscope images 641 to 643 of patch 686. For this purpose, the corresponding multichannel recordings 630 and 640 each include reference microscope images 634 and 644 with nonspecific contrast. Therefore, the multichannel recording 630 includes four channels, that is, three microscope images 631 to 633 with different specific contrasts; and a reference microscope image 634 with nonspecific contrast.

[0141] Typically, different specific contrasts are used in different staining rounds. For example, it is conceivable to use one or more non-fluorescent tags in the first staining round to capture the corresponding specific contrast. Then, one or more fluorescent tags can be used in the second staining round.

[0142] In different staining rounds, different multichannel recordings do not necessarily have the same number of channels. For example, in the first staining round, a multichannel recording can be made using two channels (e.g., H&E contrast (specific contrast) and digital phase contrast (non-specific contrast)). In the second staining round, a multichannel recording can then be made using three or more channels (e.g., with different fluorescence contrasts and digital phase contrasts).

[0143] Therefore, in Figure 9In the example, patch-to-patch registration 651 yields corresponding mosaic microscopy images 661-663 for each specific contrast captured in the patch scan (i.e., a total of three specific contrasts); and patch-to-patch registration 652 yields corresponding mosaic microscopy images 671-673 for each specific contrast captured in the patch scan. Furthermore, in Figure 9 In the process, two mosaic reference microscope images 664 and 674 are also obtained, one for each patch scan in the two patch scans. These mosaic microscope images 661-663 and 671-673 (as described above, these mosaic microscope images have been registered with the corresponding mosaic reference microscope images) can then be registered to each other: this is scan-to-scan registration 655. This can be done, for example, by using a single-modal registration algorithm that registers the two mosaic reference microscope images 664 and 674 to each other. Using a single-modal registration algorithm is suitable, especially when the two mosaic reference microscope images have the same and specific contrast.

[0144] Figure 10 System 500 is illustrated schematically. System 500 includes an electronic data processing device 540. The electronic data processing device 540 may be, for example, a "personal computer" (PC). The electronic data processing device 540 includes a processor 542 and a memory 543. The electronic data processing device 540 also includes a communication interface 541. For example, the processor 542 may receive image data of microscope images via the communication interface 541. For example, the processor 542 may receive image data from a microscope 551 or an external image database (…). Figure 10 (Not shown) receives image data. However, processor 542 can also load image data from memory 543 and execute such image data. Processor 542 can send control commands to microscope 551 that trigger the capture of microscope images and / or set specific illumination geometry and / or specific defocus values ​​(e.g., for TIE, or DPC, or a combination thereof). Processor 542 can load program code from memory 543 and execute such program code. When the processor executes such program code, it causes processor 542 to perform techniques such as those described herein, such as: determining registration parameter values ​​by comparing microscope images; determining registration parameter values ​​by combining other registration parameter values; controlling microscope 551 to capture microscope images with and without fluorescence contrast; determining microscope images with phase contrast or phase gradient contrast; and so on.

[0145] In summary, techniques for resolving registration between two microscope images with different fluorescence contrasts (or, in general, different specific contrasts) have been described above using a reference microscope image that does not have specific contrast. For example, a digital phase-contrast image (e.g., via TIE or DPC) can be used as a reference for two-dimensional or three-dimensional co-registration of fluorescence channels. This is relevant because microscope images with different fluorescence contrasts may have lateral shifts due to imperfect adjustment of filters and beamsplitters, and axial shifts due to chromatic aberration. Furthermore, some microscopes operate in a mode that involves capturing all patches per channel first. This means that samples with lateral and axial shifts are fully scanned for each fluorescence contrast before measuring the next fluorescence channel. This mode is advantageous for minimizing mechanical wear on filter wheels and beamsplitter wheels. However, the multi-directional repeatability accuracy of mechanical sample scanners is limited, which can cause additional shifts between microscope images with different fluorescence contrasts.

[0146] Furthermore, the fact that various non-specific contrast ratios can be used has already been described above. In this regard, instead of digital phase contrast, phase gradient contrast can also be used on reference microscope images. Phase gradient contrast can be generated in a computationally particularly efficient manner, i.e., by calculating the pixel-by-pixel difference between two intensity images captured, for example, with different illumination geometries.

[0147] Furthermore, techniques have been described for registering microscopic images with specific contrast and reference microscopic images with non-specific contrast using multimodal registration algorithms. These can be patch images from different patch scans or patch images from a common multichannel recording. In particular, techniques using NGF-based registration algorithms have been described. This NGF-based multimodal registration algorithm is particularly efficient and can be implemented computationally efficiently even with limited computer hardware.

[0148] Elastic multimodal registration algorithms can be used. Elastic registration can also take into account nonlinear differences (e.g., deformation or distortion) between image pairs.

[0149] To further summarize, the following examples have been specifically disclosed.

[0150] Example 1. A computer-implemented method for registering a first microscope image (81, 91) with a second microscope image (82, 92), wherein the first microscope image images a sample with a first specific contrast, and wherein the second microscope image images the sample with the first specific contrast or with a second specific contrast. The method includes: - Obtain reference microscope images (85, 86, 99) that image the sample with non-specific contrast, such as phase-like contrast, and - Use the reference microscope image to perform registration between the first microscope image and the second microscope image.

[0151] Example 2. The computer-implemented method according to Example 1, wherein registering the first microscope image and the second microscope image includes: - Based on the comparison between the second microscope image (92, 82) and the reference microscope image (85, 99), determine (930) the registration parameter values ​​(72, 311, 312) used for the registration.

[0152] Example 3. The computer-implemented method according to Example 2, wherein registering the first microscope image and the second microscope image includes: - Based on a comparison of the first microscope image (91) with the reference microscope image (85, 99) or another reference microscope image, additional registration parameter values ​​(71, 311, 312) for the registration are determined (935) to image the sample with the nonspecific contrast.

[0153] Example 4. The computer-implemented method according to Example 3, wherein registering the first microscope image and the second microscope image includes: - Based on the combination of these registration parameter values ​​(72, 311, 312) and these additional registration parameter values ​​(71, 311, 312), determine (940) the resulting registration parameter values ​​for registering the first microscope image (91) with the second microscope image (92).

[0154] Example 5. A computer-implemented method according to any one of the foregoing examples, wherein the method further comprises: - Obtain an additional reference microscope image (86) that images the sample with the nonspecific contrast. The registration of the first microscope image (81, 91) and the second microscope image (82, 92) includes: - The registration parameter value (75) for the registration is determined by comparing the reference microscope image (85) with the other reference microscope image (86).

[0155] Example 6. The computer-implemented method according to Example 5, wherein registering the first microscope image and the second microscope image includes: - Determine additional registration parameter values ​​(76) for the registration based on a comparison between the first microscope image (81) and the reference microscope image (85), and / or - Determine additional registration parameter values ​​(77) for the registration based on the comparison between the second microscope image (82) and the other reference microscope image (86).

[0156] Example 7. A computer-implemented method according to any one of Examples 1 to 6, Wherein, the reference microscope image and the first microscope image are part of the same multichannel recording, and wherein, the reference microscope image and the second microscope image are not part of the same multichannel recording, and / or The sample is manipulated between capturing the first microscope image and capturing the second microscope image.

[0157] Example 8. A computer-implemented method according to any one of Examples 1 to 6, The first microscope image, the second microscope image, and optionally the reference microscope image are part of the same multichannel recording, and the sample is not manipulated between capturing the first microscope image and capturing the second microscope image.

[0158] Example 9. A computer-implemented method according to any one of the foregoing examples, Among them, this type of phase contrast is selected from the following group: phase contrast; digital phase contrast; digital phase gradient contrast.

[0159] Example 10. A computer-implemented method according to any one of the foregoing examples, wherein the method further comprises: - Obtain a first temporary reference microscope image and a second temporary reference microscope image, wherein the first temporary reference microscope image and the second temporary reference microscope image image the sample with the same intensity contrast and different defocus values, and - The reference microscope image is determined based on the pixel-by-pixel difference between the first temporary reference microscope image and the second temporary reference microscope image.

[0160] Example 11. A computer-implemented method according to any one of the foregoing examples, The registration parameter values ​​are determined by a registration algorithm that uses an index to determine the similarity measure between corresponding microscope image pairs. This registration algorithm is robust to brightness histogram inversion in the microscope image pairs, and / or The registration parameter values ​​for this registration are determined using a multimodal registration algorithm.

[0161] Example 12. A computer-implemented method according to any one of the foregoing examples, The registration parameter values ​​are determined by a registration algorithm selected from the following group: mutual information; normalized gradient field; structural similarity index; segmentation-based similarity measure; feature detection; and artificial neural network.

[0162] Example 13. A computer-implemented method according to any one of the foregoing examples, The registration parameter values ​​for this registration are determined using an elastic registration algorithm, and / or The registration parameter values ​​for this registration are determined using a landmark-related registration algorithm.

[0163] Example 14. A computer-implemented method according to any one of the foregoing examples, wherein the method further comprises: - Based on this registration, the first microscope image and the second microscope image are stitched together.

[0164] Example 15. An electronic data processing apparatus (540) configured to register a first microscope image (81, 91) with a second microscope image (82, 92), wherein the first microscope image images a sample with a first specific contrast ratio, and wherein the second microscope image images the sample with the first specific contrast ratio or with a second specific contrast ratio. The electronic data processing device includes a processor and a memory, wherein the processor is configured to load program code from the memory and execute the program code, and wherein, when executing the program code, the processor performs the following steps: - Obtain reference microscope images (85, 86, 99) that image the sample with non-specific contrast, and - Use the reference microscope image to perform registration between the first microscope image and the second microscope image.

[0165] Example 16. The electronic data processing apparatus according to Example 10, wherein, when the program code is executed, the processor performs the method according to any one of Examples 1 to 14.

[0166] It goes without saying that the features of the embodiments and aspects of the present invention described above can be combined with each other. In particular, without departing from the scope of the present invention, these features can be used not only in the described combinations, but also in other combinations or individually.

[0167] For example, techniques for processing and registering two-dimensional images have been described above. However, in principle, all the techniques described can also be used for three-dimensional images to correct lateral and axial translation errors (alignment of overlapping focus points due to mutual displacement, or longitudinal chromatic aberration), scaling errors (variable pixel size of the camera, or variable objective magnification), and rotation errors (camera rotation).

[0168] Furthermore, the techniques described above involve capturing microscopic images of samples with specific contrast. The samples can be, for example, cell samples or tissue samples.

Claims

1. A computer-implemented method for registering first microscopic images (81, 91, 611-613, 631-633, 661-663) with second microscopic images (82, 92, 621-623, 641-643, 671-672), wherein, The first microscope image images the sample using a first specific contrast ratio, wherein the second microscope image images the sample using either the first specific contrast ratio or a second specific contrast ratio. The method includes: - Obtain reference microscope images (85, 99, 614, 634, 664) that image the sample with non-specific contrast. - Use the reference microscope image to perform registration of the first microscope image with the second microscope image (651, 652, 655).

2. The computer-implemented method according to claim 1, in, Implementing this registration involves applying multimodal registration algorithms (651, 652). The non-specific contrast is an unlabeled contrast, which may be selected from the following group: phase-like contrast; phase contrast; bright field contrast; contrast due to oblique illumination; dark field contrast; autofluorescence contrast.

3. The computer-implemented method according to claim 2, in, The first microscopic image is associated with the first staining cycle (904, 910) of the sample. The second microscopic image is associated with the second staining cycle (904, 910) of the sample.

4. The computer-implemented method according to claim 3, in, The reference microscope image is associated with this first staining cycle of the sample. The method further includes: - Obtain additional reference microscopic images (86, 624, 644) that image the sample with this non-specific contrast and are associated with this second staining cycle. The implementation of this registration includes: - Compare the reference microscope image (85, 614, 624) with the other reference microscope image (86, 634, 644).

5. The computer-implemented method according to claim 4, in, Implementing this registration also includes: - Compare the first microscopic image (81, 611-613, 631-633) with the reference microscopic image (85, 614, 634), and / or - Compare the second microscope image (82, 621-623, 641-643) with the other reference microscope image (86, 624, 644).

6. The computer-implemented method according to claim 4, in, The first microscope image and the reference microscope image are inherently registered, and / or The second microscope image and the additional reference microscope image are inherently registered.

7. The computer-implemented method according to any one of claims 4 to 6, in, The first microscopic image is a first mosaic image (661-663) composed of first patch images (611-613, 621-623). The second microscope image is a second mosaic image (671-673) composed of second patch images (631-633, 641-643). The multimodal registration algorithm (651, 652) is applied between the first patch images (611-613, 621-623) and the corresponding reference patch images (614, 624) of the reference microscope image (664), and also between the second patch images (631-633, 641-643) and the corresponding additional reference patch images (634, 644) of the additional reference microscope image (674). The registration process further includes applying a single-modal registration algorithm (655) between the reference microscope image (664) and the other reference microscope image (674).

8. The computer-implemented method according to claim 3, in, The reference microscope image is associated with this first staining cycle of the sample. Implementing this registration includes: - Compare the second microscope image with the reference microscope image.

9. The computer-implemented method according to claim 8, in, The first microscope image and the reference microscope image are inherently registered.

10. The computer-implemented method according to any one of claims 1 to 4, in, The first microscopic image is the first patch image from the patch scan. The second microscope image is a second patch image scanned from the first patch image, and the second patch image partially overlaps with the first patch image. The reference microscope image is the third image of the scanned patch.

11. The computer-implemented method according to claim 10, in, For each location of the scanned patch, a corresponding reference microscope image can be obtained.

12. An electronic data processing apparatus (540) having a processor (542) and a memory (543), wherein, The processor (542) is configured to load program code from the memory and execute the program code, wherein the processor is configured, based on the execution of the program code, to obtain a reference microscope image (85, 99, 614, 634, 664) that images the sample with non-specific contrast, and to use the reference microscope image to perform registration (651, 652, 655) of a first microscope image and a second microscope image, wherein the first microscope image images the sample with a first specific contrast, and wherein the second microscope image images the sample with the first specific contrast or with a second specific contrast.

13. The electronic data processing apparatus according to claim 12, wherein, The processor is configured to perform the method according to any one of claims 1 to 11 based on the execution of the program code.

Images