Registration of microscopic images with specific contrast

Abstract

This provides registration for microscope images with specific contrast. [Solution] A technique is described for aligning a first microscope image (81) with a second microscope image (82). The first microscope image visualizes the specimen with a first specific contrast, and the second microscope image visualizes the specimen with either the first or second specific contrast. A reference microscope image (85) is used for registration, which visualizes the specimen with a non-specific contrast.

Description

[Technical Field]

[0001] Various examples of this disclosure relate to methods for aligning two microscope images having specific contrasts. [Background technology]

[0002] In biology, fluorescence-contrast optical microscopes are extremely important because they allow for molecular-level observation of the inside of cells and tissue sections using fluorescent markers. Particular interest lies in the colocalization and quantification of fluorescence concentrations, as this can provide information about the temporal progression of disease and its treatment with pharmaceuticals. Corresponding information can also be obtained using other specific contrasts that selectively stain specific molecules or cellular structures. Another example of a common specific contrast without fluorescent labeling is hematoxylin-erosine (H&E) staining.

[0003] In recent years, fluorescent dyes have become virtually freely selectable in terms of their spectral dye properties. For example, primary antibody markers that can be specifically adapted to tissue types and organelles are available. Flexible secondary antibodies can be bound to or very close to them, thereby allowing users to target and specify the spectral characteristics of the measured fluorescence signal.

[0004] In fluorescence microscopy, selected fluorescent dyes can be made to emit light using a characteristic excitation signature or at a characteristic excitation wavelength. Therefore, the corresponding microscopic image has fluorescence contrast. Fluorescence typically emitted at longer wavelengths can then be separated from the excitation light using a particularly selected color filter. By appropriate combinations with different color spectra of the excitation light, and by appropriately selecting specific color filters in the microscope's detection arm, multiple fluorescence contrasts and / or bright-field contrasts can be recorded sequentially, in parallel, or in a temporal context. For example, multi-channel recording can be obtained, in which case multiple microscopic images are taken sequentially with different contrasts, but no optical elements are exchanged, or only as few optical elements as possible, between the acquisition of individual microscopic images in the multi-channel recording. Generally, multiple microscopic images of a sample with different fluorescence contrasts are obtained in this way. Different fluorescence contrasts specifically mark different structures in the sample. Exemplary samples examined are cell samples or tissue section samples (e.g., in histopathology).

[0005] Microscopic images with different specific contrasts (different microscopic images may be stained with different fluorescent or non-fluorescent labels, e.g., H&E) often appear differently. An example is shown in Figure 1. Figure 1 shows microscopic image 91 of a cell sample imaged with a first fluorescence contrast and microscopic image 92 of the same cell sample imaged with a second fluorescence contrast different from the first fluorescence contrast. Both microscopic images 91 and 92 are obtained by their respective tile scans. In particular, it is conceivable that microscopic images 91 and 92 may be acquired with a common tile scan, in which case a corresponding multi-channel recording with two fluorescence contrasts from different channels is captured for each tile, and then the next scan step is performed.

[0006] Because the two fluorescence contrasts specifically label different types of cellular structures, structures that are particularly visible in microscopic image 91 are different from those that are particularly visible in microscopic image 92. These become particularly visible in the superimposed image 95 (corresponding to the superimposition of microscopic image 91 and microscopic image 92).

[0007] However, the different appearances of the two microscope images 91 and 92 are not necessarily due to the use of different fluorescence contrasts or, more generally, different specific contrasts. This is because the image planes of the two microscope images 91 and 92 may be displaced relative to each other. This means there is a systematic shift in the lateral (xy direction) or axial (z direction) direction. The different appearances arise not only from the lateral offset but also from different distortions and / or rotations of the microscope images relative to each other.

[0008] There are various reasons that can cause such displacement and / or distortion and / or rotation of the image plane. Different microscope images with different specific contrasts are generally acquired using different optical channels. Different optical channels have different optical beam paths. For example, in recording microscope images with fluorescence contrast, a real problem is that the color filters in the detection arm cause microscope images with different fluorescence contrasts relative to each other in the image plane to be displaced laterally due to misalignment. This is true even if different microscope images are part of the same multi-channel recording (although in this case it is typically less pronounced). Another reason why channels with different contrasts may be laterally offset is that so-called "tile scans" are recorded channel by channel, and the exact tile position is not reached again in consecutive scans. These tiles are then combined to form a mosaic image, which is sometimes called "stitching." Yet another reason is that a microscope may have multiple recording sensors (e.g., cameras) that image slightly different sample areas with slightly different optics. This means that multiple optical channels are used. In this regard, it is also conceivable to use multiple microscopes with different optical properties to record individual fluorescence channels. In addition to lateral offset (xy-plane), axial offset (z-direction) of different fluorescence channels can also occur, for example, due to chromatic aberration in the longitudinal direction of the optical system. Similarly, relative rotation of images can occur if they are recorded with different microscopes or different cameras. It may also be possible to manipulate the sample between microscopic image captures, i.e., stain or destain it, to sequentially reveal specific structures, in order to avoid interference between microscopic signals.

[0009] In order to evaluate the information contained in the microscope images 91 and 92 with high reliability, it is often desirable to perform image registration between microscope image 91 and microscope image 92, taking into account possible displacements, rotations, and / or distortions in the image plane. This registration quantifies or compensates for such displacements and / or distortions and / or rotations, for example, by identifying common image features in the microscope images. Therefore, such image registration makes it possible to evaluate only the cell biological properties of the cell sample that can be derived, for example, from a comparison of the two microscope images 91 and 92.

[0010] However, it has been found that performing direct image registration between microscope image 91 and microscope image 92, or between microscope images with generally different fluorescence contrasts, does not provide reliable results.

[0011] Therefore, prior art discloses methods for performing corresponding registration between microscope images 91 and 92 indirectly or with the use of assistive means. For example, it is possible to capture another microscope image with yet another fluorescence contrast labeling similar cell structure types. However, registration by additional staining has a major drawback: the reference fluorescent dye fades with prolonged exposure. Ideally, it would be desirable to use a method that allows the sample to reference the reference channel, and indeed other fluorescence channels, without fading. Increased fading eliminates the reference of the sample, making repeated measurements and prolonged experiments impossible. In addition, reference staining can degrade the quality of the fluorescence channel of interest. For example, additional fluorescence staining can cause spectral overlap with other channels, resulting in spectral separation requiring complex calculations. Furthermore, the reference stain may glow more brightly than the weakly emitting region of the stain of interest. The microscope must also be equipped with additional excitation wavelengths (e.g., additional color LEDs or laser diodes) and, on the hardware side, additional color filters / beam splitters. Additional dyes are required in the sample itself. Overall, this leads to increased system and sample fabrication costs. Due to the low fluorescence yield (ratio of fluorescence emission to excitation emission), acquiring fluorescence channels typically takes longer than with transmission light modalities such as bright-field imaging. Therefore, the acquisition time during experiments increases significantly with the addition of fluorescence channels.

[0012] Another technique is known from (Patent Document 1), which discloses the registration of two microscope images with fluorescence contrast using a bright-field image as a reference. Such techniques have the drawback that some structures in the bright-field image are either not visible or only faintly visible.

[0013] Further techniques are known from (Patent Document 2), which describes how a first image of a biological sample on a first substrate and a second image of a biological sample on a second substrate are obtained. The second substrate has one or more spatial reference points. Then, registrations of the first and second images can be identified, and the registration is performed using patterns in the two images. Subsequently, using this registration, the first image can be overlaid on a spatial dataset containing spatial analysis data, and the reference system of the spatial dataset can be determined with respect to the second image based on the spatial reference points. Such techniques have the drawback that a specific transparent substrate must be used along with the spatial reference points (called references). This is relatively costly and complex. Prior knowledge of how spatial reference points appear in microscopic images must be available. Corresponding markings must be found using an appropriate image processing algorithm. [Prior art documents] [Patent Documents]

[0014] [Patent Document 1] U.S. Patent Application Publication No. 2012 / 0257811 [Patent Document 2] International Publication No. 2023 / 044071A1 Pamphlet [Patent Document 3] European Patent No. 24184623.7 [Patent Document 4] German Patent Application Publication No. 102015208084A1 Specification [Patent Document 5] International Publication No. 2021 / 198241A1 Pamphlet [Non-patent literature]

[0015] [Non-Patent Document 1] Streibl, Norbert. “Phase imaging by the transport equation of intensity.” Optics communications 49.1 (1984): 6 - 10 [Non - Patent Document 2] Mehta, Shalin B., and Colin JR Sheppard. “Quantitative phase - gradient imaging at high resolution with asymmetric illumination - based differential phase contrast.” Optics letters 34.13 (2009): 1924 - 1926 [Non - Patent Document 3] Haber, Eldad, and Jan Modersitzki. “Intensity gradient based registration and fusion of multi - modal images.” Methods of information in medicine 46.03 (2007): 292 - 299 [Summary of the Invention] [Problems to be Solved by the Invention]

[0016] Therefore, an improved technique for identifying registration parameter values for the registration of microscopic images that image the structure of a specimen with specific contrast is required. [Means for Solving the Problems]

[0017] This object is achieved by the features of the independent patent claims. The features of the dependent claims define embodiments.

[0018] With the technology described in this specification, channels with specific contrast, particularly channels that label a structure that is completely complementary to the specimen, i.e., channels with different specific contrasts, can be clearly aligned with each other, for example, aligned in two dimensions (2D) or three dimensions (3D).

[0019] The corresponding microscope images can be part of a common multi-channel recording. For example, the microscope images can be tile images at specific tile positions of a tile scan. The corresponding multi-channel recording can be captured at each tile position.

[0020] However, it is also conceivable that the microscope images are part of different multi-channel recordings. For example, the microscope images can be tile images adjacent to the positions of a common tile scan, and different multi-channel recordings are captured at different tile positions.

[0021] It is also conceivable that the specimen has to be manipulated between taking the corresponding microscope images, i.e., for example, stained and / or de-stained during a staining cycle. In such cases, the microscope images can be, for example, mosaic images each composed of a plurality of tiles.

[0022] The technology described in this specification can be used to recover a specific region of interest (ROI) of a sample, for example, captured using different imaging modalities and / or at different microscope magnifications. The technology described in this specification can be used to perform reliable stitching of microscope images, which means that a plurality of smaller, overlapping specimen images can be fused to form a larger mosaic image.

[0023] The techniques described herein do not require reference staining or substrate-based references / landmarks / reference points. The techniques described herein enable particularly accurate registration between microscopic images with different fluorescence contrasts / different specific contrasts. In particular, the techniques described herein are also robust to displacement, distortion, or rotation of the image planes of the two microscopic images relative to each other.

[0024] A computer implementation method is disclosed. This method is used to align a first microscopic image with a second microscopic image. The first microscopic image visualizes the specimen with a first specific contrast, and the second microscopic image visualizes the same specimen with either the first specific contrast or the other second specific contrast.

[0025] For example, a specimen can be a cell specimen or a tissue section specimen. A specimen may contain cellular structures.

[0026] For example, a first specific contrast specifically labels at least one first cellular structure type. A second specific contrast labels at least one second cellular structure type. The at least one first cellular structure type is, for example, at least partially different from the at least one second cellular structure type. In other words, this therefore means that the first specific contrast labels at least one cellular structure type that is not similarly labeled by the second specific contrast. Overlap of labeling between different specific contrasts can occur even if the first specific contrast is different from the second specific contrast.

[0027] For example, the first specific contrast can be a specific fluorescence contrast, or it can be obtained by a non-fluorescent label (e.g., H&E).

[0028] The second specific contrast can be a specific fluorescence contrast, or it can be obtained by a non-fluorescent label (e.g., H&E).

[0029] The first and second microscope images may either completely overlap or have at least one overlapping region, i.e., they may at least partially image the same region of the specimen.

[0030] The first microscope image can be either a tiled image or a mosaic image.

[0031] The second microscope image can also be represented as either a tiled image or a mosaic image.

[0032] If the first microscopic image is a tiled image and the second microscopic image is also a tiled image, they can be taken at the same tile position in a common tile scan or at different tile positions in a common tile scan. Alternatively, the first microscopic image can be part of the first tile scan and the second tiled image can be part of another second tile scan, and these can be optionally associated with different staining cycles of the specimen. The first and second microscopic images can be taken at corresponding tile positions in the two tile scans.

[0033] The method includes acquiring a reference microscope image, which images the specimen with nonspecific contrast. In addition, the method includes performing registration of the first microscope image and the second microscope image using the reference microscope image.

[0034] The reference microscope image can be a mosaic image (if the first and second microscope images are mosaic images). However, the reference microscope image can also be a tiled image (if the first and second microscope images are tiled images). For example, a tile scan can be performed in each case, in which one or more microscope images and their associated reference microscope images can be taken at multiple tile locations. For example, the first microscope image, the second microscope image, and the reference microscope image can all be taken at the same tile location and be part of a common multi-channel recording.

[0035] Non-specific contrast should be distinguished from primary and secondary specific contrasts. Non-specific contrast does not label specific molecules or cellular structures, which distinguishes it from primary and secondary specific contrasts. Typically, staining of the specimen is not necessary to obtain non-specific contrast. Non-specific contrast can be a label-free contrast.

[0036] For example, nonspecific contrast can be selected from the following group: phase-like contrast, phase contrast, bright-field contrast, oblique incidence illumination, dark-field contrast, and autofluorescence contrast.

[0037] The non-specific contrast of the reference microscope image allows for the observation of different labeled cell structure types in the reference image than in either the first or second microscope image. This makes reference microscope image-based registration robust and reliable, based on image features. The lack of fluorescence contrast in the reference microscope image further reduces exposure of the cell specimen. Reduced exposure of cells to fluorescent dyes lessens the impact of the measurement on cell biology. For example, performing registration can involve applying a multimodal registration algorithm. In particular, multimodal registration algorithms can be applied between microscope images with different contrasts.

[0038] For example, a multimodal registration algorithm can be applied between a microscope image with specific contrast and a (reference) microscope image with nonspecific contrast.

[0039] For example, a multimodal registration algorithm can be applied between different tile images in a tile scan with overlapping areas. It would also be possible to run a multimodal registration algorithm between images from different (i.e., sequential) tile scans.

[0040] For example, the first microscopic image can be associated with the first staining cycle of the specimen. For instance, the first staining cycle can stain the specimen with at least one non-fluorescent label. This non-fluorescent label can define the first specific contrast of the first microscopic image. For example, H&E can be used as the non-fluorescent label. However, it is also conceivable to stain the specimen with a fluorescent label in the first staining cycle.

[0041] The second microscopic image can be associated with a second staining cycle of the specimen. For example, in the second staining cycle, the specimen can be stained with at least one non-fluorescent label. This non-fluorescent label can define a second specific contrast in the second microscopic image. For example, H&E can be used as a non-fluorescent label. In the second staining cycle, the specimen can also be stained with one or more fluorescent labels. For example, only fluorescent labels can be used in the second staining cycle, i.e., no non-fluorescent labels are used as in the first staining cycle. One of the one or more fluorescent labels in the second staining cycle can define a second specific contrast in the second microscopic image.

[0042] If different fluorescence contrasts, i.e., multiple different fluorescent labels, are used in the second staining cycle, multi-channel recording can be performed to capture different fluorescence contrasts at different wavelengths.

[0043] For example, the second staining cycle can be performed after the first staining cycle.

[0044] Then, the reference microscope image can be associated with the first staining cycle of the specimen. This means that the reference microscope image is captured as a multi-channel recording, for example, together with the first microscope image. For example, the first microscope image and the reference microscope image could each be a mosaic image from a common tile scan.

[0045] Reference microscope images can be taken before the second staining cycle is performed.

[0046] In some cases, the method further involves acquiring another reference microscope image. This other reference microscope image can image the specimen with the same specific contrast as the reference microscope image when imaging the specimen. However, it is also conceivable that the other reference microscope image may use a different non-specific contrast than that of the reference microscope image.

[0047] Another reference microscope image can be associated with the second staining cycle. For example, another reference microscope image can be acquired together with the second microscope image as a multi-channel recording. For example, the second microscope image and the other reference microscope image could each be mosaic images from a common tile scan.

[0048] Therefore, this means that in one variation, there are two reference microscope images with nonspecific contrast. Then, it becomes possible to compare or align the first microscope image with the reference microscope image, and in addition, the second microscope image can also be compared and aligned with yet another reference microscope image. Furthermore, it is possible to compare or align one reference microscope image with yet another reference microscope image.

[0049] However, such comparisons between the first microscope image and the reference microscope image, and between the second microscope image and yet another reference microscope image, are optional. It is possible that the first microscope image is already inherently aligned with the reference microscope image, and / or that the second microscope image is already inherently aligned with yet another reference microscope image. For example, if the first microscope image and the reference microscope image were recorded as part of a common multi-channel recording or sequentially, the microscope would be able to record exactly the same portion again (e.g., without moving the stage between recordings and without beam splitter / filter changes causing offsets between images). Then only the reference microscope image would be compared with yet another reference microscope image from the next staining cycle, or (if yet another reference microscope image does not exist) with the second microscope image.

[0050] The various techniques disclosed are typically based on the finding that the imaging differences of specimens are relatively small, on the one hand, between a first microscopic image and a reference microscopic image, and on the other hand, between a second microscopic image and yet another reference microscopic image. This is because each of these image pairs is associated with the same staining cycle and can be acquired, for example, as part of a common multi-channel recording, particularly by a common tile scan.

[0051] For example, the first microscopic image could be a first mosaic image composed of the first tile images. The second microscopic image could be a second mosaic image composed of the second tile images. In particular, the first and second tile images can be captured in different tile scans because they are associated with different staining rounds. The multimodal registration algorithm can then be performed between the first tile image and each of the reference tile images of the reference microscopic image, and again between the second tile image and each of the yet another reference tile images of yet another reference microscopic image. For example, a reference tile image may exist for each tile position in each tile scan. In other words, this is therefore the multimodal registration algorithm is applied twice: once for the first tile scan (which captures the first tile image and each of the reference tile images) and once for the second tile scan (which captures the second tile image and each of the yet another reference tile images). The multimodal registration algorithm can therefore be applied at the tile hierarchy level.

[0052] As mentioned above, in not all variations, a comparison is necessarily made between the first tile image and each of its reference tile images, and / or between the second tile image and each of its other reference tile images. This is the case when the corresponding image pairs are already inherently aligned.

[0053] Whether a multimodal registration algorithm is applied at the tile level, or whether different tile images are already inherently aligned at the tile level, performing registration may involve applying a unimodal registration algorithm. A unimodal registration algorithm can be applied between one reference microscope image and another (i.e., after stitching at the mosaic level).

[0054] However, it is also possible to perform stitching only after applying a unimodal registration algorithm. In such a case, for example, by combining the registration parameter values ​​of two multimodal registration algorithms with the registration parameter values ​​of a unimodal registration algorithm, specific contrast can be obtained by aligning each tile of the first tile scan with each tile of the second tile scan.

[0055] For example, performing registration of the first and second microscope images may involve determining registration parameter values ​​for registration based on a comparison of the second microscope image with a reference microscope image. At the same time, it is also possible that the first microscope image and the reference microscope image are already inherently aligned, and therefore, a comparison between the first microscope image and the reference microscope image is unnecessary. This is useful, for example, when the second microscope image and the reference microscope image are not acquired in the same temporal context, i.e., they are not part of the same multichannel recording or the same tile scan. For example, the first and second microscope images may be associated with different staining cycles. In addition, this is useful when the first and second microscope images are acquired using significantly different optical channels, for example, optical channels with different objective lenses.

[0056] An optical channel generally describes the specifications of optical imaging from object space to image space. Therefore, an optical channel describes the optical imaging of a specimen from a microscope to a camera. The optical channel is determined by the optical components used, such as the objective lens and filters.

[0057] Multiple microscope images can collectively be part of a multi-channel recording. This means that multiple microscope images are captured in a temporal context, and that as many optical components as possible remain unchanged between the acquisition of microscope images in a multi-channel recording. For example, between the acquisition of different microscope images in a multi-channel recording, only a color filter can be inserted into or removed from the beam path.

[0058] Generally, registration parameter values ​​may indicate displacement in the x-direction, displacement in the y-direction, rotation, scaling and / or shearing, or deformation, distortion, compression, etc., between two images.

[0059] When the first microscope image and the reference microscope image are part of the same multi-channel recording, typically the optical channels used are very similar, and the specimen is not manipulated between the acquisition of the first microscope image and the acquisition of the reference microscope image (i.e., the first microscope image and the reference microscope image are acquired within a narrow temporal context). In particular, as already explained, the first microscope image and the reference microscope image can be inherently aligned when they are part of the same multi-channel recording, meaning that the registration parameter values ​​represent an identity map or are at least static and known in advance. Then it may become unnecessary to identify the corresponding registration parameter values ​​by comparing the first microscope image with the reference microscope image. The first microscope image and the second microscope image can also be directly aligned with each other by identifying the registration parameter values ​​by comparing the second microscope image with the reference microscope image.

[0060] However, sometimes, even if the first microscope image and the reference microscope image are part of the same multi-channel recording, proper alignment may not be achieved. For example, chromatic aberration from different color filters used to select different wavelengths may cause displacement, distortion, etc., between the two microscope images. In such cases, a multimodal registration algorithm can be used, for example, as described above.

[0061] However, it is possible that the reference microscope image and the first microscope image are not part of the same multi-channel recording. For example, the reference microscope image and the first microscope image may be acquired at different times or with respect to completely different optical channels. For example, the reference microscope image and the first microscope image may be associated with different tile scans. In such cases, in particular, another registration parameter value for registration can be identified based on a comparison between the first microscope image and the reference microscope image (for example, by applying a multimodal registration algorithm). This also aligns the first and second microscope images with each other by identifying the registration parameter value by comparing the second microscope image with the reference microscope image. Then, for example, by combining the registration parameter value with another registration parameter (again, without considering image features), an imaging specification, so-called "resulting registration parameter value," that specifies the final registration of the first and second microscope images can be obtained.

[0062] The above describes an example in which a comparison (to determine registration parameter values) is performed between at least one microscope image with specific contrast and a reference microscope image with nonspecific contrast. In some examples, such a comparison may be performed between two reference microscope images with nonspecific contrast, or only between them. For example, the registration parameter values ​​for registration between the first and second microscope images may be determined based on a comparison between the reference microscope image and another reference microscope image that also images the specimen with nonspecific contrast.

[0063] For example, the reference microscope image and the first microscope image may be part of the same multichannel recording, and another reference microscope image and the second microscope image may be part of the same (other) multichannel recording. However, on the other hand, the two reference microscope images may be taken at different times, for example, after specimen processing in a staining cycle, and / or with different optical channels, for example, with different microscopes. The two reference microscope images can therefore be part of different multichannel recordings. Thus, the first microscope image and the second microscope image become part of different multichannel recordings. In situations where different microscope images of a particular multichannel recording are inherently aligned, it is unnecessary to perform yet another comparison between the first microscope image and the reference microscope image (because these two microscope images are already inherently aligned), and in addition, it is unnecessary to perform yet another comparison between the second microscope image and yet another reference microscope image (because these two microscope images are also already inherently aligned). In other words, this means that registration between the first and second microscope images can therefore be obtained by determining the registration parameter values ​​based on a comparison of the two reference microscope images.

[0064] When multiple reference microscope images exist, they may have the same nonspecific contrast, such as a specific digital phase contrast.

[0065] In one variation, yet another registration parameter value is identified by comparing a reference microscope image with a first microscope image, and / or yet another registration parameter is identified by comparing yet another reference microscope image with a second microscope image. Such scenarios are useful, for example, when the reference microscope image and the first microscope image are not acquired in the same optical channel and / or in the same temporal context, i.e., they are not part of the same multichannel recording (the same applies to yet another reference microscope image and a second microscope image). In addition, yet another identification of such registration parameter values ​​may also be useful if there is chromatic aberration in the optical channel used to acquire each pair of reference and microscope images of the same multichannel recording.

[0066] For example, nonspecific contrast in a reference microscope image may be phase-like contrast. Phase-like contrast can include, for example, phase contrast. Examples include Zernike phase contrast and Nomarski phase contrast. In this case, specific optical elements are used in the light beam path, such as a phase ring in the objective lens and an annular aperture of a condenser lens. In this way, interference between background and object light can be made visible. Image contrast can be enhanced by using phase contrast. This means that cellular structures are particularly visible. Cells are phase objects in which the amplitude of light does not decrease at all or significantly as the light passes through the cell specimen, and therefore phase contrast is preferred to make the phase shift visible.

[0067] However, digital phase contrast can also be used as phase-like contrast. Here, multiple images are recorded and then computationally combined to form a single phase contrast image. Therefore, such a technique can be called digital phase contrast. Phase contrast is obtained by digital post-processing of recorded intensity images. Examples include the transport of intensity equation (TIE) and differential phase contrast (DPC). TIE is described in (Non-Patent Literature 1). DPC is described in (Non-Patent Literature 2). To record a TIE dataset, the sample is displaced along the optical axis (z-direction), i.e., displaced axially, and a z-stack consisting of at least two images is recorded. The data is then computationally combined to obtain a phase contrast image. For this purpose, a diffuse partial differential equation is solved. In DPC, the sample is illuminated from at least two different directions (oblique incidence illumination), while the sample remains in a fixed z-position. All types of segmented light sources can serve as light sources for oblique incidence illumination, including segmented diodes, light-emitting diode arrays, digital micromirror devices (DMDs), liquid crystal displays (LCDs or SLMs), or variable aperture diaphragms of condenser lenses. The recorded data is then converted into a phase contrast image by solving a deconvolution problem. Combinations of TIE and DPC are also conceivable, as described, for example, in Patent Document 3 dated June 26, 2024. The use of digital phase contrast (as opposed to hardware-based phase contrast) has the advantage that it does not require the insertion or removal of light from the beam path of complex objects when capturing the digital phase contrast. Rather, the illumination can be varied in a targeted manner, for example, by a switchable light-emitting diode array placed in the illumination pupil plane. This can be done quickly and easily.

[0068] However, it is also conceivable to use digital phase gradient contrast instead of phase-like contrast. For example, corresponding techniques for various oblique incidence illuminations are described in (Patent Document 4). Intensity images of cell specimens are taken from different illumination directions, and then the difference between the intensity images is calculated. For example, a normalized difference can be calculated. Such phase gradient contrasts are particularly easy to calculate. For example, deconvolution is not required.

[0069] For example, a first time-reference microscope image and a second time-reference microscope image can be obtained. Both the first and second time-reference microscope images can image the cell culture using the same intensity contrast, but with different defocus values ​​and / or illumination geometric conditions. The reference microscope image can then be identified based on the generation of a pixel-by-pixel difference image between the first and second time-reference microscope images. Normalization can be performed by choice. Phase gradient contrast can be generated based on such microscope images, using the captured intensity contrast in combination with different defocus values ​​and / or illumination geometric conditions. Generating a pixel-by-pixel difference image between two time-reference microscope images is a simple computational operation and can be performed particularly quickly without requiring large computational resources. Therefore, the corresponding technique is particularly suitable for real-time or near-real-time registration.

[0070] For example, specific registration parameter values ​​for affine transformations between corresponding image pairs can be determined by the techniques described herein. Affine transformations include translation, rotation, scaling, and shearing of matrix forms. However, nonlinear deformations using elastic registration algorithms are also conceivable. When elastic registration algorithms are used, for example, a cell specimen may be changed between the acquisition of different microscopic images, and such changes may appear in the different morphologies of individual cells. Such changes in the geometry of the structure can be taken into account by corresponding strains or deformations with respect to the elastic registration algorithm.

[0071] As already explained, multimodal registration algorithms can be used in a variety of examples. Such multimodal registration algorithms are robust to the inversion of luminance histograms between the compared microscope images. This is particularly useful when a reference microscope image with non-specific contrast (e.g., phase-like contrast) is compared to a microscope image with specific contrast (e.g., fluorescence contrast or H&E contrast).

[0072] For example, registration algorithms selected from the following groups can be used: mutual information, normalized gradient fields, structural similarity index, segmentation-based similarity index, feature detection, and artificial neural networks. For example, a multimodal registration algorithm using one of the similarity indexes mentioned above allows a first or second microscope image (each with specific contrast, e.g., fluorescence contrast) to be compared with a reference microscope image (with non-specific contrast, e.g., no fluorescence contrast, but phase-like contrast). Typically, fluorescence contrast makes a particular pair of cells in a corresponding specimen appear particularly bright. However, because cells are phase objects that do not significantly attenuate the amplitude of transmitted light, the same part of the cell may appear darker against the background in the reference microscope image. Multimodal registration algorithms can take into account such inversions of the brightness histogram.

[0073] Landmark-based registration algorithms can also be used. In a landmark-based registration algorithm, specific characteristic landmarks, i.e., characteristic structures or patterns found in both images, are searched for by an appropriate object recognition algorithm. For example, predefined structures can be searched for, that is, structures whose presence is expected in the image and whose appearance in the image is known to some extent in advance. However, it is also possible to use an object recognition algorithm that does not include any explicit indication of the type of structure to be searched for. Such landmark-based registration algorithms can be particularly robust with respect to changes in luminance values. This is because they do not focus so much on the luminance of different pixels, but rather on specific real-space patterns.

[0074] An electronic data processing device is described. The electronic data processing device is used to align a first microscope image with a second microscope image. The electronic data processing device includes a processor and memory, the processor being configured to load program code from memory and execute it. Once the program code is executed, the processor performs the steps of the method described above.

[0075] The features described above and those described below can be used not only in the specified corresponding combinations, but also in other combinations or individually, and this does not deviate from the scope of protection of the present invention. [Brief explanation of the drawing]

[0076] [Figure 1] The first microscope image with a primary fluorescence contrast, the second microscope image with a secondary fluorescence contrast, and a superimpose of the first and second microscope images are shown. [Figure 2] Figure 1 shows a phase-contrast image of the cell specimen. [Figure 3] This is a flowchart illustrating one example method. [Figure 4] This is a flowchart illustrating one example method. [Figure 5] The distance functions of NGF registration algorithms are shown for various examples. [Figure 6] The registration between two microscope images with different specific contrasts is schematically shown using various examples. [Figure 7] The registration between two microscope images with different specific contrasts is schematically shown using various examples. [Figure 8] The registration between two microscope images with different specific contrasts is schematically shown using various examples. [Figure 9] Various examples illustrate tile-to-tile registration and scan-to-scan registration. [Figure 10]A schematic diagram of a system containing data processing equipment and data sources is provided using various examples. [Modes for carrying out the invention]

[0077] The aforementioned characteristics, the features and advantages of the present invention, and the methods by which they are realized will become clearer and easier to understand when considered in relation to the following description relating to exemplary embodiments, which are described in more detail with the drawings.

[0078] The present invention will be described in more detail below with reference to the drawings, based on preferred embodiments. In the drawings, the same reference numerals refer to the same or similar elements. The drawings are schematic representations of various embodiments of the present invention. The elements shown in the drawings are not necessarily drawn to the correct scale. Rather, the various elements shown in the drawings are depicted in a manner that makes their function and general purpose clear to those skilled in the art. The functional units and connections and links between elements shown in the drawings can also be implemented as indirect connections or links. The connections or links can be implemented wired or wireless. Functional units can be implemented as hardware, as software, or as a combination of hardware and software.

[0079] A method for aligning two microscope images is disclosed below. The two microscope images may have different specific contrasts, or they may have the same specific contrast. The two microscope images may be part of a common multichannel recording or part of different multichannel recordings. The two microscope images may each be a mosaic image or a tiled image. The two microscope images may be associated with different tile scans or with the same tile scan.

[0080] In these various examples, no direct registration is observed based on image comparison between the two microscope images; rather, one or more "auxiliary registrations" are identified based on one or more reference microscope images, for example, having nonspecific contrast. Thus, the two microscope images can be represented as different fluorescence channels within a single microscope image.

[0081] For example, consider Figure 1, which shows a microscopic image 91 with a first fluorescence contrast (as an example of a first nonspecific contrast) and a microscopic image 92 with a second fluorescence contrast (as an example of a second nonspecific contrast). A related reference microscopic image 99 with phase contrast is shown in Figure 2, where different cellular structure types can all be seen together. The two microscopic images 91 and 92 may be displaced laterally with respect to each other, or even nonlinearly deformed and / or rotated. Such and other registration parameters can be quantified in the registration to enable a broader evaluation.

[0082] In one exemplary modification, microscope images 91 and 92 are first individually aligned with a reference microscope image 99 by corresponding multimodal registration. Initially, microscope images 91 and 92 are displaced and / or rotated and / or otherwise distorted with respect to the reference microscope image 99. If the multimodal registration is successful, all microscope images 91, 92, and 99 are aligned with respect to each other both axially and transversely. Possible rotation errors, etc., are corrected in the same way.

[0083] Registration of the reference microscope image 99 for the microscope images 91 and 92 can be performed on the specimen after multichannel recording (in which case the two microscope images 91 and 92 are acquired simultaneously or at least in rapid succession, i.e., both microscope images 91 and 92 are part of the same multichannel recording), and also during the process of updated multichannel recording after intermediate specimen processing, such as periodic staining (staining cycle) of the specimen. In this case, repeated recording of the reference microscope image can be performed during the process of updated multichannel recording, and this image can then be used to match the phase contrast image of the first staining round with both this and / or another staining round's fluorescence channels. This, therefore, means that each of the multiple multichannel recordings has its own reference microscope image. For example, each of the multichannel recordings can be acquired by tile scanning. It may also be advantageous to use elastic image registration, for example, when the specimen is deformed by mechanical effects during the imaging step.

[0084] Figure 3 is a flowchart of one exemplary method. The method in Figure 3 can be performed by an electronic data processing device. For example, the method in Figure 3 can be performed by a processor, which loads program code from memory and executes it. The method in Figure 3 is therefore a computer implementation method.

[0085] The method shown in Figure 3 is used to acquire a microscope image and one or more reference microscope images. The reference microscope images have nonspecific contrast, such as phase-like contrast.

[0086] Microscope images have different specific contrasts, such as different fluorescence contrasts. For example, different dyes and / or different wavelengths can be used for fluorescence excitation. Different labels can be used.

[0087] Microscopic images with different specific contrasts can be acquired in one or more "staining cycles," i.e., consecutive iterations 904, in each of which the specimen is stained with a corresponding (e.g., fluorescent) dye or label in box 905, and / or a specific dye or label is removed. For example, multiple microscopic images can be acquired for multi-channel recording for each iteration 904, i.e., different iterations correspond to different multi-channel recordings. For each iteration 904, each tile scan can be performed, in which, for example, a multi-channel tile image is acquired for each tile (i.e., at each tile position).

[0088] In box 905, the specimen is manipulated, i.e., stained. For example, the specimen can be stained with one or more dyes. Different dyes specifically label certain types of cellular structures, for example, with one or more fluorescent labels and / or one or more non-fluorescent labels. For example, the first dye labels organelles, and the other dyes label the cell membrane or the cell nucleus. Different fluorescent dyes can be excited by light of different wavelengths and / or emit fluorescence at different wavelengths.

[0089] Other operations besides staining are also possible. For example, specimens can be destained alternatively or additionally. This means that specific fluorescent dyes are removed.

[0090] Box 910 involves taking one or more microscopic images, each having nonspecific contrast. If fluorescent labels are used, for this purpose, light is typically used to excite the fluorescence of the fluorescent dye used to stain the specimen in one or more previous iterations of Box 905. Autofluorescence may also be used.

[0091] When multiple microscope images are taken with Box 910, they can be taken using different optical channels, and therefore, there may be lateral offset, distortion, and / or rotation relative to each other. For example, a beam splitter or filter can cause offset when modified. The optical system may also have slightly different magnifications for each color, or other chromatic aberrations may occur.

[0092] Multiple microscope images can be part of a multi-channel recording. This means, in principle, that the same optical channel is used each time, with only, for example, the color filter being changed between each image capture.

[0093] Microscope images can be converted into so-called mosaic images, where each tile is scanned and then combined.

[0094] Box 915 includes capturing image data that corresponds to, or is based on, a reference microscope image, which can later be used to identify the reference microscope image.

[0095] For example, a reference microscope image may be part of the same multichannel recording from which one or more microscope images are taken in box 905 within the same iteration 904. For example, a reference microscope image may be part of the same tile scan used to take one or more microscope images in box 905 within the same iteration 904.

[0096] For example, hardware-based phase contrast can be captured. Then, in box 915, the reference microscope image can be directly acquired with this phase contrast, such as Zernike phase contrast or Nomarski phase contrast. Other examples include brightfield contrast or darkfield contrast, or autofluorescence contrast. However, it is also possible to acquire multiple intensity images in box 915, each with different defocus values ​​and / or different illumination geometry conditions (contrast due to oblique incidence illumination). Then, based on such temporal reference microscope images, the reference microscope image can later be identified with digital phase contrast or digital phase gradient contrast. This means that the temporal reference microscope image is further processed to obtain the (final) reference microscope image. More details will be provided later with respect to box 925.

[0097] In some examples, as already described, the (reference) microscope images taken in box 910 and / or box 915 may consist of multiple tiles, i.e., mosaic images. Here, the microscope used to take the microscope images may have an electric specimen holder. For example, an electric microscope stage may be used, which can be translated, for example, in two or three dimensions. In such a case, each tile image (e.g., a multichannel image) is taken at a specific position on the electric specimen holder, and then the electric specimen holder is moved to the next position, where yet another tile image (e.g., a multichannel image) is taken. This is repeated until a given specimen area is covered with tile images. Since each different tile image may have an overlap with respect to each other, registration between adjacent tile images is possible in this overlap. The resulting microscope image can then be synthesized from the tile images (so-called "stitching"), i.e., a mosaic image. The above describes the case where multiple microscope images are taken in a common tile scan, and therefore this means that each of the different tile images is a multichannel recording. In other examples, even in a single iteration 904, i.e., for a single staining cycle, multiple tile scans may be performed sequentially to acquire different microscopic images with specific and / or nonspecific contrasts. As has been known, there may be significant offset and / or rotation between the microscopic images of different tile scans, especially when different tile scans are used to acquire different microscopic images. Box 920 includes determining whether another staining or destaining or other operation of the specimen is intended. This means determining whether another staining cycle is intended to be performed. If another staining or destaining is intended to be performed, another iteration 904 of Box 905 is performed.Subsequently, one or more additional microscopic images with one or more corresponding specific contrasts can be taken in box 910, and if appropriate, one or more additional reference microscopic images can be taken in box 915.

[0098] When microscope images are taken in different iterations (904), they have a temporal offset relative to each other (they lack a common temporal context), and such a temporal offset can be converted into a lateral offset between the corresponding acquired microscope images by various temporal drifts. In addition, the specimen may have been processed during that time, and therefore the appearance of the specimen may also have changed.

[0099] In principle, it is not always necessary to perform box 915 in every iteration 904. For example, data for a reference microscopy image may be captured in box 915 only in the first iteration 904, i.e., in one staining cycle, and not in subsequent iterations. Alternatively, multiple iterations 904 may be performed with box 915, resulting in multiple reference microscopy images, each used for registration of the microscopy image with specific contrast for other iterations.

[0100] Figure 3 illustrates the techniques related to the acquisition of microscope images. In the various examples described here, it is conceivable that, in principle, various microscope images have already been acquired and stored in a database at an earlier stage. In this regard, the method in Figure 3 is, in principle, based on arbitrary selection.

[0101] Figure 4 is a flowchart of one exemplary method. The method in Figure 4 can be performed by an electronic data processing device. For example, the method in Figure 4 can be performed by a processor when it loads program code from memory and executes it. The method in Figure 4 is therefore a computer implementation method.

[0102] The method shown in Figure 4 is used to align the first and second microscope images. The first microscope image has, for example, a first specific contrast, which differs from the second specific contrast of the second microscope image.

[0103] For example, the first and second microscope images and one or more reference microscope images can be acquired by the method shown in Figure 3. In this regard, the method shown in Figure 4 can be performed after the method shown in Figure 3, or it can be performed so as to partially overlap in time with the method shown in Figure 3.

[0104] Box 925 optionally includes generating one or more reference microscope images having digital phase contrast or digital phase gradient contrast (if these do not already exist). For example, multiple temporal reference microscope images taken in iteration 904 of Box 915 (see Figure 3) may be combined with each other. Techniques such as those known for DPC or TIE can be used for this purpose. Pixel-by-pixel difference image generation can also be performed to obtain a corresponding reference microscope image with phase gradient contrast. If a reference microscope image with phase contrast has already been taken directly in Box 915, it is not necessary to run Box 925.

[0105] In all variations, it is not always necessary to have one or more reference microscope images with phase-like contrast available. Other types of nonspecific contrast can also be used. In particular, unlabeled contrast can be used. Other examples of phase-like contrast, such as phase contrast, include bright-field contrast, contrast resulting from oblique incidence illumination, dark-field contrast, and autofluorescence contrast.

[0106] Box 930 includes performing registration between the first and second microscope images. The registration is performed here using one or more reference microscope images with nonspecific contrast.

[0107] Box 945 allows for optional applications based on registration between a first and second microscopic image. For example, a specimen can be evaluated considering corresponding registration parameter values ​​based on a comparison between the first and second microscopic images. For instance, an ROI can be labeled in one of the microscopic images and then displayed in the other microscopic images, taking registration into account. Colocalization and quantification of labeled concentrations can be performed to quantify disease progression. Multiple microscopic images (e.g., those taken in a common temporal context) can be fused to form a multichannel image.

[0108] However, it is also conceivable to perform stitching of the first and second microscopic images (then these will become tiled images). In such cases, the first and second microscopic images typically have the same specific contrast. Stitching broadens the field of view. A mosaic image is obtained. For this purpose, the first and second microscopic images image different regions of the specimen, but there is overlap. When the first and second microscopic images are aligned with each other, precise alignment of the two microscopic images can be performed as part of the stitching.

[0109] Next, we will describe in detail the identification of registration parameter values ​​in box 930. Different registration algorithms can be used to identify registration parameter values. For example, an elastic registration algorithm can be used. It is also possible to use a landmark-related registration algorithm. In particular, it is beneficial if a registration algorithm that can handle the differences in how specimens appear in different microscopic images is used in box 930. For example, the luminance histograms of these images being compared to each other may be inverted. In other words, this therefore means that a particular structure that appears bright in a microscopic image with specific contrast may appear dark in a reference microscopic image (and vice versa). The corresponding registration should be robust to both local and global inversion of the luminance histogram. Furthermore, it is beneficial if the first and / or second registration parameter values ​​are identified using a multimodal registration algorithm. Such a multimodal registration algorithm differs from a unimodal registration algorithm. Unimodal registration is typically used to match images with a common modality, for example, two absorption images recorded at wavelengths close to each other. However, since a microscope image with specific contrast is compared to a reference microscope image with nonspecific contrast, a multimodal registration algorithm can yield better results. While a unimodal registration algorithm can use a simple metric (e.g., least squares) as a performance index for matching the channels of each other, a multimodal registration algorithm is computationally more complex. This is because the similarity index between the images being matched must be able to withstand multiple modalities. For example, as already mentioned above, the luminance histogram can be locally inverted (fluorescence: dark background, brightfield / phase: bright / gray background). Table 1 shows typical similarity indices for multimodal registration algorithms that can be used according to the examples disclosed herein.

[0110] [Table 1]

[0111] First, let's explain Example 1, which is "mutual information" as shown in Table 1. Assume that image A is an image with grayscale levels, and image B is a second image with grayscale levels, which in the most common case has a nonlinear relationship with image A. Without restricting generality, the functional relation B = 1 - A 2 It can be used. All numerical pairs of pixels (A k |B k When a (where k is a linear pixel index) is formed and expressed relative to each other in a two-dimensional coordinate system, a nonlinear functional relationship B = 1 - A 2 This becomes visible. In particular, this nonlinear relationship allows us to understand that, for example, a bright pixel in image A appears dark in image B, and vice versa (this corresponds to an inversion of the luminance histogram). This means that there is a rule that allows us to predict the pixel grayscale value of image B if we know the corresponding pixel grayscale value in image A. Here, if we assume that image A is slightly displaced with respect to image B, and that image B is displaced one pixel to the right, the simple functional relationship is disrupted. The correspondence between image A and displaced image B becomes more complex. It is no longer possible to easily predict what the grayscale value of image B is, even if we know the grayscale value of image A. The predictability of the second image from the first image can be quantified using the mathematical concept of mutual frequency. This is done by first forming a relative frequency in which a pixel in image A takes a grayscale value a, and at the same time, the corresponding pixel (with the same linear index k) in image B takes a grayscale value b. The relative frequency in which images A and B take grayscale values ​​a and b respectively is a bivariate probability p A,B When expressed as (a,b), mutual information (MI) is given by the following formula

number

[0112] Next, Example 2 "Normalized Gradient Field" (NGF) in Table 1 will be described. Multimodal images regarding the same structure may have different luminance values, but their corners and edges should be superimposed in an aligned state. The corresponding means using this observation regarding image alignment is described by Haber and Modersitzki (Non-Patent Document 3), and here it is called the normalized gradient field NGF. The normalized gradient field of image A is

Number

[0113] Regarding the similarity based on segmentation as shown in Example 4 of Table 1, for example, if a specific structure, i.e., a cell, is masked, and for example, yeast cells are present, the masked image will acquire a value of 1, and if they are not present, it will acquire a value of 0. Segmentation allows multimodal images to be converted into unimodal images, and both the microscope image with fluorescence contrast and the reference microscope image become binary images of the same modality through segmentation. Therefore, after segmentation into binary images, conventional unimodal similarity indicators such as least squares distance can be used for registration.

[0114] According to Table 1, the artificial neural network in Example 6 can be used to fuse images of different contrast types into one. In the context of microscopy, this is called virtual staining (see, for example, Patent Document 5) or image2image learning. For this purpose, UNET or cycleGAN architectures are primarily used for the artificial neural network. As a result, for example, a phase image (as a reference microscope image) can be converted into a virtual fluorescence image. Then, other fluorescence channels can be unimodally aligned with the virtual fluorescence image.

[0115] Figure 6 schematically shows the first microscopic image 81 with the first specific contrast and the second microscopic image 82 with the second specific contrast.

[0116] Microscope images 81 and 82 can be, for example, tiled images or mosaic images. Mosaic images 81 and 82 may overlap each other at least partially. Typically, the degree of overlap is greater in mosaic images than in tiled images.

[0117] For example, two mosaic images 81 and 82 can be acquired using different optical channels and / or in separate temporal contexts (e.g., in different iterations 904 in Figure 3). This means that lateral displacement and / or rotation and / or distortion may exist between the two microscope images 81 and 82. Figure 6 shows that the two microscope images 81 and 82 are acquired using different optical channels 61 and 62.

[0118] Furthermore, Figure 6 schematically shows a reference microscope image 85 taken using the third optical channel 63. Therefore, lateral displacement, rotation, distortion, etc., may also exist between the reference microscope image 85 and the microscope image 81, and between the reference microscope image 85 and the microscope image 82. For example, the reference microscope image 85 can be taken in the same iteration 904 (see Figure 3) as the microscope image 81.

[0119] Figure 6 shows that the first registration parameter value 71 is determined based on a comparison between the microscope image 81 and the reference microscope image 85, and in addition, the second registration parameter value 72 is determined based on another, namely, a comparison between the microscope image 82 and the reference microscope image 85. In order to also perform registration between the microscope images 81 and 82, the registration parameters 71 and 72 can be combined (or generally synthesized) to obtain yet another registration parameter 73, which directly mediates between the two microscope images 81 and 82.

[0120] Figure 7 shows one variation of Figure 6. Here, the reference microscope image 85 and microscope image 81 are part of the same multichannel recording (and, for example, a mosaic image). The optical channels of the reference microscope image 85 and the optical channels of the microscope image 81 are particularly similar (for example, the two optical channels may differ only in terms of the color filters used), and therefore, the reference microscope image 85 and microscope image 81 are inherently aligned, and the corresponding registration parameter value 71 does not need to be determined separately by comparing microscope image 81 with the reference microscope image 85. On the other hand, a comparison between the reference microscope image 85 and a second microscope image 82 can be performed to determine the registration parameter value 72 (microscope image 82 is not part of the multichannel recording).

[0121] However, in some cases, the registration parameter value 71 can also be determined by comparing the reference microscope image 85 with the first microscope image 81, for example, if chromatic aberration exists due to different color filters used, this can cause lateral offset, compression, rotation, etc., of the microscope images 81 and 85 taken at different wavelengths.

[0122] Figure 8 shows yet another variation of Figure 6. This involves taking reference microscope images 85, 86 with nonspecific contrast, which are originally aligned with their respective microscope images 81, 82. For example, microscope image 81 and reference microscope image 85 can be part of the same multichannel recording, and in addition, microscope image 82 and reference microscope image 86 can be part of the same multichannel recording. The two reference microscope images 85, 86 can be taken, for example, in each staining cycle or iteration 904 in which their respective associated microscope images 81, 82 are taken. By comparing the reference microscope images 85, 86 with each other, the corresponding registration parameter values ​​75 can be identified, which correspond to the registrations of the two microscope images 81, 82. Simply or selectively (e.g., if significant chromatic aberration is present), it is possible to identify yet another registration parameter value 76, 77 between microscope images 81, 82 and their associated reference microscope images 85, 86 in each case by corresponding comparison. For comparing reference microscope images 85 and 86, a unimodal registration algorithm can typically be used, whereas for comparing microscope image 81 with reference microscope image 85, a multimodal registration algorithm should be used, for example.

[0123] In principle, the microscope images 81 and 82, and the reference microscope images 85 and / or 86, can all be considered (though not necessarily) part of a common multi-channel recording, as previously mentioned in relation to the various drawings.

[0124] In the various examples described herein, registration can be performed in separate stages of image processing. In the first stage, registration can be performed between different tile images of a tile scan. These tile images can then be combined using this registration to form a mosaic image. Then, in the second stage, registration can be performed on different mosaic images (e.g., from different staining cycles). A corresponding technique is shown in Figure 9.

[0125] Figure 9 illustrates a two-stage registration configuration. Figure 9 shows that in the first staining cycle (see iteration 904 in Figure 3), a multichannel recording 610 is captured for the first tile 680, and another multichannel recording 620 is captured for yet another tile 681. In this case, the multichannel recording 610 in the example in the figure contains four channels: three microscopic images 611-613 with specific contrast and a reference microscopic image 614 with nonspecific contrast. The multichannel recording 620 then contains the same four channels: corresponding microscopic images 621-624 with the same specific contrast as the corresponding microscopic images 611-613 and another reference microscopic image 624 with the same nonspecific contrast as reference microscopic image 614. Each reference microscopic image is captured for each tile position in each tile scan.

[0126] Instead of multi-channel recording for each tile as shown in Figure 9, it is also possible to perform separate tile scans for each contrast. This means, for example, that microscope images 611 and 612 are taken in the first tile scan, microscope images 612 and 622 are taken in the second tile scan, and microscope images 613 and 623 are taken in the third tile scan. The second tile scan is started after the completion of the first tile scan, and so on. Reference contrasts can then be recorded for one tile scan, and even for each tile scan.

[0127] The two multi-channel recordings 610 and 620 are captured in a common tile scan. This means that the microscope images 611-614 and 621-624 image different regions of the specimen (the specimen stage is moved between them), but have overlapping areas. Based on the overlapping areas, for example, microscope image 611 can be aligned with the corresponding microscope image 621 (tile-to-tile registration 651). For example, if microscope image 611 is intended to be aligned with microscope image 621, this can be done as shown in Figure 8.

[0128] More specifically, microscope image 611 can be aligned with reference microscope image 614 using, for example, a multimodal registration algorithm (see registration parameter value 76 in Figure 8), and furthermore, microscope image 621 can be aligned with reference microscope image 624 using, for example, the same multimodal registration algorithm (see registration parameter value 77 in Figure 8). However, this would only be necessary if microscope image 611 and reference microscope image 614 are not aligned at all, or if microscope image 621 and reference microscope image 624 are not aligned at all. Furthermore, it is possible to align reference microscope image 614 with reference microscope image 624 by executing another unimodal registration algorithm (registration parameter value 75) to align reference microscope image 614 with reference microscope image 624 in the overlapping region. As a result, registrations of microscope image 611 and microscope image 621 are obtained, which allows them to be accurately stitched together. This is the first stage of registration.

[0129] The first stage of registration, namely another tile-to-tile registration 652, can also be performed on the microscopic images 631-633 of tile 685 and the microscopic images 641-643 of tile 686. For this purpose, each of the corresponding multichannel recordings 630, 640 has reference microscopic images 634, 644 with nonspecific contrast. The multichannel recording 630 therefore includes four channels, namely three microscopic images 631-633 with different specific contrasts and a reference microscopic image 634 with nonspecific contrast.

[0130] Typically, different specific contrasts are used in different staining rounds. For example, the corresponding specific contrast can be obtained by using one or more non-fluorescent labels in the first staining round. Then, one or more fluorescent labels can be used in the second staining round.

[0131] Different multi-channel recordings do not need to have the same number of channels in different staining rounds. For example, in the first staining round, a multi-channel recording can be made with two channels, e.g., H&E contrast (specific contrast) and digital phase contrast (non-specific contrast). Then, in the second staining round, the multi-channel recording can be made with three or more channels, e.g., different fluorescence contrasts and digital phase contrasts.

[0132] As a result, in the example in Figure 9, for each specific contrast captured by this tile scan (i.e., a total of three specific contrasts), corresponding mosaic microscope images 661-663 are obtained by tile-to-tile registration 651, and for each specific contrast captured by this tile scan, corresponding mosaic microscope images 671-673 are obtained by tile-to-tile registration 652. Furthermore, in Figure 9, two mosaic reference microscope images 664 and 674 are also obtained, one for each of the two tile scans. Then, it becomes possible to align these mosaic microscope images 661-663 and 671-673 (which, as mentioned above, have already been aligned with their respective mosaic reference microscope images) with each other, which is done by scan-to-scan registration 655. This can be done, for example, by a unimodal registration algorithm that aligns the two mosaic reference microscope images 664 and 674 with each other. The use of a unimodal registration algorithm is particularly advantageous when the two mosaic reference microscope images have the same specific contrast.

[0133] Figure 10 schematically shows the system 500. The system 500 includes an electronic data processing unit 540. The electronic data processing unit 540 may be, for example, a “personal computer” (PC). The electronic data processing unit 540 includes a processor 542 and memory 543. The electronic data processing unit 540 also includes a communication interface 541. For example, the processor 542 can receive image data for microscope images via the communication interface 541. For example, the processor 542 can receive image data from the microscope 551 or from an external image database (not shown in Figure 10). However, the processor 542 can also load image data from memory 543 and execute them. The processor 542 can send control commands to the microscope 551 that trigger the acquisition of microscope images and / or set specific illumination geometric conditions and / or specific defocus values ​​(e.g., for TIE or DPC or a combination thereof). The processor 542 can load program code from memory 543 and execute it. When the processor executes such program code, this has the effect that the processor 542 performs techniques as described herein, such as identifying registration parameter values ​​by comparing microscope images, identifying registration parameter values ​​in combination with other registration parameter values, controlling the microscope 551 to capture microscope images with and without fluorescence contrast, identifying microscope images with phase contrast or phase gradient contrast, and so on.

[0134] In summary, the above describes techniques for coordinating registration between two microscope images with different fluorescence contrasts (or, generally, different specific contrasts) via a reference microscope image that does not have a specific specific contrast. For example, a digital phase contrast image (e.g., by TIE or DPC) can be used as a reference for two- or three-dimensional co-registration of fluorescence channels. This is reasonable because microscope images with different fluorescence contrasts may have lateral offsets due to improperly adjusted filters and beam splitters, and axial offsets due to chromatic aberration. Furthermore, some microscopes operate in a mode that includes first capturing all tiles for each channel. This means that a specimen with lateral and axial offsets is fully scanned for each fluorescence contrast before the next fluorescence channel is measured. This mode is advantageous for minimizing mechanical wear on the filter and beam splitter wheels. However, the multi-directional repeatability of mechanical specimen scanners is limited, which can lead to further displacement between microscope images with different fluorescence contrasts.

[0135] Furthermore, as already explained, various nonspecific contrasts can be used. In this regard, phase gradient contrast can also be used as an alternative to digital phase contrast for reference microscope images. Phase gradient contrast can be generated in a particularly computationally efficient manner, i.e., by computing the generation of a pixel-by-pixel difference image between two intensity images taken under different illumination geometric conditions.

[0136] Furthermore, a technique for aligning microscope images with specific contrast and reference microscope images with nonspecific contrast using a multimodal registration algorithm was described. These can be tiled images from different tile scans or from a common multi-channel recording. In particular, a technique using an NGF-based registration algorithm was described. Such an NGF-based multimodal registration algorithm is especially efficient and can be implemented with high computational efficiency even with limited computer hardware.

[0137] Elastic multimodal registration algorithms can be used. Elastic registration can also take into account nonlinear differences between image pairs, such as deformation or distortion.

[0138] To summarize further, the following embodiments are disclosed in particular. [Examples]

[0139] Example 1. A computer implementation method for aligning a first microscope image (81, 91) with a second microscope image (82, 92), wherein the first microscope image visualizes the specimen with a first specific contrast, and the second microscope image visualizes the specimen with either the first or second specific contrast. The method is, - Obtain reference microscope images (85, 86, 99) of the specimen with non-specific contrast, such as phase-like contrast, - Registration of the first and second microscope images is performed using a reference microscope image, Includes.

[0140] Example 2. Performing registration of the first and second microscope images is: - Based on a comparison of the second microscope image (92, 82) and the reference microscope image (85, 99), identify the registration parameter values ​​(72, 311, 312) for registration (930). A computer implementation method according to Example 1, including the above.

[0141] Example 3. Performing registration of the first and second microscope images is: - Identify additional registration parameter values ​​(71, 311, 312) for registration based on a comparison of the first microscopic image (91) with a reference microscopic image (85, 99) or another reference microscopic image of the specimen with nonspecific contrast (935). A computer implementation method according to Example 2, including the above.

[0142] Example 4. Performing registration of the first and second microscope images is: - Based on registration parameter values ​​(72, 311, 312) and other registration parameter values ​​(71, 311, 312), determine the resulting registration parameter values ​​for the registration of the first microscopic image (91) and the second microscopic image (92) (940). A computer implementation method according to Example 3, including the above.

[0143] Example 5. The method is: - Further including obtaining another reference microscope image (86) of the specimen with nonspecific contrast, Performing registration of the first microscope image (81, 91) and the second microscope image (82, 92) is A computer implementation method according to any one of Examples 1 to 4, comprising determining registration parameter values ​​(75) for registration based on a comparison of a reference microscope image (85) with another reference microscope image (86).

[0144] Example 6. Performing registration of the first and second microscope images is: - Based on a comparison of the first microscopic image (81) and the reference microscopic image (85), identify another registration parameter value (76) for registration and / or - Identify another registration parameter value (77) for registration based on a comparison of a second microscopic image (82) with yet another reference microscopic image (86). A computer implementation method according to Example 5, including the above.

[0145] Example 7. The reference microscope image and the first microscope image are part of the same multichannel recording, and the reference microscope image and the second microscope image are not part of the same multichannel recording, and / or The specimen is manipulated between the acquisition of the first and second microscope images. A computer implementation method according to any one of Examples 1 to 6.

[0146] Example 8. The first microscope image, the second microscope image, and optionally a reference microscope image are all part of the same multi-channel recording, and the specimen is not manipulated between the acquisition of the first microscope image and the acquisition of the second microscope image. A computer implementation method according to any one of Examples 1 to 6.

[0147] Example 9. Phase-like contrast is selected from the following group: phase contrast, digital phase contrast, and digital phase gradient contrast. A computer implementation method according to any one of Examples 1 to 8.

[0148] Example 10. Furthermore, - To acquire a first time-reference microscope image and a second time-reference microscope image, where both the first and second time-reference microscope images are acquired by imaging the specimen with the same intensity contrast and different defocus values. - Identifying the reference microscope image based on the generation of a pixel-by-pixel difference image between the first temporal reference microscope image and the second temporal reference microscope image, A computer implementation method according to any one of Examples 1 to 9, including the above.

[0149] Example 11. The registration parameter values ​​for registration are determined by a registration algorithm that identifies a similarity index between each pair of microscope images based on a robust metric for brightness histogram inversion within the microscope images. The registration parameter values ​​for registration are determined by the multimodal registration algorithm. A computer implementation method according to any one of Examples 1 to 10.

[0150] Example 12. The registration parameter values ​​for registration are determined by a registration algorithm selected from the following group: mutual information, normalized gradient field, structural similarity index, segmentation-based similarity index, feature detection, and artificial neural network. A computer implementation method according to any one of Examples 1 to 11.

[0151] Example 13. The registration parameter values ​​for the registration are determined by the elastic registration algorithm. The registration parameter values ​​for registration are determined by the landmark-related registration algorithm. A computer implementation method according to any one of Examples 1 to 12.

[0152] Example 14. Furthermore, - Stitching the first and second microscope images based on registration. A computer implementation method according to any one of Examples 1 to 13, further including the above.

[0153] Example 15. An electronic data processing device (540) configured to align a first microscope image (81, 91) with a second microscope image (82, 92), wherein the first microscope image visualizes the specimen with a first specific contrast, and the second microscope image visualizes the specimen with either the first or second specific contrast. An electronic data processing device includes a processor and memory, the processor is configured to load program code from memory and execute it, and once the program code is executed, the processor proceeds through the following steps, namely: - A step to obtain reference microscope images (85, 86, 99) that image the specimen with nonspecific contrast, - A step of performing registration of the first and second microscope images using a reference microscope image, Execute this.

[0154] Example 16. An electronic data processing device according to Example 10, wherein the processor, when program code is executed, performs a method according to any one of Examples 1 to 14.

[0155] Needless to say, the embodiments and aspects of the present invention described above can be combined with each other. In particular, the features can be used not only in the combinations described, but also in combination or individually, and this does not depart from the scope of the present invention.

[0156] For example, we have so far described techniques for processing and aligning 2D images. However, in principle, all the techniques described can also be used for 3D images to correct lateral and axial translation errors (alignment of mutually displaced focus stacks or longitudinal chromatic aberration), scaling errors (changes in camera pixel size or variable objective lens magnification), and rotation errors (camera rotation).

[0157] Furthermore, we have described techniques that include taking microscopic images with specific contrast related to the specimen. The specimen may be, for example, a cell or tissue specimen. [Explanation of symbols]

[0158] 61 optical channels 62 optical channels 63 Third Optical Channel 71 First registration parameter value 72 Second registration parameter value 75 Registration parameter values 76, 77 Another registration parameter value 81 First Microscope Image 82 Second Microscope Image 85, 86 Reference microscope images 91 First Microscope Image 92 Second Microscope Image 99 Reference Microscope Image 311, 312 Registration parameter values 500 Systems 540 Electronic data processing equipment 541 Communication Interface 542 processors 543 memory 551 Microscope 610 Multi-channel recording 611-613 Microscope images 614 Reference microscope image 620 multi-channel recording 621-623 Microscope images 624 Reference microscope image 630 Multi-channel recording 631-633 Microscope images 634, 644 Reference microscope images 640 multi-channel recording 641-643 Microscope images 651 Registrations 652 Tile-to-tile registration 655 Scan-to-Scan Registration 661-663 Mosaic microscope images 664, 674 Mosaic reference microscope images 671-673 Mosaic Microscope Images 685 tiles 686 tiles

Claims

1. In a computer implementation method for aligning a first microscope image (81, 91, 611-613, 631-633, 661-663) with a second microscope image (82, 92, 621-623, 641-643, 671-672), the first microscope image visualizes the specimen with a first specific contrast, and the second microscope image visualizes the specimen with either the first or second specific contrast. The aforementioned method, - To obtain reference microscope images (85, 99, 614, 634, 664) of the aforementioned specimen with nonspecific contrast, - The registration of the first microscope image and the second microscope image is performed using the reference microscope image, Computer implementation methods including

2. Performing the registration described above includes applying a multimodal registration algorithm (651, 652), The aforementioned nonspecific contrast is a labelless contrast, optionally selected from the following group: phase-like contrast, phase contrast, bright-field contrast, contrast resulting from oblique incidence illumination, dark-field contrast, and autofluorescence contrast. The computer implementation method according to claim 1.

3. The first microscopic image is associated with the first staining cycle (904, 910) of the specimen, The second microscopic image is associated with the second staining cycle (904, 910) of the specimen. The computer implementation method according to claim 2.

4. The aforementioned reference microscope image is associated with the first staining cycle of the specimen, The above method further, - Includes imaging the specimen with the nonspecific contrast and obtaining another reference microscope image (86, 624, 644) associated with the second staining cycle, Performing the aforementioned registration means - This includes comparing the aforementioned reference microscope images (85, 614, 624) with the aforementioned other reference microscope images (86, 634, 644), The computer implementation method according to claim 3.

5. Performing the aforementioned registration further means, - Comparing the first microscope images (81, 611-613, 631-633) with the reference microscope images (85, 614, 634), and / or - Compare the second microscope images (82, 621-623, 641-643) with the other reference microscope images (86, 624, 644), including, The computer implementation method according to claim 4.

6. The first microscope image and the reference microscope image are inherently aligned, and / or The second microscope image and the other reference microscope image are inherently aligned. The computer implementation method according to claim 4.

7. The first microscope image is a first mosaic image (661-663) composed of first tile images (611-613, 621-623), The aforementioned second microscope image is a second mosaic image (671-673) composed of a second tile image (631-633, 641-643), The multimodal registration algorithm (651, 652) is also applied between the first tile images (611-613, 621-623) and each of the reference tile images (614, 624) of the reference microscope image (664), and between the second tile images (631-633, 641-643) and each of the yet another reference tile images (634, 644) of the yet another reference microscope image (674). Performing the registration further includes applying a unimodal registration algorithm (655) between the reference microscope image (664) and the other reference microscope image (674). The computer implementation method according to any one of claims 4 to 6.

8. The aforementioned reference microscope image is associated with the first staining cycle of the specimen. Performing the aforementioned registration means - Compare the second microscope image with the reference microscope image. The computer implementation method according to claim 3, including the method described in claim 3.

9. The first microscope image and the reference microscope image are inherently aligned. The computer implementation method according to claim 8.

10. The first microscope image is the first tile image of the tile scan, The second microscope image is a second tile image of the tile scan that partially overlaps with the first tile image. The aforementioned reference microscope image is the third tile image of the tile scan. The computer implementation method according to any one of claims 1 to 4.

11. Corresponding reference microscope images are available for each tile position in the tile scan. The computer implementation method according to claim 10.

12. An electronic data processing device (540) including a processor (542) and a memory (543), wherein the processor (542) is configured to load program code from the memory and execute it, the processor is configured to acquire reference microscope images (85, 99, 614, 634, 664) that image a specimen with nonspecific contrast based on the execution of the program code, and to perform registration (651, 652, 655) of a first microscope image and a second microscope image using the reference microscope image, the first microscope image images the specimen with a first specific contrast, and the second microscope image images the specimen with either the first or second specific contrast, the electronic data processing device.

13. The electronic data processing apparatus according to claim 12, wherein the processor is configured to perform the method according to claim 1 based on the execution of the program code.

Images