Method for creating a microscope image

Abstract

Method for creating a complete image (10) of an object (11) from a multitude of, in particular digital, image recordings (12), each depicting a small area of ​​the object (10), characterized in that the overall image (10) is or is divided into partial images (13), wherein each partial image (13) of the overall image (10) is or is assigned a part of the multitude of image recordings (12), wherein the image recordings (12) are combined to form the overall image (10) such that first the image recordings (12) of a partial image (13) are recorded and combined, and then another partial image (13) is combined from image recordings (12) that are recorded, wherein, in order to assemble the image recordings (12) into a partial image (13), at least two edges (17-21) of the respective image recording (12) overlap with at least one edge (17-21) of at least two image recordings (12) adjacent to the image recording (12), and wherein the overlapping image areas (17-21) of the image recordings (12) are correlated with each other, where a weighting of the correlation is applied for the relative positioning of the adjacent image recordings (12) to each other, wherein the sequence for creating and assembling the partial images (13) from image recordings (12) is automatically determined, wherein a focus is determined for each partial image (13) before the image recordings (12) of the partial image (13) are taken.

Description

[0001] The invention relates to a method for creating a complete image of an object from a plurality of, in particular digital, image recordings, each of which depicts a small area of ​​the object.

[0002] Particularly in microscopy and microscope image processing, a problem arises that with high-resolution microscope images, the image to be created of an object or sample, such as a blood sample, is much larger than the area that can be captured with a single image, for example, from a digital camera. This results in a kind of high-resolution mosaic image. The sample or object from which the mosaic image is to be acquired is moved along a grid under the control of the image acquisition device, and individual images are captured at pre-calculated positions. These images are then combined to form a microscope image.

[0003] This ensures that the individual images can be assembled almost seamlessly.

[0004] Known are US 4 673 988 A1, US 2009 / 0 196 526 A1 and Bradley, Andrew P.; Wildermoth, Michael; Mills, Paul. Virtual microscope with extended depth of field. In: Digital Image Computing: Techniques and Applications (DICTA'05). IEEE, 2005. pp. 35-35.

[0005] Furthermore, it is known to process a digital microscope image by loading the entire image into the computer's main memory and then processing it with the computer's processor. For example, see document DE 10 2006 034 996 A1. However, since the entire microscope image is too large to be loaded in its entirety into the computer's (potentially virtual) main memory, parts of the image are swapped back to the mass storage device. This is called "swapping" or "paging." Before the calculations can be performed, the image parts must first be loaded back into main memory, which is time-consuming.

[0006] If the microscope image is so large that it cannot be held entirely in memory, the known method processes sub-images individually. This results in either the information about neighboring sub-images being disregarded, leading to inaccuracies at the edges, or the need to develop special algorithms to address these inaccuracies. Often, these inaccuracies are simply accepted, in which case separate algorithms must be found to solve the accuracy problems at the edges.Document DE 10 2006 034 996 A1 solves this problem by specifying a microscope image processing method for execution on a computer, wherein the computer has a working memory with a predetermined available storage capacity and a mass storage device that has a higher access time than the working memory, wherein a digital microscope image consisting of pixels is n-dimensional with n > 1, consists of at least two sub-images and has a size that exceeds the available storage capacity of the working memory, wherein a computational operation is applied to at least one part of the microscope image with the following steps: a) Provision of the microscope image in mass storage b) Decomposing the microscope image into at least two image sections that can be loaded into the working memory and have a dimension m, where m ≤ n, c) For a section of an image, determine all pixels that lie within the section of the image and in at least one of the sub-images, such that a filled section of the image is created. d) Provision of the filled image area in working memory, e) Applying the arithmetic operation to the pixels located in the filled image area, so that an image area result is produced, f) Repeat steps c), d) and e) for all image sections and g) Combining all image section results into one overall result.

[0007] However, this patent application remains completely silent on the creation of a complete image of an object, i.e., for example, a microscope image.

[0008] The object of the present invention is to provide an efficient method for creating a complete image of an object from a multitude of, in particular digital, image recordings, by means of which very precise complete images can be created.

[0009] This task is solved by a method for creating a complete image of an object from a multitude of, in particular digital, image recordings, each depicting a small area of ​​the object, which is further developed by dividing the complete image into partial images, wherein each partial image of the complete image is assigned a part of the multitude of image recordings, wherein the image recordings are assembled into the complete image in such a way that first the image recordings of a partial image are recorded and assembled and then another partial image is assembled from image recordings that are recorded.

[0010] The inventive method very efficiently captures a highly accurate overall image of an object or sample. For this purpose, a multitude of images, particularly digital ones, are successively acquired, each belonging to a sub-image, and then precisely combined. By dividing the overall image into sub-images, the sub-images can be acquired very quickly, resulting in very sharp images, since, for example, thermal drift of the focus is negligible due to the short acquisition time of the sub-images. Preferably, the method steps are performed for all sub-images or relevant sub-images to obtain the overall image.

[0011] A particularly accurate overall image, and also a partial image, is achieved when, in order to combine the images into a partial image, at least two edges of each image overlap with at least one edge of at least two adjacent images. This allows the images to be aligned precisely.

[0012] However, this type of image compositing can lead to problems. These problems can arise, for example, when compositing images, such as when an image positioned in the upper left is arranged or combined with the image to its right. Similarly, when compositing the image positioned to the left and below the one in the upper left, the adjustment of the image below the upper right image may not harmonize with the adjustment or combination of the image in the lower left.

[0013] For this reason, it is preferred to correlate the overlapping image areas of the image acquisitions with each other, weighting the correlation based on the relative positioning of the adjacent image acquisitions. A normalized cross-correlation of the overlapping image areas is thus performed. In this process, normalization is preferably based on image intensity.

[0014] The following is preferably performed during correlation: The positions of maximum correlation are preferably aligned. This is also referred to as image registration or template matching. For this purpose, the overlapping area of ​​the image pair to be correlated is preferably decomposed into overlapping sub-areas, preferably quadratically.

[0015] The overlapping sub-areas are processed in contrast-sorted order. Areas with the highest contrast are prioritized. Image registration with normalized cross-correlations is calculated on the overlapping sub-areas until at least two overlapping sub-areas have been processed and a minimum correlation value has been found. This minimum value can be predefined. The positions of maximum correlation from the overlapping sub-area registrations are weighted and averaged by the square of their corresponding correlation value.

[0016] This method preferably involves decomposing each overlapping sub-area to be registered into a pattern and a displacement boundary. The so-called pattern is extracted from an overlapping sub-area of ​​the earlier of the two image acquisitions. From the other image acquisition, the normalized cross-correlation with the pattern is calculated at each possible displacement position within its overlapping sub-area. The result is the displacement position with the maximum correlation value.

[0017] The width of the displacement edge or the overlap area should preferably cover the positioning error of the microscope stage and is preferably dependent on the magnification of the microscope.

[0018] Preferably, the roles of the two image captures for the pattern and the displacement search area or displacement boundary are reversed, and the cross-correlation process is repeated. This further increases the accuracy and robustness of the system and the algorithm in terms of pattern recognition.

[0019] Preferably, for the further procedure, the displacement positions between the overlapping sub-areas are determined for all image recordings of a partial image or also of the neighboring partial images, insofar as these are required for merging the newly recorded partial image, and an origin of a later recorded image recording in the earlier recorded image recording is calculated.

[0020] Preferably, all images of a sub-image are combined simultaneously; that is, correlation and weighting of the correlation are performed for all images of that sub-image. This reduces the overall error of combining the images of a sub-image. The weighting of the correlation is performed such that edge areas or overlapping image areas of the images, which exhibit high contrast, are preferably given higher weight. Area contrast is of particular interest here, i.e., high contrast over a predefined large area. An example of how the cross-correlation is performed is described in the figure examples.

[0021] Preferably the images are square, in particular rectangular, and have 2 n × 2 mImage points, where n and m are preferably ≥ 9. The sub-images are preferably also quadrilateral, in particular rectangular, and can contain several thousand image frames. Preferably, the sub-images are compact, i.e., they have as similar a side length as possible and / or border on compact sub-areas.

[0022] Preferably, the size of the sub-images is adapted to the size of a memory directly accessible to a processor (RAM). Memory directly accessible to the processor is the working memory of a computer. How the size can be precisely adjusted in an example will be described in more detail in relation to the figure description.

[0023] Preferably, before taking images, the entire image of an object is first scanned to determine its location. The image is then divided into sub-images of the object or sample so that, preferably, the entire object can be measured or captured. Where practical, sub-images of different sizes can be used. However, sub-images of the same size are preferred. For specially shaped samples or objects, the sub-images can be larger, for example, if only a small portion of the sub-image is occupied by an object. In such cases, the background does not need to be captured or saved. Only the images that capture at least part of the object are then taken.

[0024] Preferably, the image acquisition is performed sequentially, such that adjacent images are taken, with the contour of the object or sample preferably being traced, at least in the partial image. This ensures, in particular, that no artifacts or significant stitching errors occur when the images are combined. The sample area is kept compact in the resulting overall image or partial image. Registration errors remain small because long, thin chains of individual images are avoided. Furthermore, the storage requirement for the image edge is lower than with a fragmented structure of partial areas or images.

[0025] Preferably, the portion of the image data that is no longer needed to assemble the images into a sub-image is stored in compressed form. This compressed storage preferably takes place in RAM, but can also be done, for example, on a hard drive that is slower than RAM.

[0026] Preferably, after the image recordings have been combined to form a partial image, the partial image is stored, in particular compressed form, in a further storage medium, such as a hard drive, except for the edges of image recordings that are needed for combining with another partial image.

[0027] Preferably, overlapping image areas from image acquisitions arranged in adjacent sub-images are correlated to assemble partial images, with the correlation weighted according to the relative positioning of the adjacent sub-images. Again, a higher correlation carries more weight than a lower correlation. Preferably, a cross-correlation normalized to an image intensity is used.

[0028] The assembling of partial images can be achieved in two ways: firstly, by having the images to be assembled already integrated within each partial image, requiring only the partial images to be aligned relative to one another; or, preferably, by assembling an already assembled partial image with another partial image, integrating the images of the second partial image with the first. This minimizes assemblage errors.

[0029] Preferably, the sequence for creating and assembling partial images from image acquisitions is determined automatically. First, the contour or area distribution of the object or sample is determined, and the sequence for creating partial images is then established based on this contour. A preferred criterion for this sequence is the presence of an object in the partial area, the area coverage of the partial area with an object, the number of adjacent partial images created and assembled, and / or the number of image acquisitions of an adjacent partial image available for merging with the partial image.

[0030] For example, a sub-image can move further up in the sequence of creation and assembly from image captures if the area covered by the sub-image is relatively large. Similarly, a sub-image moves up in the sequence if the number of created and assembled adjacent sub-images is high. Furthermore, a sub-image moves up in the sequence if it contains an object. If the sub-image contains no object or only a small portion of an object, it moves down in the sequence. Finally, a sub-image moves up in the sequence if there is a large number of image captures from an adjacent sub-image available for merging with it.For example, if three adjacent sub-images already exist that are needed to merge them with the sub-image yet to be created, this sub-image will be placed very high up in the creation sequence. The illustrated variations for a possible sequence of creating and merging sub-images are examples. Several of the criteria mentioned above can be applied, or only a single criterion. All criteria can also be applied, and a corresponding evaluation or weighting function can be created. The individual criteria can be weighted differently.

[0031] Preferably, a first partial image, in which images are to be taken and which is then assembled from these images, has the largest area covered by the object and / or exhibits the greatest overall contrast across its area. This ensures that, wherever possible, the part of the object to be photographed, or from which a composite image is to be created, begins with the partial image that is most distinctive for the sample.

[0032] According to the invention, a focus is determined for each partial image before the images of that partial image are captured. This makes it possible to avoid, in particular, thermal shifts of the focus(s). If the creation and assembly of the respective partial image is fast enough so that no drift in the focus(s) due to thermal effects is to be feared, the focus(s) of two or more partial images can also be determined before the images of those partial images are captured.

[0033] Preferably, the focus is determined at several support points, and interpolation is performed between these points. For this purpose, for example, triangular surfaces are created between the support points with respect to the elevation contour. This is called triangulation, or a method according to Delaunay is used.

[0034] Preferably, a computer program is provided with program code resources adapted to execute the method according to the invention or a preferred method.

[0035] Preferably, the computer program is stored on a data carrier that can be read by a computer or can be downloaded as a data stream from the Internet.

[0036] According to the invention, a microscope is provided with a computer system that is configured to carry out a method according to the invention.

[0037] Further features of the invention will become apparent from the description of embodiments according to the invention, together with the claims and the accompanying drawings. Embodiments according to the invention may fulfill individual features or a combination of several features.

[0038] The invention is described below, without limiting the general concept, with reference to exemplary embodiments and the drawings, whereby for all details of the invention not explained in detail in the text, explicit reference is made to the drawings. The drawings show: Fig. 1. A schematic top view of an overall image of an object with partial images, Fig. 2. A schematic representation explaining how to create a partial image. Fig. 3. A schematic representation to explain how to combine a second partial image with a first partial image and Fig. 4 A schematic top view of an overall image with four partial images.

[0039] In the drawings, identical or similar elements and / or parts are provided with the same reference numbers, so that a re-presentation is omitted.

[0040] Fig. Figure 1 shows a schematic top view of an object 11, which is mounted on a slide and from which a microscope image is to be created as a composite image 10. The object 11 could be, for example, a tissue sample from a human, an animal, or a plant sample. To take a microscope image, for example, a thin layer of the sample is applied to a glass slide and placed in a microscope to create a composite image.

[0041] The procedure is as follows: first, the contour of object 11 is captured for the overall image, and then an area is defined from which images are to be taken. For this purpose, the overall image is divided into partial images 13, which preferably depict parts of object 11. Digital images 12 are then taken of the respective partial images and, as described later, combined to create a partial image from several digital images.

[0042] The second image from the top, viewed from the left side, of the Fig. Figure 1 schematically shows 35 images taken of the sample or object 11, although this number is only an example. In reality, several thousand images per sub-image may be used. The size of the sub-images, i.e., the number of images per sub-image, depends significantly on the amount of storage space or RAM available for the processor that performs the image processing and combines the digital images into a sub-image 13. A good example of such sizes is also shown below.

[0043] It is unnecessary to take photographs of the background 14, as it is completely irrelevant to the object 11 itself. Therefore, image 13 is not entirely covered with photographs. As in Fig. As can be seen in Figure 1, not all sub-images are the same size. The size of sub-images 13 and 13''' depends primarily on the area of ​​the digital images to be captured, and thus on the amount of data. In the case of sub-images 13'' and 13''', which are shown somewhat elongated, there is a comparatively small area of ​​the object, so only a portion of sub-image 13'' or 13''' needs to be filled with the corresponding images.

[0044] In connection with Fig. Section 2 will now explain an example of combining digital images into a partial image. For the sake of simplicity, it is assumed here that four digital images 1-4 are in the Fig. The two partial images shown are present. These four digital images are labeled with the reference numbers 1, 2, 3, and 4. The images have corresponding borders that are used to assemble or join the images in the manner of a mosaic or superimposed tiles. Image 1 has a bottom border 17, image 3 a border 18, image 2 a border 21, image 3 also a border 19, and image 4 a border 20. Image 1 also has a corresponding border on the right side, which is, however, completely obscured. The overlapping borders of images 2 and 4 are not shown.

[0045] Fig. Figure 2 also shows, in a very schematic way, how the image captures are combined. For this purpose, springs between the origin points of the image captures, i.e., in this example the upper left corners of each image capture, are schematically indicated and labeled ω. ijThe spring is characterized by the number "i" being the image from which the spring begins and "j" is the image number at which the spring ends. The spring representation is intended to suggest an analogy to Hooke's Law. When combining image images in pairs, the spring energy corresponds to the square of the displacement of the images relative to each other from the positions expected by the mechanics. It is initially assumed that the drive of the sample stage or the microscope is quite precise and that the sample 11 is moved relative to the microscope lens in such a way that relatively precisely positioned image images with predictable overlaps are taken of the sample. However, due to mechanical inaccuracies, the image images must then be combined as provided for in the invention.

[0046] When stitching the images together, 17 to 21 high-contrast areas are selected at the edges. Matching, or stitching, begins in these areas. A normalized cross-correlation, specifically a cross-correlation normalized to image intensity, is then performed. Areas with higher contrast are weighted more heavily than areas with lower contrast for the stitching process. The aim is to overlay the images as completely as possible at the edges, ensuring that the same structures are superimposed. Finally, a minimum function is calculated using a mathematical model of the values ​​according to the following formula: min{A}∑i,j∈Adjacentpairswij2‖a→j−a→i−s→ij‖2

[0047] Here, A is the set of all image acquisitions, a is the origin of each image acquisition, s are the displacement vectors, i.e., the displacement (i.e., the change in coordinates from the initial position of the image acquisitions) of the respective image acquisitions due to the cross-correlation performed, and w is the weighting factor of the respective cross-correlation. This achieves optimal positioning of all image acquisitions of a sub-image. Adjacent pairs are the neighboring image acquisitions on which the cross-correlation is performed.

[0048] Fig. Figure 3 shows how another partial image is assembled and combined with a first partial image. The origin of the first partial image, comprising images 1, 2, 3, and 4, is represented by 15. The reference number 15 can also be denoted as a1 when using the formula above. The second partial image comprises images 5, 6, 7, and 8. Corresponding combinations of the second partial image, comprising images 5, 6, 7, and 8, take place based on the strength of the respective cross-correlation, which is determined by the springs w. 56 , w 57 , w 68 and w 78 are depicted or indicated. At the same time, however, there is also a correlation between the right edge of image 2 and image 4 with the left edge of image 5 and image 7, as shown by the schematically depicted feathers w. 25 and w 47This is shown. This also results in an optimal merging result when transitioning from one partial image to the next.

[0049] The same procedure is then applied to partial images that have even more fully assembled partial images surrounding them.

[0050] Ultimately, the inventive method provides an optimized compositing matrix or an optimized placement matrix of the image recordings, which shows a minimal error when assembling.

[0051] By using partial images, the image acquisition time is reduced to such an extent that thermal drift of the focus is no longer a problem. Furthermore, the alignment of all images—that is, of all partial images—is optimized before the images are combined into the resulting final image. Incidentally, the x,y coordinates of three-dimensional space lie on the surface of the microscope slide.

[0052] Preferably, the image acquisition is performed in such a way that adjacent images are always captured successively. Similarly, it is also preferred that adjacent sub-images are always created. This minimizes artifacts during merging. Furthermore, the edge areas of the image acquisitions, which are still required for merging the images or sub-images, should be stored uncompressed to avoid artifacts as well.

[0053] Fig. Figure 4 shows a schematic top view of a sample or object 11 and serves to explain the determination of the sequence of partial images to be recorded.

[0054] Sample 11 has a gap 16 in sub-image 13. If the process were to start with this sub-image 13, difficulties would arise when attaching sub-image 13" to sub-image 13' and sub-image 13'''. For this reason, a sensible sequence must be determined. The sub-image with which the process begins should contain as much sample or object as possible. Sub-image 13''' is suitable for this purpose, as it has the greatest area coverage of the sample and the contrast is at least as high as in sub-images 13" and 13'. Therefore, the process will automatically begin with sub-image 13'''.

[0055] Once all necessary images of sub-image 13''' have been acquired and stitched together, the algorithm continues to check the amount of sample area present in the sub-images. It appears that sub-image 13''' has a slightly larger sample area than sub-image 13, but less than sub-image 13'. The algorithm for determining the sequence can be programmed to prioritize sub-images whose neighboring sub-images have already been acquired. In this case, sub-image 13''' would receive priority. Furthermore, the algorithm can be configured to prioritize sub-images containing samples with complete breaks in the sample sequence, placing them further down the order. In this embodiment, sub-image 13''' would be the next to be processed.Since there is more sample material in sub-image 13' and an adjacent sub-image has already been completed for each of the two sub-images, sub-image 13' will be completed next.

[0056] As a further algorithm component for determining the order in which the sub-image is created, the number of adjacent image captures used for merging can also be taken into account. The more image captures that can be used for merging, the higher up the sub-image ranks in the sequence.

[0057] As mentioned above, the object or sample is scanned before taking the images to perform autofocus. Since the objects have varying heights, different thicknesses, or are undulating, the focus must be set differently at different positions on the object. For this reason, it is advantageous to determine the focus at multiple reference points. This is preferably done in only one or a few partial images to counteract thermal drift. To perform autofocus, the system moves the object or slide under the camera so that the focus can be determined at the reference points. To reduce the time required, the system or algorithm checks whether corresponding focus points have already been determined, for example, by verifying that the focus or foci have already been determined in an adjacent partial image.The corresponding focus can then be used as long as its recording time is within a predefined time, i.e., not too long ago.

[0058] Interpolation then takes place between the positions where a focus has been determined. For example, a triangle is drawn between every three focus points, angled in space according to the focus height. This method is roughly equivalent to the Delaunay method.

[0059] In the preferred method used, a triangulation is performed that yields a complete and non-overlapping set of triangles, with the triangles lying between the predefined support points. This ensures that the plane interpolation works and is unique at every point on the sample surface. As a result, a focus can be determined at any point on the sample surface.

[0060] However, the pre-determined focus values ​​of adjacent sub-images can only be used if a predetermined time has not yet elapsed between the moment the focus is determined and the use of that focus value. This time could be, for example, 10 minutes. This prevents thermal drift.

[0061] Within the scope of the invention, the term "composing" also includes arranging image recordings next to each other or partial images relative to each other, or matching, arranging, adjusting, or aligning them relative to each other. The term "registering" can also be used instead of "composing." The edges of the image recordings are thus registered, or image registration of the edges of the overlapping image recordings is performed.

[0062] In cases where a partial image, or the images of a partial image, are aligned with each other, all images for the entire partial image are stored in a cache. Here, all images, including the edges of images no longer needed for merging with other images, can be stored in compressed form. The edge areas of images still required for merging with other images from adjacent partial images are stored separately and without compression, or can be stored separately, preferably also without compression.

[0063] The size of the partial images is adapted to the available RAM of the microscope's computer system. Preferably, the size of a partial image is chosen so that all images from two partial images fit into the RAM. This allows for the acquisition of further images of another partial image while combining a partial image with a previously combined one. Additionally, a certain amount of RAM can be reserved for other tasks, for example, in the range of 1 to 5 gigabytes, preferably 3 gigabytes.

[0064] All features mentioned, including those discernible from the drawings alone as well as individual features disclosed in combination with other features, are considered essential to the invention, both individually and in combination. Inventory embodiments may be fulfilled by individual features or a combination of several features. Within the scope of the invention, features marked "in particular" or "preferably" are to be understood as optional features. Reference symbol list 1 - 8 Image capture 10 Overall picture 11 object / sample 12 Image capture 13 - 13''' partial image 14 Background 15 Origin 16 gap 17 - 21 Rand 22 Part of the image without border w ij Weighting value i, j Image capture number

Images