Image registration

Abstract

A method for 3D-image registration comprises smoothing two volumes, determining a translation by registering the smoothed volumes, applying the translation to the original (pre-smoothed) volumes, and registering the original volumes based on the translation. Those coarsely registered volumes can each then be divided into subvolumes that are further registered, leading to creation of parametrized model accounting for translations, rotations, scaling, and shearing.

Description

IMAGE REGISTRATIONRELATED APPLICATIONS[ o o o i ] This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 63 / 585,230, filed on September 26, 2023, which is incorporated herein by reference in its entirety.BACKGROUND OF THE INVENTION

[0002] X-ray microscopy (XRM) is an imaging technique that allows scientists and researchers to visualize the internal structure of materials and biological samples with exceptional detail. X-ray microscopy utilizes X-rays to penetrate and provide insights often at the micro and nano scale.

[0003] The basic principle behind X-ray microscopy involves directing a beam of X- rays onto the sample of interest. The X-rays are absorbed, scattered, and refracted in ways that provide information about the sample's internal features. These interactions are detected and translated into detailed images or projections and volumes using specialized detectors and advanced computational reconstruction algorithms.

[0004] Applications for X-ray microscopy span a wide range of fields, from materials science and nanotechnology to biology and medicine. Researchers can investigate intricate structures within cells, study the growth of nanomaterials, and examine the distribution of elements in geological samples.

[0005] In X-ray microscopy data, registration is often crucial to piece together a complete and accurate view of the object or sample being studied. In X-ray microscopy, researchers capture images with different resolutions and different acquisition settings and more generally angles to reveal the internal structure of tiny things like cells or materials. However, due to these differences, in addition to slight shifts during imaging and / or slight movements of the sample, these images might not line up perfectly, and distortions may be introduced. This problem also arises when aligning masks.

[0006] Registration solves this problem by adjusting the images to fit together seamlessly. This alignment process ensures that all the information from different images is correctly compared, resulting in a comprehensive and clear image. Without registration,the images could be misaligned, leading to confusion and incomplete insights. By registering the data, a user can accurately analyze and interpret the X-ray microscopy images, enabling the user to make meaningful discoveries about the intricate details of the objects being studied.

[0007] An affine registration model is a mathematical transformation used in image processing and computer vision to align and transform one image or dataset to match another. Specifically, the affine transformation preserves straight lines, parallelism, and ratios of distances between points, making it a versatile tool for correcting translations, rotations, scaling, and shearing in images.

[0008] The affine registration model involves a set of parameters that define how the transformation should be applied. These parameters include translations (shifting the image), rotations (turning the image), scaling (resizing the image), and shearing (distorting the image). By adjusting these parameters, an affine transformation can be tailored to match the features of one image to those of another.

[0009] In the context of X-ray microscopy or other imaging techniques, affine registration can be particularly useful when trying to align images of the same object or sample taken from different angles or positions. It helps correct for small misalignments or distortions that might arise due to experimental setups, equipment variations, sample movements, etc. By applying an affine registration model, researchers can ensure that images are properly aligned, enabling accurate analysis, comparisons, and visualization of the underlying structures within the images.

[0010] Fourier space registration is a technique used to align and correct images taken from different angles or positions, resulting in a more accurate and coherent composite image. In simple terms, it involves manipulating the mathematical representation of the images in a way that helps match their features and details. This technique is particularly effective because it takes advantage of the unique properties of the Fourier transform, which converts an image from its spatial domain (like a regular picture) to a frequency domain (where patterns and features are represented by different frequencies).

[0011] Fourier space registration works by first converting the images into their Fourier space representations. In this frequency domain, the images can be adjusted and aligned more easily because shifts or rotations appear as distinct patterns. By applyingmathematical transformations, the images are manipulated to minimize these shifts and bring the patterns into better alignment.

[0012] After the adjustments are made in Fourier space, the transformed images are converted back to the spatial domain, resulting in images that are properly aligned and coherent. This alignment is crucial for accurate interpretation and analysis of X-ray microscopy data. Fourier space registration enables researchers to correct for imperfections in the images and create a more detailed and accurate representation of the sample's internal structure. Fourier space registration is described in U.S. Pat. No. 8,693,805 Determination of the Relative Position of Two Structures and U.S. Pat. No. 8,694,929 Method and Apparatus for the Position Determination of Structures on a Mask for Microlithography, both of which are incorporated herein in their entirety.SUMMARY OF THE INVENTION

[0013] Previously, Fourier space registration has been applied to two-dimensional (2D) images to register multiple images using pure translation.

[0014] The present approach adds a number of innovations. It extends Fourier space registration to three (3) dimensions. It also can further utilize a multiscale technique, wherein Fourier space registration is first applied to the multiple images or volumes that were previously downsampled or smoothed. The images or volumes are then registered, and a Fourier space registration is again applied to the coarsely registered volumes in the native resolution. Previous approaches did not work well for larger deviations or high noise as they could get stuck at local minima.

[0015] Then a fine registration can be performed with both scaling and rotation. Fine registration is performed on subvolumes with the advantage that speed is increased while the computational load is reduced. The subvolumes may then be stitched back together. Or, an affine transformation is computed using sub-volumes. Then the transformation can be applied to the full image or it can be integrated directly in the reconstruction method. The fine registration process may also be performed on volumes registered by other than the Fourier space registration.

[0016] In general, according to one aspect, the invention features splitting a first volume A into n sub-volumes Ai, A2, ... A„ for a predetermined number w; splitting a second image B into n sub-volumes Bi, B2, ... B„; registering each sub-volume Ax= Ai,A2, ... An with its corresponding sub-volume Bx= Bi, B2, ... B„; and estimating a parametrized model based on the registrations of said sub-volumes.

[0017] The first image A may be a high-resolution volume and the second image B may be a low-resolution image.

[0018] Furthermore, registering each sub-volume may comprise, for each pair of corresponding subvolumes Ax, Bxwhere x is between 1 and n smoothing or downsampling subvolumes Axand Bxto form coarse subvolumes Ax', Bx'; determining a transformation Txfor registering coarse subvolumes Ax', Bx'; and based on transformation Tx, registering original subvolumes Axand Bx.

[0019] In another embodiment, a method for 3D-image registration may comprise smoothing or downsampling a first volume A to produce a first smoothed volume A'; smoothing or downsampling a second volume B to produce a second smoothed volume B'; determining a translation by performing a 3D Fourier registration on smoothed volumes A' and B'; applying the translation determined to the first and second volumes A and B; and determining a registered volume by performing a 3D Fourier registration on volumes A and B based on the applied translation.

[0020] The first volume A may be a high-resolution volume while the second B may be a low-resolution volume.

[0021] Smoothing or downsampling of the volumes may be accomplished in Fourier space.

[0022] The estimated parametrized model may be an affine linear model.

[0023] According to another aspect, the invention features a computer software product for registering a two original 3D images Ao and Bo, the product comprising a computer- readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to:

[0024] for each of m iterations for a predetermined integer m, smooth x = 1, ..., m of m iterations where m is a predetermined integer and m> 1 : smooth volumes Ao and Bo to produce coarse volumes A.vand Bvrespectively; determine a translation using a 3D Fourier registration on volumes A.vand B.v; and determine an updated registration of volumes Ao and Bo based on the translation determined in the previous step. For every iteration exceptfor x = 1, at least one of Axand Bxis less coarse than its corresponding image Av-i and Bx-i respectively.

[0025] The first image Ao may be a high-resolution volume and the second image Bo may be a low-resolution volume. Smoothing volumes Ao and Bo may be performed in Fourier space.

[0026] The above and other features of the invention including various novel details of construction and combinations of parts, and other advantages, will now be more particularly described with reference to the accompanying drawings and pointed out in the claims. It will be understood that the particular method and device embodying the invention are shown by way of illustration and not as a limitation of the invention. The principles and features of this invention may be employed in various and numerous embodiments without departing from the scope of the invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings, reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale; emphasis has instead been placed upon illustrating the principles of the invention. Of the drawings:

[0028] FIG. l is a flow diagram illustrating a scheme for a multiscale registration for two steps.

[0029] FIG. 2 is a schematic diagram illustrating a process similar to that shown in FIG. 1 wherein the first volume is a high-resolution reference volume, and the second volume is a low-resolution measurement volume.

[0030] FIG. 3 illustrates a full volume being split into sub-volumes.

[0031] FIG. 4 is a flowchart illustrating registration of volumes that have been split into sub-volumes for registration.

[0032] FIG. 5 is schematic diagram illustrating a process similar to that illustrated in FIG. 4, wherein the first volume is a high-resolution reference volume, and the second volume is a low-resolution measurement volume.

[0033] FIGS. 6-9 illustrate demonstrating an analysis of small deformations in a sample of limestone with no correction (FIG. 6), translation only corrected (FIG. 7), translation and scale both corrected (FIG. 8) and with full linear correction (FIG. 9).DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The invention now will be described more fully hereinafter with reference to the accompanying drawings, in which illustrative embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0035] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Also, all conjunctions used are to be understood in the most inclusive sense possible. Thus, the word "or" should be understood as having the definition of a logical "or" rather than that of a logical "exclusive or" unless the context clearly necessitates otherwise. Further, the singular forms and the articles "a", "an" and "the" are intended to include the plural forms as well, unless expressly stated otherwise. It will be further understood that the terms: includes, comprises, including and / or comprising, when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Further, it will be understood that when an element, including component or subsystem, is referred to and / or shown as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present.

[0036] It will be understood that although terms such as “first” and “second” are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Thus, an element discussed below could be termed a second element, and similarly, a second element may be termed a first element without departing from the teachings of the present invention.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0038] BASE METHOD : FOURIER-BASED REGISTRATION

[0039] An efficient and accurate translation-based registration method is already used in other applications such as microlithography. Such a method was patented in 2014 by Carl Zeiss SMS GmbH in 2D (U.S. Pat. No. 8,693,805 Determination of the Relative Position of Two Structures,' and U.S. Pat. No. 8,694,929 Method and Apparatus for the Position Determination of Structures on a Mask for Microlithography). This can be extended to an arbitrary number of dimensions.

[0040] The squared L2 distance between two volumes such as a high resolution volume A and a shifted low resolution volume B(s), where 5 denotes the (3D) shift with an additional mask AT to avoid boundary issues and exclude potentially bad regions

[0042] Note that the additional parameters (affine shift model) are estimated in a second step, the Fourier based method can only be used for pure translation.

[0043] This equation can easily be expressed in the Fourier domain using the Fourier shift theorem.= arg= arg= arg= arg= arg min (s) s (11)

[0044] Other obj ective functions like correlation between the two images / volumes instead of squared difference end up in a similar structure, i.e., a different kernelbut still:

[0045] Any suitable optimization method can be used to minimize this with respect to s. A gradient descent algorithm is used on f2(s). The gradient is given by:

[0046] Embodiments of the present invention extend the basic Fourier approach. First, the solutions are applied to 3D volumes. Second, the application is applied in other domains beyond microlithography, such as x-ray microscopy. Furthermore, a multiscale approach is used for large displacements. Finally, the volume is split into sub-volumes for parametrized models.

[0047] To solve the full problem, it is necessary to handle large shifts and compute at least an affine registration model. Neither the pure translation based method is sufficient nor the base version which can only handle small displacements. The present approach combines the Fourier based approach with a multiscale approach to handle larger deviations and uses multiple sub-volume registration results to estimate a parametrized registration model (affine).

[0048] Multiscale approach

[0049] Often, it is not possible to recover large shifts. To cope with large deviations, a multiscale approach is employed. A Fourier based method, for example, is first applied to a smoothed or downsampled version of the volumes. That makes the objective function smoother, so that it avoids getting stuck in local optima. To achieve this, the approach starts with heavily smoothed images, followed by a registration step, and then continues with the result on the next scale with less smoothed images until the original images are processed.

[0050] In a current implementation, two steps were sufficient. That is, after a first registration with smoothed / downsampled images, the result is taken as an initial value for a second registration step without smoothing. FIGS. 1 and 2 represent such a simplified scheme.

[0051] Affine registration model

[0052] The method is furthermore applied to many smaller sub-volumes, so that the translation is computed for many points or sub-volumes of the full volume.

[0053] FIG. 1 is a flowchart illustrating a scheme of multiscale registration in two steps. In step 101, the two volumes to be registered are smoothed. In step 103, a first 3D Fourier registration is performed on the smoothed or downsampled volumes, resulting in a translation that in step 105 is applied to the original volumes. At step 107, a second 3D Fourier registration is performed on the original volumes based on the translation that was determined in step 105, resulting in 3D Fourier registered volumes 109.

[0054] FIG. 2 is another flowchart illustrating a similar process wherein a first volume is typically a high-resolution reference or target or fixed volume 201 A, and a second volume is a high- or low-resolution measurement volume 20 IB that will be transformed to match the high-resolution reference or target or fixed volume 201 A. In steps 203A and 203B, the reference volume and measurement volume respectively are smoothed or downsampled. In one example, the resolution or bit depth of the image is reduced. In some examples, the bit depth is reduced by 50%. In other examples, it is higher such as 75% for more. A first registration is performed at step 205, resulting in a determined displacement or translation 207. The translation 207 is applied to the second volume 20 IB to thereby coarsely register the first and second volumes with respect to each other.

[0055] Then, based on that translation 207, a second registration is performed on the first volume 201 A and the now coarsely registered second volume 20 IB at step 209.

[0056] Sub-volume registration

[0057] FIG. 3 for illustrative purposes, shows a large volume 300 split into subvolumes 301a, 301b, 301c, etc. The number of subvolumes shown in FIG. 3 is merely representative; a volume may be divided into more or less subvolumes than are shown.

[0058] To make this method more robust, it can be extended by criteria to restrict the minimization to those points x, where the registration result is trustworthy. This can bedone, e.g., by intensity thresholding or a uniformity criteria to exclude those sub-volumes that are completely void or uniform and thus not suitable to derive the registration parameters.

[0059] In an embodiment, parametrization can be restricted for example to scale (diagonal matrix) and rotation. The full procedure is shown in FIGS. 4 and 5.

[0060] FIG. 4 is a flowchart illustrating registration of two volumes that have been coarsely registered according to the approach described with respect to FIGs. 1 and 2, or registered using an alternate approach.

[0061] Each of the two volumes is then split into sub-volumes for registration. At step 401, the volumes are split into sub-volumes, as was depicted in FIG. 3. In step 403, corresponding sub-volumes are registered in both scale and rotation. Finally in stop 405, the sub-volumes are stitched together.

[0062] FIG. 5 is another flowchart illustrating a similar process wherein a first volume is a high-resolution reference volume 501 A, and a second volume is a high- or low- resolution measurement or target or fixed volume 50 IB that will be transformed to match the high-resolution reference or target or fixed volume 501 A. In steps 503A and 503B, the reference volume and measurement volumes respectively are split into subvolumes. At 505, registrations are performed for each subvolume, or a subset of subvolumes, resulting in an estimated parametrized registration model 507. This subvolume registration preferably includes translations, rotations, scaling, and shearing. More specifically, each subvolume registration is still preferably translation based, but from a set of subvolumes translations, more complex transformations can be derived: such as rotations, scaling, shearing, etc.

[0063] Further embodiments may include setting up an objective function to measure similarity between images / volumes. The present invention as described above provides a very fast method for translation-based registration, for example by minimization in Fourier space to simplify the single problems.

[0064] Further embodiments may include a multiscale extension, thereby increasing the possible displacement range and increasing robustness. Reduction to smaller subvolumes may reduce or even avoid dependence on data size. Moreover, the method is parallelizable and remains fast even for large data.

[0065] Other variations may include using different objective functions; applying the method to a different dimensionality of data (images, volumes, ...); or applying the method to other applications besides X-Ray microscopy, such as microlithography, and so on. Furthermore, a different number of scales and amount of smoothing, e.g., > 2 scales might be useful / necessary for larger displacements.

[0066] Furthermore, the splitting into sub-volumes, such as the number and or size of the sub-volumes, may be varied.

[0067] Different types of parametrized models may be used. For example, while the description herein mainly used an affine linear model, arbitrary combinations of translation, rotation, scale, skew are possible, and even non-linear models could be used.

[0068] Instead of using all sub-volumes, an intensity thresholding or uniformity criteria could be applied to exclude those sub-volumes that are completely void or uniform and thus not suitable to derive the registration parameters.

[0069] An extension towards an iterative procedure could be tailored for X-ray microscopy: For example, A and B could be registered as described, and then transformation / offset parameters can be derived and passed to an image reconstruction algorithm, to generate a new shifted reconstructed volume Bl. Then registration may be rerun on a new set of images A and Bl, leading to even better precision.

[0070] FIGS. 6-9 illustrate demonstrating an analysis of small deformations in a sample of limestone with no correction (FIG. 6), translation only corrected (FIG. 7), translation and scale both corrected (FIG. 8) and with full linear correction (FIG. 9).

[0071] While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A method for image registration, comprising: splitting a first image A into n sub-volumes Ai, A2, ... A„ for a predetermined number n splitting a second image B into n sub-volumes Bi, B2, ... B„; registering each sub-volume Ax= Ai, A2, ... Awwith its corresponding subvolume Bx= Bi, B2, ... Bn, and estimating a parametrized model based on the registrations of said sub-volumes.

2. The method claim 1, wherein the first image and the second image have different resolutions and / or different acquisition parameters (magnification, kVp, filters, etc).

3. The method of either of claims 1 or 2, wherein the step of registering each subvolume comprises: for each pair of corresponding subvolumes Ax, Bxwhere x is between 1 and n smoothing subvolumes Axand Bxto form coarse subvolumes Ax', Bx'; determining transformation Txfor registering coarse subvolumes Ax', Bx'; and based on transformation Tx, registering original subvolumes Axand Bx.

4. The method of any of claims 1-3, wherein the step of estimating a parametrized model further comprises the step of stitching back together the registered subvolumes.

5. A method for 3D-image registration, comprising: smoothing a first volume A to produce a first smoothed volume A'; smoothing a second volume B to produce a second smoothed volume B'; determining a translation by performing a 3D Fourier registration on smoothed volumes A' and B'; applying the translation determined to the first and second volumes A and B; anddetermining a registered volume by performing a 3D Fourier registration on volumes A and B based on the applied translation.

6. The method claim 5, wherein the first volume A is a high-resolution volume and the second B is a low-resolution volume.

7. A computer software product for registering a first original 3D Ao image and a second original 3D image Bo, the product comprising a non-transitory computer- readable medium in which program instructions are stored, which instructions, when read by a computer, cause the computer to: for each iteration x = 1, . . . , m ofm iterations where m is a predetermined integersmooth volume Ao to produce a coarse volume A.v; smooth a volume Bo to produce a coarse volume B.v; determine a translation using a 3D Fourier registration on volumes A.vand B.v; and determine an updated registration of volumes Ao and Bo based on the translation determined in the previous step; wherein for every iteration, except for x = 1, at least one of AAand Bxis less coarse than its corresponding image Av-i and BA.i respectively.

8. The computer software product of claim 7, wherein the first image Ao is a high- resolution volume and the second image Bo is a low-resolution volume.

9. The computer software product of either of claims 7 or 8, wherein: smoothing the first volume Ao comprises applying Fourier space registration to volume Ao; and smoothing the second volume Bo comprises applying Fourier space registration to volume Bo.

10. The computer software product of claim 9, wherein m = 2.

Images