Image registration

Abstract

One method for 3D image registration includes smoothing two volume data sets, determining translations by registering the smoothed volume data, applying the translations to the original (pre-smoothed) volume data, and registering the original volume data based on the translations. These coarsely registered volume data sets can then be divided into sub-volume data sets for further registration, resulting in a parametric model that takes into account translation, rotation, scaling, and shearing.

Description

Related applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 585,230, filed September 26, 2023, pursuant to 35 USC §119(e), which is incorporated herein by reference in its entirety. Background Technology

[0002] X-ray microscopy (XRM) is an imaging technique that allows scientists and researchers to visualize the internal structure of materials and biological samples in extraordinary detail. X-ray microscopy uses X-rays to penetrate and provide insights typically at the micrometer and nanometer scale.

[0003] The fundamental principle behind X-ray microscopy involves directing an X-ray beam onto a sample of interest. X-rays provide information about the sample's internal features through absorption, scattering, and refraction. These interactions are detected using specialized detectors and advanced computational reconstruction algorithms, and then converted into detailed images or projections and volume data.

[0004] X-ray microscopy has applications across a wide range of fields, from materials science and nanotechnology to biology and medicine. Researchers are able to explore complex structures within cells, study the growth of nanomaterials, and examine the elemental distribution in geological samples.

[0005] In X-ray microscopy data, registration is often crucial to producing a complete and accurate view of the object or sample under study. In X-ray microscopy, researchers use different resolutions, acquisition settings, and more general angles to capture images to reveal the internal structure of tiny objects such as cells or materials. However, due to these differences, these images cannot be perfectly aligned, except for slight offsets during imaging and / or slight movement of the sample, and distortion may be introduced. This problem also arises when aligning masks.

[0006] Registration solves this problem by adjusting images to fit together seamlessly. The alignment process ensures that all information from different images is correctly compared to obtain a comprehensive and clear picture. Without registration, images may be misaligned, leading to confusion and incomplete insights. With registered data, users can accurately analyze and interpret X-ray microscope images, enabling them to make meaningful discoveries about the complex details of the objects under study.

[0007] Affine registration is a mathematical transformation used in image processing and computer vision to align and transform one image or dataset to match another. Specifically, affine transformations preserve lines, parallelism, and the ratio of distances between points, making them a versatile tool for correcting translations, rotations, scaling, and cropping in images.

[0008] Affine registration models involve defining a set of parameters that define how the transformation should be applied. These parameters include translation (offsetting the image), rotation (flipping the image), scaling (resizing the image), and cropping (distorting the image). By adjusting these parameters, affine transformations can be cropped to match the features of one image with those of another.

[0009] In the context of X-ray microscopy or other imaging techniques, affine registration can be particularly useful when attempting to align images of the same object or sample taken from different angles or positions. This helps correct for minute misalignments or distortions that may arise due to experimental setup, equipment differences, sample movement, etc. By applying affine registration models, researchers can ensure proper image alignment, enabling accurate analysis, comparison, and visualization of the underlying structures within the images.

[0010] Fourier spatial registration is a technique used to align and correct images taken from different angles or positions to produce more accurate and coherent synthetic images. In short, it involves manipulating the mathematical representation of an image in a way that helps match image features and details. This technique is particularly effective because it leverages the unique properties of the Fourier transform, which transforms an image from its spatial domain (such as a regular picture) to the frequency domain (where patterns and features are represented by different frequencies).

[0011] Fourier space registration works by first transforming images to their Fourier space representation. In this frequency domain, images can be more easily adjusted and aligned because offsets or rotations manifest as different patterns. By applying mathematical transformations, the images are manipulated to minimize these offsets and bring these patterns into a better alignment.

[0012] After adjustment in Fourier space, the transformed image is converted back to the spatial domain to produce a properly aligned and coherent image. This alignment is crucial for the accurate interpretation and analysis of X-ray microscopy data. Fourier space registration allows researchers to correct imperfections in images and create a more detailed and accurate representation of the internal structure of a sample. Fourier space registration is described in U.S. Patent No. 8,693,805, “Determination of the Relative Position of Two Structures,” and U.S. Patent No. 8,694,929, “Method and Apparatus for the Position Determination of Structures on a Mask for Microlithography,” both of which are incorporated herein by reference. Summary of the Invention

[0013] Previously, Fourier space registration had been applied to two-dimensional (2D) images to register multiple images using pure translation.

[0014] This invention adds several innovations. It extends Fourier space registration to three (3) dimensions. It also enables further utilization of multi-scale techniques, where Fourier space registration is first applied to multiple image or volume data that have been previously downsampled or smoothed. The image or volume data is then registered, and the Fourier space registration is applied again to the coarsely registered volume data at the original resolution. Previous methods have not worked well for large biases or high noise levels because they may get stuck in local minima.

[0015] Fine registration can then be performed using scaling and rotation. Fine registration of sub-volume data offers the advantage of increased speed while reducing computational load. The sub-volume data can then be stitched together. Alternatively, an affine transformation can be computed using the sub-volume data. This transformation can then be applied to the entire image or directly incorporated into the reconstruction method. Fine registration can also be performed on volume data registered using non-Fourier space registration.

[0016] Generally, according to one aspect, the present invention is characterized by dividing the first body data A into n sub-body data A1, A2...A1 according to a predetermined number n. n The second image B is divided into n sub-data units B1, B2...B n ; Transfer each child data A x =A1, A2, …A n Its corresponding sub-data B x =B1, B2, … Bn Registration is performed; and a parameterized model is estimated based on the registration of the subbody data.

[0017] The first image A can be high-resolution volumetric data, and the second image B can be a low-resolution image.

[0018] In addition, registering each child body data can include for each pair of corresponding child body data A x B x Where x is between 1 and n: for sub-data A x B x Smoothing or downsampling is performed to form coarse subvolume data A. x '、B x '; Determine the data A used for registration coarse body data' x '、B x Transformation T x ; and based on transformation T x Register the original daughter body data A x and B x .

[0019] In another embodiment, a method for 3D image registration may include smoothing or downsampling a first volume data A to produce a first smoothed volume data A'; smoothing or downsampling a second volume data B to produce a second smoothed volume data B'; determining a translation by performing 3D Fourier registration on the smoothed volume data A' and B'; applying the determined translation to the first volume data A and the second volume data B; and determining registered volume data by performing 3D Fourier registration on the volume data A and B based on the applied translation.

[0020] The first volume data A can be high-resolution volume data, while the second volume data B can be low-resolution volume data.

[0021] Smoothing and downsampling of volume data can be achieved in Fourier space.

[0022] The estimated parameterized model can be a ray-like model.

[0023] According to another aspect, the present invention is characterized by a computer software product for registering two original 3D images A0 and B0, the product comprising a computer-readable medium storing program instructions, which, when read by a computer, cause the computer to:

[0024] For each of the m iterations according to a predetermined integer m, smooth the m iterations of x = 1, …, m, where m is a predetermined integer and m ≥ 1; smooth the bulk data A0 and B0 respectively to generate the bold data A. x and B x Using volume data A x and Bx The 3D Fourier registration is used to determine the translation; and based on the translation determined in the previous step, the updated registration of volume data A0 and B0 is determined. For each iteration except x=1, A x and B x The roughness of at least one of them is smaller than that of its corresponding image A. x-1 and B x-1 .

[0025] The first image A0 can be high-resolution volumetric data, and the second image B0 can be low-resolution volumetric data. Smoothing of the volumetric data A0 and B0 can be performed in Fourier space.

[0026] The above and other features of the present invention, including various novel construction details and component combinations, and other advantages, will be described in more detail with reference to the accompanying drawings and are pointed out in the claims. It should be understood that the specific methods and apparatus embodied in the present invention are for illustrative purposes only and are not intended to limit the invention. The principles and features of the present invention can be applied to a wide variety of embodiments without departing from the scope of the invention. Attached Figure Description

[0027] In the accompanying drawings, reference numerals refer to the same parts in different views. These drawings are not necessarily drawn to scale; rather, the focus is on illustrating the principles of the invention. In the drawings:

[0028] Figure 1 This is a flowchart illustrating a two-step multi-scale registration scheme.

[0029] Figure 2 It is shown that... Figure 1 The diagram illustrates a similar process, where the first volume data is high-resolution reference volume data and the second volume data is low-resolution measurement volume data.

[0030] Figure 3 This shows that the complete data is divided into sub-data.

[0031] Figure 4 This is a flowchart illustrating volume data registration, where the data has been divided into sub-volumes for registration.

[0032] Figure 5 It is shown that... Figure 4 The diagram illustrates a similar process, where the first volume data is high-resolution reference volume data and the second volume data is low-resolution measurement volume data.

[0033] Figures 6-9 It shows no correction ( Figure 6 Translation correction only Figure 7 Translation and scaling correction () Figure 8 ) and full linear correction ( Figure 9Demonstration of small deformation analysis of limestone samples. Detailed Implementation

[0034] The present invention will be described in more detail below with reference to the accompanying drawings, in which illustrative embodiments of the invention are shown. However, the invention can be embodied in many different forms and should not be construed as limiting the embodiments described herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0035] As used herein, the term “and / or” includes any and all combinations of one or more of the related listed items. Furthermore, all conjunctions used should be understood in the broadest possible sense. Therefore, the word “or” should be understood as having the logical definition of “or” rather than the logical definition of “exclusive or”, unless the context explicitly requires otherwise. Additionally, unless otherwise explicitly stated, the singular form and the articles “a,” “an,” and “this” also include the plural form. It should be further understood that, when used in this specification, the terms “comprising,” “including,” “constituting,” and “or forming” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and combinations thereof. Furthermore, it should be understood that when an element (including a component or subsystem) relates to and / or is shown as connected to or coupled to another element, that element is capable of being directly connected to or coupled to that other element, or there may be intermediate elements.

[0036] It should be understood that while terms such as “first” and “second” are used herein to describe various elements, these elements are not limited to these terms. These terms are used only to distinguish one element from another. Thus, an element discussed below may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the teachings of the invention.

[0037] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should be further understood that terms such as those defined in common dictionaries should be understood to have the same meaning as they have in the relevant technical context, and should not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0038] Basic method: Fourier-based registration

[0039] An efficient and accurate translation-based registration method has been used in other applications, such as photolithography. This method was patented in 2014 by Carl Zeiss SMS LLC in 2D (US Patent No. 8,693,805 Determination of the Relative Position of Two Structures and US Patent No. 8,694,929 Method and Apparatus for the Position Determination of Structures on a Mask for Microlithography). This method can be extended to any number of dimensions.

[0040] Two volume data (such as high-resolution volume data) and offset low-resolution volume data The squared distance L2 between ) is , where This indicates an additional mask that avoids boundary issues and eliminates potential undesirable areas. 3D offset (1)

[0041] Note that the additional parameters (affine migration model) are estimated in the second step, and the Fourier-based method can only be used for pure translation.

[0042] The equation can be easily expressed in the Fourier domain using the Fourier shift theorem. (2) (3) (4) (5) (6) (7) (8) (9) (10) (11)

[0043] Other objective functions, such as the correlation between two image / volume data points instead of the squared difference, ultimately end up with a similar structure, i.e., different kernel functions. But it is still:

[0044] Any suitable optimization method can be used to minimize the value of this function with respect to s. Apply the gradient descent algorithm. The gradient is given by the following equation:

[0045] Embodiments of this invention extend the basic Fourier method. First, these solutions are applied to 3D volumetric data. Second, the application is extended to fields beyond photolithography, such as X-ray microscopy. Furthermore, a multi-scale method is used for large displacements. Finally, the volumetric data is divided into sub-volume data of a parameterized model.

[0046] To address the entire problem, large offsets must be handled and at least one affine registration model must be computed. Simple translation-based methods and basic versions that can only handle small displacements are insufficient. This method combines Fourier-based and multi-scale approaches to handle larger offsets and uses registration results from multiple subvolumes to estimate the parametric registration model (affine).

[0047] Multiscale methods

[0048] Large shifts are typically impossible to recover. To handle large biases, multi-scale methods are employed. For example, Fourier-based methods are first applied to a smoothed or downsampled version of the volume data. This makes the objective function smoother, thus avoiding getting trapped in local optima. To achieve this, the method starts with a highly smoothed image, followed by a registration step, and then continues using less smoothed images at the next scale until the original image is processed.

[0049] In the current implementation, two steps are sufficient. That is, after the first registration using a smoothed / downsampled image, the result is used as the initial value for the second registration step without smoothing. Figure 1 and Figure 2 This simplification scheme is illustrated.

[0050] Affine registration model

[0051] This method can be further applied to many smaller sub-volumes to calculate the translation of many points in the whole volume data or sub-volume data.

[0052] Figure 1This is a flowchart illustrating a two-step multi-scale registration scheme. In step 101, the two volume data to be registered are smoothed. In step 103, the smoothed or downsampled volume data undergoes a first 3D Fourier registration, resulting in a translation applied to the original volume data in step 105. In step 107, based on the translation determined in step 105, a second 3D Fourier registration is performed on the original volume data, generating 3D Fourier registered volume data 109.

[0053] Figure 2 This is another flowchart illustrating a similar process, where the first volume data is typically a high-resolution reference or target or stationary volume data 201A, and the second volume data is high-resolution or low-resolution measurement volume data 201B that will be transformed to match the high-resolution reference or target or stationary volume data 201A. In steps 203A and 203B, the reference volume data and the measurement volume data are smoothed or downsampled, respectively. In one example, the resolution or bit depth of the image is reduced. In some examples, the bit depth is reduced by 50%. In other examples, it is reduced by more, such as 75% or more. A first registration is performed at step 205, resulting in a determined displacement or translation 207. The translation 207 is applied to the second volume data 201B, thereby coarsely registering the first volume data and the second volume data relative to each other.

[0054] Then, based on translation 207, a second registration is performed on the first body data 201A and the currently coarsely registered second body data 201B at step 209.

[0055] Subbody data registration

[0056] For the purpose of explanation, Figure 3 The general data 300 is shown, which is divided into sub-data 301a, 301b, 301c, etc. Figure 3 The multiple sub-body data shown are only representative; the body data can be divided into more or fewer sub-body data than shown.

[0057] To make the method more robust, it can be extended by standards to limit the minimum to points x where the registration result is reliable. This can be achieved by excluding those sub-volume data that are completely empty or homogeneous and therefore unsuitable for deriving registration parameters, using criteria such as intensity thresholds or homogeneity.

[0058] In one embodiment, parameterization can be limited, for example, to scaling (diagonal matrix) and rotation. The entire process is as follows: Figure 4 and Figure 5 As shown.

[0059] Figure 4 It shows that it has been based on Figure 1 and Figure 2The flowchart describes the registration of two volume data using the method described above, or the registration using an alternative method.

[0060] Then, each of the two volume data sets is divided into sub-volume data for registration. At step 401, the volume data is divided into sub-volume data, such as... Figure 3 As shown. In step 403, the corresponding sub-data is registered by scaling and rotation. Finally, in step 405, the sub-data is stitched together.

[0061] Figure 5 This is another flowchart illustrating a similar process, where the first volume data is high-resolution reference volume data 501A, and the second volume data is high-resolution or low-resolution measurement or target or stationary volume data 501B that will be transformed to match the high-resolution reference or target or stationary volume data 501A. In steps 503A and 503B, the reference volume data and measurement volume data are respectively divided into sub-volume data. At 505, each sub-volume data or subset of sub-volume data is registered to generate an estimated parameterized registration model 507. This sub-volume data registration preferably includes translation, rotation, scaling, and shearing. More specifically, each sub-volume data registration is still preferably based on translation, but more complex transformations such as rotation, scaling, and shearing can be derived from the translation of a set of sub-volume data.

[0062] Further embodiments may include setting an objective function to measure the similarity between image / volume data. As described above, the present invention provides a very fast method for translation-based registration, for example, to simplify a single problem by minimizing in Fourier space.

[0063] Further embodiments may include multi-scale scaling, thereby increasing the possible range of displacements and improving robustness. Reducing to smaller sub-data volumes can reduce or even eliminate dependence on data size. Furthermore, the method is parallelized, remaining fast even for large datasets.

[0064] Other variations may include using different objective functions; applying the method to data of different dimensions (images, volume data, etc.); or applying the method to applications other than X-ray microscopy, such as photolithography. Furthermore, different numbers of scales and smoothing amounts (e.g., >2 scales) may be useful / necessary for larger displacements.

[0065] Furthermore, the data can be divided into sub-data, and the number and / or size of the sub-data can be variable.

[0066] Different types of parametric models can be useful. For example, although this specification primarily uses affine linear models, arbitrary combinations of translation, rotation, scaling, and skewing are possible, and even nonlinear models can be used.

[0067] Instead of using all sub-data, it is possible to apply intensity thresholds or uniformity criteria to exclude sub-data that is completely empty or uniform and therefore unsuitable for deriving registration parameters.

[0068] The iterative process can be tailored for X-ray microscopy: for example, A and B can be registered as described, then the transform / offset parameters can be derived and passed to an image reconstruction algorithm to generate new offset reconstructed volume data B1. Registration can then be rerun on the new image sets A and B1, resulting in higher accuracy.

[0069] Figures 6-9 It shows no correction ( Figure 6 Translation correction only Figure 7 Translation and scaling correction () Figure 8 ) and full linear correction ( Figure 9 Demonstration of small deformation analysis of limestone samples.

[0070] While the invention has been specifically described and illustrated through its preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as covered by the appended claims.

Claims

1. A method for image registration, comprising: The first image A is divided into n sub-data A1, A2...A1 according to a predetermined number n. n ; Divide the second image B into n sub-data units B1, B2...B n ; Data A of each child x =A1, A2, … A n Its corresponding sub-data B x =B1, B2, … B n Perform registration; and The parameterized model is estimated based on the registration of the subbody data.

2. The method according to claim 1, wherein, The first image and the second image have different resolutions and / or different acquisition parameters (magnification, kVp, filter, etc.).

3. The method according to claim 1 or 2, wherein, The steps for registering each child body's data include: For each pair of corresponding child data A x B x Where x is between 1 and n: Data A of the subbody x and B x Smoothing is performed to form coarse subbody data A. x '、B x '; Determine the coarse body data A used for registration x '、B x Transformation T x ;as well as Based on transformation T x Register the original daughter body data A x and B x .

4. The method according to any one of claims 1-3, wherein, The step of estimating the parameterized model further includes the step of stitching together the registered sub-body data.

5. A method for 3D image registration, comprising: Smooth the first volume data A to produce the first smoothed volume data A'; Smooth the second volume data B to produce a second smoothed volume data B'; The translation was determined by performing 3D Fourier registration on the smooth volume data A' and B'. The determined translation is applied to the first volume data A and the second volume data B; and The registration data is determined by performing 3D Fourier registration based on the applied translational volume data A and B.

6. The method according to claim 5, wherein, The first volume data A is high-resolution volume data, and the second volume data B is low-resolution volume data.

7. A computer software product for registering a first original 3D image A0 and a second original 3D image B0, the product comprising a non-transitory computer-readable medium storing program instructions that, when read by a computer, cause the computer to: For each iteration in m iterations, x = 1, …, m, where m is a predetermined integer and m ≥ 1: Smooth volume data A0 to generate bold data A x ; Smooth volume data B0 to generate bold data B x ; Using body data A x and B x 3D Fourier registration is used to determine translation; and Based on the translation determined in the previous steps, the updated registration of volume data A0 and B0 is determined; For each iteration except x=1, A x and B x The roughness of at least one of them is smaller than that of its corresponding image A. x-1 and B x-1 .

8. The computer software product according to claim 7, wherein, The first image A0 is high-resolution volumetric data, and the second image B0 is low-resolution volumetric data.

9. The computer software product according to claim 7 or 8, wherein: Smoothing the first volume data A0 includes applying Fourier spatial registration to the volume data A0; as well as Smoothing the second volume data B0 involves applying Fourier space registration to the volume data B0.

10. The computer software product according to claim 9, wherein, m=2。

Images