Image registration method and apparatus

Abstract

Described herein are deep learning based systems, methods, and apparatus for registering images from the same imaging modality and different imaging modalities. A neural ordinary differential equation (ODE) network comprising a plurality of layers is used to determine transformation parameters associated with an image registration task, each layer configured to determine a respective gradient update for the transformation parameters based on a current state of the transformation parameters received by the layer. The gradient updates determined by the plurality of ODE layers are then integrated and applied to an initial value of the transformation parameters to obtain final parameters for completing the image registration task. The operation of the ODE network can be facilitated by a feature extraction network that is pre-trained to determine content features shared by the input images. The input images can be resampled to different scales, which are then iteratively processed by the ODE network to improve the efficiency of the ODE operation.

Description

TECHNICAL FIELD

[0001] The present application relates to image registration techniques. BACKGROUND

[0002] Image registration plays an important role in medical imaging applications. With image registration, medical images taken at different times, from different angles, and / or across multiple imaging modalities can be spatially aligned to facilitate diagnostic analysis, treatment planning, radiotherapy, etc. Conventional image registration methods formulate the problem as an optimization problem, where the transformation parameters for registering two images are solved by minimizing the dissimilarity between a first image (e.g., a moving image) and a second image (e.g., a fixed image). These conventional methods are inherently iterative and can require online optimization. As a result, conventional methods can be time-consuming and computationally expensive. In contrast, deep learning-based image registration systems can inherently be faster because they can obtain the ability to register images using large datasets through offline learning, and once online, they can complete the image registration task in only one forward pass. Thus, it is highly desirable to leverage deep learning-based techniques for image registration to achieve improved adaptability, continuous time series modeling, increased memory and parameter efficiency, etc. SUMMARY

[0003] Systems, methods, and apparatuses associated with registering a first image and a second image of an anatomical structure are described herein. The registration can be performed using an artificial neural network (ANN), such as a neural ordinary differential equation (ODE) network, and based on a plurality of transformation parameters determined by the ANN. Such an ANN can be configured to receive initial values of the transformation parameters and determine respective updates (e.g., gradient updates) of the transformation parameters based on at least a respective current state (e.g., a hidden state) of the plurality of transformation parameters associated with each of one or more iterations. Then, a final value of the transformation parameters can be obtained based on the respective updates and used to register the first image with the second image. The final value of the transformation parameters can be solved, for example, with an ordinary differential equation (ODE) solver.

[0004] In examples, the ANN described herein can include a neural ODE network trained using an adjoint sensitivity based approach, and the final values of the transformation parameters can be obtained by integrating the respective updates determined by the ANN via an ODE solver. In some examples, the first and second images being registered can be associated with the same imaging modality, while in other examples, the first and second images can be associated with different imaging modalities, such as magnetic resonance imaging (MRI) and computed tomography (CT). In examples, the ANN can also include a generative adversarial network (GAN) that is pre-trained to extract features shared by the first and second images (e.g., when the images are captured by different imaging modalities), such that the images can be registered based on the shared features (e.g., the extracted features can be used to determine a similarity metric that can facilitate performance of the image registration task).

[0005] In examples, the ANN can include at least a first subnetwork (e.g., a first neural ODE network) and a second subnetwork (e.g., a second neural ODE network). The first subnetwork can be configured to determine a first set of transformation parameters for registering the first and second images based on respective versions of the first and second images having a first scale (e.g., a first resolution), and the second subnetwork can be configured to determine a second set of transformation parameters for registering the first and second images based on the first set of transformation parameters and respective versions of the first and second images having a second scale (e.g., a second resolution). The first and second subnetworks can be characterized by different transformation (e.g., optimization) step sizes and / or different levels of fault tolerance, which can allow the ANN to complete the image registration task with a reduced number of evaluations and / or a smaller parameter search space.

[0006] In examples, the plurality of transformation parameters determined by the ANN can include deformable transformation parameters, and the ANN can include a first neural ODE subnetwork and a second neural ODE subnetwork cascaded with the first ODE subnetwork. The first neural ODE subnetwork can be configured to determine a set of rigid transformation parameters for registering the first and second images, and the second neural ODE subnetwork can be configured to determine the deformable transformation parameters based on the set of rigid transformation parameters determined by the first neural ODE subnetwork. In examples, the deformable transformation parameters can be determined without first determining rigid (or affine) transformation parameters (e.g., without using a cascaded network structure). BRIEF DESCRIPTION OF DRAWINGS

[0007] A more detailed understanding of the examples disclosed herein can be obtained by reference to the following description in connection with the accompanying drawings, which are described briefly below.

[0008] Figure 1is a block diagram illustrating an example of image registration using artificial neural networks in accordance with one or more embodiments described herein.

[0009] Figure 2 is a block diagram illustrating an example operation of a neural ordinary differential equation (ODE) network in accordance with one or more embodiments described herein.

[0010] Figure 3 is a block diagram illustrating an example structure of a neural ODE network in accordance with one or more embodiments described herein.

[0011] Figure 4 is a flow diagram illustrating an example process for training a neural ODE network in accordance with one or more embodiments described herein.

[0012] Figure 5 is a graph illustrating an example of multi-modal image feature extraction in accordance with one or more embodiments described herein.

[0013] Figure 6 is a block diagram illustrating an example of a neural network that can be used to extract content and / or style features from images in accordance with one or more embodiments described herein.

[0014] Figure 7A is a graph illustrating an example technique for training a neural network to extract content and / or style features from images associated with a multi-modal domain in accordance with one or more embodiments described herein.

[0015] Figure 7B is a graph illustrating an example technique for training a neural network to extract content and / or style features from images associated with the same modal domain in accordance with one or more embodiments described herein.

[0016] Figure 8 is a graph illustrating an example technique for training a neural ODE network to complete an image registration task using a pre-trained feature encoder in accordance with one or more embodiments described herein.

[0017] Figure 9 is a graph illustrating an example of a multi-scale neural ODE network in accordance with one or more embodiments described herein.

[0018] Figure 10 is a graph illustrating an example of an image registration framework using two cascaded neural ODE networks in accordance with one or more embodiments described herein.

[0019] Figure 11 is a block diagram illustrating an example device that can be configured to complete an image registration task in accordance with one or more embodiments described herein. DETAILED DESCRIPTION

[0020] The present disclosure is illustrated by way of example rather than limitation in the figures.

[0021] Figure 1 This illustrates the use of an artificial neural network (ANN) 102 to register two images I mov (e.g., source image) and I fix An example image (e.g., a target image). Image I fix and I mov It can be a two-dimensional (2D) or three-dimensional (3D) medical image of an anatomical structure (e.g., the human heart or brain) obtained from one or more imaging modalities. For example, Image I fix and I mov Images can be captured at different times using the same imaging modality (such as a magnetic resonance imaging (MRI) scanner, X-ray machine, computed tomography (CT) scanner, etc.). Image I fix and I mov It can also be images captured using different imaging modalities (e.g., at the same time or at different times). For example, image I fix This can be an MRI image (e.g., including film), while image I mov It can be a CT image, or vice versa.

[0022] The neural network 102 can be configured to receive image I fix and I mov (For example, as input), image I mov From the moving image domain (e.g., with image I) mov (Associated) transformation to a fixed image domain (e.g., with image I) fix (Associated), and generate image I reg (For example, as image I) mov (spatial transformation version), this image I reg Similar to image I fix (For example, in I) fix with I reg There is a minimized dissimilarity between them (104). A neural network 102 can be trained to determine the method used to process image I. mov Transform into image I reg Multiple transformation parameters θ T .

[0023] This operation can be explained by the following formula:

[0024] I reg =I mov (θ(x)) (1)

[0025] Where x can represent the coordinates in the moving image domain, θ(x) can represent the mapping of x to the fixed image domain, and I mov (θ(x)) can represent one or more grid sampling operations (e.g., using sampler 106). θ can include parameters associated with an affine transformation model that allow translation, rotation, scaling, and / or skew of the input image. θ can also include parameters associated with a deformable field (e.g., a densely deformable field) that allows deformation of the input image. For example, θ can include stiffness parameters, B-spline control points, deformable parameters, etc.

[0026] Neural network 102 can be configured to determine the value θ of the transformation parameter based on a set of initial values ​​θ0 of the transformation parameter and an integral of the update (e.g., gradient update) of the transformation parameter determined by neural network 102. T In the example, the initial value θ0 of the transformation parameter can be obtained from a normal distribution (e.g., randomly) based on an existing image registration model, etc. In the example, the neural network 102 may include a neural ordinary differential equation (ODE) network configured to determine the transformation parameter θ by solving the ordinary differential equation associated with the transformation parameter. T Such a neural ODE network may include one or more ODE layers or ODE modules, each of which can be configured to determine (e.g., predict or estimate) a corresponding update (e.g., gradient update) to the transformation parameters based on the current or current state (e.g., current value) of the transformation parameters. For example, neural network 102 may be configured to determine the corresponding update to the transformation parameters through one or more iterations (e.g., through one or more ODE layers or modules), and each update may be determined based on the current state of the transformation parameters associated with each of the one or more iterations. The update can then be used to obtain (e.g., solve) the final value θ of the transformation parameters using an ODE solver. T For example, an ODE solver can be used to integrate a corresponding update (e.g., gradient update) determined (e.g., predicted) by one or more ODE layers or modules, and apply the integral of the update to the initial parameter value θ0 to obtain the final value θ. T .

[0027] The operation of the aforementioned neural ODE network can be explained below. The image registration task is formulated as follows: Where θ can represent the transformation parameter described in this paper, and C can represent the parameter designed to indicate I. fix with I mov The loss (or cost) function (θ(x)) of the dissimilarity between 104 can be used to find the transformation parameter θ using gradient descent-based optimization techniques (such as those exemplified below):

[0028]

[0029] Where t can represent the iteration in the optimization process, η t It can represent the optimization step size, and It can represent the current or present state θ of the transformation parameters. t The derivative of the loss function C (for example, representing the current value).

[0030] A neural ODE network can be trained to predict the expression shown in equation (2). The corresponding update (e.g., gradient update), and the update predicted by such a network, can be expressed as:

[0031] θ t+1 =θ t +f(θ t ,μ t (3)

[0032] Where f can be represented by μ t A parameterized neural ODE network. For sufficiently small t, updates can occur continuously (e.g., approximately continuously), as represented by the following ordinary differential equation:

[0033]

[0034] Among them, f μ It can represent a neural ODE network parameterized with μ.

[0035] Therefore, starting from the initial parameter value θ0, a neural ODE network can be trained, for example, by using an ODE solver to generate an output θ corresponding to the solution of the ordinary differential equation shown in (4). T (For example, the final values ​​of the transformation parameters) (For example, the function of an ODE network can be understood as solving an initial value problem over a time interval [0, T]). When the input of a neural ODE network includes images (such as...) Figure 1 Image I shown fix and I mov When the gradient update predicted by the neural ODE network (e.g., by the layers of the ODE network) is in the form of:

[0036]

[0037] Furthermore, the solution to the ordinary differential equation can be:

[0038]

[0039] and

[0040] θ t+dt =θ t +f μ (Imov (θ t (x)),I fix ,t)*dt (7)

[0041] Among them, (6) can represent the continuous calculation of registration parameters at time T, and (7) can represent a one-step solution process (e.g., from t to t+dt).

[0042] Once the registration parameter θ is obtained T The values, which can be used, for example, via one or more resampling operations that can be performed using sampler 106, to resample the input image I. mov Transform into I reg During the training of the neural ODE network, image I can be used. reg With input image I fix The images are compared, and the dissimilarity 104 between them can be determined based on a loss function (e.g., a loss function based on Euclidean distance, cross-correlation, normalized cross-correlation, etc.). The dissimilarity 104 can then be used to guide the tuning of network parameters, for example, with the aim of minimizing the dissimilarity 104.

[0043] Figure 2 This illustrates the neural ODE network described in this paper (e.g., Figure 1 The diagram illustrates an example operation of neural network 102. This example operation can be performed by neural ODE network 204 to determine the values ​​used for registering two images I. mov and I fix This is associated with step 202 of the process of transforming the parameter θ (e.g., iteration from t to t+dt). As illustrated, the operation of step 202 may include, at the beginning of the step, based on the current value θ of the transformation parameter θ. t Input image I mov Transform into image I t For example, the transformation can be performed using sampler 206 based on the transformation parameter θ. t For input image I mov Resampling is performed to execute this step. Step 202 may also include concatenating (e.g., combining) images I. t and I fix The concatenated images are then fed into the neural ODE network 204 to obtain (e.g., predict) gradient updates for the transformation parameters θ. Such gradient updates can be achieved by considering the current state θ of the transformation parameters. t The gradient update is obtained by integrating with the step size (e.g., the optimization step size) dt to obtain the updated parameter value θ. t+dtFurthermore, the neural ODE network 204 can perform similar operations in one or more additional steps or iterations (e.g., via corresponding ODE modules or layers) to find the solution θ of the ordinary differential equation. T For example, as illustrated by equations (4)-(7).

[0044] Depending on the specific problem being solved, the neural ODE network described in this paper (e.g., Figure 1 Neural network 102 and / or Figure 2 The neural network (204) is designed with different structures (e.g., layer depth, layer type, etc.). For example, to solve for rigid transformation parameters (e.g., which may be associated with 3D rigid motion), a neural ODE network may include one or more convolutional layers, followed by one or more fully connected layers. Such an ODE network may be referred to herein as a rigid ODE network. As another example, to solve for transformation parameters associated with deformable motion, a neural ODE network may include a fully convolutional network, and such an ODE network may be referred to herein as a deformable ODE network. For mixed motion (e.g., rigid and deformable motion), a neural ODE network may include rigid ODE networks and deformable ODE networks, which may be cascaded together (e.g., sequentially).

[0045] Figure 3 The neural ODE network 302 is shown (e.g., Figure 1 Neural network 102 and / or Figure 2 An example structure of a neural network 204 is shown, which can be configured to determine image registration parameters (e.g., transformation parameters) for registering a first image of an anatomical structure with a second image of the anatomical structure. The figure illustrates an ODE layer or module 304 associated with a hidden state θ(t) of the transformation parameters; however, those skilled in the art will understand that the neural ODE network 302 may include multiple such layers or modules, and the transformation parameters can be tuned through a series of transformations involving multiple hidden states. Figure 3 As shown, the ODE layer or module 304 may include one or more convolutional layers 306, one or more batch normalization (BN) layers 308, one or more activation functions 310 (e.g., modified linear unit (ReLU) activation functions), one or more pooling layers (not shown), and / or one or more fully connected layers (not shown). Each convolutional layer 306 may include multiple convolutional kernels or filters with corresponding weights (e.g., kernels with a size of 9×1 or 1×1), configured to receive images (e.g., source image I described herein) from the neural ODE network 302. mov and / or target image I fixFeature extraction. The operations of convolutional layer 306 can be followed by batch normalization (e.g., via BN layer 308) and / or linear or non-linear activation (e.g., using ReLU 310), and the features extracted by convolutional layer 306 can be downsampled via a shrinking path (e.g., including one or more pooling layers and / or one or more fully connected layers) to reduce redundancy and / or size of the extracted features. In some examples, the downsampled features can then be processed via a dilation path (e.g., including one or more transposed convolutional layers and / or one or more non-pooling layers), during which the features can be upsampled to a higher resolution.

[0046] The neural ODE network 302 can determine transformation parameters for spatial alignment of an image of interest using features extracted via the convolution operations described herein. The neural ODE network 302 can predict the transformation parameters, for example, by means of one or more ODE layers or modules 304 that continuously transform the hidden states θ(t). Each transformation may correspond to transforming the hidden states of the parameters from θ(t) to θ(t+Δt), where Δt may represent the transformation or optimization step or size. When ΔT approaches zero (e.g., when the transformation step is sufficiently small), the final state of the transformation parameters (e.g., θ(t=T)) can be obtained by solving the ODE associated with the transformation parameters (e.g., as illustrated in Equations 4-7). An ODE solver can be used to evaluate the amount of transformation (e.g., adjustment) determined and / or applied by the ODE module 304. (For example, as illustrated in Equations 6 and / or 7 as described herein), and the fault tolerance level of the ODE solver can determine the number of transformations and / or evaluations to be performed before obtaining the final values ​​of the transformation parameters. The ODE solver can be implemented using various numerical analysis techniques. For example, the ODE solver may include an Euler solver (e.g., based on the Euler method for solving ODEs), a Runge-Kutta (RK) solver such as an RK2 or RK4 solver (e.g., based on the Runge-Kutta (RK) method for solving ODEs), an adaptive step-size solver (e.g., based on an adaptive step-size method for solving ODEs), etc. The ODE solver can be a standalone solver (e.g., separate from the neural ODE network 302) or it can be part of the neural ODE network 302 (e.g., the ODE solver itself can be learned through training). The fault tolerance level of the ODE solver can be configurable (e.g., as a hyperparameter of the neural ODE network 302) and can be assigned the same or different values ​​for training and inference purposes.

[0047] Therefore, using Figure 3 As shown in the example structure, the neural ODE network 302 can be trained to learn continuous dynamic properties (e.g., updates or adjustments) associated with the transformation parameters. Figure 4An example procedure 400 for training the neural ODE network described herein is illustrated. Procedure 400 may begin at 402, and at 404, initial parameters of the neural network (e.g., weights associated with various filters or kernels of the neural network) may be set. The initial network parameters may be obtained, for example, based on samples of one or more probability distributions or parameter values ​​from another neural network with a similar architecture. At 406, the neural ODE network may obtain (e.g., be provided with) a first input training image I. mov Second input training image I fix and used to train the first training image I mov With the second training image I fix The initial set of registered transformation parameters θ0. At 408, the neural ODE network can predict one or more hidden states θ with respect to the transformation parameters. t (e.g., for t = 1...T) the corresponding updates to the transformation parameters (e.g., gradient updates dθ(t) / dt), and the predictions can be evaluated using an ODE solver (e.g., the RK2 or RK4 ODE solver as described herein). For example, the updates can be based on the changes from the input image I. mov and I fix The hidden state θ of the extracted features and / or transform parameters t To predict. The update of the prediction can be integrated using the ODE solver, and the integral can be used to obtain the distorted image (e.g., I). reg Then the distorted image can be compared with the input image I. fix The results are compared to determine the effectiveness of the predictions made by the neural network.

[0048] At 410, the neural ODE network can determine whether one or more training termination criteria are met. For example, if the neural network has completed a predetermined number of training iterations, or if the difference between the predicted result and the expected result is below a predetermined threshold, the neural network can determine that the training termination criteria have been met. If the training termination criteria are determined to be met at 410, the neural ODE network can terminate the training process at 412. Otherwise, the neural ODE network can adjust the neural network parameters at 414 with the aim of minimizing the distortion image I. reg With input image I fix The differences between them (e.g., based on any differentiable loss function such as the L2 loss function). Adjustment can be performed, for example, using an adjoint sensitivity approach, during which the gradient of the loss function can be computed by solving a second (e.g., extended) ODE in reverse time. For example, if the loss function is represented as L, then it is related to the hidden state θ. t (For example, t = 1...N) The gradient dL / dθ of the associated loss function tThis can be expressed as a(t) = dL / dθ t (For example, representing the adjoint matrix). The dynamic properties of the adjoint matrix a(t) (e.g., representing the gradient of the loss function) can be given by another ODE, which can be solved by calling a second ODE solver. This second ODE solver can run in reverse, from dL / dθ N Starting with the initial value, the hidden state θ is recalculated in reverse time along with the adjoint matrix. t (For example, using the final hidden state θ) N Therefore, using this adjoint matrix method, backpropagation through the ODE solver is not required, and since the accuracy of the predictions made by the neural ODE network can be controlled by the number of evaluations performed (e.g., depending on the fault tolerance level of the ODE solver), the neural ODE network described in this paper can be used to achieve memory and / or parameter efficiency (e.g., the neural ODE model can be very deep without incurring significant memory and / or parameter overhead).

[0049] In the example (e.g., for registering images generated from different imaging modalities), the image registration operation described herein can be facilitated by a neural network pre-trained to extract features from the input image, enabling the learning (e.g., based on the extracted features) of modality-independent metrics to facilitate the registration operation. This feature extraction neural network can be trained to extract at least two types of features: style features that reflect the global contrast of the image and content features that represent structural information about the anatomical structures depicted in the image. Style features can be modality-specific (e.g., MR and CT images of the same anatomical structure can have different style features), while content features can be modality-independent (e.g., images share content features even if they were generated from different imaging modalities such as MRI and CT). In the example, the feature extraction neural network can additionally include a generator neural network trained to reconstruct the image based on the extracted content and / or style features.

[0050] Figure 5 An example of multimodal image feature extraction (e.g., image transformation) using the techniques described above is illustrated. As shown, X and X' can represent two images of an anatomical structure captured by different imaging modalities D and D' (e.g., MRI and CT) with corresponding modal domains. A content feature encoder E can be trained. c (For example, it could be part of a GAN network) to extract content features C shared by images from the input image. A style feature encoder E can be trained. s(For example, it can be part of the same GAN network) to extract corresponding style features S and S' of the image from the input image. Then, the image generation network G can use the extracted content features and style features to reconstruct an image corresponding to the input image X or X' (for example, the output image can be a transformed version of the input image).

[0051] Figure 6 Example structures of GANs that can be used to perform the feature extraction operations described herein are shown, along with example methods for training GANs. As shown in the figure, a GAN may include a content feature encoder E. c and style feature encoder E s Content Feature Encoder E c It can be a fully convolutional network pre-trained to extract content features C from an input image X, while the style feature encoder E... s The GAN may include one or more convolutional layers, one or more pooling layers (e.g., global average pooling layers), and / or fully connected layers pre-trained to extract style features (e.g., global style features) from the input image X. Modal codes may be converted into one-hot vectors d concatenated with the input tensor along the channel dimensions. The GAN may also include a decoder that uses a multilayer perceptron (MLP) network to generate a set of adaptive instance normalization (AdaIN) parameters from the encoded style features, and upsamples the encoded content / style features to a higher resolution, for example, using one or more transposed convolutional layers and / or non-pooling layers. The GAN may also include a generator network G, which may be pre-trained to receive the extracted content features C and style features S, and reconstruct the image X' based on these features. The generator G may include one or more deconvolutional layers (e.g., transposed convolutional layers) configured to output the image X' at the same resolution (or size) as the input image X. Training of the GAN may be performed in an unsupervised manner, during which the generator G may be trained to force the discriminator network D... S and D C Distinguish between generated samples (e.g., labeled "fake") and real samples (e.g., labeled "true").

[0052] Figure 7A An example technique is illustrated for training a GAN to extract content features and / or style features from images associated with n (e.g., n>=2) modal domains. During one or more training iterations, features can be extracted from different domains (e.g., D ... a and D b Select two sample images X a and X b And provide it to the GAN. In response, the GAN can use the content feature encoder described in this paper (e.g., Figure 5 or Figure 6E shown c ) and style feature encoder (e.g., Figure 5 or Figure 6 E shown s From sample image X a and X b Extract the corresponding style feature S a S b and content features C a C b Utilizing the extracted content features C b and style features S a GANs can utilize the generator networks described in this paper (e.g., Figure 5 or Figure 6 The generator network G shown generates (e.g., reconstructs) image X. b-a It can represent the input image X b Transformed to domain D a Similarly, GANs can utilize the extracted content features C a and style features S b To generate image X a-b It can represent the input image X a Transformed to domain D b GANs can then decompose image X. b-a (For example, using the feature encoder described herein) to obtain style feature S' a and content features C' b This makes the style feature S' a With S a Style consistency loss L S cc and content features C b With C' b Content consistency loss L C cc This can be used to force a GAN to adapt its parameters (e.g., the goal is to minimize these losses). In a similar way, a GAN can decompose an image X. a-b To obtain style feature S' b and content features C' a This makes the style feature S' b With S b Style consistency loss L S cc and content features C a With C' a Content consistency loss L C cc This can be used to force GANs to further adapt their parameters (e.g., with the aim of minimizing these losses).

[0053] GANs can reconstruct image X separately. a-b-a It can represent image X a Transformed to domain D b And then convert back to domain D a Then, image X a With reconstructed image X a-b-a Reconstruction consistency loss L CX cc This can be used to further optimize GAN operations. GANs can also reconstruct images that can represent X. b Transformed to domain D a And then it is transformed back to domain D. b Image X b-a-b And using image X b With reconstructed image X b-a-b Reconstruction consistency loss L CX cc Let's learn more about the parameters of GAN.

[0054] The cross-domain training of the aforementioned GAN can be accompanied by image reconstruction training within each domain. Figure 7B and Figure 7A One example illustrates the use of reconstruction loss L X cc (For example, for cycle consistency) and L X ad An example of such a training operation (e.g., for adversarial loss). It should be noted that the loss function described herein is not limited to cross-correlation loss. Euclidean distance loss or other suitable loss functions can also be used to minimize the dissimilarity between transformed images. And once the GAN is trained, the GAN's content feature encoder (e.g., Figure 5 or Figure 6 E shown c It can be used to extract features from images (e.g., multimodal images) and to determine similarity or dissimilarity metrics that can be used to facilitate image registration.

[0055] Figure 8 An example technique is shown that utilizes a pre-trained GAN (e.g., a pre-trained feature encoder of a GAN) to train a neural ODE network for image registration. As shown, the image registration task can involve two input images X. a (e.g., moving images) and X b (e.g., a fixed image). As part of the training and / or inference operation, the transformation parameters θ determined by the neural ODE network 802 can be used. T Use sampler 804 (e.g., Figure 1 Sampler 106 or Figure 2Sampler 206 in the image X a Resampling to obtain the transformed image X' a A pre-trained content encoder E can be used. c (For example, Figure 5 or Figure 6 E shown c From image X b Extracting content features C b And from the resampled image X' a Extracting content features C' a (For example, the encoder E can be pre-learned and fixed during the training of the neural ODE 802.) c (Parameters or weights). This can be found in the extracted content features C' a With C b Determine the loss L between content And it was used to optimize the parameters of the neural ODE network 802. Because encoder E c Able to obtain from image X' a and image X b Both extract shared content features (even if the images belong to different modal domains (e.g., MRI and CT)), so the neural ODE network 802 can also learn to register images regardless of whether the images come from the same or different modal domains.

[0056] In the example, one or more additional loss functions can be used for training. Figure 8 The neural ODE802 is shown. For example, during training, a random perturbation can be added to the transformation parameter θ to obtain the perturbation parameter θ'". Then, the moving image X can be processed based on the perturbation parameter θ'". a Resampling to obtain the perturbed image X” a The disturbed image X” can be a and the original image X a Provided to neural ODE 802 for registration (e.g., to determine the transformation parameter θ based on the image pair). T Since the value of the perturbation parameter θ” ​​is known, the perturbation parameter can be used as the gold standard to determine the loss L. self It can be used to minimize the network prediction parameter θ” T The dissimilarity between the standard and the gold standard θ was investigated, and the parameters of the neural ODE 802 were optimized.

[0057] In the example, the neural ODE network described herein can be a multi-scale neural ODE network (e.g., multiple scales can correspond to different sizes or resolutions of the image to be registered). Figure 9An example of such a neural ODE network 900 is shown. As illustrated, the neural ODE 900 may include multiple blocks or subsets 902, each block or subset being configured to process images of a specific scale, and the characteristics of each block or subset may lie in a corresponding step size (e.g., an approximate step size) and / or a corresponding fault tolerance level. For example, by processing the previous image X... L-1 Perform downsampling (e.g., 1 / 2 downsampling) to obtain each scaled image X. L Therefore, after downsampling the input image L times, the transformation parameter predicted by the corresponding Lth ODE module or sub-network can be θ. T / L This reflects the scale level L. Then, these parameters θ can be used. T / L This initializes the image registration operation at scale level L-1 and the corresponding (L-1)th ODE module or subnetwork. The same or similar operations can be performed iteratively (e.g., cyclically) for other scale levels until level 1 (e.g., corresponding to the original image size) is reached. The output parameters θ of the neural ODE 900 can then be determined by integrating the corresponding outputs at all scale levels. T Since the time cost of function evaluation can be much smaller at low resolutions, the multi-scale architecture of Neural ODE 900 allows the network to complete image registration tasks with a reduced number of function evaluations. The search space can also be smaller, resulting in faster convergence and lower sensitivity to local optima.

[0058] The neural ODE network described in this paper can be adapted to predict rigid, affine, and / or deformable transformation parameters. In the example, rigid transformation parameters can be estimated first, and then used to estimate deformable transformation parameters. Figure 10 An exemplary registration framework using two cascaded neural ODE networks 1002a and 1002b is shown. Neural ODE 1002a can be configured to estimate the rigid transformation parameters θ based on the initial values ​​θ0 of the parameters. T Predicted rigid transformation parameters θ T This can then be used to initialize the neural ODE 1002b so that it is based on the initial values ​​of the deformable transformation parameters. and image pair [X' a X b To predict deformable transformation parameters (For example, sampler 1004 can be used based on θ) T To obtain X' a (As described in this article). Rigid transformations performed by the neural ODE 1002a can move the image X. a Closer to a fixed image X bThis reduces the parameters used for optimal deformable transformation. This expands the search space and accelerates the registration process. It should be noted that... Figure 10 The network structures shown are for illustrative purposes and are not intended to limit how transformation parameters can be determined. For example, the neural ODE network described herein can be used to determine deformable transformation parameters (e.g., deformable transformation parameters only) without first determining rigid (or affine) transformation parameters (e.g., without using cascaded networks as described herein).

[0059] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 11 This is a block diagram illustrating an example device 1100 that can be configured to perform the image registration operations described herein. As shown, device 1100 may include a processor (e.g., one or more processors) 1102, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 1100 may also include communication circuitry 1104, memory 1106, mass storage device 1108, input device 1110, and / or communication link 1112 (e.g., communication bus) through which one or more components shown in the figure may exchange information.

[0060] Communication circuitry 1104 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 1106 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 1102 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 1108 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 1102. Input device 1110 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from device 1100.

[0061] It should be noted that device 1100 can operate as a standalone device or can be connected to other computing devices (e.g., networked or clustered) to perform the functions described herein. And even in Figure 11 Only one example of each component is shown in the figure, and those skilled in the art will understand that device 1100 may include multiple instances of one or more components shown in the figure.

[0062] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data representing physical (e.g., electronic) quantities within the registers and memories of the computer system into other data representing physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0063] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A method of image registration, the method comprising: obtaining a first image of an anatomical structure; obtaining a second image of the anatomical structure; determining a plurality of transformation parameters for registering the first image with the second image, wherein the plurality of transformation parameters are determined at least by: obtaining initial values for the plurality of transformation parameters; determining, using an artificial neural network (ANN), respective updates for the plurality of transformation parameters through one or more iterations, wherein each of the respective updates is determined based on a respective current state of the plurality of transformation parameters associated with each of the one or more iterations; and obtaining final values for the plurality of transformation parameters based on the respective updates determined by the ANN using an ordinary differential equation (ODE) solver; and registering the first image with the second image using the final values for the plurality of transformation parameters; wherein the ANN comprises at least a first subnetwork and a second subnetwork, the first subnetwork is configured to determine a first set of transformation parameters for registering the first image with the second image based on respective versions of the first image and the second image characterized by a first resolution, and the second subnetwork is configured to determine a second set of transformation parameters for registering the first image with the second image based on the first set of transformation parameters and respective versions of the first image and the second image characterized by a second resolution.

2. The method of claim 1, wherein, the ANN comprises a neural ordinary differential equation (ODE) network, and wherein obtaining the final values for the plurality of transformation parameters based on the respective updates determined by the ANN comprises integrating the respective updates determined by the ANN using the ODE solver.

3. The method of claim 2, wherein, the final values for the plurality of transformation parameters are obtained based on an integration of the updates and the initial values for the plurality of transformation parameters.

4. The method of claim 1, wherein, the ANN is trained using an adjoint sensitivity based method.

5. The method of claim 1, wherein, the first image and the second image are of a same imaging modality; or, the first image and the second image are of different imaging modalities, wherein the method further comprises determining one or more features shared by the first image and the second image using a generative adversarial network (GAN), and wherein the first image is registered with the second image based on the one or more features.

6. The method of claim 5, wherein, the first image is obtained via magnetic resonance imaging (MRI) and the second image is obtained via computed tomography (CT).

7. The method of claim 1, wherein, the first subnetwork comprises a first neural ordinary differential equation (ODE) solver characterized by a first step size or a first fault tolerance level, and the second subnetwork comprises a second ODE solver characterized by a second step size or a second fault tolerance level.

8. The method of claim 1, wherein, The plurality of transformation parameters for registering the first image with the second image includes a deformable transformation parameter, the ANN includes a first subnetwork and a second subnetwork cascaded with the first subnetwork, the first subnetwork is configured to determine a set of rigid transformation parameters for registering the first image with the second image, and the second subnetwork is configured to determine the deformable transformation parameter based on the set of rigid transformation parameters determined by the first subnetwork.

9. An image registration device comprising: one or more processors configured to: obtain a first image of an anatomical structure; obtain a second image of the anatomical structure; determine a plurality of transformation parameters for registering the first image with the second image, wherein the one or more processors are configured to determine the plurality of transformation parameters at least by: obtaining initial values for the plurality of transformation parameters; determining respective updates for the plurality of transformation parameters using an artificial neural network (ANN) through one or more iterations, wherein each of the updates is determined based on a respective current state of the plurality of transformation parameters associated with each of the one or more iterations; and obtaining final values for the plurality of transformation parameters using an ordinary differential equation (ODE) solver based on the respective updates determined by the ANN; and registering the first image with the second image using the final values for the plurality of transformation parameters; wherein the ANN includes at least a first subnetwork and a second subnetwork, the first subnetwork is configured to determine a first set of transformation parameters for registering the first image with the second image based on features of the first image and the second image at respective versions of a first resolution, and the second subnetwork is configured to determine a second set of transformation parameters for registering the first image with the second image based on the first set of transformation parameters and features of the first image and the second image at respective versions of a second resolution.

10. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an image registration device, cause the one or more processors to perform operations comprising: obtaining a first image of an anatomical structure; obtaining a second image of the anatomical structure; determining a plurality of transformation parameters for registering the first image with the second image, wherein the one or more processors are configured to determine the plurality of transformation parameters at least by: obtaining initial values for the plurality of transformation parameters; determining respective updates for the plurality of transformation parameters using an artificial neural network (ANN) through one or more iterations, wherein each of the updates is determined based on a respective current state of the plurality of transformation parameters associated with each of the one or more iterations; and obtaining final values for the plurality of transformation parameters using an ordinary differential equation (ODE) solver based on the respective updates determined by the ANN; and registering the first image with the second image using the final values for the plurality of transformation parameters; wherein the ANN includes at least a first subnetwork and a second subnetwork, the first subnetwork is configured to determine a first set of transformation parameters for registering the first image with the second image based on features of the first image and the second image at respective versions of a first resolution, and the second subnetwork is configured to determine a second set of transformation parameters for registering the first image with the second image based on the first set of transformation parameters and features of the first image and the second image at respective versions of a second resolution.

Images