Preserving tumor volumes for unsupervised medical image registration

Abstract

A similarity-based deformable registration of a medical image containing a tumor region typically undesirably results in an unproportionate volume change in the tumor region. An image registration framework for preserving tumor volume while facilitating similarity in non-tumor portions is provided. The framework involves a two-stage process. In a first stage, similarity-based registration is used to identify potential tumor regions by their volume changes, thereby generating a soft tumor mask accordingly. In a second stage, a volume-preserving registration network is used. The network uses a novel adaptive volume retention loss that adaptively treats a change in size based on the mask calculated in the previous stage. The framework balances image similarity and volume retention in different regions, i.e., non-tumor regions and tumor regions, by using a soft tumor mask to adjust the volume retention loss applied to each region. It ensures that the tumor volume is retained during the registration process.

Description

[0001] abbreviation

[0002] 2D Two Dimensions

[0003] 3D

[0004] CNN Convolutional Neural Network

[0005] CT Computerized tomography

[0006] MRI

[0007] MRI Magnetic Resonance Imaging

[0008] RCN Recursive Cascade Network

[0009] STM soft tumor mask

[0010] STSR squared tumor size ratio

[0011] TSR tumor size ratio Technical Field

[0012] The present disclosure generally relates to image registration of 3D medical images. In particular, the present disclosure relates to warpable registration of a moving image and a fixed image, wherein the moving image is a medical image having one or more tumor regions, and the one or more tumor regions are volumetrically preserved when the moving image is warped to align with the fixed image. Background Art

[0013] Deformable image registration is a fundamental task of estimating the nonlinear spatial correspondence between two images. It is useful for medical imaging studies involving 3D images of organs or tissues, such as MR brain scans and CT liver scans. Recently, various deep learning based methods have been proposed in this field, which have shown better performance and significant speedup in inference time compared to traditional registration methods [4],

[35] ,

[29] ,

[13] ,

[14] .

[0014] Currently, most learning-based methods [4],

[35] train registration models to achieve high similarity in image intensities

[35] , [4] or anatomical landmark maps

[13] . Although it is useful in various medical applications, such as atlas-based segmentation and image fusion [7], it becomes problematic when studying tumor images [3],

[11] ,

[22] . Tracking tumor growth is a core task in cancer treatment, which can be used to evaluate the results of radiotherapy and chemotherapy, and to plan optimal postoperative treatment

[11] . This process requires registration to align the anatomical structures of patient images at different times while preserving tumor characteristics, such as size. Traditional models using conventional registration have difficulty in achieving this task

[11] . We observe that this remains a problem for current learning-based methods and no feasible solution has yet been proposed.

[0015] In the present disclosure, a deformable registration technique is developed to avoid tumor volume changes during registration. Summary of the invention

[0016] Mathematical equations cited in this summary can be found in the detailed description.

[0017] One aspect of the present disclosure is to provide a computer-implemented method for performing deformable registration of a moving image and a fixed image. The moving image is a medical image that images an organ having one or more tumor regions. Each of the moving image and the fixed image is a 3D image.

[0018] The method includes performing a first-stage computational process for generating an STM. An STM is a probability distribution map indicating the likelihood that a single voxel of an organ in a moving image is located in one or more tumor regions. The method also includes performing a second-stage computational process for generating a warped image. The warped image is a 3D image obtained by warping a moving image to align with a fixed image. The second-stage computational process is a machine learning process that can be trained under the goal of minimizing a training loss calculated as a weighted sum of multiple component losses. In particular, the multiple component losses include: a volume preservation loss for measuring the loss caused by failing to preserve the volume of one or more tumor regions unchanged during warping of the moving image; and a weighted similarity loss for measuring the similarity loss of a non-tumor portion of an organ between the warped image and the fixed image; so that one or more tumor regions are retained in volume during the generation of the warped image from the moving image, while promoting the similarity of the non-tumor portion between the moving image and the fixed image during the alignment of the moving image with the fixed image. Each of the volume preservation loss and the weighted similarity loss is calculated at least according to the STM.

[0019] Preferably, the first stage calculation process includes: using a first stage registration network to generate a first stage deformation field, which is used to distort the moving image to align with the fixed image; estimating the volume change at each voxel of the organ when distorting the moving image according to the first stage deformation field, thereby generating a first stage volume change distribution map on the organ; and mapping the first stage volume change distribution map to a probability distribution map, thereby generating an STM.

[0020] Preferably, the first stage registration network is a similarity-based registration network that is modeled by the first stage multiple parameters and is trainable. The first stage calculation process also includes training the first stage registration network to fix the first stage multiple parameters before using the first stage registration network to generate the first stage deformation field.

[0021] In certain embodiments, the first phase volume change distribution map on the organ is calculated by EQN.(16).

[0022] In the process of mapping the first stage volume change distribution map to the probability distribution map, a transformation function is used to process the volume change distance to generate a synthetic probability value of a single voxel in the probability distribution map. In some embodiments, the transformation function is a sigmoid function.

[0023] Preferably, the second stage calculation process includes: using a second stage registration network to generate a second stage deformation field for warping the moving image to align with the fixed image, wherein the second stage registration network is trainable and modeled by multiple parameters of the second stage; spatially transforming the moving image according to the second stage deformation field to generate a distorted image; training the second stage registration network by optimizing multiple parameters of the second stage to minimize the training loss; estimating the volume change at each voxel of the organ when the moving image is distorted according to the second stage deformation field, thereby generating a second stage volume change distribution map on the organ; when calculating the training loss, calculating the volume preservation loss according to the second stage volume change distribution map and the STM; and when calculating the training loss, calculating the weighted similarity loss according to the distorted image, the fixed image and the STM.

[0024] Preferably, the multiple component losses also include a regularization loss for measuring the loss caused by overfitting. The second stage calculation process also includes, when calculating the training loss, calculating the regularization loss according to the second stage deformation field.

[0025] In certain embodiments, the second phase volume change distribution map on the organ is calculated by EQN.(18).

[0026] The volume preservation loss and weighted similarity loss can be calculated by EQNS.(10) and (12) respectively. The training loss can be calculated by EQN.(13).

[0027] In some embodiments, a pre-registration step is performed before performing the first stage calculation process. The pre-registration step includes: filtering the moving image with a bilateral filter to generate a filtered moving image, wherein the bilateral filter is configured to preserve the edges of the moving image and smooth its internal area; using the first stage registration network to align the filtered moving image and the fixed image to generate a third stage deformation field; and spatially transforming the moving image according to the third stage deformation field to generate an unfiltered warped image, so that the edges of the unfiltered warped image can be aligned with the fixed image without introducing volume changes to the edges. In addition, during the first stage calculation process, the first stage deformation field is generated by aligning the unfiltered warped image (rather than the moving image) with the fixed image using the first stage registration network.

[0028] As shown in the following examples, other aspects of the disclosure are disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 As an illustrative example, fixed and moving images are depicted, along with a first phase image indicating a tumor location determined by similarity-based registration, and a second phase image indicating a corresponding tumor location determined by volume-preserving registration using the disclosed framework.

[0030] Figure 2 A schematic diagram of a framework for unsupervised deformable registration of moving and fixed images as disclosed in the present disclosure is exemplarily depicted. The framework has two stages. In the first stage, the STM indicating the tumor region is estimated by analyzing the Jacobian matrix of the deformation field obtained by the registration network pre-trained by the similarity loss. In the second stage, the STM is used to guide the calculation of both volume preservation and similarity loss to train the volume preservation registration network. This deformation field estimated by the second stage ensures the preservation of both image similarity and tumor size.

[0031] Figure 3 Examples of visualizing an STM with and without performing a pre-registration step before generating the STM are provided, where image (a) is the estimated STM without pre-registration and image (b) is the STM with pre-registration.

[0032] Figure 4 A table presenting a comprehensive comparison of the disclosed framework with two traditional methods and three learning-based methods under different network architectures is provided.

[0033] Figure 5 Tables presenting the results of our evaluation of the effectiveness of various versions of volume preservation loss are provided.

[0034] Figure 6The robustness of the disclosed framework in the presence of noisy ground truth segmentation masks is illustrated.

[0035] Figure 7 A flow chart showing exemplary steps of a computer-implemented method for performing deformable registration of moving and fixed images developed based on the disclosed framework is depicted.

[0036] Figure 8 A flow chart showing preferred steps used in a first phase computational process of the disclosed method according to a first phase of the disclosed framework is depicted.

[0037] Fig. 9 Depicted is a flow chart showing preferred steps used in the second phase calculation process of the disclosed method according to the second phase of the disclosed framework.

[0038] Fig.10 Depicted is a flow chart showing exemplary steps taken in an optional pre-registration step performed prior to the first stage computation process.

[0039] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. DETAILED DESCRIPTION

[0040] As used in this specification and the appended claims, the term "avoid" or "avoiding" refers to any method that partially or completely excludes, avoids, eliminates, prevents, stops, hinders or delays the consequences or phenomena after the term "avoid" or "avoiding" occurs. The term "avoid" or "avoiding" does not mean that it is necessarily absolute, but effectively provides a certain degree of avoidance or prevention or improvement of the consequences or phenomena after the term "avoid" or "avoiding".

[0041] As used herein, a "3D image" is an ordered sequence of 2D images, wherein the 2D images are respectively obtained at different positions along a straight line, and the plane of each 2D image is perpendicular to the straight line. It should be noted that 3D images as used herein do not mean that computer-generated graphics provide a human observer with visual depth perception similar to viewing real-world objects.

[0042] As used herein, "registration" of a first stage image and a second stage image or "registering" a first stage image and a second stage image means finding a spatial transformation (e.g., scaling, rotation, and translation) that aligns the two images. "Deformable registration" of a first stage image and a second stage image means registering the first stage image and the second stage image when the spatial transformation is nonlinear. Nonlinear spatial transformations can be performed using a spatially varying deformation model.

[0043] In the context of image registration, a "moving image" (also called a "source image") is spatially transformed or warped to align with a "fixed image" (also called a "target image"), thereby forming a "warped image".

[0044] This paper discloses a two-stage framework for unsupervised deformable registration of moving and fixed images. Section 1 provides an overview of the framework. Section 2 compares the framework with the prior art. The methods used in the framework for registering moving and fixed images are provided in Section 3. Section 4 provides experimental results obtained when testing the framework. Based on various aspects of the framework disclosed in Sections 1 to 4, embodiments of the present disclosure are developed in Section 5.

[0045] 1. Overview of the disclosed framework

[0046] First, with the help of Figure 1 To illustrate the advantages of the disclosed framework. Figure 1 Depicted are a fixed image 110 and a moving image 120, as well as a first stage image 130 indicating a tumor location determined by similarity-based registration and a second stage image 140 indicating a corresponding tumor location determined by volume-preserving registration as disclosed herein. Figure 1 In FIG. 1 , the boxes on the first stage image 130 and the second stage image 140 represent the original position of the tumor in the moving image 120 before distortion. In image registration with the tumor, similarity-based registration usually results in significant changes in the size of the tumor, while our volume-preserving registration as disclosed herein is able to preserve the tumor anatomical structure. Figure 1As shown in , when a tumor image is aligned with an atlas image using a popular unsupervised registration network, the volume of the tumor is significantly reduced. This problem is very common because mainstream registration networks focus on optimizing the similarity between image pairs and ignore the tumor region. Tumor regions often lack corresponding parts, and the size and location of tumors can vary greatly even for images of the same patient. This happens because tumors change shape, size, location, or disappear over time. Therefore, in a deformable setting, the size of the tumor changes disproportionately to maximize image similarity. This problem may be more severe when the registration network has better distortion capabilities, resulting in worse performance than previous work. The disproportionate change of tumor volume in deformable registration is fatal for clinical assessment of tumor growth and greatly hinders the application of registration in the clinic. The disclosed framework aims to avoid this disproportionate change of tumor volume in deformable registration.

[0047] In the present disclosure, the deformable registration problem is reformulated as a constrained problem of preserving the characteristics of the tumor in the original image, namely the shape and size of the tumor, while maximizing the image similarity in non-tumor regions. For example, in tumor image registration with an atlas, in addition to the alignment of anatomical structures, the tumor should also have a deformation behavior similar to that of its surrounding organs. That is, the size of the tumor should vary proportionally to the size of the organ. However, it poses two major challenges. First, due to the severe shortage of labeled data, most existing registration methods rely on unsupervised learning. Therefore, a key issue is how to identify tumor regions in an unsupervised setting. Second, after the tumor region is identified, the challenge of preserving the tumor volume while maximizing the similarity of other regions remains. Finding a way to balance these two competing objectives is crucial to achieving effective tumor image registration.

[0048] To address the above two challenges, this paper discloses a novel two-stage framework that operates in an unsupervised manner. In the first stage, a similarity-based registration network is used to identify regions that experience excessive volume changes during registration. Such regions indicate the presence of tumors because tumors tend to exhibit large volume changes due to the lack of correspondence between image pairs. The result of the first stage is an STM that distinguishes normal regions from possible tumor regions. In the second stage, we introduce an adaptive volume-preserving loss to train the volume-preserving registration network. Based on the STM obtained from the first stage, this loss is designed to adaptively and unfavorably treat (or penalize) volume changes in different regions of the image. Possible tumor regions are assigned larger loss weights, and the similarity loss weights are adjusted accordingly in the opposite direction. The application of the adaptive volume-preserving loss effectively balances the similarity and volume preservation in different parts of the image.

[0049] It is important to note that the first-stage STM does not need to be highly accurate to achieve the desired results, as the second stage is robust, as demonstrated by the ablation experiments in Section 4. In addition to evaluating the robustness of the disclosed framework on various datasets and frameworks (including CNNs and transformers and different types of imaging data, such as brain and abdomen, MRI and CT images), we also introduced the STSR metric to measure the preservation of tumor volume. The results demonstrate that the disclosed framework achieves comparable warping performance metrics, such as Dice coefficient and average landmark distance, while efficiently preserving tumor characteristics measured by STSR.

[0050] 2. Comparison of the published framework with related works

[0051] 2.1. On Learning-Based Deformable Image Registration

[0052] Compared to conventional registration methods, deep networks using various network structures such as CNNs and Transformers have achieved remarkable registration accuracy and speed in applications such as atlas-to-image registration

[26] , [6],

[15] , [5],

[29] ,

[35] , [4],

[13] ,

[14] , [7]. They generally adopt unsupervised learning, using image similarity on image intensity

[26] , [6], [5],

[35] or synthetic label maps

[13] ,

[14] and smoothness regularization to supervise the registration network

[26] , [6], [5],

[13] ,

[14] ,

[35] . However, existing methods do not fully consider the presence of tumors. Tumors usually have no corresponding parts between image pairs, so optimizing for similarity will excessively change their volume, limiting the application of registration networks in critical clinical applications such as cancer treatment. In this paper, we study this problem using a learning-based approach and propose a metric that specifically preserves tumor measurement volume.

[0053] 2.2. Deformable Image Registration for Tumors

[0054] Conventional registration models typically address image-tumor registration by: excluding tumor similarity metrics

[10] ,

[17] , which requires extensive manual segmentation

[10] or initial seeds to explicitly model tumor growth

[11] ,

[17] ; jointly segmenting and registering images [8], which is time-consuming as it requires iteratively detecting image intensity dissimilarity and performing resection and retraction; and reconstructing quasi-normal images from pathological images

[12] ,

[18] , which is also slow as the reconstruction needs to be iteratively improved.

[0055] Recently, learning-based tumor registration methods

[22] ,

[20] ,

[30] have been proposed for the release of the BraTSReg dataset [3]. These methods aim to improve the spatial correspondence of non-tumor regions. Therefore, these methods either directly annotate landmarks to supervise the registration

[20] or introduce reversibility in the registration to mask out non-reversed regions and ensure high correspondence in other regions

[22] ,

[30] . However, these methods assume that there is no tumor in one image of the registration pair and may not be applicable in practical applications where two tumor images obtained from one patient need to be compared to track tumor growth. In this scenario, reversibility of the tumor is still possible and, therefore, it is crucial to consider the preservation of tumor properties during registration. The disclosed framework achieves this requirement.

[0056] 2.3. About Unsupervised Tumor Segmentation

[0057] To ensure that the tumor volume is preserved during registration, it is crucial to first determine the location of the tumor. Recent studies have demonstrated that current deep learning methods have made significant progress in unsupervised tumor segmentation

[31] ,

[32] ,

[19] . These methods generally use generative adversarial network models or simulate tumors on healthy images to generate synthetic data, which is then used to train the segmentation network. Although a well-trained network can achieve high segmentation accuracy, it often requires complex training strategies and additional model or testing steps to achieve the desired performance, as pointed out in recent studies

[32] ,

[19] . However, in our approach disclosed in this disclosure, we do not rely on complex segmentation methods to accurately segment the tumor. Instead, our experiments, as presented in Section 4, demonstrate that our tumor mask estimation is sufficient to enable the disclosed framework to learn volume-preserving registration of tumors.

[0058] 2.4. Volume-preserving deformable image registration

[0059] Volume-preserving registration has been widely studied to improve the accuracy of anatomical registration. For example, in lung CT image registration, the change in tissue intensity between the inspiration and expiration phases can be used as prior knowledge, and adding volume-preserving constraints has been shown to be effective

[28] ,

[34] . In other cases, such as when soft tissue is incompressible or when images are from different modalities, volume-preserving constraints can also serve as useful regularization

[23] ,

[27] .

[0060] In contrast, the work disclosed in this paper focuses on preserving tumor characteristics using volume preservation during image registration. The disclosed framework adaptively imposes this constraint on different parts of the image in order to preserve tumor volume and ensure anatomical accuracy, while previous works either preserve the volume of the entire image

[23] ,

[27] or use simple methods to determine the area to preserve

[28] ,

[34] .

[0061] 3. Methodology

[0062] 3.1. Preparation

[0063] Problem setting. Given a moving image I defined in a hyper-d-dimensional (d=3 as used here) space Ω m and a fixed image I f , warpable image registration aims to find a dense deformation field φ:Ω→Ω that distorts I m With I f In this disclosure, the task is to construct an algorithm that accepts input I m and I f The registration network F is used to predict the deformation field φ θ Network F θ That is to say,

[0064] φ=F θ (I m ,I f ), (1)

[0065] I w (x) = I m (φ(x)+x), (2)

[0066] as well as

[0067] I w ~I f (3)

[0068] Evaluation Criteria. In general, successful deformable image registration involves accurately aligning the anatomical structures while also ensuring a smooth deformation field. It is usually achieved by using a similarity-based objective and a regularization method that promotes smoothness. However, in the case of image registration with tumors, simply aligning the anatomical structures is not enough. As discussed previously, preserving the tumor volume is critical to accurately track tumor growth. In particular, given a tumor with T m and surrounding organs m Moving Image I m and with corresponding tumor T w and corresponding peripheral organs O w Distorted moving image I w , tumor T m The distortion should be similar to O m Therefore, in terms of volume preservation, the change in tumor size |T m |Should be related to the changes of the included organs|O m | proportional, that is:

[0069]

[0070] as well as

[0071] TSR(I m )≈TSR(I w ). (5)

[0072] Therefore, we define TSR as |T| / |O|, which should be preserved after registration. The reformulated registration goal is now to maximize image similarity while preserving the tumor volume, i.e., TSR.

[0073] 3.2. Overall framework

[0074] Figure 2 A schematic diagram of a framework (labeled 200) disclosed in the present disclosure is exemplarily depicted. The disclosed framework 200 has two stages, namely a first stage 210 of unsupervised tumor mask estimation and a second stage 220 of volume-preserving registration.

[0075] The first stage 210 estimates potential tumor regions and generates an STM 251 indicating these tumor regions. This is done via analysis of the volume changes in the deformation field (referred to as the first stage deformation field 212) obtained by the existing similarity-based registration network 211, and in particular via calculation of the Jacobian matrix 213. Since tumor regions tend to undergo excessive volume changes during registration, they can be identified through this analysis.

[0076] In the second stage 220, a volume preserving registration 221 is performed on the tumor using an adaptive volume preserving loss 261. This loss 261 is designed to effectively guide the volume preserving process and is associated with the STM 251 estimated in the first stage 210. By assigning different weights to different regions (including both normal regions and tumor regions), it is possible to preserve the tumor volume while also ensuring a high similarity of other regions.

[0077] 3.2.1. Stage 1 210: Unsupervised Tumor Mask Estimation

[0078] Following previous work [4], [5],

[35] , we first train a similarity-based registration network F with parameters θ1. θ1 211 and fix its parameters. Figure 1 As shown in , tumors in medical images can be detected by analyzing volume changes through similarity-based registration. Our goal is to exploit this property to facilitate unsupervised estimation of STM 251 to preserve volume during registration.

[0079] To determine the volume change at voxel x, we compute the Jacobian matrix 213 (J θ). (In the first stage 210, θ is given by θ1). Our goal is to ensure that the volume change 214 of each point in the tumor region is similar to the volume change of the organ, which is an indicator of the change in organ size (as shown in EQN.(6) shown below). To measure the distance in volume change between the two, we use EQNS.(8) and (7) to count the number of times one ratio exceeds the other, denoted as D1. EQNS.(6) to (8) is given by

[0080] From EQN.(5),(6)

[0081]

[0082] as well as

[0083] D θ (x) = max(D′) θ (x),1 / D′ θ (x)). (8)

[0084] It should be noted that D θ (x) is the volume change distance for voxel x, which is calculated from the deformation field obtained under the parameters. θ (x) is calculated under θ = θ1. EQN. (8) is also applied to the second stage 220, giving D θ (x) is calculated under θ=θ2.

[0085] In practice, we estimate the organ mask O by warping the reference image with the ground truth organ segmentation m and O w The reference images can be randomly selected from the training dataset and the choice of reference images does not significantly affect the results, since the registration of organs achieves high accuracy, as shown in Figure 2. Figure 4 This is demonstrated by the results of the Dice values ​​listed in the table shown in . By warping the reference image, we obtain the moving image O m Then, we calculate the distorted image O w organ segmentation.

[0086] In order to get the distance D defined by EQN.(8) from θ=θ1 θ1 214 To obtain STM 251, we use the transformation function Tr 215. This function 215 can be expressed as

[0087] STM(x)=Tr(D θ1 (x)) (9)

[0088] Where D θ1(x) 214 represents the distance value at voxel x, and Tr 215 transforms the value from the range of [1,∞) to the range of [0,1]. A value of 1 indicates the presence of a tumor, while a value of 0 indicates a normal area. It can be seen that STM 251 can be regarded as a probability distribution map indicating the likelihood that an individual voxel is a tumor. In the actual implementation, we use Tr(x) = sigm(5·(x-1.5)) because the sigmoid function is widely used to convert values ​​to the range of [0,1], and according to our experience, the size of the tumor region varies by about 1.5 times on average. However, this choice of transformation function 215 is not exclusive, as we demonstrate in the experiments in Section 4. We also set the area outside the organ mask to 0.

[0089] 3.2.2. Second stage 220: Volume-preserving registration of the tumor

[0090] After obtaining STM 251, we can adaptively select the tumor region to maintain the size ratio. Here, we use the volume change distance D calculated under θ = θ2 as shown in EQN. (8) θ2 (x) is used to define the volume preservation loss 261. By minimizing the distance, we can achieve smaller volume changes, indicating better preservation of the tumor volume. The volume preservation loss 261 is multiplied by the STM 251. It ensures that the volume preservation constraint is adaptively applied to the tumor region instead of the normal region, which can be formulated as:

[0091]

[0092] Where: STM(x) is the value of the STM of voxel x of the organ in the moving image; and |I m | is the volume of the moving image. Here, in the second stage, the registration network takes as input the STM 251, which is used as an indicator of the regions that need to be preserved in terms of volume. We follow the similarity and regularization loss formulations in

[35] , but we use a weighted similarity loss to adapt the volume preservation loss, which can be formulated as:

[0093]

[0094] as well as

[0095]

[0096] in: Yes I w Average value; I w (x) is the I at voxel x w value; isI f The average value of I f (x) is the I at voxel xf Value; Cov′[I w ,I f ] is I w and I f The weighted covariance of Cov[I w ,I w Yes w covariance; and Cov[I f ,I f ] is I f The covariance of .

[0097] Therefore, the training loss of the second stage 220, denoted as L, is given by

[0098] L=L w_sim +α1·L VP +α2·L reg , (13)

[0099] where α1 and α2 are hyperparameters that control the relative importance of volume preservation loss and regularization loss.

[0100] 3.2.3. Pre-registration

[0101] We observed that some normal regions, especially those near organ boundaries, may also show some volume changes after registration. Therefore, these regions may be erroneously included in STM 251, such as Figure 3 As shown in image (a) in . To address this issue, we introduced a pre-registration step that first aligns the edges and excludes the volume changes caused by edge alignment. In particular, in the pre-registration step, a bilateral filter is applied to the moving image 282, which preserves the edges while smoothing the internal regions. We then use the registration network 211 to register the filtered moving image with the fixed image 281 in order to align their edges. Thereafter, an unfiltered warped image is obtained by spatially transforming the unfiltered moving image with the deformation field. The first stage 210 of the framework 200 is then performed on the unfiltered warped image, which focuses on the registration of the internal regions and is able to accurately estimate the tumor mask, as shown in FIG. Figure 3 As shown in the image (b) in . Therefore, this pre-registration pipeline effectively selects the tumor region while filtering out the non-tumor region.

[0102] 4. Experiment

[0103] Experimental setup

[0104] Implementation. Following the implementation of RCN

[35] , we adopted PyTorch and used the same similarity loss and regularization loss (for affine and deformable, respectively) as the baseline method. The weight of the volume preservation loss was set to 0.1, while the weight of the similarity loss was 1 and the weight of the regularization loss was 0.1. The model was trained on an NVIDIA GeForce RTX 3090 GPU with a batch size of 4. During the training phase, we used the Adam optimizer

[33] and ran a total of 5 epochs with 20,000 iterations in each epoch. The learning rate was set to 10 -4 .

[0105] Baselines. We reimplement our framework on three state-of-the-art learning-based models with different network architectures: 1) Recursive Cascade Networks with VTN Base Network (VTN)

[35] , which stacks CNN networks to achieve higher registration accuracy; 2) VoxelMorph [4], which is a widely used CNN network that utilizes static vector fields to register images; and 3) TransMorph [5], which is a hybrid Transformer-ConvNet network with good performance. We fit these learning-based networks into our framework with relatively little effort and achieve good performance. In addition, following

[35] , we compare our proposed method with two state-of-the-art traditional methods for deformable image registration, SyN [1] (integrated with the affine stage in ANTs [2]) and B-Spline

[24] (integrated with the affine stage in Elastix

[16] ).

[0106] Evaluation Metrics. The evaluation of the proposed framework consists of five metrics: Dice score, landmark distance to measure anatomical alignment, and square tumor size ratio metric to measure tumor volume preservation. More specifically, Dice points are calculated using the following formula:

[0107]

[0108] Additionally, we use landmark annotations to measure the anatomical alignment between the warped and fixed images. We compute the average distance between the landmarks of the fixed image and the warped landmarks of the moving image, as introduced in RCN

[35] . To quantify the diffeomorphism and smoothness of the deformation field, we follow the methodology of previous works such as [6], [5],

[36] . Two metrics are used here: the number of non-positive Jacobians (|J θ |<0%), and the Jacobian determinant (Std.|J θ |).

[0109] To measure tumor volume retention, we proposed STSR, which is calculated from the change in TSR defined in EQN.(4):

[0110]

[0111] Here, I m and I w refers to moving images and distorted moving images. The squared term is used to account for larger changes in tumor volume, which may have a more significant impact on the accuracy of tumor growth estimation in clinical applications. By utilizing the STSR metric, we are able to accurately evaluate the performance of our proposed framework.

[0112] Preprocessing. In our work, we follow the standard preprocessing steps as outlined in VTN

[36] and VoxelMorph [4]. The original scans are resampled to 128×128×128 voxels after cropping any unnecessary regions around the target object.

[0113] Dataset

[0114] Training. To train our model, we leveraged two publicly available datasets: (1) LiTS17[9]: The LiTS dataset consists of 201 CT scans acquired using various CT scanners and devices. The resolution of the images in this dataset ranges from 0.56 mm to 1.0 mm in the axial direction and from 0.45 mm to 6.0 mm in the z direction. The number of slices in the z direction ranges from 42 to 1026. (2) BraTS20

[21] : It consists of 369 MRI scans of brain tumors, each with a ground truth segmentation of four different tumor components. To train our neural network, we leverage the T1ce modality and consider all four components as tumors, while considering the entire brain as surrounding organs.

[0115] Testing. For testing, we utilized a set of 10 CT liver scans of metastatic tumors and 10 MRI brain scans of metastatic tumors. The mode of the MRI brain scans was T1ce. Each case included both pre- and post-treatment images of the same patient, resulting in a total of 20 scans per set. Subsequently, three experts in the field carefully annotated the scan segmentation containing the organ (liver or brain) and the internal tumor

[25] . Three experts in the field also annotated the landmarks to evaluate the registration accuracy of our framework.

[0116] 4.3. Results and Ablation Studies

[0117] Quantitative comparison

[0118] Figure 4Tables presenting a comprehensive comparison of our proposed framework with two traditional methods and three learning-based methods under different network architectures (CNN, cascaded CNN, and hybrid transformer-CNN) are provided. We compare these methods using a similarity-based approach (conventional) following the setting in

[35] and our volume-preserving approach. Our results demonstrate that our proposed framework outperforms all other methods in terms of STSR metrics, while achieving comparable results in Dice and Landmark Dist, and in fold detection (|J θ |<0) and smoothness quality (Std.|J θ |) generally achieves better results. Furthermore, we observe that registration networks that achieve better performance in terms of previous metrics (such as Dice) tend to have worse performance in terms of preserving tumor volume (STSR). This observation highlights the necessity of balancing these two objectives in order to achieve competitive results.

[0119] 4.3.2. Volume retention loss

[0120] Figure 5 Tables presenting our evaluation results on the effectiveness of various versions of volume-preserving loss are provided. For each sub-table, the first row (#1, #6, and #11) represents the use of only similarity loss, while the second row (#2, #7, and #12) utilizes the ground truth as a weighted similarity loss. The third row (#3, #8, and #13) uses the ground truth binary mask as the tumor mask, while the fourth row (#4, #9, and #14) uses the mask of the entire organ as the tumor mask. The fifth row (#5, #10, and #15) represents our proposed framework.

[0121] By comparing the first and second rows of each sub-table with the fifth row, we observe that our proposed volume-preserving loss can effectively preserve details and achieve excellent performance across various metrics. The improvement from the third and fourth rows to the fifth row correspondingly indicates the importance of adaptivity in preserving organ and tumor volumes. It should be noted that we do not need pixel-wise ground truth to preserve tumor volume.

[0122] 4.3.3. Robustness of Tumor Mask Estimation

[0123] exist Figure 6 In , we show the robustness of the disclosed framework in the presence of noisy ground truth segmentation masks. To evaluate this, we randomly select points within the organ to ensure that the tumor has a certain Dice score relative to the ground truth segmentation of the tumor. Figure 6As shown, despite the variations in the noisy ground truth segmentation mask, the performance of the disclosed framework remains consistent. In particular, when the Dice score of the tumor mask is above a certain threshold, such as 0.15, the estimated tumor mask consistently achieves a Dice score of approximately 0.17. In contrast, using the entire organ as a mask in the volume preservation loss leads to degraded performance as it only has a Dice score of approximately 0.05. This highlights the necessity and efficiency of performing STM estimation in the first stage 210.

[0124] 5. Details of the embodiments of the present disclosure

[0125] Based on the details, examples, applications, etc. of the two-stage framework 200 for solving the deformable registration problem as described above, embodiments of the present disclosure are described in detail as follows.

[0126] One aspect of the present disclosure is to provide a computer-implemented method for performing deformable registration of a moving image and a fixed image. The moving image is a medical image that images an organ having one or more tumor regions. Each of the moving image and the fixed image is a 3D image.

[0127] With the help of Figure 7 and Figure 2 The disclosed method is illustrated. Figure 7 A flow chart (labeled 700) showing exemplary steps of the disclosed method is depicted.

[0128] Exemplarily, the method 700 includes executing a first-stage calculation process 710 and a second-stage calculation process 720. The first-stage calculation process 710 and the second-stage calculation process 720 correspond to the first stage 210 and the second stage 220, respectively.

[0129] The first stage computation process 710 is arranged to generate an STM 251. The STM 251 is a probability distribution map indicating the likelihood that individual voxels of an organ in the moving image 282 are located in one or more tumor regions.

[0130] The second stage computational process 720 is arranged to generate a warped image 252. The warped image 252 is a 3D image obtained by warping the moving image 282 to align with the fixed image 281. The second stage computational process 720 is a machine learning process that is trainable under the goal of minimizing the training loss. The training loss is calculated as a weighted sum of multiple component losses. See EQN. (13). The multiple component losses include a volume preservation loss 261 and a weighted similarity loss 262. The volume preservation loss 261 is used to measure the loss caused by the failure to preserve the volume of one or more tumor regions unchanged during the warping of the moving image 282. The weighted similarity loss 262 is used to measure the similarity loss of the non-tumor part of the organ between the warped image 252 and the fixed image 281. It should be noted that each of the volume preservation loss 261 and the weighted similarity loss 262 is calculated at least according to the STM 251. By including the volume preservation loss 261 and the weighted similarity loss 262 in the calculation of the training loss, which are minimized in the training second stage calculation process 720, the trained second stage calculation process 720 is advantageously configured such that one or more tumor regions are volumetrically preserved during the generation of the warped image 252 from the moving image 282, while promoting the similarity of non-tumor portions between the moving image 282 and the fixed image 281 during the alignment of the moving image 282 with the fixed image 281.

[0131] Figure 8 A flowchart depicts the steps used in a preferred embodiment of a first stage computational process 710 of the first stage 210 according to the disclosed framework 200. Adopting the process of the first stage 210 to the first stage computational process 710 enables the first stage computational process 710 to advantageously achieve unsupervised tumor estimation without involving complex segmentation algorithms, such as the segmentation techniques of

[32] and

[19] . The first stage computational process 710 includes steps 810, 820, and 830. In step 810, the first stage registration network 211 is used to generate a first stage deformation field 212. The first stage deformation field 212 is a dense deformation field used to warp the moving image 282 to align with the fixed image 281. After the first stage deformation field 212 is generated, the volume change at each voxel of the organ when the moving image 282 is warped is estimated in step 820 based on the first stage deformation field 212. Therefore, a first stage volume change distribution map 214 on the organ is generated in step 820. In step 830 , the first stage volume change distribution map 214 is mapped to the first stage probability distribution map 214 , thereby generating the STM 251 .

[0132] Preferably, the first stage registration network 211 is a similarity-based registration network modeled by the first stage multiple parameters and is trainable. The first stage calculation process 710 also includes step 805: training the first stage registration network 211 to fix the first stage multiple parameters before using the first stage registration network 211 to generate the first stage deformation field 212 in step 810.

[0133] In some embodiments, the first stage volume change distribution map 214 of the organ is calculated by:

[0134] D θ1 (x) = max(D′) θ1 (x),1 / D′ θ1 (x)),x∈Ω, (16)

[0135] in

[0136] D′ θ1 (x)=|J θ1(x) | / β (17)

[0137] Where: D θ1 (x) represents the volume change distance obtained from the first-stage volume change distribution map 214 and is calculated for the voxel x of the organ in the moving image 182; Ω represents the 3D space; |J θ1(x) | is the Jacobian 213 of the first stage deformation field 212, where the Jacobian 213 is calculated for voxel x; and β is the predicted ratio of the first stage volume of the organ in the warped image 252 to the second stage volume of the organ in the moved image 282. Note that EQNS.(16) and (17) are adapted from EQNS.(8) and (7), respectively. As seen from EQN.(7), β is the predicted value and is given by β=|O w | / |O m | is given by, where the organ mask O is predicted / estimated by warping a randomly selected reference image from the training dataset with the ground truth organ segmentation m and O w .

[0138] When mapping the first stage volume change distribution map 214 to the probability distribution map (i.e., STM 251), a transformation function 215 is used to process the volume change distance to generate a synthetic probability value for a single voxel in the probability distribution map. In some embodiments, the transformation function 215 is a sigmoid function. In some embodiments, the transformation function 215 is selected as Tr(x)=sigm(5·(x-1.5)).

[0139] Fig. 9A flow chart depicting steps used in a preferred embodiment of a second stage computation process 720 according to the second stage 220 of the disclosed framework 200 for volume-preserving registration is shown. The second stage computation process 720 includes steps 905, 910, 920, 930, 940, and 950. The second stage computation process 720 is implemented as an iterative process.

[0140] In the second stage computation process 720, the second stage registration network 221 is used to register the moving image 282 and the fixed image 281. The second stage registration network 221 is trainable and is modeled by a second stage plurality of model parameters.

[0141] In step 905, a plurality of model parameters of the second stage are initialized or updated.

[0142] Then, the second stage registration network 221 is used in step 910 to generate a second stage deformation field 222 for warping the moving image 282 to align with the fixed image 281 .

[0143] In step 920, the moving image 282 is spatially transformed according to the second stage deformation field 222 to generate the warped image 252. See EQN. (2) Performing the spatial transformation 226.

[0144] In addition to generating the warped image 252, the second stage deformation field 222 is also used to generate the second stage volume change distribution map 224. In step 930, the volume change at each voxel of the organ when the movement image 282 is warped is estimated according to the second stage deformation field 222, thereby generating the second stage volume change distribution map 224 on the organ.

[0145] In step 940, a volume preservation loss 261 is calculated based on the second stage volume change distribution map 224 and the STM 251; a weighted similarity loss 262 is calculated based on the warped image 252, the fixed image 281 and the STM 251; and a training loss is calculated from the multiple component losses.

[0146] The training loss is checked in step 950 to determine whether the optimization constraint of minimizing the training loss is considered to be satisfied. If the constraint is satisfied, the obtained warped image 252 is the final result of the volume-preserving registration of the moving image 182 and the fixed image 181; otherwise, steps 905, 910, 920, 930, 940 and 950 are repeated, wherein step 905 is performed to update the second stage multiple model parameters in the direction of reducing the training loss.

[0147] Steps 905 and 950 together constitute the following action: training the second stage registration network 22 by optimizing multiple parameters of the second stage to minimize the training loss.

[0148] Preferably, the multiple component losses further include a regularization loss 263 for measuring the loss due to overfitting. In step 940 , the regularization loss 263 is additionally calculated according to the second stage deformation field 222 .

[0149] In some embodiments, the second stage volume change distribution map 224 of the organ is calculated by:

[0150] of

[0151] D θ2 (x) = max(D′) θ2 (x),1 / D′ θ2 (x)),x∈Ω, (18)

[0152] in

[0153] D′ θ2 (x)=|J θ2(x) | / β(19)

[0154] Where: D θ2 (x) represents the volume change distance obtained from the second stage volume change distribution map 224 and is calculated for the voxels of the organ in the moving image 182; Ω represents the 3D space; |J θ2(x) | is the Jacobian 223 of the second stage deformation field 222, where the Jacobian 223 is calculated for voxel x; and, as described above, β is the predicted ratio of the first stage volume of the organ in the warped image 252 to the second stage volume of the organ in the moved image 282.

[0155] For various component losses, volume preservation loss 261 and weighted similarity loss 262 can be calculated by EQNS.(10) and (12) respectively. Training loss can be calculated by EQN.(13).

[0156] In certain embodiments of method 700, a pre-registration step 705 (e.g., Figure 7 ). Fig.10 A flow chart showing exemplary steps taken in the pre-registration step 705 is depicted. In step 1010, the moving image 282 is first filtered with a bilateral filter to produce a filtered moving image. The bilateral filter is configured to preserve the edges of the moving image and smooth its interior regions. In step 1020, the first stage registration network 211 is used to register the filtered moving image and the fixed image 281, thereby producing a third stage deformation field ( Figure 2). Thereafter, in step 1030, the moving image 282 is spatially transformed according to the third stage deformation field to generate an unfiltered warped image. See EQN.(2) to perform the spatial transformation. The advantage given by step 1030 is that the edges of the unfiltered warped image can be aligned with the fixed image 281 without introducing volume changes into the aforementioned edges. After obtaining the unfiltered warped image, the first stage calculation process 710 can be performed. In the first stage calculation process 710, the first stage deformation field 212 is generated by using the first stage registration network 211 to register the unfiltered warped image (rather than the moving image 282) and the fixed image 281.

[0157] Without departing from the spirit or essential features of the present disclosure, the present disclosure may be implemented in other specific forms. Therefore, the present embodiment is considered to be illustrative rather than restrictive in all aspects. The scope of the present invention is indicated by the appended claims rather than by the preceding description, and therefore, all changes within the meaning and equivalent range of the claims are intended to be included therein.

[0158] References

[0159] The following is a list of references occasionally cited in the specification. The disclosures of each of these references are incorporated herein by reference in their entirety.

[0160] [1] Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1): 26–41, 2008.

[0161] [2] Brian B Avants, Nick Tustison, Gang Song, et al. Advanced normalization tools(ants). Insight J, 2(365): 1–35, 2009.

[0162] [3] Bhakti Baheti, Diana Waldmannstetter, Satrajit Chakrabarty, Hamed Akbari, Michel Bilello, Benedikt Wiestler, Julian Schwarting, Evan Calabrese, Jeffrey Rudie, Syed Abidi, et al. The brain tumor sequence registration challenge: establishing correspondence between pre-operative and follow-up mri scans of diffuse glioma patients. arXiv preprint arXiv:2112.06979, 2021.

[0163] [4] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. An unsupervised learning model for deformable medical image registration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9252–9260, 2018.

[0164] [5]Junyu Chen,Eric C Frey,Yufan He,William P Segars,Ye Li,and Yong Du.Transmorph: Transformer for unsupervised medical image registration.Medical image analysis,82:102615,2022.

[0165] [6] Jiashun Chen, Donghuan Lu, Yu Zhang, Dong Wei, Munan Ning, Xinyu Shi, Zhe Xu, and Yefeng Zheng. Deformer: Towards displacement field learning for unsupervised medical image registration. In Medical Image Computing and Computer Assisted Intervention–MICCAI2022:25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VI, pages 141–151. Springer, 2022.

[0166] [7] Xiang Chen, Yan Xia, Nishant Ravikumar, and Alejandro F Frangi. Joint segmentation and discontinuity-preserving deformable registration Application to cardiac cine-mr images. arXiv preprint arXiv:2211.13828, 2022.

[0167] [8]Nicha Chitphakdithai and James S.Duncan.Non-rigid registration with missing correspondences in preoperative and postresection brain images.Lecture Notes in Computer Science,2010.

[0168] [9] Patrick Christ. Lits–liver tumor segmentation challenge (lits17). 2017.

[0169]

[10] Olivier Clatz, Hervé Delingette, Ion-Florin Talos, Alexandra J. Golby, Ron Kikinis, Ferenc A. Jolesz, Nicholas Ayache, and Simon K. Warfield. Robust nonrigid registration to capture brain shift from intraoperative MRI. IEEE Transactions on Medical Imaging, 2005.

[0170]

[11] Ahmed Elazab, Yousry M Abdulazeem, Ahmed M Anter, Qingmao Hu, Tianfu Wang, and Baiying Lei. Macroscopic cerebral tumor growth modeling from medical images: A review. IEEE Access, 6: 30663–30679, 2018.

[0171]

[12] Xu Han, Xiao Yang, Stephen R. Aylward, Roland Kwitt, and Marc Niethammer. Efficient registration of pathological images: A joint PCA / image-reconstruction approach. arXiv: Computer Vision and Pattern Recognition, 2017.

[0172]

[13] Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V Dalca. Synthmorph: learning contrast-invariant registration without acquired images. IEEE transactions on medical imaging, 41(3): 543–558, 2021.

[0173]

[14] Malte Hoffmann, Andrew Hoopes, Douglas N Greve, Bruce Fischl, and Adrian V Dalca. Anatomy-aware and acquisition-agnostic joint registration with synthmorph. arXiv preprint arXiv:2301.11329, 2023.

[0174]

[15] Boah Kim, Dong Hwan Kim, Seong Ho Park, Jieun Kim, June-Goo Lee, and Jong Chul Ye. Cyclemorph: cycle consistent unsupervised deformable image registration. Medical Image Analysis, 71: 102036, 2021.

[0175]

[16] Stefan Klein, Marius Staring, Keelin Murphy, Max A Viergever, and Josien PW Pluim. Elastix: a toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 29(1):196–205, 2009.

[0176]

[17] Dongjin Kwon, Marc Niethammer, Hamed Akbari, Michel Bilello, Christos Davatzikos, and Kilian M. Pohl. Portr: Preoperative and post-recurrence brain tumor registration. IEEE Transactions on Medical Imaging, 2014.

[0177]

[18] Dongjin Kwon, Ke Zeng, Michel Bilello, and Christos Davatzikos. Estimating patient specific templates for preoperative and follow-up brain tumor registration. Lecture Notes in Computer Science, 2015.

[0178]

[19] Fei Lyu, Mang Ye, Andy J Ma, Terry Cheuk-Fung Yip, Grace Lai-Hung Wong, and Pong C Yuen. Learning from synthetic CT images via test-time training for liver tumor segmentation. IEEE Transactions on Medical Imaging, 41(9): 2510–2520, 2022.

[0179]

[20] Mingyuan Meng, Lei Bi, Dagan Feng, and Jinman Kim. Brain tumor sequence registration with non-iterative coarse-to-fine networks and dual deep supervision. arXiv preprint arXiv:2211.07876, 2022.3

[0180]

[21] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging, 34(10): 1993–2024, 2014.

[0181]

[22] Tony CW Mok and Albert CS Chung. Unsupervised deformable image registration with absent correspondences in pre-operative and post-recurrence brain tumor MRI scans. In Medical Image Computing and Computer Assisted Intervention–MICCAI2022:25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VI, pages 25–35. Springer, 2022.

[0182]

[23] Torsten Rohlfing, Calvin R Maurer, David A Bluemke, and Michael A Jacobs. An alternating-constraints algorithm for volume-preserving non-rigid registration of contrast-enhanced MR breast images. In Biomedical Image Registration: Second International Workshop, WBIR 2003, Philadelphia, PA, USA, June 23-24, 2003. Revised Papers 2, pages 291–300. Springer, 2003.

[0183]

[24] Daniel Rueckert, Luke I Sonoda, Carmel Hayes, Derek LG Hill, Martin O Leach, and David J Hawkes. Nonrigid registration using free-form deformations: application to breast mr images. IEEE Transactions on Medical Imaging, 18(8): 712–721, 1999.

[0184]

[25] Siyuan Shan, Wen Yan, Xiaoqing Guo, Eric I Chang, Yubo Fan, Yan Xu, et al. Unsupervised end-to-end learning for deformable medical image registration. arXiv preprint arXiv:1711.08608, 2017.

[0185]

[26] Jiacheng Shi, Yuting He, Youyong Kong, Jean-Louis Coatrieux, Huazhong Shu, Guanyu Yang, and Shuo Li. Xmorpher: Full transformer for deformable medical image registration via cross attention. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022:25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VI, pages 217–226. Springer, 2022.

[0186]

[27] Christine Tanner, Julia A. Schnabel, Andreas Degenhard, Andy D. Castellano-Smith, Carmel Hayes, Martin O. Leach, D. R. Hose, Derek L. Gill, and David J. Hawkes. Validation of volume-preserving non-rigid registration: Application to contrast-enhanced mr-mammography. Lecture Notes in Computer Science, 2002.

[0187]

[28] Di Wang, Yue Pan, Oguz C. Durumeric, Joseph M. Reinhardt, Eric A. Hoffman, Joyce D. Schroeder, and Gary E. Christensen. Plosl: Population learning followed by one shot learning pulmonary image registration using tissue volume preserving and vesselness constraints. Medical Image Analysis, 2022.

[0188]

[29] Yibo Wang, Wen Qian, Mengqi Li, and Xuming Zhang. A transformer-based network for deformable medical image registration. In Artificial Intelligence: Second CAAI International Conference, CICAI 2022, Beijing, China, August 27–28, 2022, Revised Selected Papers, Part I, pages 502–513. Springer, 2022.

[0189]

[30] Marek Wodzinski, Artur Jurgas, Niccolo Marini, Manfredo Atzori, and Henning Muller. Unsupervised method for intra-patient registration of brain magnetic resonance images based on objective function weighting by inverse consistency: Contribution to the brats-reg challenge. arXiv preprint arXiv:2211.07386, 2022.

[0190]

[31] Qingsong Yao, Li Xiao, Peihang Liu, and S Kevin Zhou. Label-free segmentation of covid-19 lesions in lung CT. IEEE Transactions on Medical Imaging, 40(10): 2808–2819, 2021.

[0191]

[32] Xiaoman Zhang, Weidi Xie, Chaoqin Huang, Ya Zhang, Xin Chen, Qi Tian, ​​and Yanfeng Wang. Self-supervised tumor segmentation with sim2real adaptation. IEEE Journal of Biomedical and Health Informatics, 2023.

[0192]

[33] Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 IEEE / ACM 26th international symposium on quality of service (IWQoS), pages 1–2. IEEE, 2018.

[0193]

[34] Bowen Zhao, G. Christensen, Joo Hyun Song, Yue Pan, Sarah E. Gerard, Joseph M. Reinhardt, Kaifang Du, Taylor J. Patton, John E. Bayouth, and Geoffrey D. Hugo. Tissue-volume preserving deformable image registration for 4D CT pulmonary images. 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 481–489, 2016.

[0194]

[35] Shengyu Zhao, Yue Dong, Eric I Chang, Yan Xu, et al. Recursive cascaded networks for unsupervised medical image registration. In Proceedings of the IEEE / CVF international conference on computer vision, pages 10600–10610, 2019.

[0195]

[36] Shengyu Zhao, Tingfung Lau, Ji Luo, I Eric, Chao Chang, and Yan Xu. Unsupervised 3D end-to-end medical image registration with volume tweening network. IEEE Journal of Biomedical and Health Informatics, 24(5): 1394–1404, 2019.

Claims

1. A computer-implemented method for performing deformable registration of a moving image and a fixed image, the moving image being a medical image imaging an organ having one or more tumor regions, each of the moving image and the fixed image being a three-dimensional (3D) image, the method comprising: performing a first stage computational process for generating a soft tumor mask (STM), the STM being a probability distribution map indicating the likelihood that individual voxels of the organ in the moving image are located in the one or more tumor regions; as well as performing a second stage computational process for generating a warped image, the warped image being a 3D image obtained by warping the moving image to align with the fixed image, the second stage computational process being a machine learning process trainable under the objective of minimizing a training loss computed as a weighted sum of a plurality of component losses, wherein the plurality of component losses include: a volume preservation loss for measuring a loss due to a failure to preserve the volume of the one or more tumor regions during warping of the moving images, the volume preservation loss being calculated based on at least the STM; and a weighted similarity loss for measuring a similarity loss of a non-tumor portion of the organ between the warped image and the fixed image, the weighted similarity loss being calculated at least according to the STM; One or more tumor regions are volumetrically preserved during generation of the warped image from the moving image, while similarity of the non-tumor portion between the moving image and the fixed image is promoted during alignment of the moving image with the fixed image.

2. The method according to claim 1, wherein the first stage calculation process comprises: generating a first-stage deformation field using a first-stage registration network, the first-stage deformation field being used to warp the moving image to align with the fixed image; estimating a volume change at each voxel of the organ when the moving image is distorted according to the first stage deformation field, thereby generating a first stage volume change distribution map on the organ; as well as The volume change distribution map of the first stage is mapped to the probability distribution map, thereby generating the STM.

3. The method according to claim 2, wherein: The first-stage registration network is a similarity-based registration network that is modeled by a first-stage plurality of parameters and is trainable; and The first-stage computation process further includes training the first-stage registration network to fix the first-stage plurality of parameters before using the first-stage registration network to generate the first-stage deformation field.

4. The method according to claim 2, wherein the first stage volume change distribution map on the organ is calculated by D θ1 (x)=max(D′ θ1 (x),1 / D′ θ1 (x)),x∈Ω, in D′ θ1 (x)=|J θ1(x) | / b in: D θ1 (x)d represents the volume change distance obtained from the volume change distribution map of the first stage and calculated for the voxel x of the organ in the moving image; Ω represents 3D space; |J θ1(x) | is the Jacobian of the deformation field of the first stage, wherein the Jacobian is calculated for the voxel x; as well as β is the predicted ratio of the first phase volume of the organ in the warped image to the second phase volume of the organ in the moved image.

5. The method according to claim 4, wherein: In the process of mapping the first stage volume change distribution map to the probability distribution map, a transformation function is used to process the volume change distance to generate a synthetic probability value of the individual voxel in the probability distribution map; and The transformation function is a sigmoid function.

6. The method according to claim 1, wherein the second stage calculation process comprises: generating a second-stage deformation field for warping the moving image to align with the fixed image using a second-stage registration network, wherein the second-stage registration network is trainable and modeled by a second-stage plurality of parameters; spatially transforming the moving image according to the second stage deformation field to generate the distorted image; training the second stage registration network by optimizing the second stage plurality of parameters to minimize the training loss; estimating a volume change at each voxel of the organ when the moving image is distorted according to the second stage deformation field, thereby generating a second stage volume change distribution map on the organ; When calculating the training loss, calculating the volume retention loss according to the second stage volume change distribution map and the STM; as well as When calculating the training loss, the weighted similarity loss is calculated according to the distorted image, the fixed image and the STM.

7. The method according to claim 6, wherein: The plurality of component losses also include a regularization loss for measuring a loss due to overfitting; as well as The second stage calculation process also includes: When calculating the training loss, the regularization loss is calculated according to the second stage deformation field.

8. The method according to claim 6, wherein the second stage volume change distribution map on the organ is calculated by D θ2 (x)=max(D′ θ2 (x),1 / D′ θ2 (x)),x∈Ω, in D′ θ2 (x)=|J θ2(x) | / b in: D θ2 (x) represents the volume change distance obtained from the volume change distribution map of the second stage and calculated for the voxel x of the organ in the moving image; Ω represents 3D space; |J θ2(x) | is the Jacobian of the second stage deformation field, wherein the Jacobian is calculated for the voxel x; and β is the predicted ratio of the first phase volume of the organ in the warped image to the second phase volume of the organ in the moved image.

9. The method of claim 8, wherein the volume retention loss, L VP , is calculated by in: STM(x) is the value of the STM for said voxel x; and |I m | is the volume of the moving image.

10. The method according to claim 6, wherein the weighted similarity loss, L w_sim , is calculated by in in: Ω represents 3D space; STM(x) is the value of the STM for voxel x of the organ in the moving image; I w represents the distorted image, isI w The average value of I w (x) is the I at the voxel x w value; I f represents the fixed image, isI f The average value of I f (x) is the I at the voxel x f value; Cov′[I w ,I f ] is I w and I f The weighted covariance of , with the STM as weight; Cov[I w ,I w ] is I w The covariance of Cov[I f ,I f ] is I f The covariance of .

11. The method according to claim 2, wherein the second stage calculation process comprises: generating a second-stage deformation field for warping the moving image to align with the fixed image using a second-stage registration network, wherein the second-stage registration network is trainable and modeled by a second-stage plurality of parameters; spatially transforming the moving image according to the second stage deformation field to generate the distorted image; training the second registration network by optimizing the second stage multiple parameters to minimize the training loss; estimating a volume change at each voxel of the organ when the moving image is distorted according to the second-stage deformation field, thereby generating a second-stage volume change distribution map on the organ; When calculating the training loss, calculating the volume retention loss according to the second stage volume change distribution map and the STM; as well as When calculating the training loss, the weighted similarity loss is calculated according to the distorted image, the fixed image and the STM.

12. The method according to claim 11, further comprising: Before executing the first stage calculation process, a pre-registration step is performed, wherein the pre-registration step includes: filtering the moving image with a bilateral filter to produce a filtered moving image, wherein the bilateral filter is configured to preserve edges of the moving image and smooth interior regions thereof; registering the filtered moving image and the fixed image using the first stage registration network to produce a third stage deformation field; and spatially transforming the moving image according to the third stage deformation field to generate an unfiltered warped image, so as to align an edge of the unfiltered warped image with the fixed image without introducing a volume change to the edge; In the first stage calculation process, the first stage deformation field is generated by registering the unfiltered distorted image and the fixed image using the first stage registration network.

Images