Multi-modal clinical image alignment method and apparatus using joint synthesis, segmentation and registration

Abstract

An image processing method for performing image alignment, comprising: acquiring a motion image generated by a first imaging modality; acquiring a fixed image generated by a second imaging modality; jointly optimizing a generative model, a registration model and a segmentation model applied to the motion image and the fixed image according to a plurality of cost functions; applying a spatial transformation corresponding to the registration model after optimization to the motion image to align the motion image with the fixed image; wherein the generative model generates a synthetic image from the motion image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthetic image with the fixed image; and the segmentation model estimates segmentation maps of the motion image, the fixed image and the synthetic image.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 029,470, filed May 23, 2020. This application also claims priority to U.S. Patent Application No. 17 / 110,859, filed December 3, 2020, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the field of image processing, and in particular to a method and apparatus for multi-modality clinical image alignment using joint synthesis, segmentation, and registration. BACKGROUND

[0004] Image registration seeks to find a spatial transformation between a pair of images that registers points in one image to homologous points in the other, thus aligning the pair of images. In medical imaging, registration often focuses on inter-patient / inter-study single modality alignment. Another important common focus is multi-channel imaging, such as dynamic contrast computed tomography (CT), multi-parametric magnetic resonance imaging (MRI), or positron emission tomography (PET) combined with CT / MRI. In this case, the need for intra-patient multi-modal registration is critical given the inevitable patient movement or shift between subsequent imaging scans. For scenarios where there is deformable misalignment, such as the abdomen, the correspondence can be very complex. Since different modalities provide complementary visual / diagnostic information, correct and precise anatomical alignment is beneficial for radiological observations / decisions by human readers; and is critical for any downstream computerized analysis. However, due to the complex appearance variations of different modalities, which can depend on anatomy, pathology, or other complex interactions, finding correspondence between homologous points is often not trivial. Therefore, there is a need to develop improved methods for aligning clinical images. SUMMARY

[0005] A first aspect of the present disclosure provides an image processing method for performing image alignment. The method comprises: obtaining a moving image generated by a first imaging modality; obtaining a fixed image generated by a second imaging modality; jointly optimizing a generative model, a registration model, and a segmentation model applied to the moving image and the fixed image according to a plurality of cost functions; and applying a spatial transformation corresponding to the optimized registration model to the moving image to align the moving image with the fixed image; wherein the generative model generates a synthesized image from the moving image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthesized image with the fixed image; and the segmentation model estimates segmentation maps of the moving image, the fixed image, and the synthesized image.

[0006] According to some embodiments of the image processing method, the jointly optimizing the generative model, the registration model and the segmentation model comprises: supervised learning of the segmentation model using an image dataset with ground truth segmentation maps; and performing unsupervised learning of the generative model, the registration model and the segmentation model using the moving images generated by the first imaging modality and the fixed images generated by the second imaging modality.

[0007] According to some embodiments of the image processing method, the performing unsupervised learning of the generative model, the registration model and the segmentation model comprises: jointly learning the generative model and the registration model according to a first objective function; and jointly learning the generative model, the registration model and the segmentation model according to a weighted sum of the first objective function and a second objective function.

[0008] According to some embodiments, the image alignment method further comprises: taking a sum of a first regularization cost function of the generative model and a second regularization cost function of the registration model as the first objective function.

[0009] According to some embodiments, the image processing method further comprises: taking a weighted sum of a texture-based generative cost function and a conditional generative adversarial network (GAN) cost function as the first regularization cost function of the generative model, the texture-based generative cost function accounting for image texture differences, and the conditional GAN cost function being conditioned on the fixed images.

[0010] According to some embodiments, the image processing method further comprises: taking a weighted sum of a registration cost function and a smoothness regularization term as the second regularization cost function of the registration model, the registration cost function accounting for image registration differences, and the smoothness regularization term regularizing non-real space transformations.

[0011] According to some embodiments of the image processing method, the registration cost function is calculated based on an L1 loss between a deformed image of the moving images and the fixed images.

[0012] According to some embodiments, the image processing method further comprises: taking a sum of a joint generative-registration-segmentation cost function and a supervised segmentation cost function as the second objective function, the joint generative-registration-segmentation cost function accounting for constraints between synthesis, registration and segmentation tasks, and the supervised segmentation cost function accounting for supervised learning of the segmentation model.

[0013] According to some embodiments of the image processing method, the fixed image is a contrast computed tomography (CT) image of a patient at a venous phase; and the moving image is one of: a contrast CT image of the patient at an arterial phase, a contrast CT image of the patient at a delayed phase, or a non-contrast CT image of the patient.

[0014] According to some embodiments of the image processing method, jointly optimizing the generative model, the registration model, and the segmentation model comprises optimizing the generative model, the registration model, and the segmentation model using a first learning rate for the generative model, a second learning rate for the registration model, and a third learning rate for the segmentation model, the third learning rate being greater than the second learning rate, and the second learning rate being greater than the first learning rate.

[0015] Another aspect of the present disclosure provides an image processing device for performing image alignment. The device comprises a memory storing computer executable instructions; and a processor coupled with the memory, configured to, when the computer readable instructions are executed: obtain a moving image generated by a first imaging modality; obtain a fixed image generated by a second imaging modality; jointly optimize a generative model, a registration model, and a segmentation model applied to the moving image and the fixed image according to a plurality of cost functions; and apply a spatial transformation corresponding to the optimized registration model to the moving image to align the moving image with the fixed image; wherein the generative model generates a synthetic image from the moving image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthetic image with the fixed image; and the segmentation model estimates segmentation maps of the moving image, the fixed image, and the synthetic image.

[0016] According to some embodiments of the image processing device, the processor is further configured to: perform supervised learning of the segmentation model using an image dataset with ground truth segmentation maps; and perform unsupervised learning of the generative model, the registration model, and the segmentation model using the moving image generated by the first imaging modality and the fixed image generated by the second imaging modality.

[0017] According to some embodiments of the image processing device, the processor is further configured to: jointly learn the generative model and the registration model according to a first objective function; and jointly learn the generative model, the registration model, and the segmentation model according to a weighted sum of the first objective function and a second objective function.

[0018] According to some embodiments of the image processing device, the processor is further configured to: use a sum of a first regularization cost function of the generative model and a second regularization cost function of the registration model as the first objective function.

[0019] According to some embodiments of the image processing device, the processor is further configured to: take a weighted sum of a texture-based generation cost function and a conditional generative adversarial network (GAN) cost function as a first regularization cost function of the generation model, the texture-based generation cost function accounting for image-texture discrepancies, and the conditional GAN cost function being conditioned on the fixed image.

[0020] According to some embodiments of the image processing device, the processor is further configured to: take a weighted sum of a registration cost function and a smoothness regularization term as a second regularization cost function of the registration model, the registration cost function accounting for image registration discrepancies, and the smoothness regularization term regularizing non-real space transformations.

[0021] According to some embodiments of the image processing device, the registration cost function is calculated based on an L1 loss between a warped image of the motion image and the fixed image.

[0022] According to some embodiments of the image processing device, the processor is further configured to: take a sum of a joint generation-registration-segmentation cost function and a supervised segmentation cost function as the second objective function, the joint generation-registration-segmentation cost function accounting for constraints between the synthesis, registration, and segmentation tasks, and the supervised segmentation cost function accounting for supervised learning of the segmentation model.

[0023] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a plurality of instructions. When executed by a processor, the plurality of instructions cause the processor to perform a plurality of operations, including: obtaining a motion image generated by a first imaging modality; obtaining a fixed image generated by a second imaging modality; jointly optimizing a generation model, a registration model, and a segmentation model applied to the motion image and the fixed image according to a plurality of cost functions; and applying a spatial transformation corresponding to the optimized registration model to the motion image to align the motion image with the fixed image; wherein the generation model generates a synthesized image from the motion image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthesized image with the fixed image; and the segmentation model estimates segmentation maps of the motion image, the fixed image, and the synthesized image.

[0024] According to some embodiments of the non-transitory computer-readable storage medium, the plurality of instructions further cause the processor to perform a plurality of operations, including: performing supervised learning of the segmentation model using an image dataset with ground truth segmentation maps; and performing unsupervised learning of the generation model, the registration model, and the segmentation model using the motion image generated by the first imaging modality and the fixed image generated by the second imaging modality.

[0025] BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the embodiment description are briefly described below. The drawings in the following description are only some embodiments of the present disclosure. Based on these drawings, those skilled in the art can obtain other drawings.

[0027] FIG. 1 shows a relationship block diagram between synthesis, segmentation and registration tasks;

[0028] Figure 2 A block diagram of an image processing device for aligning multi-modal clinical images using a joint synthesis, segmentation and registration (JSSR) process according to some embodiments of the present disclosure is shown;

[0029] Figure 3 A flowchart of an image processing method for performing multi-modal clinical image alignment using a JSSR process according to some embodiments of the present disclosure is shown;

[0030] Figure 4 A flowchart of one embodiment of a JSSR process is shown;

[0031] Figure 5 A general block diagram of a JSSR process according to some embodiments of the present disclosure is shown;

[0032] Figure 6 A network structure diagram of each of a generation model, a registration model and a segmentation model according to some embodiments of the present disclosure is shown;

[0033] FIG. 7 shows a performance comparison chart of different methods on computed tomography (CT) images at three stages;

[0034] Figure 8A A JSSR processing result chart of an arterial phase patient abdominal CT scan is shown;

[0035] Figure 8B A JSSR processing result chart of a non-contrast phase patient abdominal CT scan is shown;

[0036] Figure 9A An image difference chart for evaluating an image processing method in one embodiment is shown;

[0037] Figure 9B A segmentation result chart as in Figure 9A Example;

[0038] Figure 10A An image difference chart for evaluating an image processing method in another embodiment is shown;

[0039] Figure 10Ba segmentation result map as in Figure 10A a segmentation result map as in

[0040] Figure 11A a segmentation result map as in

[0041] Figure 11B a segmentation result map as in Figure 11A a segmentation result map as in

[0042] Figure 12A a segmentation result map as in

[0043] Figure 12B a segmentation result map as in Figure 12A a segmentation result map as in DETAILED DESCRIPTION

[0044] The technical solutions of the embodiments of the present application will be described below with reference to the drawings. The described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0045] Multi-modal registration remains a challenging task, especially because ground truth deformations are difficult or impossible to obtain. Instead, methods must learn a transformation or loss image in order to more easily achieve correspondence between images. Unsupervised registration methods often use local modality-invariant features to measure similarity. However, these low-level features can not be universally applicable and can not always capture high-level semantic information. Other methods use generative models to reduce domain shift between modalities and then apply registration based on direct intensity similarity. Different strategies learn registration to maximize overlap of segmentation labels. The latter approach is promising because it resembles the registration process to a segmentation task and aligns images according to semantic classes. However, these methods first rely on supervised segmentation labels for each deployment scenario.

[0046] Multi-modal image registration has been extensively studied and applied in the medical field. Existing registration methods can be based on landmarks and surfaces, or based on voxel attributes, directly operating on image intensity values without user-introduced prior data or segmentation. For voxel-based methods, there are two typical strategies. One is to use self-similarity measures that do not require cross-modality features, such as local self-similarity (LSS). Certain works introduce a modality independent neighborhood descriptor (MIND) that can effectively measure cross-modality differences. Certain works employ discrete dense displacement sampling for self-similarity context (SSC) deformable registration. The other strategy is to map both modalities to a shared space and measure single-modality differences. Certain works introduce mutual information (MI) based on information theory that can be directly applied to cross-modality images as a similarity measure, followed by other works using normalized mutual information (NMI). However, these methods suffer from low convergence speed, loss of spatial information, and other drawbacks. Certain works employ convolutional neural networks (CNNs) to learn interpretable modality-invariant features with a small amount of supervised data. Certain works utilize Haar-like features from pairs of multi-modal images to fit a Patch-wise Random Forest (P-RF) regression for bidirectional image synthesis, while certain works apply CycleGAN to reduce the gap between modalities for better alignment.

[0047] In recent years, with the development of deep learning methods, various learning-based registration methods have been proposed. Although ground truth deformation fields are difficult to obtain, most methods utilize synthetic deformation fields as supervision for training. Unsupervised methods can use CNNs with spatial transformation functions. These unsupervised methods may primarily focus on single-modality image registration. Other methods utilize correspondences between labeled anatomical structures as weak supervision to aid the registration process. Some works also demonstrate how segmentation maps can aid registration. However, in most cases, segmentation maps are not available, which motivates us to combine registration and segmentation components.

[0048] Since registration, synthesis, and segmentation tasks are related to each other, various works have explored the advantages of combining different tasks together. Some methods use cycle-consistent adversarial networks (CycleGAN) to synthesize multi-modal images into single-modal images and apply single-modal registration methods. Some methods project multi-modal images into a shared feature space and register according to the features. Some methods utilize generative models to separate appearance space from shape space. Some methods combine segmentation and registration models, allowing them to benefit from each other, but focus on single-modality registration. Some methods perform supervised multi-phase segmentation based on pairs of multi-phase images, but do not jointly train registration and segmentation. Some methods use generative models to help guide segmentation models.

[0049] Both synthesis and segmentation methods are promising, but they are each limited when used alone, especially when fully supervised training data is unavailable, i.e., there are no paired multimodal images and segmentation labels, respectively. Figure 1 shows a block diagram illustrating the relationship between synthesis, segmentation, and registration tasks. Figure 1 As shown, the tasks of composition, segmentation, and registration are linked together, defining implicit constraints between them. These constraints apply to both ideal and real-world environments. Ideally, a spatial transformation example and its segmentation label are fully available for each domain. In a more realistic setting, only one example is available for each domain, and each domain is under a different spatial transformation. Furthermore, segmentation labels are not available. If segmentation, composition, and spatial transformation mappings are available, the constraints in the ideal case can be mapped to similar constraints in the real case.

[0050] Inspired in part by the relationship between synthesis, segmentation, and registration tasks, this disclosure provides an apparatus and method for using [methods to satisfy] Figure 1 The implicitly constrained Joint Synthesis, Segmentation, and Registration (JSSR) process is illustrated for aligning multimodal clinical images. The JSSR process combines synthesis, segmentation, and registration tasks to solve the multimodal registration problem in the most general case where deformed ground truth, paired multimodal images, and segmentation maps are unavailable. The JSSR process may include a generator, a segmenter, and a registerer, and can perform synthesis, segmentation, and registration tasks simultaneously. Given a stationary image and a moving image from different modalities for registration, the generator can synthesize the moving image into the same modality as the stationary image, conditioned on the stationary image to better reduce domain gaps. The registerer can then obtain the synthesized image from the generator and the stationary image to estimate the deformation field. Finally, the segmenter can estimate a segmentation map of the moving image, the stationary image, and the synthesized image. During training, the JSSR system can optimize several consistency losses, including similarity between the fixed image and the deformed synthetic image, similarity between the segmentation maps of the deformed motion image and the fixed image, adversarial loss for generating high-fidelity images, and smoothness loss for normalizing the deformation field. In some embodiments, to prevent the segmenter from providing meaningless segmentation maps, some data with segmentation annotations can be added to regularize the segmenter.

[0051] Figure 2 A block diagram of an image processing apparatus 100 for aligning multimodal clinical images using the JSSR process according to this disclosure is shown. Figure 2As shown, the image processing device 100 can be configured to receive multi-modal clinical images of a patient. In some embodiments, the received multi-modal clinical images can be three-dimensional (3D) clinical images acquired from two or more different modalities. In some embodiments, the multi-modal clinical images can include motion images generated by a first imaging modality and stationary images generated by a second imaging modality. In some embodiments, the multi-modal can include multiple image channels of an imaging system, such as multi-phase of a dynamic contrast computed tomography (CT) system, multi-parametric imaging of a magnetic resonance imaging (MRI) system, or positron emission tomography (PET) data set combined with CT and / or MRI. The multi-modal clinical images can be obtained from one or more imaging instruments, loaded from a memory module, or otherwise provided to the image processing device 100. The image processing device 100 can be configured to perform neural network processing to align the received multi-modal clinical images using a JSSR process.

[0052] In some embodiments, the image processing device 100 can output a spatial transformation for aligning the multi-modal clinical images. In some embodiments, the image processing device 100 can apply the spatial transformation to motion images generated by a first imaging modality to align (or register) the motion images to stationary images generated by a second imaging modality. In some embodiments, the spatial transformation can correspond to a registration model optimized jointly with a generative model and a segmentation model using a JSSR process.

[0053] In some embodiments, the image processing device 100 can be a computing device including a processor 102 and a storage medium 104. The image processing device 100 can also include a display 106, a communication module 108, and additional peripheral devices 112. Certain devices can be omitted and other devices can be included. The processor 102 can include any suitable processor. In some embodiments, the processor 102 can include multiple cores for multithreading or parallel processing. The processor 102 can execute computer program sequences to perform various processes, such as neural network processing programs. The storage medium 104 can be a non-transitory computer-readable storage medium and can include memory modules, such as ROM, RAM, flash memory modules, and erasable memory, as well as mass storage devices, such as CD-ROMs, USB drives, and hard disks, etc. The storage medium 104 can store computer programs and instructions for implementing various processes, which, when executed by the processor 102, cause the processor 102 to perform various steps of a neural network processing program of an image processing method for detecting and localizing segmentation abnormalities from medical images. The communication module 108 can include network devices for establishing connections over networks. The display 106 can include any suitable type of computer display device or electronic device display (e.g., CRT or LCD-based devices, touchscreens). The peripheral devices 112 can include additional I / O devices, such as keyboards, mice, etc. The processor 102 can be configured to execute instructions stored on the storage medium 104 and perform various operations related to the image processing method as detailed in the following description.

[0054] Figure 3 A flowchart of an image processing method 200 for performing multi-modal clinical image alignment using a JSSR process is shown. As shown, the image processing method 200 can include the following exemplary steps. Figure 3

[0055] S210, obtaining a motion image generated by a first imaging modality. According to some embodiments, the motion image can be generated for an object or a scene at a time point A. The object or the scene can experience a relative movement at the time point A compared to another time point B. In some embodiments, the relative movement can include one or more of the following: a position change or a shape deformation. For example, when a patient undergoes multiple medical imaging scans at different time points, the movement of the patient between different scans is usually inevitable. In some scenarios, for example, when the abdomen is imaged, there is a relative shape deformation between different scans.

[0056] ​According to some embodiments, the first imaging modality can refer to an imaging procedure using a particular type of imaging system or device. For example, the type of imaging or device can be a CT imaging system, an MRI system, a PET imaging system, an ultrasound imaging system, an optical imaging system, etc. According to some other embodiments, the first imaging modality can refer to a particular imaging procedure or imaging phase using an imaging system or device. For example, the motion image can be one of: an arterial phase contrast CT image of a patient, a delayed phase contrast CT image of the patient, or a non-contrast CT image of the patient. The second imaging modality can refer to a different imaging procedure or imaging phase than the first imaging modality. For example, the fixed image can be a venous phase contrast computed tomography (CT) image of a patient.

[0057] S220, obtaining a fixed image generated by the second imaging modality. According to some embodiments, the fixed image can be generated for the same object or scene at a second time point, which can be different from the first time point, when the motion image is generated for the scene or object at the first time point. The object or scene can undergo relative motion between the motion image and the fixed image. According to some embodiments, the second imaging modality can be different from the first imaging modality. For example, in some embodiments, the motion image can be generated by a particular type of imaging system or device, such as a CT imaging system, while the fixed image can be generated by another type of imaging system or device, such as an MRI system. In other embodiments, the motion image and the fixed image can be generated using the same imaging system or device through different imaging channels or phases. For example, the fixed image can be a venous phase computed tomography (CT) image of a patient, and the motion image can be one of: an arterial phase contrast CT image of the patient, a delayed phase contrast CT image of the patient, or a non-contrast CT image of the patient.

[0058] S230, jointly optimizing a generative model, a registration model, and a segmentation model applied to the motion image and the fixed image according to a plurality of cost functions. According to some embodiments, S230 can implement a JSSR process to jointly optimize the generative model, the registration model, and the segmentation model. The generative model generates a synthetic image from the motion image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthetic image with the fixed image; and the segmentation model estimates segmentation maps of the motion image, the fixed image, and the synthetic image.

[0059] Figure 4 A flowchart illustrating one embodiment of the JSSR process S230 is shown. As shown, the JSSR process S230 can include the following exemplary steps. Figure 4 ​

[0060] S232, supervised learning of the segmentation model using an image dataset with ground truth segmentation maps. According to some embodiments, to incorporate meaningful semantic information into the segmentation model, the segmentation model is trained using additional image data with segmentation annotations. The supervised learning of the segmentation model can provide a meaningful initial state for the segmentation model in the following unsupervised learning steps.

[0061] S234, unsupervised joint learning of the generative model and the registration model according to a first objective function. According to some embodiments, the first objective function can be a sum of a first regularization cost function of the generative model and a second regularization cost function of the registration model. According to some embodiments, the first regularization cost function of the generative model can be a weighted sum of a texture-based generative cost function and a conditional generative adversarial network (GAN) cost function. The texture-based generative cost function accounts for image texture differences. The conditional GAN cost function is conditioned on the fixed image. According to some embodiments, the second regularization cost function of the registration model can be a weighted sum of a registration cost function and a smoothness regularization term. The registration cost function accounts for image registration differences. The smoothness regularization term can regularize non-realistic spatial transformations. According to some embodiments, the registration cost function can be computed based on an L1 loss between the warped image of the moving image and the fixed image.

[0062] S236, unsupervised joint learning of the generative model, the registration model and the segmentation model according to a weighted sum of the first objective function and a second objective function. According to some embodiments, the second objective function can be a sum of a joint generative-registration-segmentation cost function and a supervised segmentation cost function. The joint generative-registration-segmentation cost function accounts for constraints between the joint generative, registration and segmentation tasks. The supervised segmentation cost function accounts for the supervised learning of the segmentation model.

[0063] Referring back to Figure 3 After the JSSR process, S240, a spatial transformation corresponding to the optimized registration model is applied to the moving image to align the moving image with the fixed image. According to some embodiments, the optimized registration model provides a spatial transformation. The spatial transformation can correspond to a deformation field that aligns the moving image with the fixed image.

[0064] An example of implementing the image processing method 200 is described below. Figure 1 and Figure 5 A block diagram of a JSSR process according to some embodiments of the present disclosure is shown.

[0065] Given a moving image from a different modality and fixed images The goal is to find a spatial transformation function τ to optimize the similarity between and τ ( ). In common application scenarios, neither the ground truth deformation field, nor the segmentation map, nor the paired multi-modal images are available. Therefore, the multi-modal image registration problem needs to be solved in a fully unsupervised way to meet the problem setup.

[0066] Inspired by the relationship between image synthesis, segmentation, and registration, the JSSR process is developed and consists of three components: a generator G, a registrar Φ, and a segmenter S. The constraints shown in FIG. 1 need to be optimized because they are necessary conditions for a correct joint system. By putting them together, the system is also able to perform unpaired image synthesis and multi-modal image segmentation. During the optimization process, the three tasks will benefit from each other, leading to improvements in all three tasks. As shown in FIG. 2, Figure 5 Figure 5 shows the overall block diagram of the JSSR process.

[0067] For the generative model of image synthesis, although there have been works to solve unpaired image synthesis, the existing synthesis methods are based on random sampling to generate various different target domain images. However, in the registration model, the synthesized images should have the same texture properties conditioned on the fixed images to help registration, making the paired image synthesis method a better choice. In the problem layout, we will as the moving images, as the fixed images, where and represent the image sets from different modalities. The JSSR process can learn the mapping from to using a conditional GAN with a generative model G, where τ represents the ground truth spatial transformation (also called deformation field) being estimated. is the inverse mapping of τ. Note that τ does not exist in image synthesis. It is used here for convenience. The discriminator D is used to detect fake images from the generator. The goal of the conditional GAN is: (1).

[0068] In the classic GAN setting, one should use instead of . However, based on the assumption that the spatial transformation function τ does not change the image texture, is replaced by ​Another texture-based loss can be added to benefit the GAN objective:

[0069] (2).

[0070] The final goal of synthesis can be computed as:

[0071] (3).

[0072] However, the goal can only be optimized as each provides τ, which can be introduced in conjunction with the registration part.

[0073] In multi-modal image registration, for two images , the registration model aims to learn a function : where τ is a spatial transformation (or deformation field), so it warps the moving image to make it as similar as possible to the fixed image . For single-modal registration, a loss can be used to measure the similarity between the fixed image and the warped image. For two images from different modalities, some methods suggest using cross-modal similarity measures, such as cross-correlation. According to some embodiments of the present disclosure, the JSSR process utilizes the generative model to transfer to the domain so that a single-modal similarity measure can be used. The registration objective can be computed as:

[0074] (4),

[0075] where , G is the generator from to synthetic images. Another smoothness term can be added to prevent producing unrealistic deformation fields:

[0076] (5),

[0077] where denotes the position of a voxel, the difference between neighboring voxels of is computed. The final goal of registration can be computed as:

[0078] (6).

[0079] Again, the goal in equation (1) cannot be computed without giving G. However, to accurately estimate G, accurate estimation of According to some embodiments, the JSSR process employs a method of adding the two objectives of the synthesis and registration parts to obtain:

[0080] (7).

[0081] In The calculation is not simple, but it can be proven that when τ is sufficiently smooth, and These two aspects are close to each other; they can be combined without significantly altering the overall objective. Specifically, when... When the generated τ is sufficiently smooth, it can be proven that for some constant k, ,like:

[0082]

[0083] use and identity transformation The smoothness assumption. Then we have .

[0084] However, minimization cannot guarantee success. The optimal solution is obtained. In fact, there is a minimization... The ordinary solution, that is, when and To avoid obtaining a conventional solution, skip connections can be added from the source domain to preserve spatial information within the generator's network structure. This process is as follows: Figure 6 As shown, Figure 6 Network structure diagrams of each of the generative model, registration model, and segmentation model according to some embodiments of the present disclosure are shown, with dashed lines indicating skipped connections. Figure 6 As shown, according to some embodiments, the registration model can adopt a PHNN network structure, and the generation model and the segmentation model can adopt a V-Net structure.

[0085] Regarding the segmentation model, it considers two aspects. First, additional information about the segmentation map can help guide the registration process. Therefore, when manual annotation is unavailable, additional segmentation models can be used to provide segmentation map information. Second, as some methods have shown, the synthesis and registration processes can benefit the segmentation model by providing auxiliary information, which may help develop better segmentation models on unannotated datasets.

[0086] The segmentation model is represented as a function. : ,in, denotes the segmentation domain. Based on the constraints among the synthesis, registration, and segmentation tasks, the objective of the joint synthesis-registration-segmentation model can be computed as:

[0087] (8),

[0088] where, and are two binary volumes between them, which is widely used in medical image segmentation. This loss term connects the three components, including the generator, the registrator, and the segmentator, and provides the main performance improvement for the system.

[0089] To make the consistency term work properly, the estimation of the segmentator needs to be accurate enough. However, with only the consistency loss, the segmentator can not learn meaningful semantic information. The segmentator that generates all background volumes can also optimize the consistency loss. To avoid this situation, according to some embodiments, additional data with segmentation annotations in one modality is added as supervision to provide the segmentator with a meaningful initial state. The supervised loss can be computed as: - Dice )] (9),

[0090] where, and are in the same modality but do not overlap. is the corresponding annotation. Then the final regularization term provided by the segmentator can be computed as:

[0091] (10).

[0092] For joint optimization, based on the above analysis, the final objective of the whole JSSR process can be computed as:

[0093] (11).

[0094] To provide a good initial point for all components, the JSSR process can first train on supervised data, then can learn and G using equation (7) on unsupervised data. Finally, the JSSR process can jointly learn all components using equation (11). During the optimization process of equation (7) and equation (11), the training of the GAN model can adopt a classic alternating process, first fixing to optimize , then fixing to optimize other components, and then alternating this optimization process.

[0095] In the following examples, various aspects of the image processing method according to Figs. 1 and 3-6 are implemented using available datasets to demonstrate the effectiveness of the image processing method. The main experiment was conducted on a large dataset of 3D dynamic contrast multi-phase liver CT scans extracted from the archives of Chang Gung Memorial Hospital (CGMH) in Taiwan. The dataset contains CT scans from 1485 patients, for each patient, there are image volumes of four different phases of intravenous contrast: i.e. the venous, arterial, delayed, and non-contrast phases. The patient population of the study consists of liver tumor patients who underwent CT imaging for an interventional biopsy. The ultimate goal is to develop a computer-aided diagnosis (CAD) system to identify the pathological subtype of any given liver tumor. For both human readers and computers, all phases of 3D CT images of each patient need to be precisely pre-registered so as to facilitate the downstream diagnosis process by observing and judging the visual dynamic contrast change pattern of liver tumor tissue in the order of non-contrast, arterial, venous, and delayed CT.

[0096] The different phases are obtained from the CT scanner at different time points after the injection of contrast agent. The images of different phases show different information according to the distribution of contrast agent in the human body. The intensity value of each voxel in the CT image, measured in Hounsfield Unit (HU), is an integer ranging from -1000 HU to 1000 HU, which can also be affected by the density of the contrast agent. The volume size of the CT image is 512x512xL, where L depends on the physical coordinates of the starting point of the scan CT and the axial resolution, which is 5mm in the dataset. Since the venous phase usually contains most of the diagnostic information, the venous phase is chosen as the anchor phase (the fixed image), and the images from the other three phases are synthesized and registered to the venous phase image. The experimental design divides the dataset into 1350 / 45 / 90 patients for training, validation, and testing, and annotates liver segmentation on the validation and test sets for evaluation. Note that a total of 1485x4 = 5940 3D CT scans (all containing pathological livers) were used in the experiment. To the best of our knowledge, this is the largest such clinically realistic study to date. For the supervised part, we chose a public dataset MSD, which contains 131 venous phase CT images with liver voxel annotations, and divided it into 100 / 31 for training and validation. The experimental design evaluates all registration, synthesis, and segmentation tasks to demonstrate how joint training improves each task.

[0097] The experimental design and all baselines for image synthesis, image registration, and image segmentation tasks are compared, including the following: for image synthesis, the experimental design selects Pix2Pix using reconstruction loss and cGAN loss. For image registration, the experimental design is first compared with the traditional state-of-the-art multi-modal registration Deeds method, which has the advantage of inference speed compared to traditional methods, while we can also show performance improvement, the experimental design is also compared with the learning-based method VoxelMorph with local cross-correlation to handle multi-modal image registration. For segmentation tasks, the experimental design is compared with the popular VNet framework in medical image segmentation.

[0098] To perform registration between the four different stages, several preprocessing procedures are performed before applying deformable registration. First, since even the CT images of different stages of the same patient have different volume sizes, we crop the maximum intersection of the four different stages according to the physical coordinates to make them the same size. Second, we use the venous phase as an anchor point and use the multi-lingual library to perform rigid registration of medical images between the four different stages. Third, we apply a window from -200HU to 200HU to each CT image and normalize it to -1 to 1, then adjust the CT volume to 256x256xL to accommodate GPU memory. For the public dataset, we sample along the axial direction to make the resolution 5mm, then apply the same preprocessing to the intensity values.

[0099]

[0100] Table 1: Evaluation of registration tasks on CGMHLiver dataset according to Dice score, HD (mm), ASD (mm), and running time

[0101] To optimize the objective, we solve all components using the Adam solver. We set the hyperparameters to λseg = λreg = 1, λsyn = 0.02. We choose different learning rates for different components in order to better balance the training. We set the learning rate of the generator to 0.0001, the learning rate of the registrator to 0.001, and the learning rate of the segmenter and the discriminator to 0.1. Another way to balance training is to adjust the weights of different loss terms. However, there are loss terms related to multiple components, which makes it more complicated to control each component individually. We train using instance normalization and a batch size of 1 on an Nvidia Quadro RTX 6000 GPU with 24 GB of memory, which takes about 1.4 GPU days.

[0102] The results of the registration task are summarized in Table 1. The methods are evaluated by the direct similarity of the segmentation of the fixed image, which is always in the venous phase, and the deformed segmentation of the moving image, which is selected from the arterial, delayed, and non-contrast. The similarity is measured by the Dice score, 95% Hausdorff distance (HD), and average surface distance (ASD). We report the mean and standard deviation on each measurement test set. We also report the consumed time (in seconds) on GPU / CPU for each method. In Table 1, we call Initial State the result before applying any registration, and Affine the result after rigid registration. We also compare with the traditional Deeds method and the learning-based VoxelMorph method. We call our joint system JSSR and JSSR-Reg as the registration part of JSSR. We also compare two ablations of JSSR. JSynR, which contains only the generator and the registrar, is optimized using equation (7). JSegR has the segmenter and the registrar. As shown in Table 1, the JSSR method outperforms the Deeds method by 0.83% on average Dice while taking advantage of much faster inference speed. Moreover, by taking advantage of the joint training, JSSR achieves significantly higher results than the VoxelMorph method (more than 1.28%) with comparable inference time. We can observe the step-by-step improvement from the VoxelMorph method to JSynR to JSSR, which proves the necessity of joint training. Ref. Figure 7 More detailed information about the results, where boxplots of the registration results (Dice Score) of different methods in three phases (A for arterial, D for delayed, N for non-contrast) are provided.

[0103]

[0104] Table 2: Synthesis and segmentation tasks on the CGMHLiver dataset evaluated according to the average Dice score

[0105] We show the results of the synthesis and segmentation tasks in Table 2. We evaluate the generative model by applying the segmentation models to the synthetic images. The intuition is that the better the synthetic images, the better the segmentation maps can be estimated. We evaluate three segmentation models. VNet baseline is trained with full supervision on the MSD dataset. JSegR-Seg is the segmentation part of JSegR. JSSR-Seg is the segmenter of the JSSR system. For each segmentation model, we test it on different versions of the generative model. For non-synthetic, we directly apply the segmentation model to the test data on the original images of the four phases and test the average Dice between the prediction and the annotation. For the other three generative models, we test the segmentation model on the original venous images and the synthetic images from the arterial, delay, and non-contrast phases. From the non-synthetic line, if we apply the segmenter model directly to the arterial phase, delay phase, and non-contrast phase, we can observe that the performance will decrease because the supervised data is all from the venous phase. Among the three phases, the delay phase decreases the least, and the non-contrast phase decreases the most, indicating that the domain gap between venous and delay is greater than that between venous and non-contrast. For Pix2Pix, the performance experiences different degrees of decline among different segmenters, and it is not as high as non-synthetic. This can be caused by the artifacts introduced by the GAN model, and the constraint provided by the L1 term is less due to the lack of paired data. For JSynR-Syn and JSSR-Syn, the performance is better by providing paired data from the registrator, but only comparable to non-synthetic. For JSynR-Syn, because JSynR is not learned jointly with the segmentation, the performance of the synthetic images is not necessarily improved. However, for JSSR-Syn, it means that the constraints we use to optimize the system do not bring enough communication between the generator and the segmenter. At the same time, we can see significant improvements from VNet to JSegR-Seg to JSSR-Seg on non-synthetic data, which means that although the generator and the segmenter are not well connected, the segmenter can still benefit from the connected system including the synthetic part. Refer to FIG. 8A and FIG. 8B, which are qualitative examples of JSSR registration, synthesis, and segmentation results, where Figure 8A shows the arterial phase CT results image, Figure 8B shows the non-contrast phase CT results image (in Figure 8A and Figure 8B in which the top row, left to right, first panel shows the moving image in the first modality (arterial phase image in Figure 8A and Figure 8Bthe second panel from the top left shows a fixed image in the second modality (the venous phase image); the third panel from the top left shows a synthetic image generated from the moving image in the second modality; the rightmost panel in the top row shows the moving image warped by applying the spatial transformation of the registration model to it; the first panel from the bottom left shows a segmented liver image in the first modality (the arterial phase image in Fig. 8A and the non-contrast phase image in Fig. 8B); the second panel from the bottom left shows a predicted segmented liver image in the second modality; the third panel from the bottom left shows a segmented liver image in the second modality; the rightmost panel in the bottom row shows a deformed segmented liver image in the first modality.

[0106] To perform ablation studies to compare JSegR with JSSR, we implement JSegR as another ablation. The goal is to investigate the importance of the synthesis module in the JSSR system. Since JSegR does not have a generator, the registration takes different phase images as input directly. The segmentation consistency term in Eq. (8) then becomes:

[0107] (12),

[0108] where, This framework includes co-learning the registration and the segmentation as some previous methods do. However, in this disclosure, in different domains and annotations are not available for both of them. Intuitively, this approach can work fine because are in different phases. However, as shown in Table 2, the performance drop across phases is not too severe even for the baseline V-Net, and with an imperfect segmentation, JSegR can achieve better results on registration than JSynR and close to JSSR, which demonstrates the importance of incorporating semantic information in registration. But for data with large domain gaps like CT, MRI, etc., the synthesis part still needs to regularize the JSSR system.

[0109] In the experiments, since the joint training process can be sensitive to certain constraints, we do not introduce Figure 1 all the constraints in (8). We have tried several additional constraints as ablations. One of them is to add

[0110] (13),

[0111] where, The system is regularized by This is optimized. Furthermore, if we add equation (12) to equation (8), the system has a chance to end up outputting all the blank segmentations. It is likely that with more constraints and more complex parameter settings, the system can converge to a better point. Since in JSSR each component can be separated, further enabling each component to have a more powerful sub-framework, the performance of the JSSR method is further improved.

[0112] Other examples for evaluating the JSSR process are shown in FIGS. 9A-12B. In Figures 9A-12B the figure references follow Figure 5 . As Figure 9A illustrates an example. Specifically, FIG. 9A shows the difference between the two inputs indicated at the top (left and right of the arrow) with each row of veins and arteries, delay, and non-contrast. Figure 9B The visualization of the segmentation map of the JSSR process corresponding to Figure 9A the example is shown. Similarly, Figure 10A and 10B show the difference map and the corresponding segmentation map in the second example. Figure 11A and 11B show the difference map and the corresponding segmentation map in the third example. Figure 12A and 12B show the difference map and the corresponding segmentation map in the fourth example.

[0113] As shown in FIGS. 9A-12B, the generator component is conditioned on and which has both benefits and drawbacks. In the column, we can see that the synthetic image can capture the intensity changes between and well since the checkerboard image only shows a slight difference between and . However, in the column, as shown in the artery row in Figure 9A and the artery row of FIG. 11 A, the generator also introduces additional boundary information from which will affect the registrator.

[0114] Further, as shown in FIG. 10B, the segmenter component produces unsatisfactory segmentations, but the overlap in o is still apparent, which means that the consistency is well satisfied, but can provide the registrator with incorrect supervision. Thus, a better consistency term can help in this case.

[0115] Further, although the image processing method provided by the present application is only tested on multi-phase CT images, since the method is equipped with a generator and a segmenter, it can be applied to many application scenarios, such as registration from CT to MRI, or domain adaptation for segmentation between CT and MRI, or it can help tumor detection by combining multi-modal information, and the segmenter is used to segment normal organs and tumor regions.

[0116] In summary, the image processing method and device provided by the present disclosure provides several contributions. The JSSR process for multi-modal image registration utilizes joint learning based on the intrinsic connection between synthesis, segmentation and registration tasks. Optimization can be optimized end-to-end with multiple unsupervised consistency losses, and each component can benefit from the joint training process. The JSSR process is evaluated on a large-scale multi-phase clinical real CT image dataset without segmentation annotations. After joint training, the performance of registration and segmentation is improved by 0.91% and 1.86% respectively in terms of average Dice score at all stages. The image processing method and device provided by the present disclosure is 1.28% better than the recently proposed Voxel-Morph algorithm, 0.83% better than the most advanced conventional multi-modal registration method, and has a relatively fast inference time. In addition, the provided method does not use or rely on any manual segmentation labels from the CT imaging dataset, which indicates that it is scalable and generalizable, and has great potential for wide application in practical clinical scenarios.

[0117] The above describes the method and device provided by the embodiments of the present application in detail. The principles and implementation manners provided by the present application are illustrated by specific examples. The description of the above embodiments is only used to help understand the method provided by the present application. Meanwhile, those skilled in the art will make changes to the specific embodiments and their application scope according to the concept provided by the present disclosure. In summary, the content of the specification should not be interpreted as limiting the present disclosure.

[0118] The present disclosure contains material protected by copyright. Copyright © belongs to the copyright owner. The copyright owner does not object to the copying of the patent document or the disclosure of the patent in the official records and files of the Patent and Trademark Office.

Claims

1. An image processing method for performing image alignment, characterized by, The method comprises: obtaining a motion image generated by a first imaging modality; obtaining a fixed image generated by a second imaging modality; jointly optimizing a generative model, a registration model and a segmentation model applied to the motion image and the fixed image according to a plurality of cost functions, the plurality of cost functions comprising a first objective function and a second objective function; the first objective function being a sum of a first regularization cost function of the generative model and a second regularization cost function of the registration model; the first regularization cost function being a weighted sum of a texture-based generative cost function and a conditional generative adversarial network cost function, the texture-based generative cost function accounting for image texture differences, and the conditional generative adversarial network cost function being conditioned on the fixed image; the second objective function being a sum of a joint generative-registration-segmentation cost function and a supervised segmentation cost function, the joint generative-registration-segmentation cost function accounting for constraints between the synthesis, registration and segmentation tasks, and the supervised segmentation cost function accounting for supervised learning of the segmentation model; and applying a spatial transformation corresponding to the optimized registration model to the motion image to align the motion image with the fixed image; wherein, the generative model generates a synthetic image from the motion image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthetic image with the fixed image; and the segmentation model estimates segmentation maps of the motion image, the fixed image and the synthetic image.

2. The image processing method of claim 1, wherein, jointly optimizing the generative model, the registration model and the segmentation model comprises: performing supervised learning of the segmentation model using an image dataset with ground-truth segmentation maps; and performing unsupervised learning of the generative model, the registration model and the segmentation model using the motion image generated by the first imaging modality and the fixed image generated by the second imaging modality.

3. The image processing method of claim 2, wherein, the performing unsupervised learning of the generative model, the registration model and the segmentation model comprises: jointly learning the generative model and the registration model according to a first objective function; and jointly learning the generative model, the registration model and the segmentation model according to a weighted sum of the first objective function and a second objective function.

4. The image processing method of claim 1, wherein, The method further comprises: using a weighted sum of a registration cost function and a smoothness regularization term as the second regularization cost function of the registration model, the registration cost function accounting for image registration differences, and the smoothness regularization term regularizing non-realistic spatial transformations.

5. The image processing method of claim 4, wherein: a L1 loss between a warped image based on the motion image and the fixed image is computed as the registration cost function, the L1 loss being used to measure a similarity between the fixed image and the warped image.

6. The image processing method of claim 1, wherein: the fixed image is a venous phase contrast computed tomography (CT) image of a patient; and The motion image is one of: an arterial phase contrast CT image of the patient, a delayed phase contrast CT image of the patient, or a non-contrast CT image of the patient.

7. The image processing method of claim 1, wherein, The joint optimization of the generation model, the registration model, and the segmentation model comprises: optimizing the generation model, the registration model, and the segmentation model using a first learning rate for the generation model, a second learning rate for the registration model, and a third learning rate for the segmentation model, the third learning rate being greater than the second learning rate, and the second learning rate being greater than the first learning rate.

8. An image processing device for performing image alignment, characterized by, The device comprises: a memory storing computer readable instructions; and a processor coupled with the memory and configured to, when the computer readable instructions are executed: obtain a motion image generated by a first imaging modality; obtain a fixed image generated by a second imaging modality; jointly optimize a generation model, a registration model, and a segmentation model applied to the motion image and the fixed image according to a plurality of cost functions, the plurality of cost functions comprising a first objective function and a second objective function; the first objective function being a sum of a first regularization cost function of the generation model and a second regularization cost function of the registration model; the first regularization cost function being a weighted sum of a texture-based generation cost function and a conditional generative adversarial network cost function, the texture-based generation cost function accounting for image texture differences, and the conditional generative adversarial network cost function being conditioned on the fixed image; the second objective function being a sum of a joint generation-registration-segmentation cost function and a supervised segmentation cost function, the joint generation-registration-segmentation cost function accounting for constraints between the synthesis, registration, and segmentation tasks, and the supervised segmentation cost function accounting for supervised learning of the segmentation model; and apply a spatial transformation corresponding to the optimized registration model to the motion image to align the motion image with the fixed image; wherein, the generation model generates a synthetic image from the motion image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthetic image with the fixed image; and the segmentation model estimates segmentation maps of the motion image, the fixed image, and the synthetic image.

9. The image processing apparatus of claim 8, wherein The processor is further configured to: perform supervised learning of the segmentation model using an image dataset with ground truth segmentation maps; and perform unsupervised learning of the generation model, the registration model, and the segmentation model using the motion image generated by the first imaging modality and the fixed image generated by the second imaging modality.

10. The image processing device according to claim 9, characterized by The processor is further configured to: jointly learn the generation model and the registration model according to a first objective function; and jointly learn the generation model, the registration model, and the segmentation model according to a weighted sum of the first objective function and a second objective function.

11. The image processing device according to claim 8, characterized by The processor is further configured to: use a weighted sum of a registration cost function and a smoothness regularization term as the second regularization cost function of the registration model, the registration cost function accounting for image registration differences, and the smoothness regularization term regularizing non-realistic spatial transformations.

12. The image processing device of claim 11, wherein: an L1 loss between a warped image based on the motion image and the fixed image is calculated as the registration cost function, the L1 loss being used to measure the similarity between the fixed image and the warped image.

13. A non-transitory computer-readable storage medium storing a plurality of instructions, the plurality of instructions comprising: the instructions, when executed by the processor, cause the processor to perform operations comprising: obtaining a motion image generated by a first imaging modality; obtaining a fixed image generated by a second imaging modality; jointly optimizing a generative model, a registration model, and a segmentation model applied to the motion image and the fixed image according to a plurality of cost functions, the plurality of cost functions including a first objective function and a second objective function; the first objective function being a sum of a first regularization cost function of the generative model and a second regularization cost function of the registration model; the first regularization cost function being a weighted sum of a texture-based generative cost function and a conditional generative adversarial network cost function, the texture-based generative cost function accounting for image texture differences, and the conditional generative adversarial network cost function being conditioned on the fixed image; the second objective function being a sum of a joint generative-registration-segmentation cost function and a supervised segmentation cost function, the joint generative-registration-segmentation cost function accounting for constraints between the synthesis, registration, and segmentation tasks, and the supervised segmentation cost function accounting for supervised learning of the segmentation model; and applying a spatial transformation corresponding to the optimized registration model to the motion image for spatial transformation to align the motion image with the fixed image; wherein: the generative model generates a synthetic image from the motion image conditioned on the fixed image; the registration model estimates a spatial transformation to align the synthetic image with the fixed image; and the segmentation model estimates segmentation maps of the motion image, the fixed image, and the synthetic image.

14. The non-transitory computer-readable storage medium of claim 13, wherein, the instructions further cause the processor to perform operations comprising: performing supervised learning of the segmentation model using an image dataset with ground-truth segmentation maps; and performing unsupervised learning of the generative model, the registration model, and the segmentation model using the motion image generated by the first imaging modality and the fixed image generated by the second imaging modality.

Images