Automated deformable medical image registration method

Abstract

The present invention belongs to the field of image processing and computer vision, and proposes an automated deformable medical image registration method, which simultaneously searches for the structure of the registration network and its corresponding training targets, so as to facilitate users to perform medical image analysis. The present invention mainly derives the registration network structure and training targets suitable for different registration tasks through automatic search mechanism and joint optimization, so as to automatically design the feature selection module, deformation estimation module and training target of the medical image registration deep learning solution for various types of medical data, and solves the problem that the existing method of manually designing and adjusting the network structure and training targets requires a lot of manpower and experience.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and proposes an automated deformable medical image registration method, which simultaneously searches for the structure of a registration network and its corresponding training targets, making it convenient for users to perform medical image analysis. Background Art

[0002] Medical image registration mainly seeks the correspondence between two or more images, and then uses the obtained correspondence to make the global or local consistency of the image to be registered with the target image, so as to achieve better analysis and application of multiple images. In the registration process, the image to be registered is the image that undergoes spatial transformation, also known as the source image or floating image, and the fixed image is another image that does not undergo transformation, also known as the target image. Medical image registration has a wide range of applications, such as helping doctors diagnose by combining information from different modalities, detecting changes in a certain anatomical part by registering medical images obtained at different times, and studying the progression of the disease. Existing medical image registration methods are mainly divided into traditional methods and deep learning methods.

[0003] The traditional registration method is an iterative optimization process: the image is matched and transformed through feature matching, first the feature information in the image is extracted, then the spatial transformation method is selected, the similarity between the images is calculated, and finally the appropriate optimization method is selected for continuous iterative optimization, so that the similarity of the registered images is the highest. If the accuracy of the registration needs to be improved, the pixel information of the image must first be converted into feature information of lower dimensions, and then feature extraction is performed, so feature extraction is particularly important. Commonly used features include: corner points, LPB, SURF, centroid or template. Among the methods for extracting feature information, some use automatic feature extraction methods, and some combine manual description features with automatic feature extraction, that is, extract obvious structural features in the image manually or automatically. The above methods are time-consuming for existing large-size medical images, which is not conducive to practical clinical applications.

[0004] Medical registration methods based on deep learning complete registration by solving the nonlinear correspondence between input image pairs and deformation fields that align the image pairs through a network solution. When faced with various medical scenarios, learning-based methods need to adjust the network structure and training objectives, which greatly determine the quality of the deformation field. However, the optimal registration structure and objectives vary greatly across different image modalities and anatomical structures, and they are difficult to provide a comprehensive solution. References [A.Hoopes, M.Hoffmann, B.Fischl, JVGuttag, and AVDalca, "Hypermorph: Amortized hyperparameter learning for image registration," in International Conference on Information Processing in Medical Imaging, 2021, pp. 3–17.] and [R.Liu, Z.Li, X.Fan, C.Zhao, H.Huang, and Z.Luo, "Learning deformable image registration from optimization: Perspective, modules, bilevel training and beyond," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.] proposed new registration network training strategies that eliminate the need to adjust important registration hyperparameters during training. However, existing methods only optimize registration hyperparameters and still rely on manual selection of other components, such as network structure, which may lead to suboptimal solutions. In addition, manually selecting and designing registration structures is often insufficient to explore a large number of possibilities. In summary, although learning-based registration methods have achieved considerable success, this success critically relies on computer experts to choose appropriate network structures and optimize training objectives, which requires considerable computation, human effort, and a lot of expertise. More importantly, the complexity of these tasks is often beyond the capabilities of non-computer experts (such as medical experts). Therefore, without expertise in machine learning, researchers need user-friendly automatic registration tools that can easily use off-the-shelf algorithms for various medical registration scenarios. Summary of the invention

[0005] To solve the above problems, the present invention provides an automated deformable medical image registration method, which is an automatic learning method for deformable medical image registration by jointly optimizing the network structure and training objectives. The present invention mainly derives the registration network structure and training objectives suitable for different registration tasks through automatic search mechanism and joint optimization, so as to automatically design the feature selection module, deformation estimation module and training objectives of the medical image registration deep learning solution for various types of medical data, solving the problem that the existing methods of manually designing and adjusting the network structure and training objectives require a lot of manpower and experience.

[0006] The specific technical scheme of the present invention is:

[0007] An automated deformable medical image registration method, where the registration network Ψ(ω; s, t) represents the deformation field between the source image s and the target image t in the medical image data. ω represents the learnable network weight; the vectorized form α is introduced as the network structure parameter, which represents a series of convolution operators constituting the network, also known as the candidate convolution operation of each network structure unit; at the same time, the deformation field The training objective includes a hyperparameter λ, which is used to measure the similarity and regularization target items; and a gradient-based strategy is used to sequentially optimize the network structure α of the registration network, the hyperparameter λ in the training objective, and the network weight ω, and finally obtain the registered medical image; the specific steps are as follows:

[0008] Step 1) Data preprocessing: FreeSurfer software was used to perform motion correction, NU intensity correction and normalization on all medical image data, and FSL software was used for affine normalization; the data set was divided into training set, validation set and test set.

[0009] Step 2) Construct an adaptive feature unit and a task-aware deformation unit, connect them in series to form a registration network, and introduce the vectorized form α F and α D The network structures of the adaptive feature unit and the task-aware deformation unit are respectively encoded; the adaptive feature unit is used to extract medical image features, and the task-aware deformation unit is used to generate a deformation field to act on the medical image and distort the medical image; both the adaptive feature unit and the task-aware deformation unit use a three-layer convolutional network, and each unit is represented as a four-node x (i) A directed acyclic graph, each node x (i) Connect to the next node x (i+1) , the edge connecting two nodes is a candidate convolution operation; in the network structure α of each adaptive feature unit FIn the last convolution layer, the features are downsampled; the adaptive feature units of the source image s and the target image t have the same structure, but different network weights; the network structure α of each task-aware deformation unit D In , the last convolutional layer performs an upsampling operation on the features; the input of the task-aware deformation unit is the features from the previous unit and the upsampled deformation field.

[0010] Step 3) Search the network structure α of the adaptive feature unit F , used to extract multi-scale medical image features; fix the hyperparameter λ in the training objective, initialize the network weight ω, is the registration loss of the registration network on the training set, is the registration loss of the registration network on the validation set, and the registration loss L of the registration network on the training set and the validation set obtained according to the training data reg , make sure that L reg The network structure of the smallest adaptive feature unit α F ; The specific steps are as follows:

[0011] 3-1) For the source image s in the medical image data and the registered medical image Calculate the registration loss on the training set and validation set respectively and The specific registration loss is composed of the similarity loss l sim and the regularization loss l reg The composition is calculated as follows:

[0012]

[0013] Among them, λ1,λ2,λ3,λ4 are hyperparameters that weigh different loss terms. Represents the similarity loss at full resolution, half resolution, and quarter resolution.

[0014] 3-2) Initialize network weight ω;

[0015]

[0016] 3-3) Update the network structure of the adaptive feature unit α F ; For the source image s and target image t in medical image data, combined with the registration loss and Optimize and update the network structure of the adaptive feature unit α F :

[0017]

[0018] Where ∈ is set to a scalar equal to the learning rate; from formulas (2) and (3), we can get:

[0019]

[0020]

[0021] 3-4) Obtain the network structure α of the optimal adaptive feature unit through continuous relaxation technology F ; The continuous relaxation technique transforms the time-consuming discrete search in a huge space into a continuous optimization problem; is a set of candidate convolution operations, where each convolution operation represents a convolution operation to be applied to node x. (i) A convolution operation o of ; among all possible convolution operations, the discrete selection of a specific convolution operation is relaxed to:

[0022]

[0023] Searching for a network structure is simplified to computing a set of continuous variables And get the most likely convolution operation Finally, the network structure α of the optimal adaptive feature unit is obtained F .

[0024] Step 4) Search the network structure α of the task-aware deformation unit D , used to generate the deformation field Will Act on the source image s and distort s to obtain the registered medical image. Reinitialize the network weights ω based on the learned feature pyramid and the registration loss and Search for the best performing adaptive feature unit α D , using the network structure of the task-aware deformation unit α D Generate registration deformation fields at different scales The specific steps are as follows:

[0025] 4-1) Initialize network weight ω; based on α F The generated feature pyramid generates registration deformation fields of different scales Calculate the registration loss according to step 3-1) and Update the network structure of the task-aware deformation unit α D :

[0026]

[0027]

[0028] From formula (7) and (8), we can get:

[0029]

[0030]

[0031] 4-2) Using the same continuous relaxation technique in step 3-4), searching for network structure is simplified to calculating a set of continuous variables And finally get the most likely convolution operation Get the optimal α D .

[0032] Step 5) Searching for hyperparameters λ in the training target; the training target contains hyperparameters λ that weigh different losses, and learning hyperparameters λ by evaluating the overlapping anatomical regions between the target and the registered medical image; searching for hyperparameters λ in the network training target, reinitializing the network weights ω, and based on the searched network structure α={α F ,α D}, determine the best network training target specific to the specific registration task based on the performance of the registration network on the training set and the validation set; the specific steps are as follows:

[0033] 5-1) Calculate the segmentation loss L seg ; Order s m ,t m Represent the segmentation map of the source image s and the target image t, respectively, and the deformation field Represents anatomical correspondence, quantified using the Dice score and t m The volume overlap of the same anatomical structure in the segmentation loss L is defined as seg ; A higher Dice score indicates that the anatomical structure is completely matched, indicating that the registration network can solve the medical image registration task well, corresponding to a satisfactory hyperparameter λ.

[0034]

[0035] 5-2) Calculate the registration loss according to step 3-1) Reinitialize the network weights ω and update the hyperparameters λ in the training objective:

[0036]

[0037]

[0038] The optimal hyperparameter λ can be obtained from formulas (12) and (13):

[0039]

[0040]

[0041] Step 6) Retrain the registration network. Based on the searched network unit structure α={α F ,α D} and the hyperparameter λ in the training objective, retrain the registration network, obtain the network weights, and finally obtain the registration network for medical image registration;

[0042] Step 7) Input the source image s and the target image t into the trained registration network to generate the corresponding deformation field The source image s and the target deformation field Perform spatial transformation to obtain registered medical images The registration performance of the registration network is tested on the test set.

[0043] The beneficial effects of the present invention are:

[0044] 1) Compared with the existing learning-based methods that require a lot of effort to heuristically design the registration network, the present invention establishes an automatic framework to automatically design the registration network by jointly optimizing the network structure and the target, solving the problem that the existing methods require a lot of manpower and experience to manually design and adjust the network structure and training targets.

[0045] 2) To efficiently discover architectures that are appropriate for registration scenarios of interest, we construct a search space with adaptive feature units and task-aware deformation units, which consist of various convolution operators that are effective for feature learning and deformation estimation. To deploy scenario-oriented training losses, we optimize hyperparameters of widely used similarity metrics and regularization.

[0046] 3) The present invention uses efficient gradient-based strategies to solve complex optimization problems of different natures (network structure, hyperparameters in training objectives, network weights). Compared with the debugging process of manual training experiments, these automatic search mechanisms with collaborative optimization greatly reduce time overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a framework diagram of the present invention;

[0048] Figure 2 It is the overall flow chart of the present invention;

[0049] Figure 3 This is a diagram showing the registration effect in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0051] The framework and process of the present invention are as follows Figure 1 and Figure 2As shown, this embodiment takes brain MR registration as an example, and the specific implementation scheme is as follows:

[0052] The first step is data preprocessing. The atlas published in the literature [G. Balakrishnan, A. Zhao, MR Sabuncu, J. Guttag, and AV Dalca, "Voxelmorph: A learning framework for deformable medical image registration," IEEE Transactions on Medical Imaging, vol. 38, no. 8, pp. 1788–1800, 2019.] is used as the target image, and 528 T1-weighted brain images from the ADNI, ABIDEPPMI, and OASIS datasets are used to divide the data into 377 training sets, 21 validation sets, and 130 test sets. In view of the huge differences between different image datasets, all medical image data are preprocessed using motion correction, NU intensity correction, normalization, and affine registration. In the specific implementation process, FreeSurfer software is used for motion correction, NU intensity correction, and normalization, and FSL software is used for affine registration;

[0053] The second step is to automatically learn the network structure and training objectives.

[0054] 2-1) Automatically learn adaptive feature unit α F and task-aware deformation unit α D The search space of the network unit structure of the present invention includes two unit structures, and a total of 8 convolution operators are introduced, as follows:

[0055] 1×1×1 Conv(1-Conv)

[0056] 5×5×5 Conv (5-Conv)

[0057] ·3×3×3 Separable Conv(3-SConv)

[0058] ·5×5×5 Separable Conv(5-SConv)

[0059] ·3×3×3 Dilation Conv(3-DConv)

[0060] ·5×5×5 Dilation Conv(5-DConv)

[0061] ·7×7×7 Dilation Conv(7-DConv)

[0062] 3×3×3 Conv (3-Conv)

[0063] Composition 2*8 (3+3) , about 50,000 possible network structures. The optimal network structure α learned in the example training search in the present invention is α = {α F ,α D}={{7-DConv,3-SConv,3-SConv,3-Conv,3-DConv,3-Conv},{3-Conv,3-SConv,3-SConv,7-Conv,7-DConv,7-DConv}}

[0064] 2-2) Automatically learn the hyperparameter λ and determine the training target. Based on the medical image registration effect of this example, calculate the registration loss L reg and segmentation loss L seg , update the hyperparameter λ. The optimal hyperparameter learned in the training search of this example is λ = {12.14, 12.15, 5.46, 0.37}

[0065] The network of the present invention was trained for 45 epochs using the Adam optimizer with a batch size of 1 and a learning rate of 1e for α. -4 , the learning rate for λ is set to 4e -3 When α={α F ,α D The search process stops when the structure ranking of} and the upper target of the hyperparameter λ become stable within 10 epochs. The present invention uses zero initialization for the structure variables, which means that all possible convolution operations are given equal attention at the beginning.

[0066] The third step is training evaluation. Based on the second step, training is started from scratch using the best unit structure and training target. The present invention uses the Adam optimizer for training, and the learning rate is set to 1e -4 , the batch size is set to 1, and the search registration network is trained for 50,000 iterations. The source image and the target image are input into the trained registration network, and the adaptive feature unit extracts the features of the source image and the target image to generate multiple pairs of feature maps with different resolutions (i.e., full resolution, 1 / 2 resolution, and 1 / 4 resolution feature maps). The three feature maps with different resolutions are stored in the feature pyramid as the input of the subsequent prediction deformation field network. The task-aware deformation unit generates the target deformation field based on the feature pyramid. According to the target deformation field Warp the source image s to obtain the registered image

[0067] The visualization result of the registration effect of the present invention is shown in Figure 3 , showing the registration effect of the image on T1 brain MRI to the atlas. As shown in the figure, the large deformation in adult medical brain images makes the registration challenging and difficult. In addition, since adolescent MR scans are still in the intrinsic myelination and maturation process, the white matter and gray matter show obvious differences compared to fixed images, which also makes the registration difficult. As shown in the figure, the source image is well aligned with the target image, showing the good registration performance of the method of the present invention.

Claims

1. An automated deformable medical image registration method, It is characterized in that Let the registration network Ψ(ω; s, t) represent the deformation field between the source image s and the target image t in the medical image data ω represents the learnable network weight; the vectorized form α is introduced as the network structure parameter, which represents a series of convolution operators constituting the network, also known as the candidate convolution operation of each network structure unit; at the same time, the deformation field The training objective includes a hyperparameter λ, which is used to measure the similarity and regularization target items; and a gradient-based strategy is used to sequentially optimize the network structure α of the registration network, the hyperparameter λ in the training objective, and the network weight ω, and finally obtain the registered medical image; the specific steps are as follows: Step 1) Data preprocessing: FreeSurfer software was used to perform motion correction, NU intensity correction and normalization on all medical image data, and FSL software was used to perform affine normalization; the data set was divided into training set, validation set and test set; Step 2) Construct an adaptive feature unit and a task-aware deformation unit, connect them in series to form a registration network, and introduce the vectorized form α F and α D The network structures of the adaptive feature unit and the task-aware deformation unit are respectively encoded; the adaptive feature unit is used to extract medical image features, and the task-aware deformation unit is used to generate a deformation field to act on the medical image and distort the medical image; both the adaptive feature unit and the task-aware deformation unit use a three-layer convolutional network, and each unit is represented as a four-node x (i) A directed acyclic graph, each node x (i) Connect to the next node x (i+1) , the edge connecting two nodes is a candidate convolution operation; in the network structure α of each adaptive feature unit F In the last convolution layer, the features are downsampled; the adaptive feature units of the source image s and the target image t have the same structure, but different network weights; the network structure α of each task-aware deformation unit D In the last convolution layer, the features are upsampled; the input of the task-aware deformation unit is the features from the previous unit and the upsampled deformation field; Step 3) Search the network structure α of the adaptive feature unit F , used to extract multi-scale medical image features; fix the hyperparameter λ in the training objective, initialize the network weight ω, is the registration loss of the registration network on the training set, is the registration loss of the registration network on the validation set, and the registration loss L of the registration network on the training set and the validation set obtained according to the training data reg , make sure that L reg The network structure of the smallest adaptive feature unit α F ; The specific steps are as follows: 3-1) For the source image s in the medical image data and the registered medical image Calculate the registration loss on the training set and validation set respectively and The specific registration loss is composed of the similarity loss l sim and the regularization loss l reg The composition is calculated as follows: Among them, λ 1 ,λ 2 ,λ 3 ,λ 4 is a hyperparameter that weighs different loss terms, Represents the similarity loss at full resolution, half resolution, and quarter resolution; 3-2) Initialize network weight ω; 3-3) Update the network structure of the adaptive feature unit α F ; For the source image s and target image t in medical image data, combined with the registration loss and Optimize and update the network structure of the adaptive feature unit α F : Where ∈ is set to a scalar equal to the learning rate; from formulas (2) and (3), we can get: 3-4) Obtain the network structure α of the optimal adaptive feature unit through continuous relaxation technology F ;set up is a set of candidate convolution operations, where each convolution operation represents a convolution operation to be applied to node x. (i) A convolution operation o of ; among all possible convolution operations, the discrete selection of a specific convolution operation is relaxed to: Searching for a network structure is simplified to computing a set of continuous variables And get the most likely convolution operation Finally, the network structure α of the optimal adaptive feature unit is obtained F ; Step 4) Search the network structure α of the task-aware deformation unit D , used to generate the deformation field Will Act on the source image s, distort s to obtain the registered medical image; reinitialize the network weight ω, based on the learned feature pyramid, combined with the registration loss and Search for the best performing adaptive feature unit α D , using the network structure of the task-aware deformation unit α D Generate registration deformation fields at different scales The specific steps are as follows: 4-1) Initialize network weight ω; based on α F The generated feature pyramid generates registration deformation fields of different scales Calculate the registration loss according to step 3-1) and Update the network structure of the task-aware deformation unit α D : From formula (7) and (8), we can get: 4-2) Using the same continuous relaxation technique in step 3-4), searching for network structure is simplified to calculating a set of continuous variables And finally get the most likely convolution operation Get the optimal α D ; Step 5) Searching for hyperparameters λ in the training target; the training target contains hyperparameters λ that weigh different losses, and learning hyperparameters λ by evaluating the overlapping anatomical regions between the target and the registered medical image; searching for hyperparameters λ in the network training target, reinitializing the network weights ω, and based on the searched network structure α={α F ,α D }, determine the best network training target specific to the specific registration task based on the performance of the registration network on the training set and the validation set; the specific steps are as follows: 5-1) Calculate the segmentation loss L seg ; Order s m ,t m Represent the segmentation map of the source image s and the target image t, respectively, and the deformation field Represents anatomical correspondence, quantified using the Dice score and t m The volume overlap of the same anatomical structure in the segmentation loss L is defined as seg ; A higher Dice score indicates that the anatomical structure is fully matched, indicating that the registration network can solve the medical image registration task well, corresponding to a satisfactory hyperparameter λ; 5-2) Calculate the registration loss according to step 3-1) Reinitialize the network weights ω and update the hyperparameters λ in the training objective: The optimal hyperparameter λ can be obtained from formulas (12) and (13): Step 6) Retrain the registration network; based on the searched network unit structure α={α F ,α D } and the hyperparameter λ in the training objective, retrain the registration network, obtain the network weights, and finally obtain the registration network for medical image registration; Step 7) Input the source image s and the target image t into the trained registration network to generate the corresponding deformation field The source image s and the target deformation field Perform spatial transformation to obtain registered medical images The registration performance of the registration network is tested on the test set.

Images