A carotid artery multimodal image registration method based on strongly constrained affine deformation feature learning

Abstract

A method for multimodal image registration of carotid arteries based on learning of strongly constrained affine deformation features, wherein medical image data is input into a convolutional network, and fixed images and moving images are input into a trained network to obtain a deformable displacement deformation field DVF (Deformable vector field), and on the basis of obtaining the DVF, a strongly constrained affine module is introduced to obtain a global affine transformation, and the DVF and the affine transformation are combined to obtain a new grid DA grid (DVF and affine transformation combined grid). In addition to using neural network training to generate a deformable deformation field, the present invention obtains a global affine deformation in the same network model, thereby improving the speed and accuracy of image registration.

Description

Technical Field

[0001] The present invention relates to the field of medical image registration, and in particular to a method for registering multimodal medical images using deep learning. Background Art

[0002] Establishing image correspondence through image registration is the key to many clinical tasks, such as image fusion, organ atlas creation, and tumor growth monitoring. With the development and progress of medical imaging technology, a large number of medical imaging devices have emerged. These imaging devices provide people with medical images of various modalities, such as CT, MRI, X-ray, etc. Medical images of different modalities provide different anatomical structures and organ tissue information, and the fusion of the two can provide more comprehensive diagnostic information.

[0003] Multimodal medical image registration is to find a spatial transformation correspondence between medical images of one or more modalities so that they are spatially consistent with the corresponding points on the image of another modality.

[0004] Traditional medical image registration can be divided into two categories: grayscale-based and feature-based registration methods. Grayscale-based registration methods refer to methods that use image grayscale information as the basis for registration, such as mutual information method, gradient information method, etc. Feature-based registration methods are further divided into external feature-based and internal feature-based registration methods. Registration methods using voxel intensity do not consider the local structural features of the image and use iterative optimization to search for the optimal parameters, resulting in a very slow processing speed and making it difficult to apply to real-time scenarios. Feature-based image registration methods extract invariant features in the reference image and the image to be registered, such as edge points, closed area centers, etc. Only a small amount of image features need to be extracted to greatly improve the registration rate, but the registration accuracy is more sensitive to feature extraction, and this method is difficult to provide accurate registration results.

[0005] With the application of deep learning to different technical fields, the medical imaging research community has developed deep learning-based methods and achieved state-of-the-art results in many applications, including image registration. Deep learning has been successfully used to improve the iteration and strength of registration-based methods. A deep learning network is used to estimate the similarity measure of two images and drive iterative optimization. This method only uses deep learning for similarity measurement and still requires traditional registration methods for iterative optimization. It does not fully utilize the advantages of deep learning, takes a long time, and is difficult to achieve real-time registration. Nowadays, neural networks are directly used to predict transformation parameters to achieve end-to-end image registration. The methods include deep iterative registration, supervised transformation estimation, and unsupervised transformation estimation. However, most of the multimodal registration methods using deep learning use neural networks to obtain deformable deformation fields, which fails to fully utilize the advantages. Summary of the invention

[0006] In order to overcome the defects of the existing technology, the present invention provides a multimodal carotid artery MRI registration method based on variable deformation and strongly constrained affine deformation. In addition to generating a variable deformation field by neural network training, a global affine deformation is obtained in the same network model, thereby improving the speed and accuracy of image registration.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for carotid artery multimodal image registration based on strongly constrained affine deformation feature learning, the method comprising the following steps:

[0009] (1) Data selection: Using multimodal carotid artery data, fixed images and floating images were selected. The fixed images were T1-weighted images (T1GD) that highlighted tissues, and the floating images were MRI images of the TOF sequence that highlighted the carotid artery using angiography. The carotid artery labels corresponding to the two sequences were marked respectively.

[0010] (2) Network training: According to the fixed image and the floating image and the segmented label data, the fixed image and the floating image are input into the neural network to generate a displacement field, and the carotid artery label of the floating image is interacted with the displacement field to obtain a distorted label, and the Dice loss function is obtained through the deformed label and the label of the fixed image, and the loss function of the regularization term is obtained through the displacement field;

[0011] (3) Image registration: Based on the model generated by network training, the fixed image and the floating image are input into the trained network to obtain the displacement field between the two images. The distorted image is obtained through the displacement field and the floating image originally input into the network.

[0012] Furthermore, in step (1), data selection includes the following steps:

[0013] (1.1) Image preprocessing: The images are uniformly corrected for offset fields and then resampled to obtain data with a voxel size of 0.6*0.6*0.6. Based on the resampling, we perform image cropping to obtain images of the same size. The process is as follows:

[0014] (1.1.1) Crop the two-modal images left and right, front and back, and top and bottom to expand the data. The cropping process should consider the spatial information matching relationship of the corresponding points of different modalities;

[0015] (1.1.2) Since the spatial coordinate system for image positioning of floating images uses RAS (Right, Anterior, Superior), and that of fixed images uses RAI (Right, Anterior, Inferior), the spatial position coordinates of the two sequence images are:

[0016] fixed(x,y,z)=moving(x,-z,y)

[0017] Among them, moving(x,-z,y) and fixed(x,y,z) represent the pixel positions of floating images and fixed images in the spatial coordinate system of image positioning, respectively. This requires that the spatial position coordinates fixed(x,y,z) need to correspond to moving(x,y,z) before the data is input into the network. Therefore, it is necessary to perform relevant operations of device flipping on the moving image to convert moving(x,-z,y) into moving(x,y,z) to meet the basic requirements of pixel matching during network training;

[0018] (1.2) Labeling of carotid artery labels: The labeling software ITK-SNAP was used to label the vascular locations in the two MRI sequences of the carotid artery.

[0019] Furthermore, in step (2), the network training includes the following steps:

[0020] (2.1) Multi-feature input restricted affine model structure design:

[0021] Input moving and fixed images, and after the pixel value matching operation is performed in the input network as described in (1.1.2), the value splicing operation is performed:

[0022]

[0023] Where a is the total number of pixels of the moving image and the fixed image, is the splicing symbol;

[0024] (2.1.1) Multi-feature input prediction fusion using serial convolution:

[0025] The serial structure can obtain different abstract features between each convolution layer. Affine deformation and variable deformation share the downsampling structure. We introduce the input of the affine layer between the downsampling and upsampling convolution layers, so that we can make full use of the feature parameters of different layers and linearly project them into an affine matrix to predict more accurate affine deformation.

[0026] After the three affine deformations are output, they are combined with the deformable grid and related weighted fusion is performed. The weights of affine layer C1, affine layer C2, and affine layer C3 are 0.2, 0.3, and 0.5 respectively. In this way, the input affine layer with high feature volume has a larger proportion and the parameter utilization effect is higher. Finally, the DA grid (the combination of the deformable grid and the affine grid) is obtained for image registration;

[0027] (2.1.2) Strong constraints to solve the initialization of network training scale imbalance and adaptive affine deformation:

[0028] Strong constraints are imposed on the training process of the affine matrix to constrain the change matrix, preventing excessive affine deformation during training from causing abnormal loss functions, and solving the initialization problem of unbalanced network training scales;

[0029] The affine transformation layer is implemented by a fully connected layer to regress the transformation parameters, resulting in a single 12-neuron dense layer, which represents the three-dimensional affine transformation parameters of translation, rotation, scaling, and shearing, respectively. Assuming that the input affine layer after convolution is set to x, then A m×n =affine(X,W,b)=X m×k W k×n +b 1×n , initialize the weight W k×n is a random normal distribution, and X m×k W k×n Strong constraints, Where -r≤a ij ≤r,(i=1,2,3,j=1,2,3,4), r is the constraint value, so that each parameter of the matrix can be strongly constrained, and the constraint value of each affine layer is different to increase the variable range of affine prediction. Different registered data can also set their own constraint values ​​to obtain adaptive affine deformation;

[0030] (2.2) Determination of loss function:

[0031] Dice is used as the loss to ensure the registration effect of the carotid artery. The curved image labels obtained after the fixed image labels and the moving image labels pass through the DA grid are used to calculate the arterial deformation:

[0032]

[0033] Where N is the total number of labeled voxels, a i ,b iare the binary labels of the fixed image label and the curved image label, respectively. To avoid anomalies, the artery deformation loss L ad Determined as:

[0034]

[0035] Bending energy is used as a regular penalty term to ensure the smoothness of the mesh:

[0036]

[0037] Where V is the number of voxels in the image domain. For affine transformations, this regularization term is zero, so only variability transformations are penalized;

[0038] According to the characteristics of carotid artery images, the key points at the bifurcation between the two sequences and different modes are marked, and the distance between the two marked points is calculated and introduced into the loss function:

[0039]

[0040] Where S is the total number of key points at the bifurcation, I fi and I mi are the spatial coordinates of the bifurcation key points corresponding to the fixed image and the moving image respectively.

[0041] The final loss is:

[0042] Loss = L ad +αL be +L distance

[0043] α is the regularization hyperparameter.

[0044] Furthermore, in step (3), after steps (1) and (2), the registered model is obtained through training. Although the training requires moving images, fixed images and their corresponding carotid artery labels, the verification process only requires a pair of unlabeled moving and fixed images. First, the trained network needs to be restored to predict the DA grid and the trained weight parameters, and then the DA grid between the two modalities is obtained by calculating the registered images, and finally the curved image is obtained through the DA grid and the moving image.

[0045] The technical concept of the present invention is: the data basis of the present invention is the multimodal magnetic resonance imaging data of the carotid artery. The innovation of the present invention lies in obtaining a pixel-based deformable displacement field based on the training of a deep learning neural network and obtaining a global affine displacement field based on strong constraints, and using dice as a loss function to ensure the reliability of the output.

[0046] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0047] First, compared with traditional registration methods, traditional registration methods often only use image features and grayscale information, often take a long time to register, and have poor registration accuracy. The method of the present invention uses deep learning to train from data to obtain a registration model, and calls the network to complete the registration in a few seconds, and can ensure registration accuracy. Secondly, compared with the registration method of deep learning, this method has a higher utilization rate of network feature parameters in realizing variable registration and affine registration, a simple network structure, a concise overall framework, simple training, and easy implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG. 1( a ) is a moving image of an original data image of the present invention;

[0049] FIG1( b ) is a fixed image of the original data image of the present invention;

[0050] Figure 2 A schematic diagram of the registration method provided by the present invention;

[0051] FIG3( a ) is a curved image after registration according to the present invention;

[0052] FIG3( b ) is a grid image output by the network after registration according to the present invention;

[0053] FIG. 3( c ) is a label superimposed image before and after carotid artery registration according to the present invention. DETAILED DESCRIPTION

[0054] In order to make the technical details and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] Referring to Figure 1(a) and Figure 1(b) to Figure 3(a) 3(b) and 3(c), a method for multimodal carotid artery image registration based on strongly constrained affine deformation feature learning, comprising the following steps:

[0056] (1) Data selection: Using multimodal carotid artery data, a fixed image and a floating image are selected. The fixed image is a T1-weighted image (T1GD) that highlights tissues, and the floating image is a TOF sequence of MRI images that highlights carotid arteries using angiography. The carotid artery labels corresponding to the two sequences are marked respectively. Figures 1(a) and 1(b) show the carotid artery images used in the embodiments of the present invention.

[0057] The data selection in step (1) includes the following steps:

[0058] (1.2) Image preprocessing: The images are uniformly corrected for offset fields and then resampled to obtain data with a voxel size of 0.6*0.6*0.6. Based on the resampling, we perform image cropping to obtain images of the same size. The process is as follows:

[0059] (1.1.3) Crop the two-modal images left and right, front and back, and top and bottom to expand the data. The cropping process should consider the spatial information matching relationship of the corresponding points of different modalities;

[0060] (1.1.4) Since the spatial coordinate system for image positioning of floating images uses RAS (Right, Anterior, Superior), and that of fixed images uses RAI (Right, Anterior, Inferior), the spatial position coordinates of the two sequence images are:

[0061] fixed(x,y,z)=moving(x,-z,y)

[0062] Among them, moving(x,-z,y) and fixed(x,y,z) represent the pixel positions of floating images and fixed images in the spatial coordinate system of image positioning, respectively. This requires that the spatial position coordinates fixed(x,y,z) need to correspond to moving(x,y,z) before the data is input into the network. Therefore, it is necessary to perform relevant operations of device flipping on the moving image to convert moving(x,-z,y) into moving(x,y,z) to meet the basic requirements of pixel matching during network training;

[0063] (1.2) Labeling of carotid artery labels: Use the labeling software ITK-SNAP to label the vascular locations in the two MRI sequences of the carotid artery;

[0064] (2) Network training: Based on the fixed image and floating image and the segmented label data, the fixed image and floating image are input into the neural network to generate a displacement field, and the carotid artery label of the floating image is interacted with the displacement field to obtain a distorted label. The dice loss function is obtained through the deformed label and the label of the fixed image, and the loss function of the regularization term is obtained through the displacement field. Figure 2 Shown is a schematic diagram of the network training registration method provided in step (2) of an embodiment of the present invention.

[0065] The network training of step (2) includes the following steps:

[0066] (2.1) Multi-feature input restricted affine model structure design:

[0067] Input moving and fixed images, and after the pixel value matching operation is performed in the input network as described in (1.1.2), the value splicing operation is performed:

[0068]

[0069] Where a is the total number of pixels of the moving image and the fixed image, is the splicing symbol;

[0070] (2.1.1) Multi-feature input prediction fusion using serial convolution:

[0071] The serial structure can obtain different abstract features between each convolution layer. Affine deformation and variable deformation share the downsampling structure. We introduce the input of the affine layer between the downsampling and upsampling convolution layers, so that we can make full use of the feature parameters of different layers and linearly project them into an affine matrix to predict more accurate affine deformation.

[0072] After the three affine deformations are output, they are combined with the deformable grid and fused with relevant weights. The weights of affine layer C1, affine layer C2, and affine layer C3 are 0.2, 0.3, and 0.5 respectively. In this way, the input affine layer with high feature volume has a larger proportion and the parameter utilization effect is higher. Finally, the DA grid (combination grid of Deformable grid and Affine grid) is obtained for image registration.

[0073] (2.1.2) Strong constraints to solve scale imbalance initialization and adaptive affine deformation:

[0074] Strong constraints are imposed on the training process of the affine matrix to constrain the change matrix and prevent excessive affine deformation during training from causing abnormal loss functions.

[0075] The affine transformation layer is implemented by a fully connected layer to regress the transformation parameters, resulting in a single 12-neuron dense layer representing the three-dimensional affine transformation parameters of translation, rotation, scaling, and shearing. Assume that the input to the affine layer after convolution is set to x, then A m×n =affine(X,W,b)=X m×k W k×n +b 1×n , initialize weight W k×n is a random normal distribution, and X m×k W k×n Strong constraints, Where -r≤a ij≤r,(i=1,2,3,j=1,2,3,4), r is the constraint value, so that each parameter of the matrix can be strongly constrained, and the constraint value of each affine layer is different to increase the variable range of affine prediction. Different registered data can also set their own constraint values ​​to obtain adaptive affine deformation;

[0076] (2.2) Determination of loss function:

[0077] Dice is used as the loss to ensure the registration effect of the carotid artery. The curved image labels obtained after the fixed image labels and the moving image labels pass through the DA grid are used to calculate the arterial deformation:

[0078]

[0079] Where N is the total number of labeled voxels, a i ,b i are the binary labels for the fixed image label and the curved image label, respectively. To avoid anomalies, the artery deformation loss L ad Determined as:

[0080]

[0081] Bending energy is used as a regular penalty term to ensure the smoothness of the mesh:

[0082]

[0083] Where V is the number of voxels in the image domain. For affine transformations, this regularization term is zero, so only variability transformations are penalized;

[0084] According to the characteristics of carotid artery images, the key points at the bifurcation between the two sequences and different modes are marked, and the distance between the two marked points is calculated and introduced into the loss function:

[0085]

[0086] Where S is the total number of key points at the bifurcation, I fi and I mi are the spatial coordinates of the bifurcation key points corresponding to the fixed image and the moving image respectively;

[0087] The final loss is:

[0088] Loss = L ad +αL be +L distance

[0089] α is the regularization hyperparameter;

[0090] (3) Image registration: Based on the model generated by network training, the fixed image and the floating image are input into the trained network to obtain the displacement field between the two images. The distorted image is obtained through the displacement field and the floating image originally input into the network.

[0091] As shown in Figures 3(a), 3(b) and 3(c), these are the registered images obtained in step (3) of the embodiment of the present invention, where Figure 3(a) is a curved image and Figure 3(b) is a mesh output by the network.

[0092] In order to more intuitively display the image registration and the registration effect of the present invention, we provide a label overlay image before and after the registration to display the registration effect. FIG3(c) is a visual effect diagram of the registered image, in which the left side of the diagram is an overlay image of the T1GD label and the TOF label of the carotid artery label before the registration, and the right side of the diagram is an overlay image of the T1GD label and the TOF label of the carotid artery label before the registration.

[0093] The above is an implementation case given by the present invention, which demonstrates a three-dimensional medical image registration method based on deep learning. Obviously, the present invention is not limited to the above embodiments, and various modifications can be made to it without deviating from the basic spirit of the present invention and not exceeding the scope of the essential content of the present invention. The registration scheme proposed by the present invention is effective for multimodal carotid artery medical images, and can well achieve alignment and registration between pixels, thereby facilitating the doctor's subsequent diagnosis and treatment judgment. It is easy for those skilled in the art to understand that the above description is only a preferred embodiment of the present invention, and it is not intended to limit the present invention. Any modifications, replacements and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A carotid artery multimodal image registration method based on strongly constrained affine deformation feature learning, characterized in that: The method comprises the following steps: (1) Data selection: Using multimodal carotid artery data, we selected fixed images and floating images. The fixed images were T1-weighted images that highlighted tissues, and the floating images were MRI images of the TOF sequence that highlighted the carotid arteries using angiography. The carotid artery labels corresponding to the two sequences were marked. (2) Network training: According to the fixed image and the floating image and the segmented label data, the fixed image and the floating image are input into the neural network to generate a displacement field, and the carotid artery label of the floating image is interacted with the displacement field to obtain a distorted label, and the dice loss function loss is obtained through the deformed label and the label of the fixed image, and the loss function of the regularization term is obtained through the displacement field; (3) Image registration: Based on the model generated by network training, the fixed image and the floating image are input into the trained network to obtain the displacement field between the two images. The distorted image is obtained through the displacement field and the floating image originally input into the network. In step (1), data selection includes the following steps: (1.1) Image preprocessing: The images are uniformly corrected for offset fields and then resampled to obtain data with a voxel size of 0.6*0.6*0.

6. Based on the resampling, the images are cropped to obtain images of the same size. The process is as follows: (1.1.1) Crop the two-modal images left and right, front and back, and top and bottom to expand the data. The cropping process should consider the spatial information matching relationship of the corresponding points of different modalities; (1.1.2) Since the spatial coordinate system for image positioning of floating images uses RAS, and that of fixed images uses RAI, the spatial position coordinates of the two sequence images are: fixed(x,y,z)=moving(x,-z,y) Among them, moving(x,-z,y) and fixed(x,y,z) represent the pixel positions of floating images and fixed images in the spatial coordinate system of image positioning, respectively. This requires that the spatial position coordinates fixed(x,y,z) need to correspond to moving(x,y,z) before the data is input into the network. Therefore, it is necessary to perform relevant operations on the moving image to convert moving(x,-z,y) into moving(x,y,z) to meet the basic requirements of pixel matching during network training; (1.2) Labeling of carotid artery labels: Use the labeling software ITK-SNAP to label the vascular locations in the two MRI sequences of the carotid artery; In step (2), network training includes the following steps: (2.1) Multi-feature input restricted affine model structure design: Input moving and fixed images, and perform pixel value matching as described in (1.1.2) before inputting into the network, and then perform value splicing operation: Where a is the total number of pixels of the moving image and the fixed image, is the splicing symbol; (2.1.1) Multi-feature input prediction fusion using serial convolution: The serial structure obtains different abstract features between each convolution layer. Affine deformation and variable deformation share the downsampling structure. The input of the affine layer is introduced between the downsampling and upsampling convolution layers. This makes full use of the feature parameters of different layers and linearly projects them into an affine matrix to predict more accurate affine deformation. After the three affine deformations are output, they are combined with the deformable grids and fused with relevant weights. The weights of affine layer C1, affine layer C2, and affine layer C3 are 0.2, 0.3, and 0.5 respectively. In this way, the input affine layer with high feature volume has a larger proportion and the parameter utilization effect is higher. Finally, the DA grid is obtained for image registration. (2.1.2) Strong constraints to solve the initialization of network training scale imbalance and adaptive affine deformation: Strong constraints are imposed on the training process of the affine matrix to constrain the change matrix, prevent excessive affine deformation during training from causing abnormal loss functions, and solve the initialization problem of unbalanced network training scales; The affine transformation layer is implemented by a fully connected layer to regress the transformation parameters, resulting in a single 12-neuron dense layer, which represents the three-dimensional affine transformation parameters of translation, rotation, scaling, and shearing, respectively. Assuming that the input affine layer after convolution is set to x, then A m×n =affine(X,W,b)=X m×k W k×n +b 1×n , initialize the weight W k×n is a random normal distribution, and X m×k W k×n Strong constraints, Where -r≤a ij ≤r, i=1,2,3,j=1,2,3,4, r is the constraint value, so that each parameter of the matrix is ​​strongly constrained, and the constraint value of each affine layer is different to increase the variable range of affine prediction. Different registered data set their own constraint values ​​to obtain adaptive affine deformation; (2.2) Determination of loss function: Dice is used as loss to ensure the registration effect of the carotid artery. The curved image labels obtained after the fixed image labels and the moving image labels are passed through DAgrid are used to calculate the arterial deformation: Where N is the total number of labeled voxels, a i ,b i are the binary labels of the fixed image label and the curved image label, respectively. To avoid anomalies, the artery deformation loss L ad Determined as: Use bending energy as a regularization penalty term to ensure the smoothness of the mesh: Where V is the number of voxels in the image domain. For affine transformations, this regularization term is zero, so only variability transformations are penalized; According to the characteristics of carotid artery images, the key points at the bifurcation between the two sequences and different modes are marked, and the distance between the two marked points is calculated and introduced into the loss function: Where S is the total number of key points at the bifurcation, I fi and I mi are the spatial coordinates of the bifurcation key points corresponding to the fixed image and the moving image respectively; The final loss is: Loss=L ad +αL be +L distance α is the regularization hyperparameter.

2. The method for carotid artery multimodal image registration based on strongly constrained affine deformation feature learning as claimed in claim 1, characterized in that: In step (3), after steps (1) and (2), the registered model is obtained through training. Although the training requires moving images, fixed images and their corresponding carotid artery labels, the verification process only requires a pair of unlabeled moving and fixed images. First, the trained network needs to be restored to predict the DA grid and the trained weight parameters, and then the DA grid between the two modalities is obtained by calculating the registered images. Finally, the curved image is obtained through the DA grid and the moving image.

Images