Multi-modal medical image registration optimization method and system

Abstract

The invention relates to the technical field of multi-modal medical image registration optimization methods, in particular to a medical image registration optimization method and system, and the method comprises the following steps: inputting a medical image data set comprising an MRI image data set, a CT image data set and a PET image data set into a trained deformable registration network; obtaining a deformation field by using the deformable registration network; obtaining a final registration result of the two groups of medical image data based on the deformation field, obtaining an optimized final registration result through an iterative optimization algorithm, and obtaining a quality evaluation coefficient at the same time; constructing an optimized objective function by using the quality evaluation coefficient, and realizing the adjustment of the hyper-parameter of the deformable registration network through the back propagation of the optimized objective function; prediction registration is carried out by using the deformable registration network after hyper-parameter adjustment, so that an optimized registration result is obtained, the position and the shape of a convolution kernel are adaptively adjusted based on a feature extraction network of deformable convolution, and nonlinear deformation of a local anatomical structure is better captured.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal medical image registration optimization methods, and in particular to a medical image registration optimization method and a system thereof. Background Art

[0002] In recent years, with the rapid development of modern medical imaging technologies, such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), etc., multimodal medical imaging data is growing explosively. How to effectively integrate and utilize these heterogeneous data has become an important topic in precision medicine and smart medicine. Multimodal medical image registration technology, as a key means to achieve spatial alignment and semantic fusion of image information, plays an increasingly important role in medical applications such as disease diagnosis, surgical planning, and efficacy evaluation.

[0003] Traditional multimodal medical image registration methods are mainly based on manually designed image features and optimization criteria, such as mutual information, normalized cross-correlation, Demons algorithm, etc. These methods have achieved automatic registration of multimodal medical images to a certain extent, but there are still some problems that need to be solved. First, traditional methods usually rely on manually selected features, such as gradients and textures, which are difficult to fully explore the intrinsic structural information and semantic information of image data, resulting in insufficient registration accuracy and robustness. Secondly, most traditional methods use iterative optimization solution strategies with high computational complexity and are difficult to meet real-time requirements, limiting their application in clinical practice. Thirdly, the inherent imaging principles and contrast differences between different modalities make it difficult for traditional methods to establish a unified and effective similarity metric, affecting the accuracy and reliability of registration.

[0004] In response to the above problems, some scholars have tried to introduce deep learning technology and use deep neural networks to automatically learn hierarchical and semantic feature representations from large-scale data in order to improve the performance of multimodal medical image registration. However, the existing deep learning registration methods still have some limitations. On the one hand, most methods use rigid or affine transformation models, which are difficult to characterize complex nonlinear deformations, especially in local areas such as organ edges and lesion areas, and the registration accuracy still needs to be improved. On the other hand, the existing methods mainly focus on feature representation learning, and rarely involve transformation optimization and model generalization, resulting in insufficient interpretability and adaptability of the registration process. In addition, the existing deep network structure and loss function design also need to be further optimized to balance registration accuracy, computational efficiency and training stability. Summary of the invention

[0005] The present invention aims to solve the above technical problems and provide an end-to-end multimodal medical image registration optimization method and system. The method combines deep learning and classical optimization theory, starting from feature representation, spatial transformation, and objective function, to achieve comprehensive optimization and adaptive fine-tuning of the registration process.

[0006] The present invention provides a medical image registration optimization method, comprising the following steps:

[0007] Inputting a medical imaging dataset including an MRI image dataset, a CT image dataset, and a PET image dataset into a trained deformable registration network;

[0008] Obtaining a deformation field using the deformable registration network;

[0009] Based on the deformation field, a final registration result of two sets of medical image data is obtained, and an optimized final registration result is obtained through an iterative optimization algorithm, and a quality evaluation coefficient is obtained at the same time;

[0010] Using the quality assessment coefficient to construct an optimization objective function, and adjusting the hyperparameters of the deformable registration network through back propagation of the optimization objective function;

[0011] The deformable registration network with adjusted hyperparameters is used to perform predictive registration, thereby obtaining an optimized registration result.

[0012] Preferably, training the deformable registration network comprises:

[0013] Constructing the deformable registration network including a deformable convolution module, a deformation field network module and an iterative optimization module, wherein the deformable convolution module includes a plurality of convolution layer units and a deformable convolution unit located at the last convolution layer unit, the deformation field network module is used to obtain the deformation field, and the iterative optimization module uses an average error function of the inverse affinity of the deformation field as an objective function;

[0014] The data pair to be registered is input into the deformable convolution module and output to the deformation field network module to obtain the deformation field, and then the final optimized registration result and quality coefficient are obtained through the iterative optimization module. Finally, the optimization convergence of the network weight parameters is achieved through the loss function.

[0015] Preferably, the step of inputting the data pair to be registered into the deformable convolution module and outputting it to the deformation field network module to obtain the deformation field, and then obtaining the final optimized registration result and quality coefficient through the iterative optimization module is specifically as follows:

[0016] Inputting the data pair to be registered into the deformable convolution module, extracting data features using the convolution layer in the deformable convolution module and outputting the data features to the deformable convolution layer in the deformable convolution module, and the deformable convolution layer obtains a deformation field;

[0017] The obtained deformation field is transmitted to the iterative optimization module for iterative optimization, so as to obtain an optimized registration result and a quality coefficient;

[0018] The iterative optimization includes: deforming the floating image through the deformation field, and then obtaining the inverse consistency loss. The iterative optimization module uses the inverse consistency loss to reversely obtain the deformation field and update it, and repeats the above process until the error of the optimization objective function converges or after a set number of iterations, finally obtaining the optimized registration result and the registration quality coefficient.

[0019] Preferably, the deformable convolution module uses a spatial transformation network to construct the relationship between the internal layers and the inter-layers of the network to obtain the residuals of the inputs of different layers, and deformable convolution layers are set both in the internal layers and between the layers of the network.

[0020] Preferably, the deformation field network module obtains a deformation field in any form, and a residual structure is set inside the deformation field network module to stabilize the training process; the deformation field network module includes:

[0021] An image feature extraction part for obtaining high-dimensional image features of input data, wherein the image feature extraction part includes a convolutional layer unit;

[0022] A global deformation field part of the global deformation field is obtained using the extracted features, wherein the global deformation field part includes a residual connection and a deformable convolution unit.

[0023] Preferably, the deformation field network module inputs the fixed image and the floating image into their respective image feature extraction parts, and the image feature extraction part obtains high-dimensional image features of the fixed image and the floating image, which are then spliced ​​and input into the global deformation field part to obtain and output the deformation field after the residual structure, and the deformation field is applied to the floating image to obtain the result image after registration.

[0024] Preferably, the deformation field is obtained by using the deformable convolutional network:

[0025] The fixed image and the floating image of the medical image are used as input data to perform forward propagation on the input network to obtain the predicted deformation field, and the objective function is used for back propagation to optimize the model parameters, and the quality assessment coefficient is also set in the objective function;

[0026] Applying the obtained optimized network parameters to the deformable convolutional network, inputting different medical image data into the deformable registration network, obtaining the deformation field through network prediction, and obtaining the registration result through iterative optimization;

[0027] After registering and aligning the two sets of data from different modalities, the registration quality based on pixel error is obtained, and at the same time, the objective function is obtained, thereby obtaining the quality evaluation coefficient.

[0028] Preferably, the iterative optimization module is used to obtain a consistent transformation of the image space, and the iterative optimization includes a consistency loss and an inverse consistency loss.

[0029] Preferably, the loss function also includes weight parameters of the deformable convolution module.

[0030] Multimodal medical image registration optimization system, including:

[0031] A deformable registration network prediction module is used to obtain the deformation field using the trained deformable registration network;

[0032] An optimization module, used to obtain optimized registration results and quality coefficients through iterative optimization;

[0033] The deformable registration network prediction module is also used to perform predictive registration using the deformable registration network with adjusted hyperparameters to obtain the optimized registration result, which includes the optimized final registration result and the quality assessment coefficient;

[0034] The hyperparameters are obtained by constructing an optimization objective function using quality assessment coefficients, and adjusting the hyperparameters of the deformable registration network through back propagation of the optimization objective function.

[0035] In terms of feature representation, the present invention designs a feature extraction network based on deformable convolution, which can adaptively adjust the position and shape of the convolution kernel to better capture the nonlinear deformation of the local anatomical structure. In terms of spatial transformation, the present invention introduces a deformation field generation module based on a spatial transformation network, which can predict arbitrarily complex nonlinear deformations and improve the degree of freedom and accuracy of registration. In terms of objective function, the present invention cleverly combines the similarity metric based on mutual information, the transformation regularization based on inverse consistency constraints, and the model regularization based on parameter norms to achieve unified optimization of data-driven, physical constraints, and model generalization.

[0036] The beneficial effects of the present invention are:

[0037] 1. The introduction of deformable convolution can adaptively adjust the local receptive field, extract structural features more accurately, reduce the registration error caused by position sensitivity and deformation sensitivity, and significantly improve the registration accuracy of nonlinear areas such as lesions and edges.

[0038] 2. The use of spatial transformation networks can model arbitrarily complex nonlinear deformations, breaking through the limitations of simple models such as rigidity and affine transformation in traditional methods, greatly improving the freedom and precision of registration, and is especially suitable for situations with large deformations and non-rigid organs.

[0039] 3. The mutual information similarity metric can effectively handle the contrast differences and nonlinear relationships between different modalities, providing a general and accurate registration optimization goal; at the same time, the introduction of inverse consistency constraints can ensure the reversibility and symmetry of the registration transformation, avoid local minimum traps, and further improve the registration reliability.

[0040] 4. Combine deep learning methods with classical optimization theory, incorporate prior physical constraints into end-to-end, data-driven model training, balance registration accuracy, overfitting risk and model generalization, significantly reduce dependence on data quality and quantity, shorten model training and inference time, and improve the robustness, adaptability and practicality of the system.

[0041] 5. An efficient and scalable network structure and training strategy, such as encoder-decoder structure, residual connection, multi-scale optimization, etc., are designed to ensure the convergence and stability of model learning, and can flexibly adapt to different organs, diseases and modality combinations, and has good generalization and migration capabilities.

[0042] 6. It provides a unified and standardized medical image preprocessing process, covering operations such as image resampling, interpolation, normalization, and noise removal, to ensure that the scale and quality of the input data are uniform, laying the data foundation for subsequent end-to-end alignment.

[0043] In summary, the present invention has comprehensively upgraded and optimized the existing multimodal medical image registration technology from multiple levels such as methods, models, and systems, and has specifically solved the deficiencies of traditional methods and existing deep learning methods, and has achieved significant breakthroughs in registration accuracy, computational efficiency, clinical practicality, etc. The present invention can not only improve the comprehensive utilization level of multimodal images and assist in the diagnosis and treatment of diseases, but also open up new ideas and approaches for multimodal medical image analysis and intelligent medical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a logic block diagram of the multimodal medical image registration optimization method of the present invention.

[0045] Figure 2 It is a logical block diagram of training the deformable registration network of the present invention.

[0046] Figure 3 It is a logical block diagram of the deformable registration network prediction module of the present invention.

[0047] Figure 4 It is a logic block diagram of the optimization module of the present invention.

[0048] Figure 5 This is a logic block diagram of the multimodal medical image registration optimization system of the present invention. DETAILED DESCRIPTION

[0049] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0050] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0051] The technical solution of the present invention is described in detail below in conjunction with specific embodiments.

[0052] Please refer to Figure 1-5 The present invention discloses a multimodal medical image registration optimization method, comprising the following steps:

[0053] First, medical image datasets including MRI image datasets, CT image datasets, and PET image datasets are input into the trained deformable registration network. The medical image datasets here can be medical images from different patients or different modalities of the same patient, such as MRI, CT, and PET. These datasets are input into the pre-trained deformable registration network to prepare for subsequent registration optimization.

[0054] Then, the deformation field is obtained by using the deformable registration network. The deformable registration network can establish a spatial mapping relationship between images of different modalities by learning a large amount of medical image data, thereby predicting the deformation field required to register the image of one modality to another modality. This deformation field describes the nonlinear transformation of the image space.

[0055] Next, the final registration result of the two sets of medical image data is obtained based on the deformation field, and the optimized final registration result is obtained through an iterative optimization algorithm, and a quality evaluation coefficient is obtained at the same time. Using the predicted deformation field, one set of medical image data can be deformed to the space of another set of medical image data, thereby achieving the registration of the two sets of data. However, due to certain errors in the deformation field prediction, it is necessary to further improve the registration accuracy through an iterative optimization algorithm. Preferably, in one embodiment of the present invention, an optimization algorithm based on gradient descent, such as L-BFGS, etc., can be used. While iteratively optimizing, the quality evaluation coefficient of the registration result can also be calculated to measure the precision and accuracy of the registration.

[0056] Again, the quality assessment coefficient is used to construct an optimization objective function, and the hyperparameters of the deformable registration network are adjusted by back propagation of the optimization objective function. The quality assessment coefficient reflects the quality of the current registration result and can be used to construct optimization objective functions, such as mean square error, mutual information, etc. By minimizing the optimization objective function and using the back propagation algorithm to update the hyperparameters of the deformable registration network, such as the learning rate, regularization coefficient, etc., the network model can be better adapted to the current registration task.

[0057] Finally, the deformable registration network with adjusted hyperparameters is used to perform predictive registration, thereby obtaining an optimized registration result. The deformable registration network after adjustment is used to re-predict the deformation field, and the final optimized registration result is obtained based on the new deformation field.

[0058] For example, suppose we have a set of MRI and CT image data of a patient, and we need to register the MRI image to the space of the CT image. First, we input the MRI and CT datasets into a pre-trained deformable registration network, which outputs a deformation field that deforms the MRI to the CT space. Then, use this deformation field to deform the MRI image to obtain a preliminary registration result. Next, by minimizing the mean square error between the registration result and the CT image, the deformation field is iteratively optimized using the L-BFGS algorithm, and the registration quality assessment coefficient, such as the Dice coefficient, is calculated. The Dice coefficient is then used to construct a loss function, and the network's learning rate and other hyperparameters are adjusted through back propagation. Finally, the tuned network is used to re-predict the deformation field and obtain the final MRI to CT registration result.

[0059] In an embodiment of the present invention, training the deformable registration network comprises:

[0060] First, the deformable registration network including a deformable convolution module, a deformation field network module and an iterative optimization module is constructed. The deformable convolution module includes a plurality of convolution layer units and a deformable convolution unit located at the last convolution layer unit, which is used to extract image features. The deformation field network module is used to obtain the deformation field to realize nonlinear mapping of the image space. The iterative optimization module uses the average error function of the inverse consistency of the deformation field as the objective function to improve the accuracy of the deformation field prediction.

[0061] For example, the deformable convolution module can use a classic CNN network structure such as VGG16, and add a deformable convolution unit after the last convolution layer. The deformation field network module can use an encoding and decoding structure such as U-Net to decode the convolution feature map into a deformation field. The iterative optimization module can use the inverse consistency loss function of the deformable registration, that is, calculate the inverse consistency error of the forward and reverse deformation fields respectively.

[0062] Then, the data pair to be registered is input into the deformable convolution module and output to the deformation field network module to obtain the deformation field, and then the final optimized registration result and quality coefficient are obtained through the iterative optimization module, and finally the optimization convergence of the network weight parameters is achieved through the loss function.

[0063] Specifically, the medical image data pairs of the source domain and the target domain are input into the deformable registration network, and after the steps of feature extraction, deformation field prediction and iterative optimization, the registered medical images and the corresponding quality assessment coefficients are obtained. Then, the total loss function is constructed with this quality coefficient and the inverse consistency loss, and the weight parameters of the network are optimized using the back propagation algorithm until the loss function converges.

[0064] Preferably, in one embodiment of the present invention, an Adam optimizer may be used, and the initial learning rate may be set to 1e -4 , it decays to 0.5 times of the original value every 10 epochs, and the maximum training time is 200 epochs.

[0065] In an embodiment of the present invention, the steps of inputting the data pair to be registered into the deformable convolution module and outputting it to the deformation field network module to obtain the deformation field, and then obtaining the final optimized registration result and quality coefficient through the iterative optimization module are specifically as follows:

[0066] First, the data pair to be registered is input into the deformable convolution module, and the data features are extracted by the convolution layer in the deformable convolution module and output to the deformable convolution layer in the deformable convolution module, and the deformable convolution layer obtains the deformation field. The deformable convolution layer here can adaptively adjust the position of the convolution kernel according to the local deformation of the feature map to extract more accurate features.

[0067] For example, for a pair of 256x256 MRI and CT slices, three 3x3 convolutional layers can be used to extract features, each followed by a 2x2 maximum pooling layer to downsample the feature map to 64x64. After a 3x3 deformable convolutional layer, two 64x64 offset maps are generated to represent the displacement in the horizontal and vertical directions.

[0068] Then, the obtained deformation field is transmitted to the iterative optimization module for iterative optimization, so as to obtain the optimized registration result and quality coefficient. The iterative optimization process here can be divided into the following steps: the floating image is deformed by the deformation field, and then the inverse consistency loss is obtained. The iterative optimization module uses the inverse consistency loss to reversely obtain the deformation field and update it, and repeats the above process until the error of the optimization objective function converges or after a set number of iterations, finally obtaining the optimized registration result and registration quality coefficient.

[0069] Specifically, the predicted deformation field is used to perform spatial transformation on the floating image to obtain the registered floating image. Then, the deformation fields of the forward transformation (from the floating image to the fixed image) and the reverse transformation (from the fixed image to the floating image) are calculated respectively, and the inverse consistency error between them is calculated, that is, the composite transformation of the forward transformation and the reverse transformation should be equal to the identity transformation. The error term is used to construct the optimization objective function, and then the deformation field is iteratively updated by optimization algorithms such as the gradient descent method. At the same time, the quality evaluation coefficients of the registration results, such as mutual information and Dice coefficient, are calculated after each iteration. When the error of the objective function is less than the set threshold or the maximum number of iterations is reached, the iteration is stopped and the final registration result and quality coefficient are output.

[0070] Preferably, in one embodiment of the present invention, the inverse consistency loss can be combined with the mutual information to construct a joint optimization objective function. The L-BFGS optimization algorithm is used, the maximum number of iterations is set to 100, and the error threshold is 1e -6 The Dice coefficient is calculated every 10 iterations.

[0071] In an embodiment of the present invention, the deformable convolution module uses a spatial transformation network to construct the relationship between the internal layers and the inter-layers of the network to obtain the residuals of the inputs of different layers, and deformable convolution layers are set both in the internal layers and between the layers of the network.

[0072] The spatial transformer network here can learn the spatial relationship between feature maps and establish connections between features of different scales and positions. Using this relationship, the deformation variable of the feature map of the current layer relative to the feature map of the previous layer, that is, the residual, can be predicted. At the same time, deformable convolution is introduced in each convolution layer of the network, so that each layer can autonomously adjust the convolution kernel according to the local deformation and extract more robust features.

[0073] For example, for a 64x64 input feature map, you can first use a 3x3 normal convolution layer to extract features, then connect a 3x3 deformable convolution layer, use the features of the previous layer to predict the deformation of the current layer, and use this deformation as the offset of the current layer to deform the convolution kernel. Repeat this process three times to get three 64x64 deformed feature maps. Then fuse these feature maps through a 1x1 convolution layer and add them to the original feature map to get the final residual feature map.

[0074] In an embodiment of the present invention, the deformation field network module obtains a deformation field in any form, and a residual structure is set inside the deformation field network module to stabilize the training process; the deformation field network module includes: an image feature extraction part for obtaining high-dimensional image features of input data, and a global deformation field part for obtaining a global deformation field using the extracted features. The image feature extraction part includes a convolutional layer unit; the global deformation field part includes a residual connection and a deformable convolution unit.

[0075] The deformation field network module here adopts an encoder-decoder structure, where the encoder part is used to extract image features and the decoder part is used to generate deformation fields. In the encoder part, multiple convolutional layers are used to gradually reduce the feature map size and increase the number of channels to extract high-level semantic features. In the decoder part, multiple transposed convolutional layers are used to gradually enlarge the feature map size and reduce the number of channels to restore spatial resolution. At the same time, a deformable convolutional layer is introduced after each transposed convolutional layer to capture local deformation. In addition, a residual connection is added between the encoder and decoder to directly pass the encoder features to the decoder layer of the corresponding scale, alleviating the gradient vanishing problem during training and improving the convergence speed and stability of the network.

[0076] For example, for a 256x256 input image pair, the encoder part may include 4 convolutional layers, each using a 3x3 convolution kernel with a stride of 2, reducing the feature map size to 64x64, 32x32, 16x16, and 8x8. The decoder part includes 4 transposed convolutional layers, each using a 4x4 convolution kernel with a stride of 2, enlarging the feature map size to 16x16, 32x32, 64x64, and 128x128. A 3x3 deformable convolutional layer is added after each transposed convolutional layer. Finally, a 1x1 convolutional layer is used to generate a 2-channel deformation field, representing the horizontal and vertical displacements, respectively. After the second and third transposed convolutional layers, residual connections are established with the third and second convolutional layer outputs of the encoder, respectively.

[0077] In an embodiment of the present invention, the deformation field network module inputs the fixed image and the floating image into their respective image feature extraction parts, and the image feature extraction part obtains high-dimensional image features of the fixed image and the floating image, which are then spliced ​​and input into the global deformation field part to obtain and output the deformation field after the residual structure, and the deformation field is applied to the floating image to obtain the result image after registration.

[0078] Specifically, the fixed image and the floating image are sent to two independent but identical encoders to extract their respective feature maps. The two sets of feature maps are then concatenated in the channel dimension to form a joint feature map, which is sent to the decoder to generate a deformation field. Each layer of the decoder establishes a residual connection with the corresponding layer of the encoder, and the features of the encoder are directly passed to the decoder to enhance the expressiveness of the features. Finally, the obtained deformation field is used to spatially transform the floating image to obtain the result image registered to the fixed image space.

[0079] Preferably, in one embodiment of the present invention, the encoder may adopt a PreAct-ResNet-18 structure, including 4 residual blocks, each residual block including 2 3x3 convolutional layers and 1 downsampling layer. The decoder adopts a similar structure, but replaces the downsampling layer with a transposed convolution, and adds a 1x1 convolutional layer after the last residual block to generate a deformation field.

[0080] In an embodiment of the present invention, the deformation field is obtained by using the deformable convolutional network as follows:

[0081] Firstly, the fixed image and floating image of the medical image are used as input data to perform forward propagation on the input network to obtain the predicted deformation field, and the objective function is used for back propagation to optimize the model parameters, and the quality assessment coefficient is also set in the objective function.

[0082] Specifically, the fixed image and the floating image are fed into the deformable registration network. After a series of operations such as feature extraction, deformation field generation and spatial transformation, the registered floating image and the predicted deformation field are obtained. Then, the similarity metrics between the registered floating image and the fixed image, such as mutual information and normalized cross-correlation, are calculated as the evaluation of the registration quality. The quality evaluation coefficient is combined with the inverse consistency error, smoothness regularization term, etc. to form the overall objective function. The back-propagation algorithm is used to calculate the gradient of the objective function to the network parameters, and the network parameters are updated using optimization algorithms such as the gradient descent method to minimize the objective function.

[0083] Then, the obtained optimized network parameters are applied to the deformable convolutional network, different medical image data are input into the deformable registration network, the deformation field is obtained by network prediction, and the registration result is obtained through iterative optimization.

[0084] For example, in the training phase, a large amount of paired medical imaging data (such as MRI-CT image pairs) can be used as training sets and fed into the deformable registration network for end-to-end training. The objective function can be a weighted sum of negative normalized cross-correlation and inverse consistency loss, and a second-order derivative smoothing regularization term of the deformation field is added. The Adam optimizer is used, and the initial learning rate is set to 1e -4 , decaying by 50% every 10 epochs, and training for a maximum of 200 epochs. In the test phase, for any cross-modal medical image data pair to be registered, it is input into the trained deformable registration network, and the predicted deformation field can be obtained by forward propagation, which is used to deform the floating image to obtain the registration result.

[0085] Finally, for the new paired medical image data to be registered, it is input into the optimized deformable registration network, and the predicted deformation field is obtained by forward propagation. The floating image is deformed with the deformation field to obtain the registered image. At the same time, the similarity measure such as the mutual information between the registered image and the fixed image is calculated as the evaluation coefficient of the registration quality.

[0086] Preferably, in one embodiment of the present invention, a multi-scale strategy can be adopted when calculating the mutual information. The mutual information at the original resolution is first calculated, and then the image is downsampled by 2 times and 4 times in turn and the mutual information is calculated. Finally, the weighted average of the mutual information at different scales is used as the final evaluation coefficient to more comprehensively measure the registration quality.

[0087] In an embodiment of the present invention, the iterative optimization module is used to obtain a consistent transformation of the image space, and the iterative optimization includes a consistency loss and an inverse consistency loss.

[0088] Among them, consistency loss refers to the difference measure between the registered floating image and the fixed image, which can be measured by mean square error, cross entropy, mutual information and other indicators. Inverse consistency loss refers to the difference measure between the composite transformation of the forward transformation (from floating image to fixed image) and the reverse transformation (from fixed image to floating image) and the identity transformation, reflecting the reversibility and continuity of the registration process. Combining the two losses as the objective function of iterative optimization can make the registration result optimal in terms of image similarity and transformation self-consistency.

[0089] For example, for a 256x256 MRI-CT image pair, the mean square error can be used as the consistency loss, that is:

[0090]

[0091] in and represent the registered floating image and fixed image respectively, Indicates The coordinates of the pixels, is the total number of pixels.

[0092] The inverse consistency loss can be expressed as:

[0093]

[0094] in and denote the deformation fields from floating image to fixed image and from fixed image to floating image respectively, represents the two-norm.

[0095] The final optimization objective function can be expressed as:

[0096]

[0097] in To balance the weight coefficients of the two losses, they can be adjusted according to specific tasks and data.

[0098] Preferably, in one embodiment of the present invention, Gaussian smoothing can be performed on the deformation field when calculating the inverse consistency loss to improve the smoothness and continuity of the transformation. At the same time, a pyramid-like multi-scale optimization strategy can be adopted to first adjust the deformation field in a large range at a low resolution, and then gradually increase the resolution for local fine-tuning to improve the optimization efficiency and accuracy.

[0099] In an embodiment of the present invention, the loss function also includes weight parameters of the deformable convolution module.

[0100] The loss function here can be expressed as:

[0101]

[0102] in represents the registration similarity loss, represents the inverse consistency loss, represents the weight regularization term of the deformable convolution module, Represents the weight parameters of the deformable convolution module

[0103] and is the weight coefficient for balancing different loss terms.

[0104] The weight regularization term can use L1 or L2 norm, that is:

[0105]

[0106] or

[0107]

[0108] in Indicates weight parameters, is the total number of weight parameters.

[0109] The purpose of introducing weight regularization is to control model complexity and reduce the risk of overfitting. By penalizing larger weight parameters, the model can be made more concise and robust, and the generalization ability can be improved. At the same time, weight regularization can also play a role in feature selection, automatically identifying feature channels and spatial positions that are more important for the registration task, and suppressing redundant and irrelevant features.

[0110] Preferably, in one embodiment of the present invention, the weight coefficients of each item in the loss function can be dynamically adjusted according to the difficulty of the registration task and the quality of the data. For example, in the early stage of registration, the weight of the similarity loss can be appropriately increased to accelerate the convergence of the registration; in the later stage of registration, the weight of the inverse consistency loss and the regularization term can be appropriately increased to improve the registration accuracy and robustness. In addition, the weights of each item can be adaptively adjusted according to the quality evaluation feedback in the registration process to achieve closed-loop optimization.

[0111] In an embodiment of the present invention, the medical images under different scanning modalities in the medical image dataset are preprocessed separately, including slice edge alignment, removal of slices with incomplete edges and resampling, so as to obtain a registered and aligned dataset.

[0112] Before performing multimodal medical image registration, a series of preprocessing is required on the original medical image data under different scanning modalities to eliminate the deviation and noise in the data acquisition process and improve the registration accuracy and stability.

[0113] First, for each modality of medical imaging data, slice edge alignment is usually required. Since the scanning equipment and parameters of different modalities may be different, the acquired slices may have certain deviations in spatial position and direction. In order to facilitate subsequent alignment, the edges of all slices need to be aligned to a unified coordinate system. The commonly used method is to select a reference slice, calculate the spatial transformation matrix of other slices and the reference slice, and resample the slice image to align it with the reference slice.

[0114] Secondly, slices with incomplete edges need to be removed. During the scanning process, due to factors such as patient movement and equipment vibration, the edge information of some slices may be incomplete or missing. These slices have a significant negative impact on the registration accuracy and need to be removed in the preprocessing stage. The common method is to set a threshold and calculate the difference between the pixel value and the background value of the edge area of ​​each slice. If the difference is less than the threshold, the edge of the slice is considered incomplete and needs to be removed.

[0115] Finally, all slices need to be resampled to have the same resolution and size. Medical images of different modalities often have different original resolutions. For example, the resolution of CT is usually higher than that of PET. In order to facilitate registration, images of all modalities need to be resampled to a uniform resolution and size. Common methods are based on interpolation algorithms, such as linear interpolation, spline interpolation, etc., to upsample or downsample the original image to obtain image data with uniform resolution and size.

[0116] After the above preprocessing steps, a multimodal medical imaging dataset with spatial alignment, complete edges and uniform resolution can be obtained, which provides high-quality input data for subsequent registration optimization and helps to improve the registration accuracy and efficiency.

[0117] For example, for a brain tumor image dataset containing three modalities of MRI, CT, and PET, each modality contains 240, 180, and 120 slices, respectively, with original resolutions of 1.0mm, 1.5mm, and 3.0mm, respectively. In the preprocessing stage, the middle slice of MRI can be selected as the reference slice, and the spatial transformation matrix of other slices and the reference slice can be calculated using the mutual information registration algorithm, and the slice edges are aligned. Then, the average grayscale value of the edge area of ​​each slice is calculated and compared with the average grayscale value of the entire slice. If the difference is less than 20%, the edge of the slice is considered incomplete and needs to be removed. Finally, the trilinear interpolation algorithm is used to resample all slices to an isotropic resolution of 1.0mm and crop them to a uniform size (such as 256x256x240). After preprocessing, an MRI-CT-PET multimodal brain tumor image dataset with alignment after registration, complete edges, and uniform resolution is obtained.

[0118] In summary, the multimodal medical image registration optimization method and system thereof, from an end-to-end perspective, unifies traditional feature engineering, similarity measurement, transformation optimization and other steps into a deformable convolutional network framework, and realizes fully automatic, high-precision and high-efficiency multimodal medical image registration.

[0119] In terms of network structure design, the present invention innovatively introduces components such as deformable convolution modules, spatial transformation networks, and residual connections to model image features and spatial transformations from local to global multi-scale, enhancing the network's feature extraction and transformation capabilities. In terms of loss function design, the present invention cleverly combines multiple optimization objectives such as image similarity, transformation consistency, and parameter regularization to guide the network to learn the optimal registration strategy from both data-driven and physical constraint perspectives. At the same time, the present invention also provides a complete set of preprocessing processes to ensure the quality and consistency of input data from aspects such as data collection, screening, alignment, and resampling, thereby improving the accuracy and reliability of registration.

[0120] The present invention has broad application prospects in the field of multimodal medical image analysis. By accurately registering medical images of different modalities to the same coordinate space, the fusion and complementarity of image information can be achieved, thereby providing a more comprehensive and reliable reference for tasks such as disease diagnosis, surgical planning, and efficacy evaluation. For example, in tumor radiotherapy, combining the soft tissue contrast of MRI with the electron density information of CT can accurately outline the tumor boundary and formulate a personalized irradiation plan. In the study of neurodegenerative diseases, registering structural MRI with functional PET can reveal the relationship between brain morphology and metabolism, and help understand the mechanism of disease occurrence and development.

[0121] In addition, the present invention also provides new ideas and directions for the development of medical image registration technology. Traditional registration methods based on feature engineering and iterative optimization often rely on prior knowledge and manual design, and are difficult to cope with complex registration tasks of different modalities, different organs, and different diseases. The present invention adopts the end-to-end paradigm of deep learning, and uses massive data to automatically learn registration rules, which has stronger adaptability and generalization capabilities. At the same time, the concepts advocated by the present invention, such as deformable modeling, multi-scale analysis, and joint optimization, also provide useful references for other medical image processing and analysis tasks (such as segmentation, detection, synthesis, etc.).

[0122] The present invention also provides a multimodal medical image registration optimization system, which includes a deformable registration network prediction module 1 and an optimization module 2, and is used to realize the automation and systematization of the above-mentioned multimodal medical image registration optimization method.

[0123] Among them, the function of the deformable registration network prediction module 1 is to use the pre-trained deformable registration network to forward propagate the input source domain and target domain medical image data to obtain the deformation field. The deformation field describes the spatial mapping relationship of how to deform the source domain image to the target domain image, and is the key intermediate result of the registration process.

[0124] In specific implementation, the deformable registration network prediction module 1 can adopt the network structure described in the above embodiment, including a feature extraction part, a deformation field generation part and a spatial transformation part. The feature extraction part is responsible for extracting multi-scale and multi-level feature maps from the input source domain and target domain images, the deformation field generation part is responsible for predicting the displacement vector field required for spatial transformation based on the extracted feature map, and the spatial transformation part is responsible for using the predicted displacement vector field to perform spatial transformation on the source domain image to obtain the registered image. These three parts can be jointly trained in an end-to-end manner to automatically learn the registration mapping from the source domain to the target domain.

[0125] It should be noted that the network parameters in the deformable registration network prediction module 1 are obtained through pre-training, that is, before the model is deployed, it is necessary to first train on a large-scale multimodal medical imaging dataset to learn the optimal feature representation and transformation strategy. The pre-training process can be performed offline, by minimizing the objective functions such as registration similarity loss, transformation consistency loss, and parameter regularization terms, and using optimization algorithms such as gradient descent to iteratively update the network parameters until convergence. The model parameters obtained by pre-training can be used as the initial parameters of the deformable registration network prediction module 1 to improve the registration efficiency and accuracy.

[0126] The optimization module 2 works in conjunction with the deformable registration network prediction module 1. Its function is to further improve the registration accuracy through iterative optimization based on the predicted deformation field and to quantitatively evaluate the registration quality.

[0127] Specifically, the optimization module 2 first uses the deformation field obtained by the deformable registration network prediction module 1 to perform preliminary registration on the source domain image to obtain the registered image. Then, the optimization module 2 iteratively optimizes the deformation field by minimizing the registration similarity loss function and using optimization algorithms such as gradient descent to obtain the optimized deformation field and registration result. During the optimization process, the optimization module 2 also calculates the registration quality evaluation coefficients, such as the Dice coefficient and mean square error, to quantitatively evaluate the accuracy and reliability of the current registration result. The registration quality evaluation coefficient can be used as a basis for judging the convergence of the optimization, and can also be used as a feedback basis for adjusting the network hyperparameters.

[0128] It is worth mentioning that the registration quality evaluation coefficient obtained by the optimization module 2 can also be fed back to the deformable registration network prediction module 1 to improve and fine-tune the pre-trained model. Specifically, the registration quality evaluation coefficient can be used to construct a new optimization objective function, and the hyperparameters of the deformable registration network, such as the learning rate and regularization coefficient, can be updated through the back-propagation algorithm. This closed-loop feedback mechanism can adapt the pre-trained model to the characteristics of the current registration task and data, and improve the adaptability and robustness of the entire system.

[0129] The collaborative work of the optimization module 2 and the deformable registration network prediction module 1 can be regarded as a systematic implementation of the "coarse registration + fine registration" strategy. The coarse registration stage uses the pre-trained model to quickly predict the deformation field and capture the global geometric transformation between the source domain and the target domain images; the fine registration stage uses the iterative optimization algorithm to fine-tune the deformation field and capture the local nonlinear deformation. The organic combination of the two stages can take into account the efficiency and accuracy of the registration and realize fully automatic, high-performance multimodal medical image registration.

[0130] In summary, the multimodal medical image registration optimization system provided by the present invention gives full play to the complementary advantages of deep learning technology and traditional optimization theory, and realizes end-to-end registration optimization from data-driven to physical constraints. The system has a clear structure, clear module functions, standardized interface definitions, and is easy to implement and integrate. At the same time, the system adopts a hierarchical optimization strategy of pre-training + fine-tuning, coarse registration + fine registration, taking into account the generalization of the model and the adaptability of the task, ensuring the stability and reliability of the registration performance.

[0131] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal medical image registration optimization method, characterized in that: The following steps are involved: Inputting a medical imaging dataset including an MRI image dataset, a CT image dataset, and a PET image dataset into a trained deformable registration network; Obtaining a deformation field using the deformable registration network; Based on the deformation field, a final registration result of two sets of medical image data is obtained, and an optimized final registration result is obtained through an iterative optimization algorithm, and a quality evaluation coefficient is obtained at the same time; Using the quality assessment coefficient to construct an optimization objective function, and adjusting the hyperparameters of the deformable registration network through back propagation of the optimization objective function; The deformable registration network with adjusted hyperparameters is used to perform predictive registration, thereby obtaining an optimized registration result.

2. The multimodal medical image registration optimization method according to claim 1, characterized in that: Training the deformable registration network includes: Constructing the deformable registration network including a deformable convolution module, a deformation field network module and an iterative optimization module, wherein the deformable convolution module includes a plurality of convolution layer units and a deformable convolution unit located at the last convolution layer unit, the deformation field network module is used to obtain the deformation field, and the iterative optimization module uses an average error function of the inverse affinity of the deformation field as an objective function; The data pair to be registered is input into the deformable convolution module and output to the deformation field network module to obtain the deformation field, and then the final optimized registration result and quality coefficient are obtained through the iterative optimization module. Finally, the optimization convergence of the network weight parameters is achieved through the loss function.

3. The multimodal medical image registration optimization method according to claim 2, characterized in that: The steps of inputting the data pair to be registered into the deformable convolution module and outputting it to the deformation field network module to obtain the deformation field, and then obtaining the final optimized registration result and quality coefficient through the iterative optimization module are specifically as follows: Inputting the data pair to be registered into the deformable convolution module, extracting data features using the convolution layer in the deformable convolution module and outputting the data features to the deformable convolution layer in the deformable convolution module, and the deformable convolution layer obtains a deformation field; The obtained deformation field is transmitted to the iterative optimization module for iterative optimization, so as to obtain an optimized registration result and a quality coefficient; The iterative optimization includes: deforming the floating image through the deformation field, and then obtaining the inverse consistency loss. The iterative optimization module uses the inverse consistency loss to reversely obtain the deformation field and update it, and repeats the above process until the error of the optimization objective function converges or after a set number of iterations, finally obtaining the optimized registration result and the registration quality coefficient.

4. The multimodal medical image registration optimization method according to claim 1, characterized in that: The deformable convolution module uses a spatial transformation network to construct the relationship between the internal layers and the inter-layers of the network to obtain the residuals of the inputs of different layers, and at the same time, deformable convolution layers are set both in the internal layers and between the layers of the network.

5. The multimodal medical image registration optimization method according to claim 2, characterized in that: The deformation field network module obtains a deformation field in any form, and a residual structure is set inside the deformation field network module to stabilize the training process; The deformation field network module includes: An image feature extraction part for obtaining high-dimensional image features of input data, wherein the image feature extraction part includes a convolutional layer unit; A global deformation field part of the global deformation field is obtained using the extracted features, wherein the global deformation field part includes a residual connection and a deformable convolution unit.

6. The multimodal medical image registration optimization method according to claim 5, characterized in that: The deformation field network module inputs the fixed image and the floating image into their respective image feature extraction parts respectively, and the image feature extraction part obtains high-dimensional image features of the fixed image and the floating image, and then splices them and inputs them into the global deformation field part, obtains and outputs the deformation field after the residual structure, and acts on the floating image with the deformation field to obtain the result image after registration.

7. The multimodal medical image registration optimization method according to claim 2, characterized in that: The method of obtaining the deformation field using the deformable convolutional network is as follows: The fixed image and the floating image of the medical image are used as input data to perform forward propagation on the input network to obtain the predicted deformation field, and the objective function is used for back propagation to optimize the model parameters, and the quality assessment coefficient is also set in the objective function; Applying the obtained optimized network parameters to the deformable convolutional network, inputting different medical image data into the deformable registration network, obtaining the deformation field through network prediction, and obtaining the registration result through iterative optimization; After registering and aligning the two sets of data from different modalities, the registration quality based on pixel error is obtained, and at the same time, the objective function is obtained, thereby obtaining the quality evaluation coefficient.

8. The multimodal medical image registration optimization method according to claim 2, characterized in that: The iterative optimization module is used to obtain a consistent transformation of the image space, and the iterative optimization includes a consistency loss and an inverse consistency loss.

9. The multimodal medical image registration optimization method according to claim 3, characterized in that: The loss function also includes weight parameters of the deformable convolution module.

10. A multimodal medical image registration optimization system for executing the method according to any one of claims 1 to 9, characterized in that: include: A deformable registration network prediction module is used to obtain the deformation field using the trained deformable registration network; An optimization module, used to obtain optimized registration results and quality coefficients through iterative optimization; The deformable registration network prediction module is also used to perform predictive registration using the deformable registration network with adjusted hyperparameters to obtain the optimized registration result, which includes the optimized final registration result and the quality assessment coefficient; The hyperparameters are obtained by constructing an optimization objective function using quality assessment coefficients, and adjusting the hyperparameters of the deformable registration network through back propagation of the optimization objective function.

Images