A Progressive 3D Biomedical Image Registration Method Based on Deep Self-Calibration

Abstract

The present invention relates to a progressive three-dimensional biomedical image registration method based on deep self-calibration, which solves the problem of information leakage in the recursive cascade registration strategy and the defect that the registration accuracy is limited due to the smoothing constraint on the displacement field in the one-shot registration strategy compared with the prior art. The present invention includes the following steps: acquisition and preprocessing of a three-dimensional biomedical image data set; construction of a registration model; training of the registration model; acquisition and preprocessing of an image to be registered and a template image; and obtaining of a three-dimensional biomedical image registration result. By recursively inputting the previous registration result into the network for self-calibration optimization and recording and integrating the displacement field generated by each recursion, the present invention completes the large displacement in the space in segments through the recursive network, thereby improving the registration accuracy of the three-dimensional biomedical image without information leakage and effectively saving GPU resources.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional biomedical images, and in particular to a progressive three-dimensional biomedical image registration method based on depth self-calibration. Background Art

[0002] With the development of medical imaging technology, modern medical diagnosis is inseparable from medical image analysis. In order to analyze the condition of the disease or the changes in tissue structure, medical images captured from different positions and periods should be aligned into a common space, thus giving rise to medical image registration technology, which aims to establish accurate nonlinear anatomical correspondences between two or more image voxels. Therefore, it has been widely used in various clinical tasks such as medical image fusion, treatment planning, and surgical navigation.

[0003] In the past few decades, researchers have proposed many image registration methods, which can be roughly divided into traditional methods and deep learning-based methods. Traditional methods, such as NiftyReg, ANTs, etc., iteratively optimize the predefined objective function for each pair of images while promoting the smoothness of the registration mapping. However, traditional registration methods are always computationally intensive and time-consuming in practical applications, and these characteristics also make them not suitable for clinical real-time applications.

[0004] As deep learning has been widely used in the field of computer vision, research on biomedical image registration algorithms based on deep learning has become a hot topic with important practical value. Nowadays, biomedical image registration methods based on deep learning can be divided into supervised learning and unsupervised learning based on whether labels are required.

[0005] However, the annotation of medical images usually requires the work of medical image analysis experts, so the annotation cost is too high, which limits the training of supervised learning registration methods. Therefore, the current mainstream research focuses on unsupervised methods. However, mainstream unsupervised methods also have defects. First, although the biomedical image registration based on deep learning has a fast processing speed, it is difficult to achieve the level of traditional algorithms in terms of the accuracy of image pairs with large registration differences. Although the existing multi-scale image registration methods can achieve a coarse-to-fine refinement of the deformation field to handle significant differences between image pairs and large displacements in space, if these methods produce errors at the coarse level, they are likely to be unable to recover at the fine level. This defect will become more serious as the scale increases. Second, most of the Spatial Transformer Networks used in deep learning-based one-shot image registration methods use regular grid points to define and represent the deformation field. However, using regular grid points to represent the deformation field cannot reflect the anatomical information of the brain. In the loss function, only the overall smoothness constraint can be imposed on the displacement field. If the overall smoothness constraint is too large, some brain regions cannot be registered. If the overall smoothness constraint is too small, some brain regions will be over-distorted. Therefore, the smoothness constraint on the displacement field is inconsistent with the completion of large deformation. Third: When the existing recursive cascade network is training the network, the number of deformation interpolation images to be registered is the same as the number of recursive cascades, which gradually causes the leakage of information of the images to be registered. Fourth: Since the amount of data contained in the biomedical image dataset is small, it is easy to cause the trained network to overfit and the generalization ability is not strong. At this time, a data augmentation method is needed to augment the dataset to better train the network. Therefore, achieving better three-dimensional biomedical image registration effect has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to solve the information leakage problem existing in the recursive cascade registration strategy and the defect of limited registration accuracy caused by the smoothness constraint of the displacement field in the one-shot registration strategy, and to provide a progressive three-dimensional biomedical image registration method based on deep self-calibration to solve the above problems.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A progressive three-dimensional biomedical image registration method based on deep self-calibration comprises the following steps:

[0009] 11) Acquisition and preprocessing of 3D biomedical image datasets: Obtain a single-modality 3D biomedical image dataset, sample all images to the same size, normalize pixel values ​​to 0-255, and affine pre-register all images to the same brain template as a 3D biomedical image training set;

[0010] 12) Construction of the registration model: Use Python language to build a low-scale convolutional neural network and an original-scale convolutional neural network respectively and establish a cascade of the two as a registration model, and set the loss function of the low-scale convolutional neural network and the original-scale convolutional neural network;

[0011] 13) Training of the registration model: setting the number of iterations of the registration model training and the number of recursions of the low-scale convolutional neural network and the original-scale convolutional neural network respectively, and training the registration model using a three-dimensional biomedical image training set;

[0012] 14) Acquisition and preprocessing of the image to be registered and the template image: Acquisition of single-modality three-dimensional biomedical image to be registered and the template image, and preprocessing;

[0013] 15) Obtaining the three-dimensional biomedical image registration result: input the preprocessed image pair to be registered into the trained registration model to obtain the registered result.

[0014] The construction of the registration model comprises the following steps:

[0015] 21) The constructed registration model is set to be a cascade of a low-scale convolutional neural network and an original-scale convolutional neural network;

[0016] 22) The low-scale convolutional neural network of the registration model is set to a U-Net-like architecture, which consists of an encoder and a decoder. The input of the low-scale convolutional neural network is the target image and the image to be registered of size (d / 2, w / 2, h / 2);

[0017] 23) The original scale convolutional neural network of the registration model is set to a U-Net-like architecture, which consists of an encoder and a decoder. The input of the original scale convolutional neural network is the target image and the image to be registered of size (d, w, h);

[0018] 24) The outputs of the low-scale registration network model and the original-scale convolutional neural network are set to be three displacement fields of the same size and dimension as the corresponding input images. The displacement field represents the transformation relationship from the image to be registered to the target image, specifically, the displacement of the corresponding coordinate point on the registration result image in the x-direction, y-direction, and z-direction corresponding to a point on the moving image;

[0019] 25) Set the loss function of the low-scale registration network model and the original-scale convolutional neural network to be the similarity loss L between the registration result and the target image. sim and the smoothness constraint loss L of the convolutional neural network output displacement field smooth , the total loss function of a single recursion is L = L sim +λ*L smooth, where λ is the weight coefficient that balances the proportion of the two sub-loss functions in the total loss function.

[0020] The training of the registration model comprises the following steps:

[0021] 31) Set the number of iterations of model training to N_iter, that is, in each iteration, two images are randomly selected from the 3D biomedical image training set as the image to be registered and the target image and input into the registration model for training;

[0022] 32) Set the number of low-scale training recursions to N_low, N_low>=0, and the number of original-scale training recursions to N_original, N_original>=0;

[0023] 33) The first recursion of the low-scale network: the image to be registered and the target image are downsampled by two times and then cascaded into the low-scale convolutional neural network to obtain the first output displacement field, and save the first output displacement field;

[0024] 34) The second recursion of the low-scale network, assuming that the set N_low>1: the first registration result is used as a new image to be registered and cascaded with the original target image to be input into the low-scale convolutional neural network again to obtain the second output displacement field, and save the second output displacement field;

[0025] 35) Using the displacement field integration method, the displacement field output by the convolutional neural network model for the second time is integrated with the displacement field output for the first time into a total displacement field. The original image to be registered is subjected to a deformation interpolation operation using the total displacement field to obtain the second recursive registration result. This registration result is used as a new image to be registered and is cascaded with the target image and input into the convolutional neural network for optimization. If the set N_low>2, then steps 34)-35) are repeated on the basis of recursion twice, that is, the displacement field output by each recursion is integrated with the displacement field output before, and the registration result of this time is obtained through a deformation interpolation operation, until the recursion is N_low times;

[0026] 36) Save the total displacement field obtained by low-scale recursion N_low times as prior information and input it into the original scale convolutional neural network together with the image to be registered and the target image that have not been downsampled in this iterative training;

[0027] 37) Using the total displacement field obtained by low-scale recursion N_low times to upsample twice and numerically expand twice to obtain a new displacement field, the original image to be registered of the original size is pre-deformed by deformation interpolation, and the pre-deformed image is used as the original size of the initial image to be registered;

[0028] 38) The first recursion of the original scale network: the pre-deformed original scale image to be registered and the original target image of the original size are cascaded and input into the original scale convolutional neural network model constructed by this method, the first displacement field at the original scale is output, and the displacement field outputted for the first time is saved;

[0029] 39) The original scale network is recursed for the second time, assuming that N_original>1 is set: the first registration result is used as a new image to be registered and cascaded with the original target image to be input into the convolutional neural network model again to obtain the displacement field of the second output, and the displacement field of the second output is saved;

[0030] 310) Using the displacement field integration method, the displacement field output by the convolutional neural network for the second time is integrated with the displacement field output for the first time into a total displacement field. The original image to be registered is subjected to a deformation interpolation operation using the total displacement field to obtain the second recursive registration result. This registration result is used as a new image to be registered and is cascaded with the target image and input into the convolutional neural network for optimization. If the set N_original>2, then steps 39)-310) are repeated on the basis of recursion twice, that is, the displacement field output by each recursion is integrated with the displacement field output before, and the registration result of this time is obtained through a deformation interpolation operation, until the recursion is N_original times;

[0031] 311) Randomly select two images from the three-dimensional biomedical image training set as the image to be registered and the target image, and repeat steps 31)-310) until the registration model training is completed.

[0032] The displacement field integration method comprises the following steps:

[0033] 41) Set the input of the displacement field integration strategy to Flow (n) with Flow (n+1) (n>=0),Flow (n) Represents the displacement field of the network output at the nth recursion, Flow (n+1) Represents the displacement field output by the network at the n+1th recursion;

[0034] 42) Set Flow_Agg (n,n+1) Represents the displacement field Flow of the image output for the nth time (n) And the displacement field Flow output for the n+1th time (n+1) The integrated total displacement field, the displacement field integration principle is shown in formula (1),

[0035]

[0036]

[0037] x′=x+I(x)

[0038]

[0039] x′=x+I n (x)

[0040]

[0041]

[0042] In the above formula,

[0043] Therefore, from the above formula, we can get Flow_Agg (n,n+1) equal

[0044] Among them, F(x) represents the point at position x in the Fixed image, M(x) represents the point at position x in the Moving image, M(x′) represents the point at position x′ in the Moving image, φ represents the deformation field, and φ n When the nth recursive network is represented by Flow (n) The deformation field formed, φ n+1 The n+1th recursive network is composed of Flow (n+1) The deformation field formed, Indicates the warped operation, I n represents the displacement field output by the nth recursive network, I(x) represents the point at position x in the displacement field, (x) represents the image obtained by the deformation field warping operation on the Moving image;

[0045] The above formula is expanded to integrate the displacement fields Flow of the previous m times into one displacement field (Flow (0,…,m) ), we get the following formula (2):

[0046]

[0047]

[0048] …

[0049]

[0050] Where Flow_Agg (0,…,m) It is the total displacement field integrating the displacement fields Flow of the previous m times.

[0051] Beneficial Effects

[0052] The present invention provides a progressive three-dimensional biomedical image registration method based on deep self-calibration. Compared with the prior art, the method recursively inputs the previous registration result into the network for self-calibration optimization and records and integrates the displacement field generated by each recursion. The large displacement in space is completed in segments through the recursive network, thereby improving the registration accuracy of the three-dimensional biomedical image without information leakage and effectively saving GPU resources.

[0053] The present invention utilizes non-convolutional downsampling to obtain multi-scale information of the registered image pair and utilizes recursive self-calibration and displacement field integration strategy training at each scale to enable the network to learn important information that is beneficial to the registration and is lost in the previous registration result. This solves the defects of the prior art that the coarse level of the multi-scale unsupervised learning deep registration algorithm causes errors that lead to the inability to recover the fine level and the information leakage problem in the recursive cascade network registration process. It also solves the contradiction between the overall smoothness constraint of the displacement field and the completion of complex deformation. At the same time, it also realizes the size of the dynamically augmented data set to better adapt to the data set size required for network training, avoiding the defects of underfitting and overfitting of the network caused by too large or too small data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a method sequence diagram of the present invention;

[0055] Figure 2 This is a diagram of the overall deep network registration framework involved in the present invention;

[0056] Figure 3 It is a schematic diagram of the progressive registration without information leakage of the present invention;

[0057] Figure 4a A three-dimensional fMOST image slice of the mouse brain is selected as the Fixed image;

[0058] Figure 4b A three-dimensional fMOST image slice of a mouse brain is selected as the Moving image;

[0059] Figure 4c To use the traditional image registration tool ANTs Figure 4b Register to Figure 4a The result graph of

[0060] Figure 4d To use the existing depth registration algorithm VoxelMorph Figure 4b Register to Figure 4a The result graph of

[0061] Figure 4e To use the existing depth registration algorithm VoxelMorph-diff Figure 4b Register to Figure 4aThe result graph of

[0062] Figure 4f To use the method of the present invention, Figure 4b Register to Figure 4a The result graph of

[0063] Figure 5a A three-dimensional MRI image slice of the human brain is selected as the Fixed image;

[0064] Figure 5b A three-dimensional MRI image slice of the human brain is selected as the Moving image;

[0065] Figure 5c To use the traditional image registration tool ANTs Figure 5b Register to Figure 5a The result graph of

[0066] Figure 5d To use the existing depth registration algorithm VoxelMorph Figure 5b Register to Figure 5a The result graph of

[0067] Figure 5e To use the existing depth registration algorithm VoxelMorph-diff Figure 5b Register to Figure 5a The result graph of

[0068] Figure 5f To use the method of the present invention, Figure 5b Register to Figure 5a Result graph. DETAILED DESCRIPTION

[0069] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0070] like Figure 1 As shown, the present invention provides a progressive three-dimensional biomedical image registration method based on deep self-calibration, comprising the following steps:

[0071] The first step is to obtain and preprocess the 3D biomedical image dataset: obtain a single-modality 3D biomedical image dataset, sample all images to the same size, normalize the pixel values ​​to 0-255, and affine pre-register all images to the same brain template as the 3D biomedical image training set. In the experimental phase, the above dataset can be randomly divided into a training set and a test set.

[0072] The second step is to build the registration model: use Python language to build a low-scale convolutional neural network and an original-scale convolutional neural network respectively, and cascade the two models as a registration model, and set the loss function of the low-scale convolutional neural network and the original-scale convolutional neural network. Figure 2 As shown in the figure, since the receptive fields of the feature maps obtained by convolution with the same size of convolution kernels for different sizes of the same image are different, the downsampled image can enlarge the receptive field of the feature map in disguise, so that the low-scale network can learn the large deformation information that the original-scale network cannot learn, and the downscaled image input network can reduce the consumption of GPU resources while improving the registration speed. We can fuse the information of the image to be registered through the low-scale network output to the original-scale image, and perform fast regional pre-registration before the image is input into the original-scale registration network, which can greatly reduce the pressure on the original-scale registration network.

[0073] The specific steps are as follows:

[0074] (1) The constructed registration model is set as a cascade of a low-scale convolutional neural network and an original-scale convolutional neural network.

[0075] (2) The low-scale convolutional neural network of the registration model is set to a U-Net-like architecture, which consists of an encoder and a decoder. The input of the low-scale convolutional neural network is the target (template) image and the image to be registered with a size of (d / 2, w / 2, h / 2).

[0076] (3) The original-scale convolutional neural network of the registration model is set to a U-Net-like architecture, which consists of an encoder and a decoder. The input of the original-scale convolutional neural network is the target image and the image to be registered of size (d, w, h).

[0077] (4) The outputs of the low-scale registration network model and the original-scale convolutional neural network are set to be three displacement fields of the same size and dimension as the corresponding input images. This displacement field represents the transformation relationship from the image to be registered to the target image, specifically, the displacement of the corresponding coordinate point on the registration result image in the x-, y-, and z-directions corresponding to a point on the mobile image.

[0078] (5) Set the loss function of the low-scale registration network model and the original-scale convolutional neural network to be the similarity loss L between the registration result and the target image. sim and the smoothness constraint loss L of the convolutional neural network output displacement field smooth , the total loss function of a single recursion is L = L sim +λ*L smooth , where λ is the weight coefficient that balances the proportion of the two sub-loss functions in the total loss function.

[0079] The third step is to train the registration model: set the number of iterations of the registration model training and the number of recursions of the low-scale convolutional neural network and the original-scale convolutional neural network respectively, and use the three-dimensional biomedical image training set to train the registration model.

[0080] The traditional one-shot registration method based on deep learning only learns the features of a pair of images through one forward propagation and back propagation. Based on the traditional one-shot registration method based on deep learning, this method optimizes the input pair of images multiple times, takes the result of the last registration as the new image to be registered, and combines the fixed image with the displacement field integration strategy to input the original network for recursive optimization. The difference between this method and other methods of using recursion to use the result of the last registration as the new image to be registered is that this method saves the displacement field output by each recursive optimization, integrates all the previous displacement fields into a total displacement field by using the displacement field integration strategy, and uses this total displacement field to form a deformation field to perform an interpolation deformation operation on the original image to be registered for the next recursive optimization. That is, the image to be registered that is input into the network for recursion each time is the total deformation field of the original image to be registered after the previous recursion and undergoes an interpolation operation.

[0081] In this way, the error in the previous registration can be calibrated through the displacement field output by each recursion, and large deformation can be completed in a way without information leakage. In addition, this method also has the effect of augmenting the data set. Since a pair of registered image pairs are recursively optimized multiple times, each recursion of the network is equivalent to generating a new pair of images to be registered. By modifying the number of recursions, the size of the augmented data set can be dynamically controlled, which can make up for the shortcoming of the small number of biomedical imaging data sets.

[0082] The specific steps are as follows:

[0083] (1) The number of iterations of model training is set to N_iter, that is, in each iteration, two images are randomly selected from the 3D biomedical image training set as the images to be registered and the target images and input into the registration model for training.

[0084] (2) Set the number of low-scale training recursions to N_low, N_low>=0, and the number of original-scale training recursions to N_original, N_original>=0.

[0085] (3) The first recursion of the low-scale network: The image to be registered and the target image are downsampled by two times and then cascaded into the low-scale convolutional neural network to obtain the first output displacement field, and the first output displacement field is saved.

[0086] (4) The second recursion of the low-scale network, assuming that the set N_low>1: the first registration result is used as a new image to be registered and concatenated with the original target image and input into the low-scale convolutional neural network again to obtain the second output displacement field, and save the second output displacement field.

[0087] (5) Using the displacement field integration method, the displacement field of the second output of the convolutional neural network model is integrated with the displacement field of the first output into a total displacement field. The total displacement field is used to perform a deformation interpolation operation on the original image to be registered to obtain the second recursive registration result. This registration result is used as the new image to be registered and is cascaded with the target image to input the convolutional neural network for optimization. If the set N_low>2, then steps (4)-(5) are repeated on the basis of the second recursion, that is, the displacement field of each recursive output is integrated with the displacement field output before, and the registration result of this time is obtained through a deformation interpolation operation, until the recursion is N_low times.

[0088] The displacement field integration method includes the following steps:

[0089] A1) Set the input of the displacement field integration strategy to Flow (n) with Flow (n+1) (n>=0),Flow (n) Represents the displacement field of the network output at the nth recursion, Flow (n+1) Represents the displacement field output by the network at the n+1th recursion;

[0090] A2) Set Flow_Agg (n,n+1) Represents the displacement field Flow of the image output for the nth time (n) And the displacement field Flow output for the n+1th time (n+1) The integrated total displacement field, the displacement field integration principle is shown in formula (1),

[0091]

[0092]

[0093] x′=x+I(x)

[0094]

[0095] x′=x+I n (x)

[0096]

[0097]

[0098] In the above formula,

[0099] Therefore, from the above formula, we can get Flow_Agg (n,n+1) equal

[0100] Among them, F(x) represents the point at position x in the Fixed image, M(x) represents the point at position x in the Moving image, M(x′) represents the point at position x′ in the Moving image, φ represents the deformation field, and φ n When the nth recursive network is represented by Flow (n) The deformation field formed, φ n+1 The n+1th recursive network is composed of Flow (n+1) The deformation field formed, Indicates the warped operation, I n represents the displacement field output by the nth recursive network, I(x) represents the point at position x in the displacement field, (x) represents the image obtained by the deformation field warping operation on the Moving image;

[0101] The above formula is expanded to integrate the displacement fields Flow of the previous m times into one displacement field (Flow (0,…,m) ), we get the following formula (2):

[0102]

[0103]

[0104] …

[0105]

[0106] Where Flow_Agg (0,…,m) It is the total displacement field integrating the displacement fields Flow of the previous m times.

[0107] (6) The total displacement field obtained by the low-scale recursion N_low times is saved as prior information and input into the original scale convolutional neural network together with the image to be registered and the target image that have not been downsampled in this iterative training.

[0108] (7) The total displacement field obtained by low-scale recursion N_low times is upsampled twice and numerically enlarged twice to obtain a new displacement field deformation interpolation to pre-deform the original image to be registered in the original size, and the pre-deformed image is used as the original size of the initial image to be registered.

[0109] (8) The first recursion of the original scale network: The pre-deformed original scale image to be registered and the original target image of the original size are cascaded and input into the original scale convolutional neural network model constructed by this method, the first displacement field at the original scale is output, and the first output displacement field is saved.

[0110] (9) The second recursion of the original scale network, assuming that the setting N_original>1: the first registration result is used as the new image to be registered and cascaded with the original target image to be input into the convolutional neural network model again to obtain the second output displacement field, and the second output displacement field is saved.

[0111] (10) Using the displacement field integration method, the displacement field of the second output of the convolutional neural network is integrated with the displacement field of the first output into a total displacement field. The original image to be registered is subjected to a deformation interpolation operation using the total displacement field to obtain the second recursive registration result. This registration result is used as a new image to be registered and is cascaded with the target image and input into the convolutional neural network for optimization.

[0112] If the set N_original>2, then steps (9)-(10) are repeated on the basis of recursion twice, that is, the displacement field outputted by each recursion is integrated with the displacement field outputted previously, and the alignment result of this time is obtained through one deformation interpolation, until the recursion is performed N_original times.

[0113] (11) Randomly select two images from the 3D biomedical image training set as the image to be registered and the target image, and repeat steps (1)-(10) until the registration model training is completed.

[0114] The fourth step is to obtain and preprocess the image to be registered and the template image: obtain the single-modality three-dimensional biomedical image to be registered and the target image, and perform preprocessing. The preprocessing process is the same as that of the three-dimensional biomedical image training set.

[0115] The fifth step is to obtain the three-dimensional biomedical image registration result: input the preprocessed image to be registered into the trained registration model to obtain the registered result.

[0116] Although existing convolution-based methods for multi-scale medical image registration can achieve a coarse-to-fine refinement of the deformation field to handle significant differences between image pairs and large displacements in space, these methods may not be able to recover at the fine level if errors occur at the coarse level. This defect becomes more serious as the scale increases. Figure 3It is a classic deep learning-based medical image registration algorithm. Most deep learning-based medical image registration methods use Spatial Transformer Networks to define and represent the deformation field with regular grid points. However, using regular grid points to represent the deformation field cannot reflect the anatomical information of the brain. In the loss function, only an overall smoothness constraint can be imposed on the displacement field. If the overall smoothness constraint is too large, some brain regions will not be able to be registered. If the overall smoothness constraint is too small, some brain regions will be overly distorted. Therefore, the smoothness constraint on the displacement field is inconsistent with completing large deformations.

[0117] Figure 4a and Figure 4b The fMOST images of the rat brain are used as the target image and the latter as the image to be registered. The goal is to register 4b to Figure 4a superior, Figure 4c The registration results obtained using the traditional image registration tool ANTs, Figure 4d The results obtained using the classic deep learning algorithm VoxelMorph, Figure 4e The results obtained using the classic deep learning algorithm VoxelMorph-diff, Figure 4f By comparison, the registration results obtained by the present invention are better than those obtained by the method proposed by the present invention in the parts marked by the boxes in the figure. Figure 4a The target images shown are more similar in structure and brightness distribution. The improvement effect of the present invention can also be seen from the human brain MRI image registration: Figure 5a and Figure 5b These are MRI images of the human brain. Similarly, the former is a fixed image and the latter is a floating image. Figure 5c The registration results obtained using ANTs, Figure 5d The result obtained using VoxelMorph, Figure 5e The result obtained using VoxelMorph-diff is: Figure 5f The results obtained by using the present invention are also compared. In the parts marked by the boxes in the figure, the registration effect of the method proposed by the present invention is better, that is, the registration results obtained by using the present method are better than those obtained by using the method. Figure 5a The target images shown are more similar in structure and brightness distribution. Table 1 is a table of dynamic data set augmentation effects provided by the method of the present invention.

[0118] Table 1 Dynamic data set augmentation effect table provided by the method of the present invention

[0119]

[0120] It can be seen from Table 1 that the test results with a training recursion number greater than 0 are better than the test results with a training recursion number of 0, whether at low scale (Low) or original scale (Original). This shows that the dynamically augmented dataset can indeed improve the registration accuracy of the network.

[0121] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A progressive three-dimensional biomedical image registration method based on deep self-calibration, characterized in that, it includes the following steps: 11) Acquisition and preprocessing of the three-dimensional biomedical image dataset: Obtain a single-modal three-dimensional biomedical image dataset, sample the sizes of all its images to the same size, normalize the pixel values to 0-255, and affine pre-register all the images to the same brain template as the three-dimensional biomedical image training set; 12) Construction of the registration model: Use the Python language to build a low-scale convolutional neural network and an original-scale convolutional neural network respectively and cascade the two as the registration model, and set the loss functions of the low-scale convolutional neural network and the original-scale convolutional neural network; 13) Training of the registration model: Set the number of iterations for training the registration model and the number of recursions of the low-scale convolutional neural network and the original-scale convolutional neural network respectively, and use the three-dimensional biomedical image training set to train the registration model; The training of the registration model includes the following steps: 131) Set the number of iterations for model training to N_iter, that is, randomly select two images from the three-dimensional biomedical image training set as the images to be registered and the target image for each iteration and input them into the registration model for training; 132) Set the number of low-scale training recursions to N_low, N_low >= 0, and the number of original-scale training recursions to N_original, N_original >= 0; 133) First recursion of the low-scale network: Downsample the image to be registered and the target image by a factor of two and then cascade them and input them into the low-scale convolutional neural network to obtain the displacement field output for the first time, and save the displacement field output for the first time; 134) Second recursion of the low-scale network, assuming N_low > 1 is set: Use the registration result of the first time as the new image to be registered and cascade it with the original target image and input them into the low-scale convolutional neural network again to obtain the displacement field output for the second time, and save the displacement field output for the second time; 135) Use the displacement field integration method to integrate the displacement field output for the second time of the convolutional neural network model and the displacement field output for the first time into a total displacement field, and use the total displacement field to perform a deformation interpolation operation on the original image to be registered to obtain the registration result of the second recursion. Use this registration result as the new image to be registered and cascade it with the target image again and input them into the convolutional neural network for optimization; If N_low > 2 is set, repeat steps 134)-135) on the basis of the second recursion, that is, integrate the displacement field output for each recursion and the displacement field output before to obtain the registration result of this time through a deformation interpolation until the N_lowth recursion; 136) Save the total displacement field obtained after the N_lowth recursion of the low-scale network as prior information and input it together with the image to be registered and the target image that have not been downsampled in this iteration of training into the original-scale convolutional neural network; 137) Upsample the total displacement field obtained by low-scale recursion \(N_{low}\) times by two times and numerically expand it by two times to obtain a new displacement field, and perform pre-deformation on the original image to be registered with the original size by deforming and interpolating using the new displacement field. Then, use the pre-deformed image as the first image to be registered with the original size. 138) First recursion of the original-scale network: Concatenate the pre-deformed original-scale image to be registered with the original-scale target image and input them into the convolutional neural network model at the original scale. Output the first displacement field at the original scale and save the displacement field output for the first time. 139) Second recursion of the original-scale network. Assume that \(N_{original}>1\): Use the registration result of the first time as the new image to be registered, concatenate it with the original target image again, and input them into the convolutional neural network model to obtain the second output displacement field, and save the second output displacement field. 1310) Use the displacement field integration method to integrate the displacement field output for the second time by the convolutional neural network and the displacement field output for the first time into a total displacement field. Use the total displacement field to perform one deformation and interpolation operation on the original image to be registered to obtain the registration result of the second recursion. Use this registration result as the new image to be registered and concatenate it with the target image again to input into the convolutional neural network for optimization. If \(N_{original}>2\), repeat steps 139)-1310) on the basis of two recursions, that is, integrate the displacement field output each time with the displacement field output before, and obtain the registration result of this time through one deformation and interpolation until \(N_{original}\) recursions. 311) Randomly select two more images from the three-dimensional biomedical image training set as the images to be registered and the target image, and repeat steps 131)-1310) until the training of the registration model is completed. 14) Acquisition and preprocessing of the image to be registered and the template image: Acquire the single-modal three-dimensional biomedical image to be registered and the template image, and perform preprocessing. 15) Obtaining the registration result of the three-dimensional biomedical image: Input the preprocessed pair of images to be registered into the trained registration model to obtain the registered result.

2. A progressive three-dimensional biomedical image registration method based on deep self-calibration according to claim 1, characterized in that the construction of the registration model includes the following steps: 21) Set the constructed registration model as a cascade of a low-scale convolutional neural network and an original-scale convolutional neural network. 22) Set the low-scale convolutional neural network of the registration model as a U-Net-like architecture, which consists of an encoder and a decoder. The input of the low-scale convolutional neural network is the target image and the image to be registered with the size of \((d / 2, w / 2, h / 2)\). 23) Set the original-scale convolutional neural network of the registration model as a U-Net-like architecture, which consists of an encoder and a decoder. The input of the original-scale convolutional neural network is the target image and the image to be registered with the size of \((d, w, h)\). 24) Set the outputs of both the low-scale registration network model and the original-scale convolutional neural network to be three displacement fields with the same size and dimension as the corresponding input images. This displacement field represents the transformation relationship from the image to be registered to the target image. Specifically, it is the displacement in the x, y, and z directions of a certain point on the moving image corresponding to the corresponding coordinate point on the registered result image; 25) Set the loss functions of both the low-scale registration network model and the original-scale convolutional neural network as the similarity loss L between the registration result and the target image sim and the smoothness constraint loss L of the displacement field output by the convolutional neural network smooth , and the total loss function for a single recursion is L = L sim + λ * L smooth , where λ is the weight coefficient that balances the proportions of the two sub-loss functions in the total loss function.

3. A progressive three-dimensional biomedical image registration method based on deep self-calibration according to claim 1, wherein, the displacement field integration method includes the following steps: 31) Set the input of the displacement field integration strategy as Flow (n) and Flow (n+1) (n >= 0), Flow (n) represents the displacement field output by the network at the n-th recursion, and Flow (n+1) represents the displacement field output by the network at the (n + 1)-th recursion; 32) Let Flow_Agg (n,n+1) represent the total displacement field obtained by integrating the displacement field Flow output at the n-th time (n) with the displacement field Flow output at the (n + 1)-th time (n+1) The principle of displacement field integration is shown in formula (1). In the above formula, Therefore, from the above equation, Flow_Agg (n,n+1) is equal to Among them, F(x) represents the point at the position x of the Fixed image, M(x) is the point at the position x on the Moving image, M(x′) is the point at the position x′ on the Moving image, φ represents the deformation field, φ n represents the deformation field formed by Flow at the nth recursive network (n) formed, φ n+1 is the deformation field formed by Flow at the (n + 1)th recursive network (n+1) formed, represents the warped operation, I n represents the displacement field output at the nth recursive network, I(x) represents the point at the position x in the displacement field, represents the image obtained by warping the Moving image through the deformation field; Expand the above formula to integrate the displacement fields Flow of the previous m times into a displacement field (Flow (0 ,…, m) ), and the following formula (2) is obtained, … Among them, Flow_Agg (0 ,…, m) is the total displacement field obtained by integrating the displacement fields Flow for the previous m times.

Images