Context-driven unsupervised registration system and method for pyramid structure

Abstract

A context-driven pyramid structure unsupervised registration system and method, comprising: two multi-resolution pyramid modules respectively used to receive a reference image and an image to be registered and share weights, a deformation field estimation module, an optimization module, a deformation field regularization module and a registration module, wherein: the deformation field estimation module estimates the deformation field at different resolutions according to the reference image, the image to be registered, the deformation field estimated at the previous level, and the multi-angle semantic information generated at the previous level, and generates the deformation field and the multi-angle semantic information; the optimization module adjusts the estimated deformation field to be consistent with the resolution of the original image; the deformation field regularization module performs local Gaussian smoothing filtering on the folding area of ​​the deformation field through a deformation field regularization model, thereby reducing the folding rate of the deformation field and outputting the final deformation field; the registration module performs feature mapping according to the final deformation field to obtain the registered image.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to a context-driven three-dimensional multi-resolution pyramid structure unsupervised registration system and method. Background Art

[0002] Existing unsupervised medical image registration methods have difficulty achieving a good balance between registration accuracy and anatomical structure preservation when processing images with large displacements and complex deformations. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a context-driven pyramid structure unsupervised registration system and method. In view of the large displacement and complex deformation characteristics of three-dimensional medical image registration, a deformation field is output according to a reference image and an image to be registered through a deep learning network, and feature mapping is performed on the image to be registered according to the deformation field to obtain the final registered image. The process is fully automated and is expected to achieve real-time registration without labor costs, thereby providing assistance for medical analysis work.

[0004] The present invention is achieved through the following technical solutions:

[0005] The invention relates to an unsupervised registration system based on a context-driven pyramid structure, comprising: two multi-resolution pyramid modules respectively used for receiving a reference image and an image to be registered and sharing weights, a deformation field estimation module, an optimization module, a deformation field regularization module and an interpolation transformation module, wherein: the deformation field estimation module estimates the deformation field at different resolutions according to the reference image, the image to be registered, the deformation field estimated at the previous level and the multi-angle semantic information generated at the previous level, and generates the deformation field and the multi-angle semantic information; the optimization module adjusts the estimated deformation field to be consistent with the resolution of the original image; the deformation field regularization module performs local Gaussian smoothing filtering on the folding area of ​​the deformation field through a deformation field regularization model, thereby reducing the folding rate of the deformation field and outputting a final deformation field; the interpolation transformation module performs feature mapping on the image to be registered according to the final deformation field to obtain a registered image.

[0006] The present invention relates to an unsupervised registration method based on the above system, which continuously updates network parameters in an iterative training manner, and improves the registration accuracy of the network through multi-receptive field autocorrelation processing, multi-angle semantic extraction and deformation field regularization processing while effectively retaining the topological structure of the image, and performs feature mapping on the image to be registered according to the predicted deformation field to obtain the final registered image. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is an overall schematic diagram of the present invention;

[0008] Figure 2 This is the workflow diagram of the deformation field estimation module;

[0009] Figure 3 Schematic diagram of multi-receptive field autocorrelation module;

[0010] Figure 4 This is a schematic diagram for the regularization demonstration of the deformation field;

[0011] In the figure: a is before regularization, b is after regularization;

[0012] Figure 5 Schematic diagram of the training loss function curve without regularization module / with regularization module;

[0013] Figure 6 The visualization results of registering liver CT images using different methods in the embodiment;

[0014] Figure 7 The following are visualization results of registering brain MR images using different methods in the embodiments. DETAILED DESCRIPTION

[0015] like Figure 1 As shown, this embodiment relates to a context-driven pyramid structure unsupervised registration system, comprising: two multi-resolution pyramid modules for receiving a reference image F and an image to be registered M, a deformation field estimation module, an optimization module, a deformation field regularization module and an interpolation transformation module, wherein: the two multi-resolution pyramid modules share weights, and the deformation field estimation module calculates the deformation field according to the reference image F. i , image M to be registered i , the deformation field φ estimated at the previous level i-1 , the multi-angle semantic information c generated by the previous level i-1 , estimate the deformation field at different resolutions and generate the deformation field φ i and multi-angle semantic information c i ; The optimization module keeps the resolution of the final deformation field consistent with that of the original image; the deformation field regularization module performs local Gaussian smoothing filtering on the folding area of ​​the deformation field through the deformation field regularization model, thereby reducing the folding rate of the deformation field and outputting the final deformation field φ; the interpolation transformation module performs feature mapping on the image to be registered M according to the final deformation field φ to obtain the registered image R.

[0016] like Figure 1 As shown, the two multi-resolution pyramid modules are: a four-level feature pyramid F i and M iEach level includes a convolution layer and a LeakyReLU layer. The number of channels at each level is 4, 8, 16, and 16 from high resolution to low resolution, where: i = 1, 2, 3, 4, the convolution kernel size is 3, and the step size is 2.

[0017] like Figure 2 As shown in FIG. 1 , the deformation field estimation module includes: a feature mapping (FeatureWarping) unit, a multi-receptive field autocorrelation (MRF-CM) unit, a multi-angle semantic extraction (MV-CEM) unit and a residual deformation field estimation (RDFEM) unit, wherein: the feature mapping unit converts the image feature M to be registered output by the multi-resolution pyramid module i After double upsampling, φ i-1 Distortion processing to obtain distorted features Multi-receptive field autocorrelation units calculate distorted features With the reference image F i Correlation diagram between i ; The residual deformation field estimation unit is based on the output d of the multi-receptive field autocorrelation unit i , the deformation field φ estimated at the previous level i-1 and the multi-angle semantic information c generated by the previous level i-1 Predict the residual deformation field After that, the residual deformation field Perform two times upsampling to obtain the deformation field φ i , the multi-angle semantic extraction unit is based on the deformation field φ i , the multi-angle semantic information c generated by the previous level i-1 And the output d of the multi-receptive field autocorrelation unit i Generate multi-angle semantic information i .

[0018] The feature mapping unit uses the twice upsampled φ i-1 Treat the registered image M i Mapping is performed to reduce the image M to be registered i and the reference image F i The misalignment area between them is: Among them: ο represents the twist operation, up 2 Represents an upsampling operation.

[0019] This embodiment uses a transposed convolution layer with a convolution kernel size of 4 and a step size of 2 to perform upsampling operations.

[0020] The feature mapping unit transforms the image M to be registered i and the reference image F iThe deformation field estimation module only needs to estimate a residual deformation field at each resolution, so the deformation field estimation process becomes easier, which is very helpful for estimating complex deformation fields.

[0021] like Figure 3 As shown, the multi-receptive field autocorrelation unit includes four branches: ①, ②, ③, and ④, wherein: the first branch consists of a 1×1×1 convolution layer and two 3×3×3 convolution layers, the second branch consists of a 1×1×1 convolution layer and a 3×3×3 convolution layer, the third branch consists of a 1×1×1 convolution layer, and the fourth branch consists of a pooling layer and a 3×3×3 convolution layer; the output d of the multi-receptive field autocorrelation unit is i The results of the four branches are spliced ​​together, and the global and local information of the network is effectively utilized through multi-receptive field autocorrelation units, making the overall registration of medical images more accurate and reasonable while promoting the effective registration of local structures.

[0022] The 1×1×1 convolutional layer in the multi-receptive field autocorrelation unit is used to transform the two inputs (F i and ) are fused, and the 3×3×3 convolutional layer is used to expand the receptive field area.

[0023] The number of channels of the resolution of the convolutional layers of the four branches in the multi-receptive field autocorrelation unit are 16, 32, 64, and 128 respectively.

[0024] A LeakyReLU layer is arranged after each convolutional layer in the multi-receptive field autocorrelation unit.

[0025] The multi-angle semantic extraction unit extracts the multi-angle semantic information c generated by the previous level. i-1 , the output of the multi-receptive field autocorrelation unit d i and the deformation field φ i Perform splicing processing, specifically: c i =H c ([d i ,φ i , c i-1 ]), where: [d i ,φ i , c i-1 ] means d i ,φ i , c i-1 The combination of the three, H c() represents the overall function of the multi-angle semantic extraction unit, which integrates the deep and shallow semantic information of the network to improve the registration accuracy of the network. The multi-angle semantic extraction unit includes: a transposed convolution layer and a LeakyReLU layer, where the convolution kernel size of the convolution layer is 4 and the step size is 2.

[0026] The residual deformation field estimation unit is based on the multi-angle semantic information c generated by the previous level. i-1 , related graph d i and the deformation field φ generated at the previous level i-1 Predict the residual deformation field Specifically: Where: [d i ,φ i-1 , c i-1 ] means d i ,φ i-1 , c i-1 The combination of the three, H r () indicates that φ i-1 The result after upsampling by a factor of two is Adding together, we can get the estimated deformation field φ i ,Right now: The residual deformation field estimation unit is implemented by a convolutional layer with a convolution kernel size of 3 and a stride of 1.

[0027] The optimization module is based on the deformation field φ output by the deformation field estimation module. 4 and multi-angle semantic information c 4 , the final deformation field φ is calculated, specifically: φ=R([φ 4 , c 4 ]), where: R() represents the overall effect of the optimization module; the optimization module is implemented by a convolutional layer with a convolution kernel size of 3 and a step size of 1.

[0028] like Figure 4 As shown, the deformation field regularization module performs local Gaussian smoothing filtering on the folded area, which is equivalent to unfolding and folding the voxel p, specifically: in: Where: * represents the convolution operation, g is a three-dimensional Gaussian kernel; the set of voxels whose Jacobian determinant in the deformation field φ is non-positive Where: Ω represents the grid area where the input image is located, det(φ(p) represents the value of the Jacobian determinant at voxel p; δ r It is the voxel within the range with voxel p as the center and r as the radius.

[0029] The present invention trains the unsupervised registration system by reasonably designing a loss function including a similarity loss and two regularization losses. The loss function is specifically: Among them: the similarity loss is aimed at the accuracy problem in medical image registration, and the two regularization losses are aimed at the folding problem in medical image registration; λ 1 and λ 2 are the weights of TV loss and anti-folding loss respectively; similarity loss function TV loss function Anti-collapsing loss function det(φ r )(p)), where: NCC(·) represents the normalized correlation coefficient, σ(·) is the ReLU activation function, that is, it is linear in the positive region and all zero in the negative region. det(·) represents the Jacobian determinant.

[0030] In order to prove the improvement effect of the feature mapping unit on network performance, the performance of Multi-RegNet with / without the feature mapping unit is compared on the Sliver, LiTS, and BGCV datasets, and the experimental results are shown in Table 1. As can be seen from Table 1, the feature mapping makes the dice of the Multi-RegNet of this embodiment improve by 3.6%, 3.5%, and 2.5% on the liver registration datasets Sliver, LiTS, and BGCV, respectively. Through this experiment, it can be seen that the feature mapping unit improves the network performance.

[0031] Table 1. The improvement effect of feature mapping on network performance. The standard deviation is in brackets.

[0032]

[0033] This embodiment compares the improvement effect of different branches of multi-receptive field autocorrelation units on network performance through the following experiments. The experimental results are shown in Table 2, where Sliver, LiTS, and BGCV are liver registration data sets, and OASIS is a brain registration data set. As can be seen from Table 2, when any one of the four multi-receptive field autocorrelation unit branches is removed from the network, the Dice coefficient is 1%-4% lower than the Dice coefficient when the four branches exist at the same time. Through this experiment, the role of multi-receptive field autocorrelation units in promoting the prediction of complex deformation fields can be further illustrated.

[0034] Table 2 The improvement effect of different branches of multi-receptive field autocorrelation units on network performance, with standard deviations in brackets.

[0035]

[0036] This embodiment conducts experiments on liver registration datasets Sliver, LiTS, and BGCV to compare the registration performance of the network with and without the multi-angle semantic extraction unit. The experimental results are shown in Table 3. As can be seen from Table 3, when the multi-angle semantic extraction unit is included, the Dice is improved by 5.5%, 5.2%, and 2.5% respectively compared with the case without the module. It can be seen that the multi-angle semantic extraction unit has a certain effect on improving the registration performance of the network.

[0037] Table 3. The improvement effect of multi-angle semantic extraction unit on network performance. The standard deviation is in brackets.

[0038]

[0039] This embodiment conducts experiments on the combination of different perspectives, as shown in Table 4. This is to illustrate the role of different perspectives. As can be seen from Table 4, when the multi-angle semantic extraction unit removes any perspective, the network registration performance will drop by 1%-2%. This shows the effectiveness of the design of the multi-angle semantic extraction unit.

[0040] Table 4 The impact of different perspective combinations of multi-angle semantic extraction units on network performance

[0041]

[0042] In order to prove the role of the regularization module in improving the network registration performance, this example conducts training on the brain registration dataset OASIS with and without the regularization module. As shown in Table 5, the Dice coefficient and folding rate in the two cases are as follows: Figure 5 As shown in , these are the training loss function curves for the two cases. It can be seen from Table 5 that when the anti-folding weight increases from 1e-5 to 1e-3, the folding rate of the network without the regularization module decreases from 3.30e-2 to 3.97e-3, but the Dice coefficient also decreases from 0.761 to 0.750. However, in the case of a regularization module, when the anti-folding weight increases from 1e-5 to 1e-3, the folding rate decreases and is always lower than the folding rate of the network without the regularization module under the same conditions, and the Dice coefficient remains above 0.759. This shows that adding a deformation field regularization module to the network can effectively reduce the folding rate while alleviating the impact of the anti-folding constraint on the network registration accuracy. Figure 5 It can be seen that with the regularization module, the network converges after about 20k iterations, but without the regularization module, it converges after about 50k iterations. This shows that the deformation field regularization module can accelerate the convergence of the network. The above experiments show that the deformation field regularization module can keep the folding rate at a low level without negatively affecting the registration accuracy, and can also accelerate the convergence of the network.

[0043] Table 5 Different λ 2 Dice and folding ratio under , with standard deviations in brackets.

[0044]

[0045]

[0046] In addition, this embodiment conducts further experiments to compare the performance of this system with existing methods. The experiments are conducted on liver CT images and brain MR images, respectively. In the liver CT image experiment, 1025 liver scan images from two public datasets, MSD and BFD, are used, and three public datasets with liver segmentation data, Silver, LITS, and BGCV, are used as test sets; in the brain MR image experiment, 414 brain magnetic resonance images are obtained from the OASIS-1 dataset. In the experiment, the dataset is randomly divided into 330, 14, and 100 images as training sets, validation sets, and test sets, respectively. In the liver CT registration experiment, all images are resampled to a size of 128×128×128, and λ in the loss function 1 and λ 2 Set to 1.0 and 1×10 respectively -4 ; In the brain MR image experiment, all images were resampled to a size of 128×128×128, and λ in the loss function 1 and λ 2 are set to 0.1 and 1×10 respectively. -4 The radius r in all experiments is set to 5. In training, the batch size is set to 4 and the Adam optimizer is used for 10 5 Iterations. The initial learning rate is 10 -4 , after 6×10 4 After 8×10 iterations, the number is halved. 4 After the iteration, it is halved.

[0047] This method is compared with VoxelMorph, VTN, FAIM, and CycleMorph: All methods are implemented using TensorFlow, VTN (ADD) runs on two 32G NVIDIA Teslav100 cards, and other networks run on a single 32G NVIDIA Teslav100 card and use the Dice coefficient and folding rate as two evaluation criteria.

[0048] Table 6 shows the comparison results of different methods under the two indicators of Dice coefficient and folding rate. It can be seen from the table that compared with the baseline method VoxelMorph, Multi-RegNet improves the Dice of the three data sets from 88%, 84%, and 82% to 93%, 88%, and 85%, respectively. In addition, compared with VTN, FAIM, and CycleMorph, Multi-RegNet has also greatly improved in the two indicators of Dice and folding rate. Although the cascade network VTN (ADD) has excellent Dice indicators, it has the highest folding rate. Therefore, considering the two indicators of dice and folding rate, Multi-RegNet still has better results than VTN (ADD).

[0049] Table 6 Comparison of the performance of different methods in liver CT image experiments, with standard deviations in brackets.

[0050]

[0051] like Figure 6 As shown in Figure 1, the visualization results of liver CT image registration using different methods are shown. The first row shows the reference image and the image to be registered, and the second row shows the registered image.

[0052] By observing the registered images, we can find that the registered images of VoxelMorph, VTN, FAIM and CycleMorph are quite different from the reference images. The registered images of VTN (ADD) are close to the reference images, but some details are not natural enough. In contrast, the registered images of Multi-RegNet are better and are generally close to the reference images.

[0053] 2.2 Brain MR imaging experiment

[0054] Table 7 Comparison of the performance of different methods in brain MR image experiments, with standard deviations in brackets.

[0055]

[0056] Table 7 shows the comparison results of Multi-RegNet and other methods in terms of Dice coefficient and folding rate. As can be seen from the table, Multi-RegNet has better performance than other networks in terms of Dice coefficient and folding rate. Among them, the Dice of Multi-RegNet is 0.766, and the folding rate is 2.23e-3%. The Dice of the classic registration network VoxelMorph is 0.749, the Dice of VTN is 0.721, and the Dice of VTN (ADD) is 0.754. Compared with VoxelMorph, VTN and VTN (ADD), the Dice of Multi-RegNet is improved by 0.017, 0.045 and 0.012 respectively.

[0057] The folding rate of VoxelMorph is 0.57%, the folding rate of VTN is 0.68%, the folding rate of FAIM is 1.28e-2%, and the folding rate of CycleMorph is 2.73e-2%. Compared with VoxelMorph and VTN, the folding rates of FAIM and CycleMorph are significantly reduced, while the folding rate of the method of this embodiment is 2.23e-3%, which is an order of magnitude lower than that of FAIM and CycleMorph.

[0058] like Figure 7 As shown in the figure, the visualization results of brain MR image registration by different methods are shown. The first row shows the reference image and the image to be registered, and the second row shows the registered image. The boundaries of the ventricles, thalamus, and hippocampus are drawn in the figure, and the square box marks the area where the excellent performance of Multi-RegNet can be reflected. By observing the boundaries between the structures in the square box and some brain regions, it can be found that compared with the registered images of other methods, the registered images of this method are closer to the reference images.

[0059] In summary, compared with the prior art, the present invention realizes full automation of the registration process through multi-receptive field autocorrelation processing, multi-angle semantic extraction and deformation field regularization processing, while significantly improving the registration accuracy.

[0060] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A context-driven pyramid structure unsupervised registration system, It is characterized in that include: Two multi-resolution pyramid modules, a deformation field estimation module, an optimization module, a deformation field regularization module and an interpolation transformation module, which are respectively used to receive a reference image and an image to be registered and share weights, wherein: the deformation field estimation module estimates the deformation field at different resolutions according to the reference image, the image to be registered, the deformation field estimated at the previous level, and the multi-angle semantic information generated at the previous level, and generates the deformation field and multi-angle semantic information; the optimization module adjusts the estimated deformation field to be consistent with the resolution of the original image; the deformation field regularization module performs local Gaussian smoothing filtering on the folding area of ​​the deformation field through the deformation field regularization model, thereby reducing the folding rate of the deformation field and outputting the final deformation field; the interpolation transformation module performs feature mapping on the reference image according to the final deformation field to obtain the registered image; The deformation field estimation module includes: a feature mapping unit, a multi-receptive field autocorrelation unit, a multi-angle semantic extraction unit and a residual deformation field estimation unit, wherein: the feature mapping unit converts the image feature M to be registered output by the multi-resolution pyramid module i The deformation field φ of the previous level estimate after double upsampling i-1 Distortion processing to obtain distorted features Multi-receptive field autocorrelation units calculate distorted features With the reference image F i Correlation diagram between i ; The residual deformation field estimation unit is based on the output d of the multi-receptive field autocorrelation unit i , the deformation field φ estimated at the previous level i-1 and the multi-angle semantic information c generated by the previous level i-1 Predict the residual deformation field After that, the residual deformation field Perform two times upsampling to obtain the deformation field φ i , the multi-angle semantic extraction unit is based on the deformation field φ i , the multi-angle semantic information c generated by the previous level i-1 And the output d of the multi-receptive field autocorrelation unit i Generate multi-angle semantic information i ; The multi-receptive field autocorrelation unit includes four branches, ①, ②, ③, and ④, wherein: the first branch is composed of a 1×1×1 convolution layer and two 3×3×3 convolution layers, the second branch is composed of a 1×1×1 convolution layer and a 3×3×3 convolution layer, the third branch is composed of a 1×1×1 convolution layer, and the fourth branch is composed of a pooling layer and a 3×3×3 convolution layer; the output d of the multi-receptive field autocorrelation unit is i It is the concatenation of the results of the four branches; The multi-angle semantic extraction unit extracts the multi-angle semantic information c generated by the previous level. i-1 , the output of the multi-receptive field autocorrelation unit d i and the deformation field φ i Perform splicing processing, specifically: c i =H c ([d i ,φ i ,c i-1 ]), where: [d i ,φ i ,c i-1 ] means d i ,φ i , c i-1 The combination of the three, H c () represents the overall effect of the multi-angle semantic extraction unit; The deformation field regularization module performs local Gaussian smoothing filtering on the folded area, which is equivalent to unfolding and folding the voxel p, specifically: in: Where: * represents the convolution operation, g is a three-dimensional Gaussian kernel; the set of voxels whose Jacobian determinant in the deformation field φ is non-positive Where: Ω represents the grid area where the input image is located, det(φ(p)) represents the value of the Jacobian determinant at voxel p; δ r () is the voxel function within the range with voxel p as the center and r as the radius.

2. The context-driven pyramid structure unsupervised registration system according to claim 1, Its characteristics are: The unsupervised registration system is trained by a loss function, specifically: Where: 1 and λ 2 They are the weights of TV loss and anti-folding loss respectively; Similarity loss function for accuracy problem in medical image registration TV loss function for the folding problem in medical image registration And the anti-folding loss function Where: F represents the reference image, M represents the image to be registered, NCC() represents the normalized correlation coefficient, σ() is the ReLU activation function, that is, the positive area is linear and the negative area is all zero, det() represents the Jacobian determinant, Represents a warp operation.

3. An unsupervised registration method based on the system of claim 1 or 2, It is characterized in that The network parameters are continuously updated through iterative training, and the network's registration accuracy is improved through multi-receptive field autocorrelation processing, multi-angle semantic extraction and deformation field regularization processing, while the image's topological structure is effectively preserved. Feature mapping is performed on the image to be registered according to the predicted deformation field to obtain the final registered image.

Images