Multimodal image registration network training device and training method and registration method

Abstract

The present invention discloses a multimodal image registration network training device and training method. The device estimates free deformation model parameters through a G network, wherein the G network includes a feature extraction module, a feature fusion module, and a parameter estimation module. The feature extraction module includes M stacked pyramid convolution blocks, and each two adjacent pyramid convolution blocks are provided with a downsampling module. The feature fusion module includes M-1 upsampling modules and M-1 cascade layers, and each cascade layer is used to splice output features of the same size from the pyramid convolution block and the upsampling module and input them into the next upsampling module. The free deformation model parameters are obtained through the parameter estimation module. Based on the estimated parameters, a deformation field from a floating image to a reference image and a deformation field from a reference image to a floating image are generated. The two deformation fields are respectively applied to the floating image and the reference image, and are input into two D networks to determine whether the bidirectional registration is completed simultaneously, thereby achieving symmetry constraints and improving estimation accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and analysis, and more specifically, relates to a multimodal image registration network training device and a training method thereof, as well as a registration method. Background Art

[0002] Imaging technology is widely used in the medical field. Due to different imaging principles, each imaging method has its own advantages in reflecting the human body's anatomy or function. Medical image fusion of different modalities effectively combines the advantages of different modalities and plays a vital role in assisting disease diagnosis and treatment. Multimodal medical image registration is a prerequisite for medical image fusion and is crucial for medical image analysis and clinical research.

[0003] A common image registration method currently uses deep learning to achieve end-to-end registration. This method first constructs a specific deep learning network, which takes a reference image and a floating image as input, directly estimates the deformation field, and applies the deformation to the floating image to generate a registered image. Typical examples include VoxelMorph (abbreviated as Morph), the Volumetric Tweening Network (VTN), the Adversarial Similarity Network (ASN), and the Inverse Consistency Network (ICNet).

[0004] However, current deep learning models require learning a large number of parameters, have low estimation accuracy, and the current end-to-end registration algorithm is a one-way registration, so it is difficult to avoid changes in the topological structure caused by the folding of the deformation field during the registration process. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a multimodal image registration network training device, a training method and a registration method, which aim to avoid the change of topological structure caused by deformation field folding during the registration process and improve the estimation accuracy.

[0006] To achieve the above object, according to one aspect of the present invention, a multi-modal image registration network training device is provided, comprising a G network, a deformation field calculation module, a spatial transformation module, a D fr Network and D rf The G network includes a feature extraction module, a feature fusion module and a parameter estimation module, wherein:

[0007] The feature extraction module includes an input cascade layer for merging input features and M stacked pyramid convolution blocks PyConv connected sequentially to the input cascade layer. i , PyConv i is the i-th pyramid convolution block after the input cascade layer, i = 1, 2, ..., M, and every two adjacent pyramid convolution blocks PyConv k and PyConvk+1 There is a k-th downsampling module between them, k=1, 2, ..., M-1;

[0008] The feature fusion module includes M-1 upsampling modules and M-1 cascade layers, wherein the first upsampling module is used to upsample the output features of the M-th pyramid convolution block, and the j-th cascade layer is used to concatenate the output features of the j-th upsampling module and the Mj-th pyramid convolution block and input the concatenated features into the j+1-th upsampling module, and the output feature sizes of the j-th upsampling module and the Mj-th pyramid convolution block are the same;

[0009] The parameter estimation module includes multiple convolutional layers for obtaining free deformation model parameters based on the output features of the feature fusion module;

[0010] The deformation field calculation module is used to obtain the deformation field F from the floating image to the reference image according to the free deformation model parameters. fr and the deformation field F from the reference image to the floating image rf ;

[0011] The spatial transformation module is used to represent the structure of the original reference image PSR r and deformation field F rf Get the structural representation PSR of the deformed reference image rf , and the structural representation PSR of the original floating image f and deformation field F fr Get the structural representation PSR of the deformed floating image fr ;

[0012] The D fr The network is used to calculate the structural representation (PSR) of deformable floating images. fr and the structural representation PSR of the original reference image r The degree of registration;

[0013] The D rf The network is used to calculate the structural representation PSR of the deformed reference image rf and the structural representation PSR of the original floating image f The degree of alignment.

[0014] In one embodiment, each pyramid convolution block has multiple parallel front-end convolution layers, a cascade layer for splicing output features of all convolution layers, and a back-end convolution layer for fusing the spliced ​​features, wherein the convolution kernel sizes of the front-end convolution layers in the same pyramid convolution block are different.

[0015] In one embodiment, as the size of the received features gradually decreases, the number of front-end convolutional layers in the pyramid convolution block gradually decreases.

[0016] In one embodiment, each downsampling module includes a convolution layer with the same convolution kernel size and sampling step size.

[0017] In one embodiment, each upsampling module includes two convolutional layers and one deconvolutional layer connected in sequence, and the step size of the deconvolutional layer in the j-th upsampling module is the same as the step size of the convolutional layer in the Mj-th downsampling module.

[0018] In one embodiment, the parameter estimation module includes three sequentially connected convolutional layers, wherein the convolution kernel size and sampling step size of each convolutional layer are the same.

[0019] In one embodiment, the deformation field module uses the B-spline deformation formula to calculate the deformation field F fr and deformation field F rf .

[0020] According to another aspect of the present invention, a multimodal image registration network training method is provided, comprising:

[0021] The structural representation PSR of the original reference image in the sample set r and the structural representation PSR of the original floating image f Inputting as input features the above multimodal image registration network training device;

[0022] Make G network and D fr Network, D rf The network undergoes adversarial training.

[0023] In one embodiment,

[0024] The loss function of the G network is:

[0025] L FSGAN (G) = λ1L adv (G)+λ2L MSE (G)+λ3L smooth (G);

[0026] Among them, λ1, λ2 and λ3 represent the set penalty coefficients respectively; L adv (G) represents the adversarial loss, L MSE (G) represents the mean square error term, L smooth (G) represents the deformation field smoothing penalty term, where

[0027]

[0028]

[0029]

[0030] Among them, E represents expectation, D fr (PSR fr ) means D fr The output of the network, D rf (PSR rf ) means D rf The output of the network; ||·|| F represents the Frobenius norm, N represents the number of voxels in an image; represents the gradient of the deformation field;

[0031] D fr The loss function of the network is

[0032]

[0033] D rf The loss function of the network is

[0034]

[0035] Based on G network, D fr Network and D rf The loss function of each of the three networks is converged to a preset degree for adversarial training.

[0036] According to another aspect of the present invention, a multimodal image registration method is provided, comprising:

[0037] The structural representation PSR of the original reference image r and the structural representation PSR of the original floating image f Input the G network as input features to obtain free deformation model parameters, wherein the G network is a G network trained by the multimodal image registration network training method described above;

[0038] Obtain the deformation field F from the floating image to the reference image according to the free deformation model parameters fr ;

[0039] According to the deformation field F fr The original floating image is spatially transformed to obtain a deformed floating image for registration with the reference image.

[0040] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0041] The present invention constructs a unique G network structure, in which the feature extraction module can extract multi-scale features, the feature fusion module fuses the features, and then uses the parameter estimation module to generate free deformation model (FFD) parameters, and then generates a deformation field based on the deformation parameters, and realizes the deformation of the image through the deformation field to obtain a registered image. The G network can more accurately extract effective features suitable for image registration, thereby helping to accurately estimate the complex deformation involved in the floating image relative to the reference image. The present invention also constructs two discriminator networks, which can achieve symmetry constraints through dual judgment. The introduction of symmetry constraints can ensure that the registration of the floating image to the reference image and the registration of the reference image to the floating image are simultaneously optimal, thereby helping to avoid deformation field folding, prevent abnormal changes in the topological structure of the floating image, avoid the problem of changes in the topological structure caused by deformation field folding, and further improve the registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 1. It is an architectural diagram of a multimodal image registration network training device according to one embodiment;

[0043] Figure 2 is a structural block diagram of a G network according to an embodiment;

[0044] Figure 3 is a specific structural diagram of a G network according to an embodiment;

[0045] Figure 4 is a flowchart of the steps of a multimodal image registration method according to an embodiment;

[0046] Figure 5 are images to be registered in one embodiment; wherein (a) is a floating image, and (b) is a reference image;

[0047] Figure 6 are the image registration results of Example 1 and each comparative example, wherein (a) is the T1-T2 image registration result obtained in Example 1, (b) is the T1-T2 image registration result obtained by the method of Comparative Example 1, (c) is the T1-T2 image registration result obtained by the method of Comparative Example 2, (d) is the T1-T2 image registration result obtained by the method of Comparative Example 3, (e) is the T1-T2 image registration result obtained by the method of Comparative Example 4, (f) is the T1-T2 image registration result obtained by the method of Comparative Example 5, (g) is the T1-T2 image registration result obtained by the method of Comparative Example 6, and (h) is the T1-T2 image registration result obtained by the method of Comparative Example 7;

[0048] Figure 7 is an image to be registered in another embodiment; wherein (a) is a floating image, and (b) is a reference image;

[0049] Figure 8The present invention shows the image registration results of Example 1 and each comparative example of an embodiment, wherein (a) is the T1-T2 image registration result obtained in Example 1, (b) is the T1-T2 image registration result obtained by the method of Comparative Example 1, (c) is the T1-T2 image registration result obtained by the method of Comparative Example 2, (d) is the T1-T2 image registration result obtained by the method of Comparative Example 3, (e) is the T1-T2 image registration result obtained by the method of Comparative Example 4, (f) is the T1-T2 image registration result obtained by the method of Comparative Example 5, (g) is the T1-T2 image registration result obtained by the method of Comparative Example 6, and (h) is the T1-T2 image registration result obtained by the method of Comparative Example 7. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0051] In response to the time-consuming iterative process in the design of traditional registration methods, the high dimensionality of estimated parameters involved in existing deep learning methods based on deformation field estimation, and the problem of deformation field folding in one-way registration based on deep learning, the present invention proposes a multimodal image registration network training device, namely a generative adversarial network model combined with free deformation model and symmetry constraint (Generative Adversarial Networks Combined with FFD and SymmetryConstraint, FSGAN), to achieve end-to-end unsupervised non-rigid multimodal medical image registration.

[0052] like Figure 1 The diagram shows the architecture of the multimodal image registration network training device, which mainly includes G network, deformation field calculation module, spatial transformation module, D fr Network and D rf The G network generates FFD parameters based on the original reference image and the floating image, and the deformation field calculation module calculates the deformation field F based on the FFD parameters. fr and deformation field F rf , the spatial transformation module is based on the deformation field F fr The structural representation map PSR of the deformed floating image is obtained by combining the original floating image fr , based on the deformation field F rf The structural representation map PSR of the deformed reference image is obtained by combining the original reference image rf , and then input the two structural representations into D fr Network and Drf Network, D fr Used to determine whether the registration of the floating image to the reference image is complete, D rf It is used to judge whether the registration of the reference image to the floating image is completed at the same time, and the double judgment is used to realize the bidirectional registration, thereby avoiding the problem of topological structure change caused by the folding of the deformation field. fr and D rf It can be composed of three convolutional layers with a kernel size of 5×5×5 and a stride of 2 and a fully connected layer.

[0053] Among them, the G network includes a feature extraction module, a feature fusion module and a parameter estimation module, which is used to represent the PSR according to the original reference image structure. r and the structural representation PSR of the original floating image f Extract FFD parameters. Therefore, after obtaining the original reference image I r and the original floating image I f After that, a structural representation of the corresponding image must be constructed. By calculating the structural representation of the reference image and the floating image, the multimodal medical image registration problem is transformed into a single-modal medical image registration problem, reducing the impact of nonlinear grayscale differences between multimodal medical images.

[0054] Specifically, the structural representation graph of the original image can be obtained according to the existing method. In this embodiment, the structural representation graph can be constructed by a trained three-dimensional PCANet network. Specifically, for all images in the training set, 3×3×3 image blocks are extracted without intervals, and all image blocks are vectorized; all vectors are combined into a matrix, and the matrix is ​​subjected to PCA (principal component analysis) operation to obtain the eigenvalues ​​and eigenvectors of the matrix; the eigenvalues ​​are arranged in order from large to small, the eigenvectors corresponding to the first 8 eigenvalues ​​are selected, and the eigenvectors are matrixed to obtain the convolution kernel of the first-level PCANet. The original image is convolved with the convolution kernel of the first-level PCANet, and the result is calculated in the same way as the first level to obtain the convolution kernel of the second-level PCANet and obtain the trained PCANet. The features of the original image are extracted using the trained PCANet, and all the features of the first level and all the features of the second level are fused to obtain the fused features F1 of the first level and the fused features F2 of the second level; F1 and F2 are used to construct the structural representation graph.

[0055] Select multiple pairs of medical images in the training set and obtain the original reference images I r Structural representation diagram PSR r and the structural representation PSR of the original floating image f .

[0056] Among them, such as Figure 2The following is a block diagram of the G network structure, where the feature extraction module includes an input cascade layer and M stacked pyramid convolution blocks PyConv i , PyConv i is the i-th pyramid convolution block after the input cascade layer, i = 1, 2, ..., M. The input cascade layer is used to cascade the input features. In this invention, it is the structural representation map PSR of the original reference image. r and the structural representation PSR of the original floating image f The stacked pyramid convolution blocks are used to gradually extract image features. Each adjacent two pyramid convolution blocks PyConv k and PyConv k+1 A k-th downsampling module, where k = 1, 2, ..., M-1, is provided between the blocks to downsample the input features of the pyramid convolution blocks. The feature fusion module consists of M-1 upsampling modules and M-1 cascade layers. The upsampling modules are used for upsampling, while the cascade layers are used for merging features. The first upsampling module is used to upsample the output features of the M-th pyramid convolution block. The final output of the feature extraction module serves as the input to the feature fusion module. The j-th cascade layer concatenates the output features of the j-th upsampling module and the Mj-th pyramid convolution block before inputting them into the j+1-th upsampling module. The output features of the j-th upsampling module and the Mj-th pyramid convolution block have the same size. After obtaining the concatenated features, they are fused before being input into the next upsampling module. The parameter estimation module comprises multiple convolutional layers and is used to derive FFD parameters based on the output features of the feature fusion module. The specific structure of each module in the G network can be flexibly adjusted according to needs.

[0057] like Figure 3 The figure shows a specific structural diagram of the G network in one embodiment. Each pyramid convolution block has multiple parallel front-end convolution layers, a cascade layer for splicing the output features of all convolution layers, and a back-end convolution layer for fusing the spliced ​​features. The convolution kernel sizes of the front-end convolution layers in the same pyramid convolution block are different, and different convolution kernels are used to extract features of different receptive fields of the image. At the same time, as the number of downsampling modules increases, the size of image features becomes smaller and smaller. Therefore, for stacked pyramid convolution blocks, the types of convolution kernels in the PyConv block will decrease as the feature scale decreases. In the last PyConv block, there is only one type of convolution kernel to adapt to the reduction in feature size. Preferably, each time the convolution kernel is reduced, the convolution kernel with the largest size is removed. In one embodiment, each downsampling module in the feature extraction module is the same and is used to reduce the features by the same proportion.

[0058] In one embodiment, each upsampling module includes a convolutional layer and a deconvolutional layer connected in sequence, forming a convolution-deconvolution block (Conv-DeConv). Preferably, each upsampling module in the feature fusion module is identical and is used to amplify the features at the same ratio. In one embodiment, the step size of the deconvolution layer in the jth upsampling module can be set to be the same as the step size of the convolution layer in the Mjth downsampling module to achieve the same feature size in the same cascade layer.

[0059] In one embodiment, the parameter estimation module includes three sequentially connected convolutional layers, wherein the convolution kernel size and sampling step size of each convolutional layer are the same.

[0060] In one embodiment, the deformation field F can be calculated using the B-spline deformation formula. fr and deformation field F rf , the specific calculation process can refer to the existing technology.

[0061] In a specific embodiment, Figure 3 As shown, M=4, that is, the G network includes 4 stacked pyramid convolution blocks, 3 downsampling modules, 3 upsampling modules and 3 cascade layers.

[0062] The four stacked pyramid convolution blocks are PyConv1, PyConv2, PyConv3, and PyConv4. PyConv1 includes four parallel front-end convolutional layers with kernel sizes of 3×3×3, 5×5×5, 7×7×7, and 9×9×9, respectively. PyConv2 includes three parallel front-end convolutional layers with kernel sizes of 3×3×3, 5×5×5, and 7×7×7, respectively. PyConv3 includes two parallel front-end convolutional layers with kernel sizes of 3×3×3 and 5×5×5, respectively. PyConv4 includes one front-end convolutional layer with kernel sizes of 3×3×3. Different convolution kernels can extract features of different receptive fields. Each pyramid convolution block also includes a cascade layer and a back-end convolution layer, wherein the cascade layer is used to merge the output structures of different front-end convolution layers in the current pyramid convolution block to form a merged feature, while the back-end convolution layer is used to fuse the merged features. Specifically, the convolution kernel size of the back-end convolution layer is 1×1×1, and the step size is 1. In one embodiment, in each PyConv block, a 3×3×3 dilated convolution is used instead of a large-size convolution kernel (for example, 5×5×5, 7×7×7, and 9×9×9 convolution kernels) to reduce the number of network parameters.

[0063] The three downsampling modules are all convolutional layers with a size of 3×3×3 and a stride of 2. Adjacent pyramid convolution blocks are connected using convolutional layers with a size of 3×3×3 and a stride of 2. Their function is to downsample the image to obtain features at different scales of the image.

[0064] The three upsampling modules are Conv-DeConv1, Conv-DeConv2 and Conv-DeConv3. Each upsampling module has the same structure, consisting of two convolutional layers with a convolution kernel size of 3×3×3 and a stride of 1, and one deconvolution layer with a deconvolution kernel size of 5×5×5 and a stride of 2. The convolution layer integrates the fusion features at this scale, and the deconvolution layer upsamples the features. The input of the first convolution-deconvolution block Conv-DeConv1 is the smallest feature obtained by the feature extraction module.

[0065] The three cascade layers concatenate the output of each convolution-deconvolution block with the features of the same size obtained by the feature extraction module. The output results are fused through a 1×1×1 convolution layer and input into the next convolution-deconvolution block.

[0066] The parameter estimation module consists of L convolutional layers with a convolution kernel size of 3×3×3 and a stride of 2 to obtain the FFD parameters.

[0067] Based on the registration model established above, the problem of image topology change caused by deformation field folding during the registration process can be avoided, and fast and accurate image registration can be achieved.

[0068] Accordingly, the present invention also protects a multimodal image registration network training method, which comprises:

[0069] The structural representation PSR of the original reference image in the sample set r and the structural representation PSR of the original floating image f A registration network training device that inputs the multimodal image as input features;

[0070] Make G network and D fr Network, D rf The network undergoes adversarial training.

[0071] Among them, when training the G network, the associated D fr Network and D rf The output of the network, when training the discriminator, associate the output of the G network so that the G network and D fr Network, D rf The network is trained adversarially to make the three networks reach the optimal state at the same time.

[0072] Specifically, the loss functions of the three can be set separately.

[0073] Among them, the loss function of the G network consists of three items: adversarial loss function, mean square error, and smoothness constraint:

[0074] L FSGAN (G) = λ1L adv (G)+λ2L MSE (G)+λ3L smooth (G);

[0075] Among them, λ1, λ2 and λ3 represent the penalty coefficients, which are used to balance the three components in the loss function. adv (G) represents the adversarial loss, which is constructed as follows:

[0076]

[0077] Among them, E represents expectation.

[0078] L MSE (G) indicates PSR fr and PSR r The mean square error and PSR between rf and PSR f The sum of mean squared errors is constructed as follows:

[0079]

[0080] Among them, ||·|| F represents the Frobenius norm, and N represents the number of voxels in an image.

[0081] L smooth (G) represents the penalty term related to deformation field smoothing, which is defined as follows:

[0082]

[0083] in, Represents the gradient of the deformation field.

[0084] D fr The loss function is:

[0085]

[0086] D rf The loss function is:

[0087]

[0088] According to the above loss function, FSGAN is trained and PSR is r and PSR fInput into the G network, estimate the FFD parameters, and based on the estimated parameters, use the B-spline function to generate two deformation fields: the deformation field F from the floating image to the reference image fr And the deformation field F from the reference image to the floating image rf , apply these two deformation fields to PSR respectively f and PSR r , get the deformed floating image PSR fr And the deformed reference image PSR rf . r and PSR fr Input to D fr In the process, it is determined whether the registration of the floating image to the reference image is completed; at the same time, the PSR f and PSR rf Input to D rf In the process, it is judged whether the registration of the reference image to the floating image is completed; through double judgment, the symmetry constraint is realized. Alternately train the G network and the D fr 、D rf network, making the three networks reach the optimal state at the same time.

[0089] The above training method first uses PSR to calculate the structural representation of the reference image and the floating image, thereby reducing the nonlinear grayscale differences between multimodal medical images; G is used to estimate the FFD model parameters that represent the spatial position relationship between the reference image and the floating image, thereby reducing the number of estimated parameters; finally, two D networks are used to introduce symmetry constraints to avoid the problem of image topology changes introduced by deformation field folding in one-way registration.

[0090] Accordingly, the present invention also protects a multimodal image registration method, which mainly includes:

[0091] The structural representation PSR of the original reference image r and the structural representation PSR of the original floating image f Input the G network as input features to obtain free deformation model parameters, wherein the G network is a G network trained by the above-mentioned registration network training method;

[0092] Obtain the deformation field F from the floating image to the reference image according to the free deformation model parameters fr ;

[0093] According to the deformation field F fr The original floating image is spatially transformed to obtain a deformed floating image for registration with the reference image.

[0094] After obtaining the deformed floating image, it can be aligned with the reference image.

[0095] refer to Figure 4As shown, after obtaining the trained G network, it is used for multimodal image registration, usually using the reference image as a benchmark to register the floating image with the reference image. First, obtain the original floating image I f Structural representation diagram PSR f and the original reference image I r Structural representation diagram PSR r , input the two structural representations into the trained G network at the same time, extract the FFD parameters, and since the floating image is registered with the reference image, only the deformation field F of the floating image needs to be calculated. fr , and based on the deformation field F fr Perform spatial transformation on the original floating image to obtain the registered image I that can be used for registration g , and then can register the image I g Register with the reference image.

[0096] The present invention is compared with several currently commonly used methods below to verify the advantages of the present invention.

[0097] Example 1

[0098] Build attached Figure 4 The registration network shown calculates the reference image I r and floating image I f Structural representation diagram PSR of r and PSR f The structural characterization results are input into the above-mentioned registration model to generate FFD model parameters. Based on the model parameters and the B-spline function, the deformation field from the floating image to the reference image is obtained. The deformation field is applied to the floating image to obtain the registered image.

[0099] Comparative Example 1

[0100] Registration was performed using the ICNet method with the following parameters: batch size 2, learning rate 0.0001, and momentum 0.5.

[0101] Comparative Example 2

[0102] The registration was performed according to the ASN method in (Med. Image Anal. 58(2019)101545). The specific parameters were: batch size 2, learning rate 0.0001, and momentum 0.5 during training.

[0103] Comparative Example 3

[0104] The VTN method in (IEEE J BIOMED HEALTH. 24(5)(2019)1394-1404) is used for registration. The training parameters are: batch size 2, learning rate 0.0001, and momentum 0.5.

[0105] Comparative Example 4

[0106] The registration is performed using the morph method described in (CVPR.2018, pp.9252-9260). The training parameters are: batch size 2, learning rate 0.0001, and momentum 0.5.

[0107] Comparative Example 5

[0108] The registration is performed according to the SSC method in (MICCAI.2013, pp.187-194.) with the following parameters: the image block size is 3×3.

[0109] Comparative Example 6

[0110] The registration was performed according to the MIND method in (Med. Image Anal. 16(7)(2012)1423-1435.) The specific parameters are: the image block size is 3×3.

[0111] Comparative Example 7

[0112] Registration was performed using the ESSD method described in Med. Image Anal. 16(1)(2012)1-17. The specific parameters are: a 9×9 image block was selected, and the entropy corresponding to the image block was calculated using Gaussian weights, local normalization, and Parzen window estimation, thereby obtaining the ESSD corresponding to the entire image.

[0113] Analysis of experimental results.

[0114] In a comparative experiment, to further demonstrate the advantages of the present invention, we compared the registration accuracy of Example 1 with that of Comparative Examples 1 to 7. The registration accuracy was evaluated using the TRE value, where TRE is defined as:

[0115]

[0116] Where |P| represents the number of selected landmark points, P(f) and P(r) represent the positions of corresponding landmark points selected on the floating image and the reference image respectively, and P(F fr ) represents the deformation field F fr The value at the corresponding marker position.

[0117] In this comparative experiment, a simulation experiment was conducted based on the HGG dataset to test the registration accuracy of each registration algorithm. Figure 5 As shown in (a); T2 is the reference image, as shown in Figure 5 (b) in the figure. Table 1 lists the standard deviation and mean of the TRE values ​​obtained by each registration method. As can be seen from Table 1, when registering multimodal MR images, the TRE mean and standard deviation provided by Example 1 are lower than those of the comparison algorithm, indicating that the method proposed in this invention has the highest registration accuracy and the best robustness among all the compared methods.

[0118] Table 1 TRE values ​​of the registration results based on the HGG dataset

[0119]

[0120] The registration results of Example 1 and Comparative Examples 1-7 for the floating image T1 and the reference image T2 are as follows: Figure 6 As shown in (a) to (h), Figure 6 (a) is the T1-T2 image registration result obtained in Example 1 of the present invention. Figure 6 (b) is the T1-T2 image registration result obtained by the method of comparative example 1. Figure 6 (c) is the T1-T2 image registration result obtained by the method of comparative example 2. Figure 6 (d) is the T1-T2 image registration result obtained by the method of comparative example 3. Figure 6 (e) is the T1-T2 image registration result obtained by the method of comparative example 4. Figure 6 (f) in the figure is the T1-T2 image registration result obtained by the method in comparative example 5. Figure 6 (g) is the T1-T2 image registration result obtained by the method of Comparative Example 6. Figure 6 (h) is the T1-T2 image registration result obtained by the method of Comparative Example 7. It can be seen from the registration result diagram that the registration algorithm proposed in the present invention is significantly superior to the comparison algorithm in correcting the outer contour and internal details of the image, proving that the present invention not only uses bidirectional registration to more accurately find the spatial correspondence between the reference image and the floating image, but also the extracted FFD model parameters can more effectively realize the recovery of global and local deformations, thereby generating a registered image with better correspondence to the anatomical structure of the reference image.

[0121] In another comparative experiment, to further demonstrate the advantages of the present invention, we compared the registration accuracy of Example 1 with that of Comparative Examples 1-7. The registration accuracy was evaluated using the dice value, where dice is defined as:

[0122]

[0123] Where |A| represents the number of voxels in image A, |B| represents the number of voxels in image B, and |A∩B| represents the number of voxels in the common part of image A and image B.

[0124] In this comparative experiment, a simulation experiment was conducted based on the IXI dataset to test the registration accuracy of each registration algorithm. In this embodiment, the 3D-slicer was first used to segment the white matter, gray matter and cerebrospinal fluid (CSF) in the registration image and the reference image, and their dice values ​​were calculated to judge the accuracy of the registration. Table 2 lists the standard deviation and mean of the dice values ​​of each part obtained by each algorithm. As can be seen from Table 2, compared with the comparative registration algorithm, Example 1 provides the highest mean and the lowest standard deviation, which shows that the method proposed in the present invention has the highest registration accuracy and the best robustness among all compared methods.

[0125] Table 2 Dice values ​​of registration results based on the IXI dataset

[0126]

[0127]

[0128] In order to more intuitively demonstrate the superiority of the present invention over other methods, we provide visual effect diagrams of the corresponding registration images of Example 1 and Comparative Examples 1-7. Figure 7 shown. Figure 7 (a) is the floating image T1 used in Example 1 of the present invention and Comparative Examples 1-7, Figure 7 (b) is the reference image T2 used in the embodiment of the present invention and comparative examples 1-7. Figure 8 (a) is the T1-T2 image registration result obtained by the method of Example 1 of the present invention, Figure 8 (b) is the T1-T2 image registration result obtained by the method of comparative example 1. Figure 8 (c) is the T1-T2 image registration result obtained by the method of comparative example 2. Figure 8 (d) is the T1-T2 image registration result obtained by the method of comparative example 3. Figure 8 (e) is the T1-T2 image registration result obtained by the method of comparative example 4. Figure 8 (f) in the figure is the T1-T2 image registration result obtained by the method in comparative example 5. Figure 8 (g) is the T1-T2 image registration result obtained by the method of Comparative Example 6. Figure 8(h) is the T1-T2 image registration result obtained by the method of comparative example 7. As can be seen from the figure, compared with the floating image, the anatomical spatial correspondence between the registration results of all registration methods and the reference image is improved. For the correction of deformation of the external contour of the image (for example, the area near the skull), the method proposed by the present invention is significantly better than other registration methods. For the correction of deformation of the detailed area inside the image, Figure 8 The frontal lobe of the brain is marked with a frame and then magnified. The results show that compared with other registration methods, the registration results obtained by the present invention are more consistent with the anatomical structure of the reference image.

[0129] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal image registration network training device, characterized in that: Including G network, deformation field calculation module, space transformation module, D fr Network and D rf The G network includes a feature extraction module, a feature fusion module and a parameter estimation module, wherein: The feature extraction module includes an input cascade layer for merging input features and M stacked pyramid convolution blocks PyConv connected sequentially to the input cascade layer. i , PyConv i is the i-th pyramid convolution block after the input cascade layer, i = 1, 2, ..., M, and every two adjacent pyramid convolution blocks PyConv k and PyConv k+1 There is a k-th downsampling module between them, k=1,2,...,M-1; The feature fusion module includes M-1 upsampling modules and M-1 cascade layers, wherein the first upsampling module is used to upsample the output features of the M-th pyramid convolution block, and the j-th cascade layer is used to concatenate the output features of the j-th upsampling module and the Mj-th pyramid convolution block and input the concatenated features into the j+1-th upsampling module, and the output feature sizes of the j-th upsampling module and the Mj-th pyramid convolution block are the same; The parameter estimation module includes multiple convolutional layers for obtaining free deformation model parameters based on the output features of the feature fusion module; The deformation field calculation module is used to obtain the deformation field F from the floating image to the reference image according to the free deformation model parameters. fr and the deformation field F from the reference image to the floating image rf ; The spatial transformation module is used to represent the structure of the original reference image PSR r and deformation field F rf Get the structural representation PSR of the deformed reference image rf , and the structural representation PSR of the original floating image f and deformation field F fr Get the structural representation PSR of the deformed floating image fr ; The D fr The network is used to calculate the structural representation (PSR) of deformable floating images. fr and the structural representation PSR of the original reference image r The degree of registration; The D rf The network is used to calculate the structural representation PSR of the deformed reference image rf and the structural representation PSR of the original floating image f The degree of alignment.

2. The multimodal image registration network training device according to claim 1, wherein: Each pyramid convolution block has multiple parallel front-end convolution layers, a cascade layer for splicing the output features of all convolution layers, and a back-end convolution layer for fusing the spliced ​​features. The convolution kernel sizes of the front-end convolution layers in the same pyramid convolution block are different.

3. The multimodal image registration network training device according to claim 2, wherein: As the size of the received features decreases, the number of front-end convolutional layers in the pyramid convolution block gradually decreases.

4. The multimodal image registration network training device according to claim 1, wherein: Each downsampling module includes convolutional layers with the same kernel size and sampling stride.

5. The multimodal image registration network training device according to claim 1, wherein: Each upsampling module consists of two convolutional layers and one deconvolution layer connected in sequence. The step size of the deconvolution layer in the j-th upsampling module is the same as the step size of the convolution layer in the Mj-th downsampling module.

6. The multimodal image registration network training device according to claim 1, wherein: The parameter estimation module includes three sequentially connected convolutional layers, wherein the convolution kernel size and sampling step size of each convolutional layer are the same.

7. The multimodal image registration network training device according to claim 1, wherein: The deformation field calculation module uses the B-spline deformation formula to calculate the deformation field F fr and deformation field F rf .

8. A multimodal image registration network training method, characterized in that: include: The structural representation PSR of the original reference image in the sample set r and the structural representation PSR of the original floating image f a registration network training device for a multimodal image according to any one of claims 1 to 7 input as input features; Make G network and D fr Network, D rf The network undergoes adversarial training.

9. The multimodal image registration network training method according to claim 8, wherein: The loss function of the G network is: L FSGAN (G)=λ1L adv (G)+λ2L MSE (G)+λ3L smooth (G); Among them, λ1, λ2 and λ3 represent the set penalty coefficients respectively; L adv (G) represents the adversarial loss, L MSE (G) represents the mean square error term, L smooth (G) represents the deformation field smoothing penalty term, where Among them, E represents expectation, D fr (PSR fr ) means D fr The output of the network, D rf (PSR rf ) means D rf The output of the network; ||·|| F represents the Frobenius norm, N represents the number of voxels in an image; represents the gradient of the deformation field; D fr The loss function of the network is D rf The loss function of the network is Based on G network, D fr Network and D rf The loss function of each of the three networks is converged to a preset degree for adversarial training.

10. A multimodal image registration method, characterized in that: include: The structural representation PSR of the original reference image r and the structural representation PSR of the original floating image f Input the G network as input features to obtain free deformation model parameters, wherein the G network is a G network trained by the multimodal image registration network training method according to claim 8 or 9; Obtain the deformation field F from the floating image to the reference image according to the free deformation model parameters fr ; According to the deformation field F fr The original floating image is spatially transformed to obtain a deformed floating image for registration with the reference image.

Images