An image registration method

Abstract

The present invention relates to an image registration method, and relates to the field of computer vision. The method can effectively improve the accuracy of image registration under large displacement without network cascading and multiple interpolations. The method comprises the following steps: after splicing a fixed image and a floating image, inputting a trained image registration network, using the image registration network to learn the difference between the floating image and the fixed image, and outputting a registered image; the image registration network comprises a backbone network, a multi-convolution fusion module, and a spatial transformation layer; wherein: the backbone network obtains a first displacement field based on the spliced fixed image and floating image, and interpolates the floating image based on the first displacement field to obtain a first feature map; the multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain a second displacement field, and then adds the first and second displacement fields to obtain a third displacement field; and the spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate the registered image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a three-dimensional medical image registration method, specifically to an image registration method based on multi-convolution fusion, which can be used to solve the problem of low registration accuracy when a floating image undergoes a larger degree of non-rigid deformation than a fixed image in image registration. Background Art

[0002] The emergence of medical imaging technology has fueled the development of precision medicine. The processing and analysis of medical images has gradually become a major challenge in medical treatment. The human brain, with its numerous nerves and blood vessels and relatively small size, is the control center of the human body. Therefore, accurate diagnosis and treatment are crucial. Magnetic resonance imaging (MRI), with its high resolution, diverse image types, arbitrary layering, and non-invasive imaging, has become increasingly popular in the clinical treatment of brain diseases. MRI image registration has become a research hotspot in medical image processing. MRI registration aims to match points in the same space between two images through spatial transformation, achieving information fusion. By registering MRI images of the brain at different times, the progression of lesions can be tracked, and multi-directional slices are more conducive to lesion localization. Only by accurately localizing lesions and tracking their progression can better diagnose the condition, develop more detailed treatment plans, guide surgical and radiotherapy treatments, and improve surgical and treatment accuracy.

[0003] Alexnet's success in the ImageNet 2012 challenge has led to rapid progress in deep learning in various computer vision fields. In image registration, deep learning was first used to drive the iteration of traditional registration algorithms. Soon thereafter, image registration based on reinforcement learning appeared. With the demand for fast registration, there was a one-step conversion based on deep learning algorithms. Supervised learning methods were applied to image registration to directly predict the conversion parameters instead of depth-driven iteration. However, this method requires a large number of high-quality gold standards, because the trained model depends to a large extent on the quality of the gold standard, and theoretically it is difficult for the model's effect to exceed the gold standard, and obtaining a high-quality gold standard is precisely a difficulty. Based on this, unsupervised learning methods were used in the field of image registration, avoiding the bottleneck problem of supervised learning.

[0004] The registration of human brain magnetic resonance images often suffers from the problem of low registration accuracy under large displacements. That is, when the floating image undergoes a large degree of non-rigid deformation compared to the fixed image, the spatial transformation obtained is not accurate enough due to the limited processing ability of the single-stage model for large displacements. This can lead to the inability to accurately locate the lesion position and even mismatches, which brings certain difficulties to the diagnosis of the disease and the formulation of treatment plans. Summary of the Invention

[0005] In order to address the problem of poor registration accuracy when the floating image undergoes a larger degree of non-rigid deformation compared to the fixed image in the above-mentioned image registration, the existing methods require cascading multi-level networks, layered training or multiple interpolation, which makes the network structure more complex. Multiple continuous interpolations introduce greater errors and cause interpolation artifacts. The purpose of the present invention is to propose a method that can effectively improve the image registration accuracy under large displacement without performing network cascading and multiple interpolations.

[0006] In order to achieve the above object, the present invention proposes an image registration method comprising the following steps:

[0007] After the fixed image and the floating image are stitched together, they are input into the trained image registration network, which uses the image registration network to learn the difference between the floating image and the fixed image and output the registered image;

[0008] The image registration network includes a backbone network, a multi-convolution fusion module and a spatial transformation layer; wherein:

[0009] A backbone network obtains a first displacement field based on the spliced fixed image and the floating image, and interpolates the floating image based on the first displacement field to obtain a first feature map;

[0010] The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field. The first and second displacement fields are then added together to obtain the third displacement field.

[0011] The spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate a registered image.

[0012] In the above technical solution, the backbone network is an encoder-decoder, which includes several three-dimensional convolutional layers, each of which is followed by a batch normalization layer and a nonlinear activation function Leaky ReLU.

[0013] In the above technical solution, before the floating image and the fixed image are input into the backbone network, they are first normalized using the maximum and minimum method to convert the pixel values of the image to between [0, 1].

[0014] In the above technical solution, multi-scale position information is extracted and fused based on the first feature map to obtain the second displacement field, including:

[0015] Use multiple parallel dilated convolutional layers with different dilation rates, and construct convolution kernels with different receptive fields with different dilation rates.

[0016] After fusing the features extracted by each convolution kernel, global average pooling is performed and the dimension is reduced through a fully connected layer. The fully connected layer consists of several branches, one for each convolution kernel.

[0017] After the fully connected layer, weighted fusion is performed to obtain the second displacement field.

[0018] In the above technical solution, after the fully connected layer, a normalized exponential function is applied to obtain the weighted scores of the channel information extracted by each convolution kernel, and the scores are fused to generate the second displacement field.

[0019] In the above technical solution, the spatial transformation layer is implemented based on the grid generator and sampler of the spatial transformation network;

[0020] The third displacement field φ is used as the transformation parameter. The grid generator obtains the correspondence between the pixel coordinates before and after the transformation based on the third displacement field φ, and then uses the sampler to generate the registration image in a trilinear interpolation manner.

[0021] In the above technical solution, the third displacement field is smoothed before the spatial transformation;

[0022] The smoothing process is to calculate the displacement field of the differential homeomorphism by integrating the third displacement field φ. The relationship between the displacement field positions at different times t is:

[0023] φ (t+Δt) =φ (t) +v(t+Δt)

[0024] Among them, v is the change in displacement field at different times, and Δt is the time change amount;

[0025] In the calculation, by dividing t∈[0,1] into 2 n The segment sets the number of spatial transformations to n.

[0026] In the above technical solution, the image registration network training includes the following steps:

[0027] The fixed image is used as the true value, and the image output by the image registration network is used as the predicted value;

[0028] Use the negative value of the cross-correlation coefficient as the similarity loss function, take the predicted value and the true value as the input of the similarity loss function, and calculate the similarity loss l sim ;

[0029] Calculate the smoothness loss l of the displacement field predicted by the network smooth ;

[0030] Calculate the overall loss l = l sim +λl smooth , the Adam optimization algorithm is used to update the network parameters, and λ is the regularization coefficient of the smoothness loss.

[0031] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any one of the above methods.

[0032] In a third aspect, the present invention proposes an image registration system, comprising a backbone network, a multi-convolution fusion module, and a spatial transformation layer; wherein:

[0033] A backbone network obtains a first displacement field based on the spliced fixed image and the floating image, and interpolates the floating image based on the first displacement field to obtain a first feature map;

[0034] The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field. The first and second displacement fields are then added together to obtain the third displacement field.

[0035] The spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate a registered image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 、 one Schematic diagram of image registration network under different implementation methods. DETAILED DESCRIPTION

[0037] To address the limited ability of single-stage models to handle large displacements, Shengyu Zhao, Yue Dong, Eric I-Chao Chang, Yan Xu, and others proposed a registration method based on recursive cascade networks in their paper "Recursive cascaded networks for unsupervised medical image registration." (Proceedings of the IEEE International Conference on Computer Vision, 2019). This method uses a cascaded approach to sequentially register images from coarse to fine. Each cascaded network spatially transforms the input floating image to obtain a distorted floating image, which serves as the floating image for the next cascaded network. The cascaded networks have the same structure. However, this method interpolates the floating image multiple times, which can lead to interpolation artifacts.

[0038] In their paper "Networks for Joint Affine and Non-Parametric Image Registration" (Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2019), Zhengyang Shen, Xu Han, Zhenlin Xu, Marc Niethammer, and others proposed a model that combines affine registration and vector momentum to parameterize static velocity fields. The model has three stages. In the first stage, a multi-step affine network is used to predict an affine transformation parameter; in the second stage, an encoder-decoder network is used to generate a momentum, which can be used to calculate a velocity field through smoothing; in the third stage, an independent graph-based differential homeomorphism transformation component is used to perform parameter-free fine-tuning based on the current transformation mapping. In this method, the multi-step affine network and the momentum generation network need to be trained separately, each learning their own way. The network training is complex, and the cascade effect is limited.

[0039] To address the problem of poor registration accuracy when floating images undergo larger non-rigid deformations than fixed images in human brain MRI registration, existing single-stage models have limited processing capabilities for large displacements, resulting in inaccurate spatial transformations. This can lead to inability to accurately locate lesions and even mismatches, posing certain difficulties for diagnosis and treatment planning. As mentioned above, although there are methods that attempt to address this problem, these methods require cascading multi-level networks, layered training, or multiple interpolation, making the network structure more complex. Multiple interpolation introduces greater errors and produces interpolation artifacts. The technology proposed in the present invention can effectively improve the accuracy of human brain MRI registration under large displacements without network cascading or multiple interpolation.

[0040] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] The terms "first," "second," and "third" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to implicitly specify the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," or "third" may explicitly or implicitly include one or more of such features.

[0042] See also Figure 1As shown in FIG, after the floating image and the fixed image are spliced, they are input into the trained image registration network, and the image registration network is used to learn the difference between the floating image and the fixed image, and the registered image is output. Figure 1 In [1], the backbone network in the image registration network is the encoder-decoder. In addition, the image registration network also includes a multi-convolution fusion module, a differential homeomorphism integration module, and a spatial transformation layer; where:

[0043] The floating image and the fixed image are spliced and input into the encoder-decoder to obtain a first displacement field, and the floating image is interpolated and preliminarily transformed based on the first displacement field to obtain a first feature map;

[0044] The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field. The first and second displacement fields are then added together to obtain the third displacement field.

[0045] The differential homeomorphism integration module smoothes the third displacement field φ and calculates the displacement field of the differential homeomorphism by integration. The relationship between the displacement field positions at different times t is:

[0046] φ (t+Δt) =φ (t) +v(t+Δt)

[0047] Among them, v is the change in displacement field at different times, and Δt is the time change amount;

[0048] In the calculation, by dividing t∈[0,1] into 2 n The segment sets the number of spatial transformations to n.

[0049] The spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate a registered image.

[0050] The above-mentioned first displacement field can be obtained not only by the encoder-decoder network, but also by any network structure for the purpose of image registration. The present invention can be used directly after the existing single-stage registration network without changing its original network structure, and has the advantage of convenient deployment.

[0051] If the third displacement field is not smoothed by the diffeomorphism integration module before the spatial transformation layer, image registration can also be achieved.

[0052] The displacement field obtained by the above-mentioned smoothing process is diffeomorphic, reversible and topological, avoids position mapping that does not conform to reality, and has the advantage of high precision.

[0053] The following content is the implementation steps of a method for using an image registration network to realize the registration of human brain magnetic resonance image sequences. However, this image registration network is not limited to the registration of human brain magnetic resonance images, but can be applied to any type of image registration; the multi-convolution fusion module therein can not only be inserted after the backbone network of the current registration network, but also any single-stage network for the purpose of image registration.

[0054] Step 1: Image preprocessing

[0055] (1a) The obtained human brain magnetic resonance image sequence in DICOM (Digital Imaging and Communications in Medicine) format is converted into an array type using the medical image processing toolkit SimpleITK (Simple Insight Segmentation and Registration Toolkit).

[0056] (1b) Use the maximum and minimum method to normalize the image pixel values to [0, 1], x, are the pixel values before and after normalization, x max , x min are the maximum and minimum pixel values before normalization.

[0057]

[0058] Step 2: Build an image registration network

[0059] (2a) Building a 3D image registration backbone network

[0060] In this embodiment, the encoder-decoder structure is used as the backbone network to extract features from the spliced floating image and fixed image to obtain a first displacement field. The floating image is spatially transformed according to the first displacement field to obtain a first feature map. The encoding stage of the encoder and the decoding stage of the decoder both contain several three-dimensional convolutional layers. The number of feature channels in the encoding stage is [16, 32, 32, 32], and the number of feature channels in the decoding stage is set to [32, 32, 16, 16, 16]. The channels are doubled and the size is halved in the encoding stage. The doubled-size features after upsampling in the decoding stage are spliced along the channel dimension with the features in the encoding stage and decoded layer by layer. Each convolutional layer is followed by a batch normalization layer and a nonlinear activation function Leaky ReLU (Leaky Rectified Linear Unit).

[0061] (2b) Building a multi-convolution fusion module

[0062] The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field, and then adds the first displacement field and the second displacement field to obtain the third displacement field.

[0063] In its implementation, this module uses multiple parallel atrous convolutional layers with varying dilation rates, with different dilation ratios used to construct convolutional kernels with varying receptive fields. The features extracted by each kernel are fused, then globally average pooled and passed through a fully connected layer for dimensionality reduction. Further processing occurs through several branches, each containing a fully connected layer followed by a normalized exponential function to obtain a weighted score for the channel information extracted by each kernel, which is then fused to generate the final result.

[0064] (2c) Building the spatial transformation layer

[0065] The spatial transformation layer transforms the floating image based on the third displacement field to generate the registered image. This layer is implemented using a spatial transformer network (STN), which consists of three components: a localization network, a grid generator, and a sampler. The localization network is used for parameter prediction, the grid generator for coordinate mapping, and the sampler for pixel acquisition.

[0066] The spatial transformation layer only contains a grid generator and a sampler. The third displacement field φ output by the network is directly used as the transformation parameter. The grid generator obtains the correspondence between the pixel coordinates before and after the transformation based on the third displacement field φ. Finally, the sampler is used to generate a feature map after spatial transformation by trilinear interpolation. This map is the registration image.

[0067] The first feature map obtained by the first spatial transformation layer in the image registration network is only used as the input of the multi-convolution fusion module. The final registered image is obtained through only one spatial transformation, avoiding the problem of multiple interpolations introducing greater errors and interpolation artifacts.

[0068] (2d) Building a Differentiomorphism Integration Module

[0069] Before the third displacement field is transformed into space, the differential homeomorphism displacement field is calculated by integration. Since the relationship between the displacement fields at different times is

[0070] φ (t+Δt) = φ (t) + v(t+Δt) (2)

[0071] Among them, v is the change in displacement field at different times. When Δt is small enough, the known preliminary displacement field φ is used. (0) , we can integrate over t∈[0,1] to solve the final displacement field φ (1)Since v can be expressed as a member of a Lie algebra, φ (1) It can be expressed as a member of a Lie group, which ensures that the displacement field is diffeomorphic, topologically preserving and reversible, and further constrains the smoothness of the displacement field.

[0072] In actual calculation, by selecting a suitable division scale, the integral operation can be converted into accumulation. In this invention, t∈[0,1] is divided into 2 7 The segment contains a total of seven spatial transformation layers.

[0073] Step 3: Iteratively train the 3D image registration network

[0074] (3a) The brain MRI images processed in step 1 are batch-inputted into the 3D brain MRI registration network. The fixed image is used as the ground truth, and the floating image, which is spatially transformed according to the displacement field predicted by the network, is used as the predicted value. Both are used as inputs to the similarity loss function. The negative value of the cross-correlation coefficient is used as the similarity loss function, and the smoothness loss of the displacement field predicted by the network is calculated. The weighted sum of the two losses is used as the overall loss function. The network parameters are updated using the Adam optimization algorithm.

[0075] (3b) Iterate the network on the training dataset for epochs until the overall loss function converges. The different losses and overall loss of the model are calculated as follows:

[0076]

[0077]

[0078]

[0079] l=l sim +λl smooth (6)

[0080] Among them, l sim ,l smooth are similarity loss and smoothness loss respectively, CC represents the correlation coefficient between two images, f, m, φ represent fixed image, floating image and displacement field respectively, Ω represents the space where the three-dimensional image is located, p i represents a voxel in the 3D image space, They represent the average grayscale values of the fixed image and the transformed floating image, respectively. λ is the regularization coefficient for smoothness loss. u represents the initial displacement field, i.e., the displacement field before differential homeomorphism transformation. In actual calculation, the gradient is expressed as Equation (7). The gradient in the x-direction is calculated as Equation (8). The calculation of the gradients in the other two directions is analogous to the x-direction.

[0081]

[0082]

[0083] According to the above method, a corresponding system can be implemented. In one embodiment, the system includes a backbone network, a multi-convolution fusion module and a spatial transformation layer; wherein:

[0084] A backbone network obtains a first displacement field based on the spliced fixed image and the floating image, and interpolates the floating image based on the first displacement field to obtain a first feature map;

[0085] The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field. The first and second displacement fields are then added together to obtain the third displacement field.

[0086] The spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate a registration image. Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that the present disclosure can be implemented with the help of software plus necessary general hardware. Of course, it can also be implemented through dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or dedicated circuits. However, for the present disclosure, in most cases, software program implementation is a better implementation method.

[0087] In summary, the present invention proposes a method based on multi-convolution fusion to perform coarse-to-fine registration, making the displacement field predicted by the network closer to the actual displacement field, thereby improving the accuracy of image registration. Specifically, the present invention uses multiple convolution kernels with different receptive fields to extract multi-scale position information, and calculates the contribution of the information obtained by different convolution kernels. After weighted fusion, the displacement field of the fine registration stage is obtained, which is added to the displacement field of the coarse registration to play a complementary and corrective role, fully alleviating the problem of poor registration accuracy when the floating image undergoes a larger degree of non-rigid deformation compared to the fixed image in image registration.

[0088] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. An image registration method, characterized in that: The method comprises the following steps: After the fixed image and the floating image are stitched together, they are input into the trained image registration network, which uses the image registration network to learn the difference between the floating image and the fixed image and output the registered image; The image registration network includes a backbone network, a multi-convolution fusion module and a spatial transformation layer; wherein: A backbone network obtains a first displacement field based on the spliced fixed image and the floating image, and interpolates the floating image based on the first displacement field to obtain a first feature map; The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field. This includes: using multiple parallel dilated convolutional layers with different dilation rates, with different dilations constructing convolution kernels with different receptive fields; fusing the features extracted by each convolution kernel, performing global average pooling, and reducing the dimension through a fully connected layer. The fully connected layer includes several branches, one for each convolution kernel; after the fully connected layer, weighted fusion is performed to obtain the second displacement field; then, the first and second displacement fields are added to obtain the third displacement field; The spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate a registered image.

2. The method according to claim 1, characterized in that The backbone network is an encoder-decoder, which contains several three-dimensional convolutional layers, each of which is followed by a batch normalization layer and a nonlinear activation function Leaky ReLU.

3. The method according to claim 1, characterized in that Before the floating image and the fixed image are input into the backbone network, they are normalized using the maximum and minimum method to convert the pixel values of the image to between [0, 1].

4. The method according to claim 3, characterized in that After the fully connected layer, a normalized exponential function is applied to obtain the weighted scores of the channel information extracted by each convolution kernel, and the scores are fused to generate the second displacement field.

5. The method according to claim 1, wherein The spatial transformation layer is implemented based on the grid generator and sampler of the spatial transformation network; The third displacement field ϕ is used as the transformation parameter. The grid generator obtains the correspondence between the pixel coordinates before and after the transformation based on the third displacement field ϕ, and then uses the sampler to generate the registration image by trilinear interpolation.

6. The method according to claim 1, characterized in that Smoothing the third displacement field before performing spatial transformation; The smoothing process is to calculate the displacement field of the differential homeomorphism by integrating the third displacement field φ. The relationship between the displacement field positions is: in, is the variation of displacement field at different moments, is the moment change amount; In the calculation, by Divided into The segment sets the number of spatial transformations to n.

7. The method according to claim 1, characterized in that Image registration network training includes the following steps: The fixed image is used as the true value, and the image output by the image registration network is used as the predicted value; Use the negative value of the cross-correlation coefficient as the similarity loss function, take the predicted value and the true value as the input of the similarity loss function, and calculate the similarity loss ; Compute the smoothness loss of the displacement field predicted by the network ; Calculate overall loss , use the Adam optimization algorithm to update the network parameters, is the regularization coefficient of the smoothness loss.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

9. An image registration system for executing the method as claimed in claim 1, characterized in that: The system includes a backbone network, a multi-convolution fusion module and a spatial transformation layer; wherein: A backbone network obtains a first displacement field based on the spliced fixed image and the floating image, and interpolates the floating image based on the first displacement field to obtain a first feature map; The multi-convolution fusion module extracts multi-scale position information based on the first feature map and fuses it to obtain the second displacement field. This includes: using multiple parallel dilated convolutional layers with different dilation rates, with different dilations constructing convolution kernels with different receptive fields; fusing the features extracted by each convolution kernel, performing global average pooling, and reducing the dimension through a fully connected layer. The fully connected layer includes several branches, one for each convolution kernel; after the fully connected layer, weighted fusion is performed to obtain the second displacement field; then, the first and second displacement fields are added to obtain the third displacement field; The spatial transformation layer performs spatial transformation on the floating image based on the third displacement field to generate a registered image.

Images