A medical image registration method based on deep learning and contour features

By introducing contour features and alternating optimization strategies in the unsupervised medical image registration method, the problems of data acquisition and accuracy improvement in the existing methods are solved, and medical image registration effect with higher accuracy and smoothness are achieved.

CN114332018BActive Publication Date: 2024-11-19DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111643250.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-29
Publication Date
2024-11-19
Estimated Expiration
2041-12-29

AI Technical Summary

Technical Problem

Existing unsupervised medical image registration methods have challenges in acquiring real data and improving registration accuracy, especially when the auxiliary data loss function does not match the image grayscale loss function, the effect is not significant.

Method used

A medical image registration method based on deep learning and contour features is proposed. By extracting contour features from the pre-segment results of medical images, a differentiability loss function is constructed, and combined with unsupervised methods, it is divided into deformation parameter models. and deformation field smoothing model, alternately optimize network parameters to improve registration accuracy.

Benefits of technology

By introducing contour features and alternating optimization strategies, the accuracy and smoothness of medical image registration are significantly improved, and more accurate image registration results are achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114332018B_ABST
    Figure CN114332018B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of medical image registration, and relates to a medical image registration method based on deep learning and contour features. The present invention introduces additional information into the medical image registration task, that is, extracts contour features from the pre-segmentation results of the medical image, and uses the differentiability of the feature during spatial transformation to construct a loss function, thereby improving the registration accuracy of the medical image; a network framework for medical image registration is proposed, and the network model is divided into two parts: a deformation parameter model and a deformation field smoothing model. The deformation parameter model can construct accurate deformations between corresponding pixels, and the deformation field smoothing model is used to constrain the smoothness of the constructed deformation field; the alternating optimization strategy ensures that the model can eventually converge to a smoother deformation field, and obtain a more accurate registration result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of medical image registration and relates to a medical image registration method based on deep learning and contour features. Background Art

[0002] Medical image registration refers to seeking a (or a series of) geometric space transformations between two (or more) different medical images so that the transformed image can match another (or more) images.

[0003] Medical image registration can be divided into feature-based and grayscale-based registration according to the selected image feature categories. Feature-based image registration methods mainly use extracted features to optimize spatial transformations. These features include corners, contours, ridges, etc. However, this type of method relies on the quality of the extracted features in terms of registration accuracy, so it has certain limitations. Grayscale-based image registration methods directly use the information of image pixels or voxels to guide registration. This type of method does not require the extraction of image features, so it is currently a research hotspot in this field.

[0004] In recent years, due to the rapid iteration of deep learning, using neural networks to learn complex mappings from images to spatial transformations has become an active research field. Supervised learning methods train the network by directly using sample images and their true transformations. Although it can directly predict image transformations, it relies on the quality of the generated transformations, and the true transformations are usually unavailable. Unsupervised learning methods can avoid this defect. They train the network by optimizing the similarity measure based on the image grayscale to achieve end-to-end parameter prediction. However, its training requires a large amount of real data, which is difficult to obtain in practice, and the registration accuracy of this type of method needs to be further improved.

[0005] In order to utilize the additional information of the image in the unsupervised method, some studies have added auxiliary data during the training process, such as the image segmentation label or key point information. This type of method uses the unsupervised network framework as the backbone and adds auxiliary data as supervised information for joint training. Experimental results show that it is better than the unsupervised method. However, the loss function constructed by this type of method for auxiliary data cannot explicitly conform to the loss function constructed by the unsupervised method that relies on the image grayscale, so the effect is not significant in some cases. Summary of the invention

[0006] Based on the above problems of using additional information in unsupervised methods, and in order to improve the smoothness of the deformation field, the present invention proposes a medical image registration method based on deep learning and contour features. Through the method of the present invention, contour features are extracted from the pre-segmentation results of medical images, and the loss function is constructed by using the differentiability of the features during spatial transformation. The loss function is combined with the proposed unsupervised method to jointly optimize the neural network parameters, which can achieve more accurate medical image registration.

[0007] The method of the present invention is roughly divided into three parts:

[0008] (1) Contour feature extraction module: Based on the pre-segmentation results of the medical image, the image contour features are extracted using the edge detection method, and a similarity measure for the contour features is constructed.

[0009] (2) Network model construction module: A network framework for medical image registration is proposed, which divides the network model into two parts: a deformation parameter model and a deformation field smoothing model. The function of the deformation parameter model is to obtain the precise correspondence between the pixels of the image to be registered, and the function of the deformation field smoothing model is to obtain a smooth deformation field while allowing errors to exist. Both models use the same image as input, but the deformation parameter model uses the contour features corresponding to the image as auxiliary data, so that the model parameters can be optimized more effectively. According to the input and output results of the two models, the corresponding loss function is designed.

[0010] (3) Network parameter optimization module: The optimization of the deformation parameter model and the deformation field smoothing model is performed alternately. First, the parameters of the deformation field smoothing model are fixed to optimize the deformation parameter model. Then, the parameters of the parameter optimization model are fixed to optimize the deformation field smoothing model until the loss functions of the two models converge to the minimum or the number of iterations reaches a pre-set number.

[0011] In order to achieve the above object, the present invention adopts the following technical solution:

[0012] A medical image registration method based on deep learning and contour features comprises the following steps:

[0013] Step (1): Preprocess the medical image dataset, including image cropping, normalization, and affine registration.

[0014] Step (2): Extract the contour features based on the image pre-segmentation results and construct a similarity measure for the contour features.

[0015] Step (3): Select a pair of preprocessed images and their corresponding contour features as reference images I F (excluding contour features) and reference contour G F, others as floating images I M With floating profile G M .

[0016] Step (4): The reference image I F With floating image I M As the input of the neural network, add its corresponding contour feature G F and G M As additional information to assist training, a deformation parameter model is constructed; the reference image I F With floating image I M As the input of the neural network, no contour features are added to build a deformation field smoothing model; the loss functions of the deformation parameter model and the deformation field smoothing model are constructed, and the parameters of the two network models are optimized alternately.

[0017] Step (5): Input the image to be registered and the reference image into the trained deformation field smoothing model to obtain the registration result of the image to be registered.

[0018] Furthermore, the step (2) comprises the following steps:

[0019] (2.1) Obtain the segmentation results of each medical image and store them in the form of the same category corresponding to the same grayscale value.

[0020] (2.2) For the segmentation result S of the medical image, the Sobel operator is used to perform edge detection to obtain the contour features of the medical image, that is, the gradient image G. The Sobel operator is a combination of Gaussian smoothing and differential operation, which contains two convolution kernels, namely the horizontal and vertical convolution kernels. The horizontal and vertical convolution kernels are convolved with the segmentation result S respectively, that is:

[0021]

[0022] Among them, * is the convolution operator, G x and G y They are the horizontal and vertical gradient values ​​of the segmentation results respectively.

[0023] According to the two gradient values, the gradient image of each segmentation result, that is, the contour feature, is calculated using the following formula:

[0024]

[0025] For three-dimensional medical images, the above-mentioned Sobel operator can be converted into the corresponding three-dimensional image for operation.

[0026] (2.3) Describe the similarity of contour features based on the normalized cross-correlation matching algorithm. Normalized cross-correlation (NCC) is used to describe the degree of correlation between the targets to be matched. By constructing a neighborhood matching window with a width of w at the same pixel position x of the two images I1 and I2, an objective function is established to measure the correlation of the pair of windows. The objective function is defined as follows:

[0027]

[0028] Where Ω is the set of pixel positions, and are the local grayscale averages in the neighborhood of x, i Represents the pixel position within the neighborhood of x.

[0029] By using the contour features obtained in step (2.2) as parameters to calculate NCC(G1, G2), we can obtain the similarity measure of each pair of contour features.

[0030] Furthermore, the step (4) comprises the following steps:

[0031] (4.1) Construct deformation parameter model:

[0032] (4.1.1) The reference image I F With a floating image I M Merge the image channels to get a 2-channel image as the input layer of the neural network.

[0033] (4.1.2) Downsample the network input to obtain a downsampled feature map.

[0034] (4.1.3) Upsample the downsampled feature map obtained in step (4.1.2) to obtain an upsampled feature map.

[0035] (4.1.4) Further refine the upsampled feature map to obtain the corresponding floating image I M To the reference image I F The deformation field φ t .

[0036] (4.1.5) The floating image I M and the deformation field φ t Input to the differentiable spatial transformation network, and transform the floating image I M The corresponding contour feature G M and the deformation field φ t Input into the spatial transformation network, and undergo spatial transformation of the spatial transformation network to obtain the corresponding deformed image Its contour features

[0037] The described spatial transformation network is an STN network, comprising three components, namely a localization network, a grid generator and a sampler. The STN network first uses a localization network to predict the required transformation, and then uses a grid generator and a sampler to transform the image. The grid generator can be interpolated using nearest neighbor interpolation and linear interpolation, but in the network training process, the nearest neighbor interpolation will cause the network structure to be non-differentiable, so the image segmentation result is not suitable for the training process of the STN network. The contour features obtained in step (2.2) can adapt to linear interpolation because they are gradient images, thereby making the network structure differentiable.

[0038] (4.2) Constructing the deformation field smoothing model: The process of constructing the deformation field smoothing model is the same as steps (4.1.1) to (4.1.4), and the model architecture and initialization parameters are exactly the same as those of the deformation parameter model. F With floating image I M After calculation of the deformation field smoothing model, the deformation field φ corresponding to the input image size can be obtained s .

[0039] (4.3) Alternately optimize network model parameters:

[0040] (4.3.1) According to the input reference image I F , the deformation field φ predicted by the deformation parameter model obtained in the previous steps t , the deformation field φ predicted by the deformation field smoothing model s , and the corresponding deformed image Its contour features Construct a loss function for optimizing the deformation parameter model. The loss function expression is defined as follows:

[0041]

[0042] Among them, L I Denotes the reference image I F With the deformed image The similarity measure between G Denotes the reference image I F The contour feature G F With floating image I M The contour feature G M Features after deformation The similarity measure between dist represents the deformation field φ predicted by the deformation parameter modelt The deformation field φ predicted by the deformation field smoothing model s The L2 distance between them, α and β are the hyperparameters that control the loss function, and γ is the hyperparameter that controls the difference between the two deformation fields, indicating the degree of uncertainty allowed in the deformation field.

[0043] For the similarity measure L in the loss function I With L G , and the normalized cross-correlation function described in step (2.3) is used. For the L2 distance L in the loss function dist , using mean square error (MSE), the loss function is defined as follows:

[0044]

[0045] Among them, Ω is a set of pixel positions, and x is the same pixel position of images I1 and I2.

[0046] (4.3.2) The deformation field φ predicted by the deformation parameter model obtained in step (4.3.1) t And the deformation field φ predicted by the deformation field smoothing model s , construct the loss function of optimizing the deformation field smoothing model. The loss function expression is defined as follows:

[0047] L s (φ t ,φ s )=γL dist (φ t ,φ s )+σL R (φ s )

[0048] Among them, L dist represents the deformation field φ predicted by the deformation parameter model t The deformation field φ predicted by the deformation field smoothing model s The L2 distance between R represents the regularization loss, γ is the hyperparameter that controls the difference between the two deformation fields, and σ is the regularization control parameter.

[0049] For the L2 distance L in the loss function dist , using the mean square error loss described in step (4.3.1). For the regularization term loss L in the loss function R , defined as follows:

[0050]

[0051] Where m represents the displacement at pixel position x. Taking a three-dimensional image as an example, it is expressed as

[0052] (4.3.3) Under the initialization condition, the deformation parameter model and the deformation field smoothing model parameters are exactly the same, that is, Then the alternating optimization strategy is implemented. In one iteration, the network parameters of the deformation field smoothing model are fixed, that is, (i represents the number of iterations), according to L t The loss function value is used to back-propagate the network parameters of the deformation parameter model; then the network parameters of the deformation parameter model are fixed, that is, According to L s The loss function value is used to perform back propagation of the network parameters of the deformation field smoothing model. In each iteration, the reference image I of the two network models is input. F With floating image I M The alternating optimization strategy is executed until the loss function converges to the minimum or the iteration reaches a preset number of times, and the network parameter optimization is considered to be completed, and finally two trained network models are obtained, and the deformation field smoothing model is saved as the final medical image registration network model.

[0053] Beneficial effects of the present invention:

[0054] The present invention introduces additional information into the medical image registration task, namely, extracting contour features from the pre-segmentation results of the medical image, and constructing a loss function using the differentiability of the feature during spatial transformation, thereby improving the registration accuracy of the medical image; a network framework for medical image registration is proposed, and the network model is divided into two parts: a deformation parameter model and a deformation field smoothing model. The deformation parameter model can construct accurate deformations between corresponding pixels, and the deformation field smoothing model is used to perform smoothness constraints on the constructed deformation field; the alternating optimization strategy ensures that the model can eventually converge to a smoother deformation field and obtain a more accurate registration result. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a medical image registration method based on deep learning and contour features described in the present invention.

[0056] Figure 2 This is a network framework diagram of the training phase of a medical image registration method based on deep learning and contour features described in the present invention. Figure 2 The black solid line in the figure represents the calculation process of the deformation parameter model, the black dotted line represents the calculation process of the deformation field smoothing model, the gray solid line represents the loss function of the deformation parameter model optimization process, and the gray dotted line represents the loss function of the deformation field smoothing model optimization process.

[0057] Figure 3This is a network framework diagram of the prediction stage of a medical image registration method based on deep learning and contour features described in the present invention. Specific embodiments

[0058] In order to further explain the present invention to researchers in this technical field, the following is a complete and detailed description of the specific embodiments of the present invention in combination with the drawings and technical solutions of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0059] Figure 1 The flowchart of a method according to an embodiment of the present invention includes the following steps:

[0060] (1) Image preprocessing process: The medical images used in the present invention include but are not limited to medical images obtained from public data sets, medical images obtained from the DICOM interface of medical imaging equipment such as CT or MRI, and the obtained images are annotated or image segmented. This embodiment uses the three-dimensional human brain image in the LPBA40 (The LONI Probabilistic Brain Atlas) data set and the image segmentation results of the corresponding images. The medical image is cropped, normalized, and affine-registered, and the pre-segmented label-aligned medical image is also cropped and affine-transformed to obtain a three-dimensional medical image of size 160×192×160 and its corresponding segmentation result.

[0061] (2) Contour feature extraction process: The Sobel operator is used to perform edge detection on the segmentation results of the medical image to obtain the contour features of the medical image.

[0062] (3) Model initialization process: Construct a deformation parameter model and a deformation field smoothing model. Both models include downsampling, upsampling, and feature map refinement. The deformation parameter model needs to add a spatial transformation network to perform spatial transformation on the input image and contour features. Initialize the parameters of the deformation parameter model and the deformation field smoothing model so that the parameters of the two models are exactly the same.

[0063] The downsampling process described in this embodiment includes 4 downsampling steps, each of which includes a convolution layer with a convolution kernel size of 4 and a subsequent LeakyReLU activation function layer.

[0064] The upsampling process described in this embodiment includes 4 upsampling processes, each of which alternately performs upsampling, connecting jump connections, and convolution calculations. Each convolution calculation process includes a convolution layer with a convolution kernel size of 3, and a subsequent LeakyReLU activation function layer. The jump connection serves to connect the feature maps corresponding to the downsampling and upsampling.

[0065] The feature map refinement process described in this embodiment includes three convolution layers using a convolution kernel of size 3, and a subsequent LeakyReLU activation function layer to obtain a deformation field corresponding to the input image size.

[0066] In order to make the model converge faster, a pre-trained model is used to initialize parameters in this embodiment.

[0067] (4) Model training process:

[0068] (4.1) Among the preprocessed images and their corresponding contour features, one pair is selected as the reference image and the reference contour, and the others are selected as the floating image and the floating contour.

[0069] (4.2) The reference image and one of the floating images are used as inputs of the deformation parameter model to obtain the deformation field predicted by the deformation parameter model. Then, the floating image and the deformation field are input into a differentiable spatial transformation network. At the same time, the contour features corresponding to the floating image and the deformation field are input into the spatial transformation network. After the spatial transformation of the spatial transformation network, the corresponding deformed image and its contour features are obtained. The steps correspond to Figure 2 The solid black line part.

[0070] (4.3) Fix the network parameters of the deformation field smoothing model, use the reference image and the floating image used in step (4.2) as inputs of the deformation field smoothing model, and obtain the deformation field predicted by the deformation field smoothing model. The steps correspond to Figure 2 The black dotted line part.

[0071] (4.4) Based on the input and output of step (4.2) and step (4.3), construct the loss function of the deformation parameter model and perform back propagation on its network parameters. The step corresponds to Figure 2 The solid grey line part.

[0072] (4.5) Using the reference image and the floating image used in step (4.2) as inputs of the deformation field smoothing model to obtain the deformation field predicted by the deformation field smoothing model. The steps correspond to Figure 2 The black solid line part in , but does not include the spatial transformation network and its subsequent steps.

[0073] (4.6) Fix the network parameters of the deformation parameter model, use the reference image and the floating image used in step (4.2) as inputs of the deformation parameter model, and obtain the deformation field predicted by the deformation parameter model. The steps correspond to Figure 2 The black dashed line in .

[0074] (4.7) Based on the two deformation fields output by step (4.5) and step (4.6), construct the loss function of the deformation field smoothing model and perform back propagation on its network parameters. The steps correspond to Figure 2 Medium grey dotted line.

[0075] (5) Model prediction process: After the loss functions of the two models converge to the minimum or the number of iterations reaches a preset number, the deformation field smoothing model is saved as the final medical image registration network model. The three-dimensional medical image to be registered and the reference image are input into the model for registration, and finally the registered image of the image to be registered can be obtained. The prediction process of the model is as follows: Figure 3 shown.

Claims

1. A medical image registration method based on deep learning and contour features, characterized in that: The steps include: Step (1): preprocessing the medical image dataset, including image cropping, normalization, and affine registration; Step (2): Extract the contour features based on the image pre-segmentation results. Construct a similarity measure for contour features; Step (3): Select a pair of preprocessed images and their corresponding contour features as reference images I F With reference contour G F , others as floating images I M With floating profile G M ; Step (4): The reference image I F With floating image I M As the input of the neural network, add its corresponding contour feature G F and G M As additional information to assist training, a deformation parameter model is constructed; The reference image I F With floating image I M As the input of the neural network, without adding contour features, a deformation field smoothing model is constructed; the loss functions of the deformation parameter model and the deformation field smoothing model are constructed, and the parameters of the two network models are optimized alternately; Step (5): input the image to be registered and the reference image into the trained deformation field smoothing model to obtain the registration result of the image to be registered; The step (2) comprises the following steps: (2.1) Obtain the segmentation results of each medical image and store them in the form of the same category corresponding to the same grayscale value; (2.2) For the segmentation result S of the medical image, the Sobel operator is used to perform edge detection to obtain the contour features of the medical image, that is, the gradient image G; the Sobel operator contains two convolution kernels, namely the horizontal and vertical convolution kernels; the horizontal and vertical convolution kernels are convolved with the segmentation result S respectively, that is: Among them, * is the convolution operator, G x and G y They are the horizontal and vertical gradient values ​​of the segmentation results respectively; According to the two gradient values, the gradient image of each segmentation result, that is, the contour feature, is calculated using the following formula: For three-dimensional medical images, the above Sobel operator is converted into the corresponding three-dimensional image for operation; (2.3) The similarity of contour features is described based on the normalized cross-correlation matching algorithm; a neighborhood matching window with a width of w is constructed at the same pixel position x of the two images I1 and I2, and an objective function is established to measure the correlation of the pair of windows. The objective function is defined as follows: Where Ω is the set of pixel positions, and are the local grayscale averages in the neighborhood of x, i represents the pixel position in the neighborhood of x; Use the contour features obtained in step (2.2) as parameters to calculate NCC(G1,G2) and obtain the similarity measure of each pair of contour features; The step (4) comprises the following steps: (4.1) Construct deformation parameter model: (4.1.1) The reference image I F With a floating image I M Merge the image channels to get a 2-channel image as the input layer of the neural network; (4.1.2) Downsample the network input to obtain a downsampled feature map; (4.1.3) upsampling the downsampled feature map obtained in step (4.1.2) to obtain an upsampled feature map; (4.1.4) Further refine the upsampled feature map to obtain the corresponding floating image I M To the reference image I F The deformation field φ t ; (4.1.5) The floating image I M and the deformation field φ t Input to the differentiable spatial transformation network, and transform the floating image I M The corresponding contour feature G M and the deformation field φ t Input into the spatial transformation network, and undergo spatial transformation of the spatial transformation network to obtain the corresponding deformed image Its contour features The spatial transformation network is an STN network, which includes three components, namely a localization network, a grid generator and a sampler; the STN network first uses the localization network to predict the required transformation, and then uses the grid generator and the sampler to transform the image; the grid generator uses linear interpolation to interpolate; (4.2) Constructing the deformation field smoothing model: The process of constructing the deformation field smoothing model is the same as steps (4.1.1) to (4.1.4). The architecture and initialization parameters of the model are exactly the same as those of the deformation parameter model. The input reference image I F With floating image I M After calculation of the deformation field smoothing model, the deformation field φ corresponding to the input image size is obtained s ; (4.3) Alternately optimize network model parameters: (4.3.1) According to the input reference image I F , the deformation field φ predicted by the deformation parameter model t , the deformation field φ predicted by the deformation field smoothing model s , and the image after deformation Its contour features Construct a loss function for optimizing the deformation parameter model; the loss function expression is defined as follows: Among them, L I Denotes the reference image I F With the deformed image The similarity measure between G Denotes the reference image I F The contour feature G F With floating image I M The contour feature G M Features after deformation The similarity measure between dist represents the deformation field φ predicted by the deformation parameter model t The deformation field φ predicted by the deformation field smoothing model s The L2 distance between them, α and β are the hyperparameters that control the loss function, and γ is the hyperparameter that controls the difference between the two deformation fields, indicating the degree of uncertainty allowed in the deformation field. For the similarity measure L in the loss function I With L G , and the normalized cross-correlation function described in step (2.3) is used; for the L2 distance L in the loss function dist , using mean square error loss, the loss function is defined as follows: Where Ω is the set of pixel positions, and x is the same pixel position of images I1 and I2; (4.3.2) The deformation field φ predicted by the deformation parameter model obtained in step (4.3.1) t And the deformation field φ predicted by the deformation field smoothing model s , construct the loss function of optimizing the deformation field smoothing model; the loss function expression is defined as follows: L s (f t ,f s )=γL dist (f t ,f s )+σL R (f s ) Among them, L dist represents the deformation field φ predicted by the deformation parameter model t The deformation field φ predicted by the deformation field smoothing model s The L2 distance between R represents the regularization term loss, γ is the hyperparameter that controls the difference between the two deformation fields, and σ is the regularization control parameter; For the L2 distance L in the loss function dist , using the mean square error loss described in step (4.3.1); for the regularization term loss L in the loss function R , defined as follows: Where m represents the displacement at the pixel position x; for a three-dimensional image, it is expressed as (4.3.3) Under the initialization condition, the deformation parameter model and the deformation field smoothing model parameters are exactly the same, that is, Then the alternating optimization strategy is implemented; in one iteration, the network parameters of the deformation field smoothing model are fixed, that is, Where i represents the number of iterations, according to L t The loss function value is used to back-propagate the network parameters of the deformation parameter model; then the network parameters of the deformation parameter model are fixed, that is, According to L s The loss function value is used to back propagate the network parameters of the deformation field smoothing model; in each iteration, the reference image I of the two network models is input F With floating image I M The same; execute the alternating optimization strategy until the loss function converges to the minimum or the iteration reaches a preset number of times, the network parameter optimization is considered to be completed, and finally two trained network models are obtained, and the deformation field smoothing model is saved as the final medical image registration network model.