A cross-modal medical image registration method based on signed distance function co-segmentation

Abstract

The present invention relates to a cross-modal medical image registration method based on signed distance function collaborative segmentation, comprising: dividing a cross-modal biomedical image into a floating image and a fixed image; estimating the signed distance field of the fixed image; obtaining a trained segmentation model; training a Voxel-Morph registration model; repeating steps (2) to (4) until the reconstruction model, the segmentation model, and the registration model converge; and inputting the estimated signed distance field and the calculated true signed distance field of the floating image to be registered into the converged Voxel-Morph registration model to generate a registered floating image. The present invention uses the signed distance field of the cross-modal image as a potential structural feature to effectively collaborate on segmentation and registration tasks, transforming the cross-modal image registration problem into a modality-independent single-modality registration problem, greatly improving the registration performance of cross-modal biomedical images in small sample scenarios, and improving the accuracy of cross-modal brain registration.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation and image registration, and in particular to a cross-modal medical image registration method based on signed distance function collaborative segmentation. Background Art

[0002] Image registration and image segmentation are two fundamental tasks in medicine, widely used in medical scenarios such as brain mapping, disease diagnosis, and surgical guidance. Image segmentation and image registration are two independent topics, but in the biomedical field, segmentation and registration are inherently complementary. Segmentation can be achieved through registration, and vice versa.

[0003] With the development of high-resolution optical microscopy, high-throughput, high-resolution imaging technologies have generated a large number of mouse brain images from various modalities, such as fluorescence microscopic optical sectioning tomography (FMOST), volumetric imaging with simultaneous rapid scanning and readout (VISOR), and serial two-photon tomography (STPT). Cross-modal image registration of biomedical images from these different imaging modalities enables comparison, analysis, and visualization in a unified coordinate space. In recent years, deep learning has brought improvements in the accuracy and speed of image registration. However, biomedical data is difficult to obtain, and data annotation is time-consuming and labor-intensive, requiring expert knowledge. The amount of data is often in the hundreds. Unsupervised registration methods based on deep learning have overcome the limitations of data labels. However, these tasks currently use similarity metrics such as mean square error (MSE) and non-normalized correlation coefficient (NCC) to drive network training. These statistical information metrics based on grayscale distribution are not suitable for tissues without similar anatomical structures and relatively stable brightness distributions, such as FMOST, CCF, and VISOR data. Designing a perfect cross-modal similarity metric for unsupervised cross-modal image registration is a challenging task.

[0004] Therefore, in view of the difficulties in designing effective cross-modal similarity metrics and the small amount of annotations faced by biomedical images in cross-modal registration technology based on deep learning, how to deeply explore the correlation and complementarity between segmentation tasks and registration tasks and establish efficient collaboration between segmentation and registration tasks to complete cross-modal registration is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of the present invention is to provide a cross-modal medical image registration method based on signed distance function collaborative segmentation, which can mine the potential correlation between segmentation and registration tasks through the signed distance function, couple registration and segmentation to achieve cross-modal image registration, and improve the registration efficiency and accuracy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a cross-modal medical image registration method based on signed distance function collaborative segmentation, the method comprising the following steps in sequence:

[0007] (1) Obtain cross-modal biomedical image data and segmentation labels, and divide cross-modal biomedical images into floating images and fixed images;

[0008] (2) Using a fixed image to construct a reconstruction model and perform model training, the trained reconstruction model is used to estimate the signed distance field of the fixed image;

[0009] (3) Using the estimated fixed image signed distance field as the input of the segmentation model and training the segmentation model to obtain a trained segmentation model;

[0010] (4) Calculate the true signed distance field of the floating image through the segmentation label of the floating image, use the estimated fixed image signed distance field and the calculated true signed distance field of the floating image as the input of the Voxel-Morph registration model, and train the Voxel-Morph registration model. The Voxel-Morph registration model uses the image similarity constraint and deformation field regularization to construct a cost function to drive the registration model training; use the trained segmentation model to generate segmentation labels for the Voxel-Morph registration model, and use the segmentation labels to assist the training of the Voxel-Morph registration model;

[0011] (5) Repeat steps (2) to (4) until the reconstruction model, segmentation model, and registration model converge;

[0012] (6) The fixed image to be registered is input into the converged reconstruction model to estimate the signed distance field, and its true signed distance field is calculated through the segmentation label of the floating image to be registered. Then, the estimated signed distance field and the calculated true signed distance field of the floating image to be registered are input into the converged Voxel-Morph registration model to generate the registered floating image.

[0013] The step (1) specifically refers to: obtaining two biomedical image data sets of different modalities, including X-modality biomedical image data and Y-modality biomedical image data, wherein the Y-modality biomedical image is a standard 3D brain mean atlas, called a floating image; and the X-modality biomedical image is a fixed image.

[0014] The reconstruction model in step (2) is constructed based on a 3DUnet convolutional neural network, comprising an encoding path and a decoding path, each path having four resolution stages, wherein the encoding path comprises two convolutional layers with a convolution kernel size of 3×3×3, each convolutional layer is followed by a GroupBN layer and a ReLU layer, and downsampling is achieved by Maxpooling; each layer in the decoding path comprises a deconvolutional layer with a convolution kernel size of 3×3×3, each deconvolutional layer is followed by a GroupBN layer and a ReLU layer, and layers of the same resolution in the encoding path are transferred to the decoding path through skip connections, thereby providing high-resolution features for the decoding path;

[0015] Using the reconstruction model to estimate the signed distance field, the following steps are included:

[0016] (2a) Considering the training of the reconstruction model as a regression problem, we first need to fix the true signed distance field of the image as a label to constrain the training of the model. The calculation formula of the true signed distance field is as follows:

[0017]

[0018] in, It is a three-dimensional image The brain region of interest in Ω c It is the complement of Ω in I, τ represents the boundary of the brain region of interest, x and y represent the points in the region of interest and the points on the boundary, respectively, and x-y2 represents the Euclidean distance between the two points x and y;

[0019] (2b) The fixed image is input into the reconstruction model, and the reconstruction model outputs a signed distance field with the same resolution as the fixed image. The loss function Huberloss is used to minimize the difference between the predicted signed distance field and the true signed distance field, thereby driving the training of the reconstruction network. The calculation formula of the loss function Huberloss is as follows:

[0020]

[0021] Among them, y gt is the true signed distance field, f(x) is the estimated signed distance field, and δ is a parameter.

[0022] The segmentation model in step (3) adopts the 3DUnet network framework.

[0023] The training process of the Voxel-Morph registration model in step (4) includes the following steps:

[0024] (4a) using the estimated signed distance field of the fixed image and the calculated true signed distance field of the floating image in step (2) as inputs of the Voxel-Morph registration model, training the Voxel-Morph registration model, and using the Voxel-Morph registration model to estimate the deformation field;

[0025] (4b) performing bilinear interpolation on the signed distance field of the floating image using the deformation field to obtain the signed distance field of the registered floating image;

[0026] (4c) The cost function of the Voxel-Morph registration model is constructed by combining the similarity constraint of the signed distance field of the registered floating image and the calculated true signed distance field of the fixed image with the regularization term of the deformation field. The similarity constraint is the mean square error (MSE):

[0027]

[0028] Among them, SDM f and are the signed distance function calculated for the fixed image and the signed distance field after floating image registration, respectively. p is a voxel in the region of interest Ω of the 3D image.

[0029] The regularization term formula of the deformation field is:

[0030]

[0031]

[0032] Where u(p) represents the displacement of voxel p;

[0033] (4d) The segmentation labels of the fixed image generated by the segmentation model are used as weakly supervised data to assist the training of the Voxel-Morph registration model. The Voxel-Morph registration model uses the similarity metric Dice to measure the overlap between the anatomical structures of the segmentation labels of the floating image and the segmentation labels of the fixed image after registration. The calculation formula of Dice is as follows:

[0034]

[0035]

[0036] Where K represents the number of anatomical structures, s f represents the segmentation label of a fixed image, represents the segmentation label of the floating image after registration, represents the segmentation label of the k-th anatomical structure of a fixed image, represents the segmentation label of the k-th floating image after registration.

[0037] It can be seen from the above technical solution that the beneficial effects of the present invention are: the present invention uses the signed distance field of cross-modal images as the potential structural feature to effectively coordinate segmentation and registration tasks. For cross-modal image registration, the cross-modal image registration problem is converted into a modality-independent single-modal registration problem, avoiding the use of cross-modal similarity measurement, thereby greatly improving the registration performance of cross-modal biomedical images in small sample scenarios, which not only improves the accuracy of cross-modal brain registration, but also generates labels for brain images, providing a new solution for cross-modal biomedical image registration in small sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] like Figure 1 As shown, a cross-modal medical image registration method based on signed distance function co-segmentation includes the following steps in sequence:

[0040] (1) Obtain cross-modal biomedical image data and segmentation labels, and divide cross-modal biomedical images into floating images and fixed images;

[0041] (2) Using a fixed image to construct a reconstruction model and perform model training, the trained reconstruction model is used to estimate the signed distance field of the fixed image;

[0042] (3) Using the estimated signed distance field of the fixed image as the input of the segmentation model and training the segmentation model to obtain a trained segmentation model;

[0043] (4) Calculate the true signed distance field of the floating image through the segmentation label of the floating image, use the estimated signed distance field of the fixed image and the calculated true signed distance field of the floating image as the input of the Voxel-Morph registration model, and train the Voxel-Morph registration model. The Voxel-Morph registration model uses the image similarity constraint and deformation field regularization to construct a cost function to drive the registration model training; use the trained segmentation model to generate segmentation labels for the Voxel-Morph registration model, and use the segmentation labels to assist the training of the Voxel-Morph registration model;

[0044] (5) Repeat steps (2) to (4) until the reconstruction model, segmentation model, and registration model converge;

[0045] (6) The fixed image of the cross-modal biomedical image to be registered is input into the converged reconstruction model to estimate the signed distance field, and its true signed distance field is calculated through the segmentation label of the floating image of the cross-modal biomedical image to be registered. Then, the estimated signed distance field and the calculated true signed distance field of the floating image of the cross-modal biomedical image to be registered are input into the converged Voxel-Morph registration model to generate the registered floating image.

[0046] The step (1) specifically refers to: obtaining two biomedical image data sets of different modalities, including X-modality biomedical image data and Y-modality biomedical image data, wherein the Y-modality biomedical image is a standard 3D brain mean atlas, called a floating image; and the X-modality biomedical image is a fixed image.

[0047] The reconstruction model in step (2) is constructed based on a 3DUnet convolutional neural network, comprising an encoding path and a decoding path, each path having four resolution stages, wherein the encoding path comprises two convolutional layers with a convolution kernel size of 3×3×3, each convolutional layer is followed by a GroupBN layer and a ReLU layer, and downsampling is achieved by Maxpooling; each layer in the decoding path comprises a deconvolutional layer with a convolution kernel size of 3×3×3, each deconvolutional layer is followed by a GroupBN layer and a ReLU layer, and layers of the same resolution in the encoding path are transferred to the decoding path through skip connections, thereby providing high-resolution features for the decoding path;

[0048] Using the reconstruction model to estimate the signed distance field, the following steps are included:

[0049] (2a) Considering the training of the reconstruction model as a regression problem, we first need to fix the true signed distance field of the image as a label to constrain the training of the model. The calculation formula of the true signed distance field is as follows:

[0050]

[0051] in, It is a three-dimensional image The brain region of interest in Ω c It is the complement of Ω in I, τ represents the boundary of the brain region of interest, x and y represent the points in the region of interest and the points on the boundary, respectively, and x-y2 represents the Euclidean distance between the two points x and y;

[0052] (2b) The fixed image is input into the reconstruction model, and the reconstruction model outputs a signed distance field with the same resolution as the fixed image. The loss function Huberloss is used to minimize the difference between the predicted signed distance field and the true signed distance field, thereby driving the training of the reconstruction network. The calculation formula of the loss function Huberloss is as follows:

[0053]

[0054] Among them, y gt is the true signed distance field, f(x) is the estimated signed distance field, and δ is a parameter.

[0055] The segmentation model in step (3) adopts the 3DUnet network framework.

[0056] The training process of the Voxel-Morph registration model in step (4) includes the following steps:

[0057] (4a) using the estimated signed distance field of the fixed image and the calculated true signed distance field of the floating image in step (2) as inputs of the Voxel-Morph registration model, training the Voxel-Morph registration model, and using the Voxel-Morph registration model to estimate the deformation field;

[0058] (4b) performing bilinear interpolation on the signed distance field of the floating image using the deformation field to obtain the signed distance field of the registered floating image;

[0059] (4c) The cost function of the Voxel-Morph registration model is constructed by combining the similarity constraint of the signed distance field of the registered floating image and the calculated true signed distance field of the fixed image with the regularization term of the deformation field. The similarity constraint is the mean square error (MSE):

[0060]

[0061] Among them, SDM f and are the signed distance function calculated for the fixed image and the signed distance field after floating image registration, respectively. p is a voxel in the region of interest Ω of the 3D image.

[0062] The regularization term formula of the deformation field is:

[0063]

[0064]

[0065] Where u(p) represents the displacement of voxel p;

[0066] (4d) The segmentation labels of the fixed image generated by the segmentation model are used as weakly supervised data to assist the training of the Voxel-Morph registration model. The Voxel-Morph registration model uses the similarity metric Dice to measure the overlap between the anatomical structures of the segmentation labels of the floating image and the segmentation labels of the fixed image after registration. The calculation formula of Dice is as follows:

[0067]

[0068]

[0069] Where K represents the number of anatomical structures, s f represents the segmentation label of a fixed image, represents the segmentation label of the floating image after registration, represents the segmentation label of the k-th anatomical structure of a fixed image, represents the segmentation label of the k-th floating image after registration.

[0070] Cross-modal image registration is a fundamental task in medicine. It establishes a pixel-level mapping between two images by finding a rigid (linear) or flexible (nonlinear) spatial transformation. In the task of brain mapping, brain images from different individuals, different developmental stages, different scales, and different imaging and labeling modalities can be compared, analyzed, and visualized in a unified coordinate space.

[0071] There are many traditional methods such as Elastic and Ants. Elastic and ANTS methods support multimodal registration by providing cross-modal similarity metrics such as mutual information (MI) and normalized cross correlation (NCC). However, these statistical information indicators based on grayscale distribution are not suitable for tissues without similar anatomical structures and relatively stable brightness distribution, such as fmost, ccf, and visor. Feature-based registration methods can easily combine multiple similarity indicators to weaken the heterogeneity of grayscale distribution and achieve accurate registration by key point matching. However, these methods are often limited by high computational costs and long registration time.

[0072] Image registration methods based on deep learning can currently greatly improve the accuracy and speed of registration. However, supervised methods lack labels. Unsupervised methods currently rely on similarity metrics such as MSE and NCC to drive network training. However, these methods are friendly to single-modal images. For cross-modal images, due to the huge differences in intensity distribution, anatomical structure and texture between cross-modal biomedical images, designing a perfect similarity metric becomes a challenging task.

[0073] Cross-modal image registration can be addressed using a multi-task approach, combining segmentation and registration. Image segmentation and registration are two independent medical disciplines, but in the biomedical field, they are inherently complementary. Segmentation can be achieved based on registration, and registration can be achieved based on segmentation. Establishing an efficient collaboration between segmentation and registration is crucial for completing both tasks. However, segmentation and registration take unary and binary inputs, respectively. The output of registration is the X, Y, and Z offset (displacement field) between each pixel in the floating image and the corresponding pixel in the reference image, while the output of segmentation is the label or confidence level that each pixel is assigned to a different brain region. This difference has led existing methods to primarily separate the collaboration between registration and segmentation into two tasks. Based on this, and considering that cross-modal registration requires eliminating differences in appearance and texture between images across modalities, and by deeply exploring the deep commonalities between segmentation and registration, we propose using a signed distance function as a latent structure representation to unify the two tasks. The signed distance function is defined as the distance from each voxel in a 3D image to its nearest boundary. Each voxel in the 3D image thus contains boundary information, which is most beneficial for segmentation and registration. Segmentation primarily focuses on aligning segment boundaries, while registration focuses on aligning anatomical structure boundaries. Furthermore, this signed distance function serves as a latent structure representation for cross-modal registration, eliminating appearance differences between modal images. This transforms the cross-modal problem into a single-modal registration problem, a currently mature technology.

[0074] Example 1

[0075] The present invention's cross-modal biomedical image registration method, guided by a signed distance function, was run on a single NVIDIA Tesla P40 with 24GB of video memory, using the PyTorch framework for cross-modal biomedical image registration. To verify the effectiveness of this method, this example used signed distance fields to train a cross-modal registration network and trained voxelmorph directly using raw multimodal images. The training results are shown in the following table: SDM-voxelmorph is the method proposed in this invention.

[0076]

[0077] It can be seen that after using the method proposed in the present invention, the Dice score of the registration model increased from 84.09% to 89.74%, indicating that this method improves the registration accuracy of cross-modal image registration.

[0078] In summary, compared with the existing technology, the present invention utilizes the signed distance field collaborative registration network and the segmentation network that are beneficial to segmentation and registration tasks for cross-modal registration, converts the cross-modal registration problem into a single-modal registration problem, avoids the use of challenging cross-modal similarity metrics, greatly improves the registration accuracy of cross-modal brain images, and provides a new solution for cross-modal biomedical image registration tasks.

Claims

1. A cross-modal medical image registration method based on signed distance function collaborative segmentation, characterized by: The method comprises the following steps in sequence: (1) Obtain cross-modal biomedical image data and segmentation labels, and divide cross-modal biomedical images into floating images and fixed images; (2) Using a fixed image to construct a reconstruction model and perform model training, the trained reconstruction model is used to estimate the signed distance field of the fixed image; (3) Using the estimated fixed image signed distance field as the input of the segmentation model and training the segmentation model to obtain a trained segmentation model; (4) Calculate the true signed distance field of the floating image through the segmentation label of the floating image, use the estimated fixed image signed distance field and the calculated true signed distance field of the floating image as the input of the Voxel-Morph registration model, and train the Voxel-Morph registration model. The Voxel-Morph registration model uses the image similarity constraint and deformation field regularization to construct a cost function to drive the registration model training; use the trained segmentation model to generate segmentation labels for the Voxel-Morph registration model, and use the segmentation labels to assist the training of the Voxel-Morph registration model; (5) Repeat steps (2) to (4) until the reconstruction model, segmentation model, and registration model converge; (6) The fixed image to be registered is input into the converged reconstruction model to estimate the signed distance field, and its true signed distance field is calculated through the segmentation label of the floating image to be registered. Then, the estimated signed distance field and the calculated true signed distance field of the floating image to be registered are input into the converged Voxel-Morph registration model to generate the registered floating image.

2. The cross-modal medical image registration method based on signed distance function collaborative segmentation according to claim 1, characterized in that: The step (1) specifically refers to: obtaining two biomedical image data sets of different modalities, including X-modality biomedical image data and Y-modality biomedical image data, wherein the Y-modality biomedical image is a standard 3D brain mean atlas, called a floating image; and the X-modality biomedical image is a fixed image.

3. The cross-modal medical image registration method based on signed distance function collaborative segmentation according to claim 1, characterized in that: The reconstruction model in step (2) is constructed based on a 3D Unet convolutional neural network, which includes an encoding path and a decoding path. Each path has four resolution stages. The encoding path includes two convolutional layers with a convolution kernel size of 3×3×3, each of which is followed by a Group BN layer and a ReLU layer, and downsampling is achieved by Maxpooling; each layer of the decoding path includes a deconvolution layer with a convolution kernel size of 3×3×3, each of which is followed by a Group BN layer and a ReLU layer. Through skip connections, layers of the same resolution in the encoding path are transferred to the decoding path, providing high-resolution features for the decoding path; Using the reconstruction model to estimate the signed distance field, the following steps are included: (2a) Considering the training of the reconstruction model as a regression problem, we first need to fix the true signed distance field of the image as a label to constrain the training of the model. The calculation formula of the true signed distance field is as follows: in, It is a three-dimensional image The brain region of interest in Ω c It is the complement of Ω in I, τ represents the boundary of the brain region of interest, x and y represent the points in the region of interest and the points on the boundary, respectively, and ||xy||2 represents the Euclidean distance between points x and y; (2b) The fixed image is input into the reconstruction model, and the reconstruction model outputs a signed distance field with the same resolution as the fixed image. The loss function Huberloss is used to minimize the difference between the predicted signed distance field and the true signed distance field, thereby driving the training of the reconstruction network. The calculation formula of the loss function Huberloss is as follows: Among them, y gt is the true signed distance field, f(x) is the estimated signed distance field, and δ is a parameter.

4. The cross-modal medical image registration method based on signed distance function collaborative segmentation according to claim 1, characterized in that: The segmentation model in step (3) adopts the 3DUnet network framework.

5. The cross-modal medical image registration method based on signed distance function collaborative segmentation according to claim 1, characterized in that: The training process of the Voxel-Morph registration model in step (4) includes the following steps: (4a) using the estimated signed distance field of the fixed image and the calculated true signed distance field of the floating image in step (2) as inputs of the Voxel-Morph registration model, training the Voxel-Morph registration model, and using the Voxel-Morph registration model to estimate the deformation field; (4b) performing bilinear interpolation on the signed distance field of the floating image using the deformation field to obtain the signed distance field of the registered floating image; (4c) The cost function of the Voxel-Morph registration model is constructed by combining the similarity constraint of the signed distance field of the registered floating image and the calculated true signed distance field of the fixed image with the regularization term of the deformation field. The similarity constraint is the mean square error (MSE): Among them, SDM f and are the signed distance function calculated for the fixed image and the signed distance field after floating image registration, respectively. p is a voxel in the region of interest Ω of the 3D image. The regularization term formula of the deformation field is: Where u(p) represents the displacement of voxel p; (4d) The segmentation labels of the fixed image generated by the segmentation model are used as weakly supervised data to assist the training of the Voxel-Morph registration model. The Voxel-Morph registration model uses the similarity metric Dice to measure the overlap between the anatomical structures of the segmentation labels of the floating image and the segmentation labels of the fixed image after registration. The calculation formula of Dice is as follows: Where K represents the number of anatomical structures, s f represents the segmentation label of a fixed image, represents the segmentation label of the floating image after registration, represents the segmentation label of the k-th anatomical structure of a fixed image, represents the segmentation label of the k-th floating image after registration.

Images