Multimodal registration method, device, equipment and storage medium based on prostate MR image segmentation results

Abstract

The present invention provides a multimodal registration method, device, equipment and storage medium based on the segmentation result of prostate MR image, and relates to the field of image registration. The registration method includes: segmentation: inputting MRI image and US image into respective segmentation networks, respectively, and obtaining probability maps of MRI prostate image, US prostate image, and respective tumor and feature point labels; affine transformation: inputting the probability maps of MRI prostate image and US prostate image, and respective tumor and feature point labels into affine transformation network, outputting affine transformation matrix, and obtaining probability maps of MRI prostate image, tumor and feature point labels after affine transformation; elastic transformation: inputting the probability maps of US prostate image and MRI prostate image, tumor and feature point labels after affine transformation into elastic transformation network, outputting dense deformation field, and obtaining probability maps of MRI prostate image, tumor and feature point labels after elastic transformation.

Description

Technical Field

[0001] The present invention relates to the field of image registration, and in particular to a multimodal registration method, device, equipment and storage medium based on prostate MR image segmentation results. Background Art

[0002] During radiotherapy for prostate cancer, in order to improve the effectiveness of the treatment plan, real-time ultrasound (US) images are usually preferred to guide the puncture of the tumor. However, ultrasound imaging is limited by low tissue contrast, and the puncture needle introduces artifacts and reduces image quality. Compared with US imaging, magnetic resonance (MR) imaging can better display the fine structure of soft tissue, so it has an advantage in the tumor identification process. Outlining the outlines of organs and tumors on high-quality MR images and then fusing them to US images using image registration technology will help improve the effect of prostate radiotherapy. In recent years, deep learning-based image registration methods have been proposed to automate registration. Typically, the source image and target image to be registered are input into a deep learning network, and the corresponding transformation matrix or deformation field is predicted. The source image can then be transformed with this matrix or deformation field to generate a registered image. During the network training process, the network parameters are optimized by measuring the similarity between the registered image and the target image. However, multimodal image registration remains a challenging task, mainly because it is difficult to measure the similarity of images during the optimization process. In MR and US images, due to the different imaging principles, the same anatomical structure may appear in different image intensity ranges, so traditional intensity-based measurement methods cannot be used for multimodal image registration. To solve this problem, many multimodal image registration strategies have been studied. Some studies use supervised methods to evaluate the similarity between the source and target images by measuring the difference between the predicted transformation matrix and the true transformation matrix, which is highly limited by the availability and quality of the true transformation matrix. In addition, some studies have proposed weakly supervised strategies that bypass the use of true transformation matrices and intensity-based image similarity metrics, and instead use labels representing anatomical structures to replace pixel intensity to achieve image consistency metrics. By marking anatomical structures on the source and target images and measuring the overlap between the transformed labels and the corresponding target labels, the image similarity is evaluated and the model learning process is guided. In response to the problems of poor image quality, lack of weakly supervised labels and difficulty in accurate registration, this scheme provides a new deep learning multimodal registration scheme, which combines image segmentation models to obtain labels for prostate, tumors and some feature points to achieve more accurate multimodal registration. Summary of the invention

[0003] Purpose of the invention: To propose a multimodal registration method, device, equipment and storage medium based on prostate MR image segmentation results to solve the above-mentioned problems existing in the prior art.

[0004] First, a multimodal registration method based on prostate MR image segmentation results is proposed, which includes the following stages:

[0005] S1, segmentation stage: MRI images and US images are input into their respective segmentation networks to obtain MRI prostate images, US prostate images, and probability maps of their respective tumors and feature point labels;

[0006] S2, affine transformation stage: inputting the MRI prostate image and the US prostate image, and the probability maps of the respective tumors and feature point labels into an affine transformation network, outputting an affine transformation matrix, and obtaining the probability maps of the MRI prostate image, tumor, and feature point labels after affine transformation;

[0007] S3, elastic transformation stage: input the US prostate image and the probability map of the MRI prostate image, tumor and feature point labels after affine transformation into the elastic transformation network, output a dense deformation field, and obtain the probability map of the MRI prostate image, tumor and feature point labels after elastic transformation.

[0008] In a further embodiment of the first aspect, step S2 further comprises:

[0009] After affine registration, the deformed MRI segmentation labels and US segmentation labels are input into the deformable registration network;

[0010] During network training, Dice loss is used as part of the label similarity cost function, encouraging the deformable registration network to generate overlap between the deformed MR labels and the fixed US labels.

[0011] In a further embodiment of the first aspect, the affine transformation loss function in step S2 is a Dice loss function L Dice :

[0012]

[0013] Where W p and f p represents the label probability value of pixel p, and V represents the entire 3D image.

[0014] In a further embodiment of the first aspect, in the Dice loss function L Dice On the basis of , the L2 norm of the deformation field gradient is added as a regular term to smoothly align the deformation field:

[0015]

[0016] Where, L Smooth is the regular constraint term of the deformation field, specifically the L2 norm of the spatial gradient of the deformation field, is the registration deformation field.

[0017] In a further embodiment of the first aspect, based on the Dice loss function L Dice , the regularization constraint term L of the deformation field Smooth , we get the total loss function L Total :

[0018]

[0019] Where, L Total is the total loss function, is the Dice loss function of the kth label, k = 0, 1, ... n; is the loss function weight corresponding to the kth label.

[0020] A second aspect of the present invention provides a multimodal registration device based on prostate MR image segmentation results, the device comprising a segmentation module, an affine transformation module, and an elastic transformation module.

[0021] The segmentation module is used to input the MRI image and the US image into their respective segmentation networks, and obtain the MRI prostate image, the US prostate image, and the probability maps of their respective tumor and feature point labels;

[0022] The affine transformation module is used to input the MRI prostate image and the US prostate image, and the probability maps of the respective tumors and feature point labels into the affine transformation network, output an affine transformation matrix, and obtain the probability maps of the MRI prostate image, tumor and feature point labels after affine transformation;

[0023] The elastic transformation module is used to input the US prostate image and the probability map of the MRI prostate image, tumor and feature point labels after affine transformation into the elastic transformation network, output a dense deformation field, and obtain the probability map of the MRI prostate image, tumor and feature point labels after elastic transformation.

[0024] In a third aspect of the present invention, an electronic device is proposed, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the multimodal registration method based on the prostate MR image segmentation results as described in the first aspect is implemented.

[0025] In a fourth aspect of the present invention, a computer-readable storage medium is proposed, characterized in that at least one executable instruction is stored in the storage medium, and when the executable instruction is run on an electronic device, the electronic device executes the multimodal registration method based on the prostate MR image segmentation result as described in the first aspect.

[0026] Compared with the prior art, this application combines an image segmentation model to obtain labels for the prostate, tumors, and some feature points, and achieves more accurate multimodal registration to address the problems of poor image quality, lack of weakly supervised labels, and difficulty in accurate registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is an overall workflow diagram of the MRI-US multimodal registration solution in an embodiment of the present invention.

[0028] Figure 2 Schematic diagram of a 3D MRI-US multimodal registration framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described.

[0030] The overall workflow of the multimodal registration solution is as follows: Figure 1 As shown. First, preoperative prostate magnetic resonance and ultrasound imaging is obtained, and 3D MRI and 3D US are reconstructed, and 3D MRI and 3D US are three-dimensionally registered. When real-time two-dimensional ultrasound images are used to guide the puncture of prostate tumors during surgery, the transformation matrix mapping 3DUS to 2DUS is obtained through the spatial tracking device on the ultrasound probe, and the transformation matrix is ​​applied to the registered 3DMRI to obtain the corresponding registered 2D MRI image.

[0031] The three-dimensional image registration framework of 3D MRI and 3D US consists of two parts, such as Figure 2As shown in the figure, the first part is the segmentation of prostate, tumor and other feature points of 3D MRI and 3D US. The basic method is the fully convolutional neural network segmentation model (FCN); the second part is the three-dimensional image registration of 3D MRI and 3D US. The result of the segmentation model in the first part is used as the input of the three-dimensional registration network. The three-dimensional registration model is divided into two steps: affine transformation network and elastic transformation network. The affine transformation network realizes the coarse registration of 3D MRI and 3D US positions, and the elastic transformation completes a more accurate fine registration based on the coarse registration. All the above networks are trained separately.

[0032] The implementation process of the registration framework is described as follows:

[0033] Segmentation: 3D MRI and 3D US are input into their respective segmentation networks to obtain the probability maps of the prostate, tumor and feature point labels.

[0034] Affine transformation: The probability maps of MRI and US prostate, tumor and feature point labels are input into the affine transformation network, the affine transformation matrix is ​​output, and the probability maps of MRI prostate, tumor and feature point labels after affine transformation are obtained. The affine transformation loss function is the Dice loss function.

[0035] Elastic transformation: The probability maps of prostate, tumor and feature point labels of MRI after US and affine transformation are input into the elastic transformation network, and the dense deformation field is output, and the probability maps of prostate, tumor and feature point labels of MRI after elastic transformation are obtained. The elastic transformation loss function is the Dice loss function and the deformation field regularization loss function.

[0036] FCNs have shown good performance in automatic segmentation of medical images. Pixel-level prediction enables FCN to obtain end-to-end prediction of the entire image in one forward pass. We trained two FCNs for MRI and US segmentation respectively. Prostate, tumor and feature points manually labeled in MRI and US images were used as learning targets for the two FCNs. We combined the binary cross entropy loss and Dice loss into a hybrid loss function for deep supervised training.

[0037] The MR and US segmentation results are designated as source and target labels, respectively, because our goal is to use the deformed MRI that matches the intraoperative US as an image guide during surgery. According to related studies, setting the rigid or affine registration results as the initialization input before the subsequent non-rigid registration can help obtain more accurate and stable registration performance. To this end, we designed a 3D Unet with 3D labels as input, which predicts 12 affine transformation parameters to deform the original MR labels as the initialization step of non-rigid registration. The loss function of the affine transformation network is the Dice function, which is used to measure the overlap between the deformed MRI prostate, tumor and other feature point labels and the corresponding labels of the target US in spatial position.

[0038] After affine registration, the deformed MRI segmentation labels and US segmentation labels are input into the deformable registration network, which has a structure similar to 3D Unet. During network training, Dice loss is used as part of the label similarity cost function, which encourages the network to generate a large overlap between the deformed MR labels and the fixed US labels.

[0039]

[0040] W p and f p represents the label probability value of pixel p, and V represents the entire 3D image.

[0041] Besides, we add the L2 norm of the deformation field gradient as a regularization term in the loss function to smooth the registration deformation field.

[0042]

[0043] L Smooth is the regular constraint term of the deformation field, specifically the L2 norm of the spatial gradient of the deformation field, is the registration deformation field.

[0044]

[0045] Where L Total is the total loss function, is the Dice loss function of the kth label, k = 0, 1, ... n. It is the loss function weight corresponding to the kth label.

[0046] In summary, this scheme provides a new deep learning multimodal registration scheme, which combines the image segmentation model to obtain the labels of the prostate, tumor and some feature points. It can achieve more accurate multimodal registration for the problems of poor image quality, lack of weak supervision labels and difficulty in accurate registration.

[0047] The multimodal registration scheme proposed in the present invention takes the MRI and US images of the organ prostate as an example to illustrate the specific registration process and method, but the applicable targets and image modalities of the present invention are not limited to this, and the following explanation is added. The present invention is not limited to the registration of the prostate, but also applicable to the registration of other organs or tumors such as the liver, heart, brain functional areas, etc. The multimodal registration in the present invention is not limited to the two modalities of 3D MRI and 3D US, but can also be applied to other multimodalities such as 3D MRI and 3D CT, 3D CT and 3D US, etc.

[0048] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A multimodal registration method based on prostate MR image segmentation results, characterized in that: The following stages are included: S1, segmentation stage: obtain preoperative prostate magnetic resonance and ultrasound imaging, reconstruct MRI images and US images respectively; input MRI images and US images into their respective segmentation networks, and obtain MRI prostate images, US prostate images, and probability maps of their respective tumors and feature point labels; S2, affine transformation stage: inputting the MRI prostate image and the US prostate image, and the probability maps of the respective tumors and feature point labels into the affine transformation network, outputting the affine transformation matrix, and obtaining the probability maps of the MRI prostate image, tumor and feature point labels after affine transformation; the affine transformation matrix is ​​calculated by the spatial tracking device on the ultrasound probe; After affine registration, the deformed MRI segmentation labels and US segmentation labels are input into the deformable registration network; During network training, Dice loss is used as part of the label similarity cost function to encourage the deformable registration network to generate overlap between the deformed MR labels and the fixed US labels; The affine transformation loss function is the Dice loss function. , and in the Dice loss function On the basis of , the L2 norm of the deformation field gradient is added as a regular term to smoothly align the deformation field: ； In the formula, and represents the label probability value of pixel p, and V represents the entire 3D image; ； In the formula, is the regular constraint term of the deformation field, specifically the L2 norm of the spatial gradient of the deformation field, To register the deformation field; Based on the Dice loss function , the regularization constraint of the deformation field , and get the total loss function : ； In the formula, is the total loss function, is the Dice loss function of the kth label, k=0,1,…n; is the loss function weight corresponding to the kth label; Using Dice loss function Measure the overlap between the labels of the deformed MRI prostate, tumor and feature points and the corresponding labels of the target US in spatial position; S3, elastic transformation stage: input the US prostate image and the probability map of the MRI prostate image, tumor and feature point labels after affine transformation into the elastic transformation network, output a dense deformation field, and obtain the probability map of the MRI prostate image, tumor and feature point labels after elastic transformation.

2. A multimodal registration device based on prostate MR image segmentation results, used to automatically execute the multimodal registration method based on prostate MR image segmentation results as claimed in claim 1, characterized in that: The multimodal registration device comprises: Segmentation module; the segmentation module is used to input the MRI image and the US image into their respective segmentation networks, respectively, to obtain the MRI prostate image, the US prostate image, and the probability maps of their respective tumors and feature point labels; Affine transformation module; the affine transformation module is used to input the MRI prostate image and the US prostate image, and the probability maps of the respective tumors and feature point labels into an affine transformation network, output an affine transformation matrix, and obtain the probability maps of the MRI prostate image, tumor and feature point labels after affine transformation; Elastic transformation module: The elastic transformation module is used to input the US prostate image and the probability map of the MRI prostate image, tumor and feature point labels after affine transformation into the elastic transformation network, output a dense deformation field, and obtain the probability map of the MRI prostate image, tumor and feature point labels after elastic transformation.

3. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the multimodal registration method based on prostate MR image segmentation results as claimed in claim 1 is implemented.

4. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on the electronic device, the electronic device executes the multimodal registration method based on the prostate MR image segmentation result as claimed in claim 1.

Images