Abdominal Multi-Organ Registration Method Based on Deep Probability Map and Vector Fusion

Abstract

The present invention discloses a multi-organ registration method based on depth probability map and vector fusion, which mainly solves the problems of incorrect organ correspondence and easy conflict in the fusion of different deformation fields when performing multi-organ registration on abdominal CBCT and MRI images in the prior art. The implementation scheme is as follows: preprocess the multi-organ data and use Mask-RCNN to obtain the segmentation results; construct a multi-organ registration network composed of a multi-organ deformation field and probability map generation network and a VFF vector field fusion module, and define its loss function; use the training set data to train the network; input the test set data into the trained network to obtain the fused deformation field; use the deformation field to interpolate the target image to obtain the registration result. The present invention can well correspond to the organ relationship, effectively reduce the influence of each organ on the surrounding organs, improve the accuracy of multi-organ registration, and can be used for large-deformation cross-modal medical image registration to assist doctors in radiotherapy and puncture.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for abdominal multi-organ registration, which can be used for registering medical images in a multi-modal and multi-phase scenario. Background Art

[0002] Abdominal medical images refer to the images generated during shooting through techniques such as abdominal X-ray, ultrasound, computed tomography, etc., which are used to reflect partial structural changes or partial medical information such as positioning in the abdominal part. Common abdominal organs include the liver, gallbladder, pancreas, spleen, bladder, and cervix. No matter what kind of disease, changes may occur in these abdominal organs, and abdominal medical images can effectively detect these changes.

[0003] For cervical cancer, according to different stages of its development, the lesions may infiltrate into surrounding organs. During the radiotherapy of cervical cancer, it may also affect the surrounding organs. At present, the detection and treatment of cervical cancer often rely on cytological screening or various imaging means. Among them, the imaging means include various images such as B-ultrasound, CT, X-ray examination, magnetic resonance imaging (MRI). Different imaging methods can distinguish different tissues and organs by means of different physical principles. It requires doctors to use their experience, use different modal images, and estimate the approximate location of the tumor according to the relative positions of the organs, which brings great difficulties to precise radiotherapy. Therefore, it is necessary to unify different phases into the same spatial domain and align the corresponding tissues and organs. The medical image registration method can not only help doctors accurately discover and locate lesions, but also be the basis for other multi-modal image analysis tasks.

[0004] The current medical image registration methods mainly include whole-image registration and single-organ registration. Among them, whole-image registration is to align the entire medical image so that their overall structure and appearance are consistent. However, the massive noise accompanying the massive information will make it difficult to perform whole-image registration. Single-organ registration is to align specific anatomical structures or organs between different images. However, registering individual organs separately and then fusing them may cause mutual conflicts in spatial positions. Therefore, it is urgently necessary to study both the organ registration accuracy and the interaction relationship between multiple organs during registration.

[0005] Currently, most of the research on medical image registration focuses on the brain and chest. Brain registration mainly registers different modality images of the patient's brain MRI, while chest registration mainly focuses on the patient's lungs or heart. These two types of images involve both whole-image registration and single-organ registration because the main organs in their regional images account for more than 90% of the whole image, with few cavities, adhesions, and large-scale stretching phenomena. Different from these two regions, the distribution of abdominal organs is complex, with large variations in position and shape, and there are many organs in the abdominal cavity, showing the characteristics of multiple organs arranged in a certain spatial order. Therefore, multi-organ registration is a unique problem in abdominal medical image registration.

[0006] JP et al. published a registration algorithm for a global coordinate transformation model in the journal "Medical Image Analysis". This is a traditional iterative-based algorithm that uses the gradient of the reference image and the gray-level difference between the reference image and the floating image to calculate the coordinate offset of each point, thereby obtaining the coordinate offset of the entire image of the reference image and the floating image, and using the coordinate offset to resample and interpolate the floating image. This process is repeated until the number of iterations reaches the set number of iterations or the similarity between the reference image and the floating image reaches the set threshold. Due to the long iteration time, the clinical application of this method is limited.

[0007] The article published by G Balakrishnan et al. in the journal "IEEE Transactions on Medical Imaging"

[0008] In "VoxelMorph: A Learning Framework for Deformable Medical Image Registration", a brain registration method based on a deformation field was proposed, which performs image registration by introducing a deformation field. However, since this method only learns the deformation field after splicing the base image and the floating image, it does not fully utilize the mutual information between the base image and the floating image, nor does it perform additional precision correction on the edges of key organs, resulting in poor registration performance at the organ edges. At the same time, when used for multi-organ registration, it cannot adapt to the different deformation models and structural differences of multiple organs, so the performance for multi-organ registration is not good. Summary of the Invention

[0009] The object of the present invention is to propose a method for multi-organ registration of the abdomen based on depth probability maps and vector fusion in view of the above-mentioned deficiencies of the prior art, so as to improve the registration performance at the organ edges, and achieve adaptability to multi-organ registration by fusing information, and enhance the performance of multi-organ registration.

[0010] A multi-organ registration framework based on Poisson fusion strategy first generates single-organ vector fields, and then uses the mutual attention mechanism and Poisson fusion strategy to fuse the single-organ vector fields into a unified vector field, thus effectively dealing with the multi-organ registration problem and solving the organ correspondence problem and the problem of mutual interference of vector fields in the multi-organ registration process.

[0011] The technical idea to achieve the object of the present invention is as follows: by obtaining the correction of the boundary information by the segmentation branch, fusing the depth probability map and the multi-modal vector; adopting the combination of local loss and global loss to make the task objective more clear, improving the registration accuracy of individual organs, and thus improving the registration accuracy of multi-organs after fusion; through the multi-modal fusion module, the vector fields obtained by different organs can be better fused, thereby improving the registration MI index.

[0012] According to the above technical idea, the implementation steps of the present invention are as follows:

[0013] (1) Using the medical image datasets of three organs with cross-modal large deformations, namely the uterus, bladder, and rectum, as the original data, dividing them into a training set and a test set according to 8:1. Both the training set and the test set include two parts: labels and images;

[0014] (2) Inputting the images in the training set into the Mask-RCNN network, calculating its loss value using the labels and the output of the network, and performing iterative training on it using the stochastic gradient descent method to obtain a trained Mask-RCNN network;

[0015] (3) Inputting the images in the test set and the training set into the trained Mask-RCNN network, and re-dividing the training set and the test set, and merging the re-divided training set and test set to obtain the segmentation results D1, D2, and D3 of the three organs;

[0016] (4) Using the results of step (3), constructing a multi-organ deformation field and probability map generation network:

[0017] 4a) Selecting three existing U-net network structures to respectively form three deformation field generation networks Net1, Net2, and Net3, that is, each deformation field generation network includes an encoder and a decoder;

[0018] 4b) Selecting the encoder in the existing U-net network to form the global probability map network Net4;

[0019] 4c) Inputting the segmentation results D1, D2, and D3 of the three organs into the three deformation field generation networks Net1, Net2, and Net3 respectively to obtain the deformation fields g1, g2, and g3 of the three organs;

[0020] 4d) Fuse the segmentation results D1, D2, and D3 of the three organs and input them into the global probability map encoder network Net4 to obtain the global probability map g a ;

[0021] 4e) Connect the three deformation field generation networks Net1, Net2, and Net3 and the global probability map network Net4 in parallel to form a multi-organ deformation field and probability map generation network;

[0022] (5) Construct a VFF vector field fusion module:

[0023] 5a) Establish a multi-head attention network branch1 including a linear layer, layer normalization, and a multi-head attention layer, and input the three deformation fields g1, g2, g3 and the global probability map g a into branch1 to obtain the weights W1, W2, W3 of each organ deformation field;

[0024] 5b) Establish a Laplacian convolution network branch2 composed of a Laplacian kernel to obtain the gradients of the three deformation fields g1,

[0025] g2, g3

[0026] 5c) Establish a reverse decoding fusion network U including an MLP layer, layer normalization, and a residual structure N , which is used to input the gradients of the three deformation fields and the weights W1, W2, W3 of each organ deformation field to obtain the deformation field required for registration;

[0027] 5d) Connect the multi-head attention network branch1 and the Laplacian convolution network branch2 in parallel, and then cascade them with the reverse decoding fusion network U N to form a VFF vector field fusion module;

[0028] (6) Cascade the multi-organ deformation field and probability map generation network and the VFF vector field fusion module to form a multi-organ registration network PoissonNet, and define its loss function as:

[0029] L = λ1(L s1 + L s2 + L s3 ) + λ2L g

[0030] where L s1 is the local loss function of the uterine deformation field, L s2 is the local loss function of the bladder deformation field, L s3 is the local loss function of the rectal deformation field, L g = L ncc+λ3L mse +λ4L reg is the fusion loss function, λ1 is the weight coefficient of L s1 、L s2 、L s3 ; λ2 is the weight coefficient of L g ; L ncc is the normalized cross - correlation loss function, L mse is the mean square error loss function, L reg is the regularization loss function, λ3 is the weight coefficient of L mse ; λ4 is the weight coefficient of L reg ;

[0031] (7) Use the re - partitioned training set data and the Adam optimizer to iteratively train the multi - organ registration network PoissonNet until the loss function converges, and obtain the trained multi - organ registration network;

[0032] (8) Input the segmentation results D1, D2, and D3 of the uterus, bladder, and rectum into the trained multi - organ registration network to obtain the deformation field required for multi - organ registration;

[0033] (9) Interpolate each pixel in the target image with the generated deformation field to find the corresponding position in the reference image, and obtain the registration result.

[0034] The present invention has the following advantages compared with the prior art:

[0035] 1. Since the present invention constructs a multi - organ registration network PoissonNet based on depth probability maps and vector fusion, the deformation fields of different organs can be obtained and fused, making the fusion of the deformation fields of multiple organs more natural and smooth, and improving the registration performance of organ edges.

[0036] 2. Since the present invention introduces a VFF vector field fusion module into the multi - organ registration network PoissonNet, it solves the pressure on the entire network caused by multi - organ co - registration, and is also superior to semantic segmentation based on prior knowledge. It can effectively reduce the network scale, greatly reduce the registration difficulty, achieve the adaptability to multi - organ registration, and improve the performance of multi - organ registration.

[0037] 3. By introducing network local loss and global loss, the present invention defines the loss function of the multi - organ registration network PoissonNet, which can better balance the outputs of multiple networks, make the task objective more clear, improve the registration accuracy of individual organs, and thus improve the accuracy of multi - organ registration after fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of the implementation process of the present invention;

[0039] Figure 2 It is a schematic diagram of the overall structure of the multi-organ registration network PoissonNet constructed in the present invention;

[0040] Figure 3 It is a structural diagram of the VFF vector field fusion module constructed in the present invention;

[0041] Figure 4 It is a comparison chart of the Dice similarity coefficient DICE index obtained from the registration comparison experiment of three organs using the present invention and the existing four registration methods ANTs, demons, Voxelmorph, and SMMC;

[0042] Figure 5 It is a comparison chart of the average surface distance ASD index obtained from the registration comparison experiment of three organs using the present invention and the existing four registration methods ANTs, demons, Voxelmorph, and SMMC.

[0043] Figure 6 It is a comparison chart of the registration results of three organs using the registration method of directly adding the multi-organ deformation fields in the present invention and the existing method;

[0044] Figure 7 It is a comparison chart of the visualization results of fusing the deformation fields of three organs, namely the bladder, uterus, and rectum, using the present invention and the existing fusion method of directly adding multiple single organs respectively. Detailed implementation manners

[0045] The implementation process and effects of the present invention are further described in detail below with reference to the accompanying drawings. It should be noted that: this embodiment is only used to describe the present invention and does not constitute any limitation to the present invention.

[0046] An important difficulty in cross-modal image processing problems is the problem of inconsistent gray levels. Using segmentation labels for supervised learning or information representation is a common method. At present, it is difficult to obtain medical image segmentation labels. Therefore, in this example, the weakly supervised learning Mask-RCNN network is used for organ segmentation.

[0047] Refer to Figure 1 , the implementation steps of this example are as follows:

[0048] Step 1, obtain data and divide the training set and the test set.

[0049] n pairs of medical image datasets of three organs with large cross-modal deformations, namely the uterus, bladder, and rectum, obtained from a certain hospital are used as the original data, which includes small abdomen T2 magnetic resonance imaging (MRI) data and cone beam computed tomography (CBCT) data of patients in a similar period;

[0050] After manually screening and aligning the obtained large-deformation medical image dataset, it is divided into a training set and a test set according to a ratio of 8:1. Both the training set and the test set include two parts: labels and images.

[0051] Step 2: Select an existing Mask-RCNN network and train it.

[0052] The Mask-RCNN network is an instance segmentation algorithm, which includes a convolutional neural network Backbone for extracting image features, an RPN network for generating candidate object bounding boxes, a RoI Align network for more precisely corresponding to the regions within each candidate box on the feature map, an object classifier for determining the object category in the RoI, a bounding box regressor for finely adjusting the position of the candidate box, and a Mask branch for pixel-level object instance segmentation.

[0053] The Mask-RCNN network is trained as follows:

[0054] 2.1) Input the images in the training set into the Mask-RCNN network, and calculate its loss value L using the labels and the output of the network mask :

[0055] L mask =-∑ i (y c log(p c ))

[0056] where y c is the pixel value in the label, p c is the pixel value in the network output result, and ∑ i represents the sum of the calculation results for each point in the image;

[0057] 2.2) Use the backpropagation algorithm to calculate the parameter gradients of the network model according to the loss value;

[0058] 2.3) Set the learning rate to 0.0004, and update the parameters of the network model using the stochastic gradient descent method according to the parameter gradients of the network;

[0059] 2.4) After repeating 2.1) - 2.3) 12,000 times, obtain the trained Mask-RCNN network model.

[0060] Step 3: Use the trained Mask-RCNN network model to obtain the segmentation results of three organs with large cross-modal deformations.

[0061] 3.1) Input the images in the test set and the training set into the trained Mask-RCNN network to obtain the segmentation results including multiple organs;

[0062] 3.2) Screen the above segmentation results to obtain data pairs in which each pair of data completely contains the three organs A, B, and C to be experimented, which are the uterus, bladder, and rectum respectively. Perform semantic mask operations on the data pairs to obtain the re-divided training set and test set. Merge the re-divided training set and test set to obtain the segmentation results D1, D2, and D3 of the three organs.

[0063] Step 4: Construct a multi-organ registration network PoissonNet based on depth probability maps and vector fusion.

[0064] Refer to Figure 2 , and the implementation steps of this step are as follows:

[0065] 4.1) Select three existing U-net network structures to respectively form three deformation field generation networks Net1, Net2, and Net3. Each deformation field generation network includes an encoder and a decoder;

[0066] 4.2) Select the encoder in the existing U-net network to form the global probability map network Net4;

[0067] 4.3) Connect the three deformation field generation networks Net1, Net2, and Net3 and the global probability map network Net4 in parallel to form a multi-organ deformation field and probability map generation network;

[0068] 4.4) Input the segmentation results D1, D2, and D3 of the three organs A, B, and C, that is, the uterus, bladder, and rectum, into the three deformation field generation networks Net1, Net2, and Net3 respectively to obtain the deformation fields g1, g2, and g3 of the three organs;

[0069] 4.5) Fuse the segmentation results D1, D2, and D3 of the three organs and input them into the global probability map encoder network Net4 to obtain the global probability map g a ;

[0070] 4.6) Establish a multi-head attention network branch1:

[0071] 4.6.1) Cascade a linear layer and a normalization layer to form the first linear network flow1;

[0072] 4.6.2) Cascade three linear layers and three normalization layers respectively to form the second linear network flow2, the third linear network flow3, and the fourth linear network flow4. Then parallelize these three linear networks flow2, flow3, and flow4 to form the merged linear network flow a ;

[0073] 4.6.3) In the first linear network flow1 and the merged linear network flow a are respectively connected to a multi-head attention layer to form a multi-head attention network branch1, where the first linear network flow1 serves as the key-value and input of the multi-head attention, and the merged linear network flow a serves as the query of the multi-head attention;

[0074] 4.7) Input the three deformation fields g1, g2, g3 and the global probability map g a into the multi-head attention network branch1 to obtain the weights W1, W2, W3 of each organ deformation field;

[0075] 4.8) Establish a Laplacian convolutional network branch2 composed of a Laplacian kernel,

[0076] 4.8.1) Create a convolutional kernel of size 3×3, that is, a 3×3 matrix;

[0077] 4.8.2) Initialize the convolutional kernel parameters, that is, set the values of the first row to 0, 1, 0; set the values of the second row to 1, -4, 1; set the values of the third row to 0, 1, 0 to obtain the Laplacian convolutional network branch2, which is used to obtain the gradients of the three deformation fields g1, g2, g3

[0078] 4.9) Establish a reverse decoding fusion network U N :

[0079] 4.9.1) Establish a convolutional weighting module C1 composed of a convolutional layer;

[0080] 4.9.2) Construct a residual MLP module M composed of a cascade of an MLP layer and a normalization layer, and connect the input end of the MLP layer to the output end of the normalization layer r ;

[0081] 4.9.3) Cascade the convolutional weighting module C1 and the residual MLP module M r to form the reverse decoding fusion network U N , which is used to input the gradients of the three deformation fields and the weights W1, W2, W3 of each organ deformation field to obtain the deformation field required for registration;

[0082] 4.10) Connect the multi-head attention network branch1 and the Laplacian convolutional network branch2 in parallel, and then cascade them with the reverse decoding fusion network U N to form a VFF vector field fusion module, as Figure 3 shown;

[0083] 4.11) Cascade the multi-organ deformation field and the probability map generation network with the VFF vector field fusion module to construct the multi-organ registration network PoissonNet.

[0084] Step 5, construct the loss function of the multi-organ registration network PoissonNet.

[0085] 5.1) Define the mean square loss function L mse :

[0086]

[0087] where N is the total number of pixels, and f(i) and t(i) are the values of the i-th pixel on the reference image and the floating image, respectively;

[0088] 5.2) Define the regularization loss function L reg :

[0089]

[0090] where θ j is the j-th parameter of the multi-organ registration network PoissonNet, M is the number of parameters of the multi-organ registration network PoissonNet, and Λ reg is the weight coefficient;

[0091] 5.3) Define the normalized cross-correlation loss function L ncc :

[0092]

[0093] where n is the window size, f and t are the reference image and the floating image of the multi-organ registration data, (x, y) are the position coordinates on the image, and σ and μ are the standard deviation and the mean of the image, respectively;

[0094] 5.4) Define the local loss function L s1 of the uterine deformation field, the local loss function L s2 of the bladder deformation field, and the local loss function L s3 of the rectal deformation field, which are expressed as follows:

[0095] L s1 = L mse + λ5L reg

[0096] L s2 = L mse + λ5L reg

[0097] L s3 = L mse + λ5L reg

[0098] where L mse is the mean squared loss, λ5 is the weight coefficient of L reg , and L reg is the regularization loss function;

[0099] 5.5) Define the fusion loss function L g , and the formula is as follows:

[0100] L g = L ncc + λ3L mse + λ4L reg

[0101] where λ3 is the weight coefficient of the mean squared loss L mse , and λ4 is the weight coefficient of the regularization loss L reg ;

[0102] 5.6) According to the above function, define the total loss function L of the multi-organ registration network PoissonNet as:

[0103] L = λ1(L s1 + L s2 + L s3 ) + λ2L g

[0104] where λ1 is the weight coefficient of the sum of L s1 , L s2 , and L s3 , and λ2 is the weight coefficient of L g ;

[0105] Step 6: Train the multi-organ registration network PoissonNet.

[0106] 6.1) Input the images in the re-partitioned training set into the multi-organ registration network PoissonNet, and use the loss function L to calculate the loss value between the reference image and the floating image;

[0107] 6.2) Use the backpropagation algorithm to calculate the parameter gradients of the multi-organ registration network PoissonNet according to the loss value;

[0108] 6.3) Set the learning rate to 0.0001, and update the network parameters using the Adam optimizer according to the parameter gradients of the multi-organ registration network PoissonNet;

[0109] 6.4) Repeat steps 6.1) - 6.3) until the loss function converges to obtain the trained multi-organ registration network PoissonNet.

[0110] Step 7: Obtain the registration result of magnetic resonance imaging (MRI) and computed tomography (CBCT) using the trained multi-organ registration network.

[0111] 7.1) Input the semantic mask-based training set and test set into the trained multi-organ registration network to obtain the total deformation field of multi-organ registration.

[0112] 7.2) Interpolate each pixel in the target image using the generated total deformation field to find the corresponding position in the reference image.

[0113] 7.2.1) For a pixel in the target image, determine its position in the reference image according to the information of the total deformation field. Based on the position of the target pixel in the reference image, calculate the weight of the pixel closest to the target pixel.

[0114] 7.2.2) According to the calculated weight, use bicubic interpolation to perform weighted summation of the pixel values of adjacent pixels according to the weight size to obtain the estimated pixel value of the current pixel in the target image in the reference image.

[0115] 7.2.3) Repeat the above 7.2.1) - 7.2.2) for all pixels in the target image, and fuse all processed pixels to obtain the registered target image result.

[0116] The labels of the above steps are only used to clearly illustrate the implementation scheme of the present invention, and their sequence numbers are not limited.

[0117] The effect of the present invention can be further illustrated by the following simulation experiments.

[0118] 1. Simulation conditions:

[0119] The simulation platform for this experiment is an Intel Core i7-8700K CPU and 32GB RAM, an NVIDIA GeForce GTX 1080Ti GPU, the operating environment is windows10, the CUDA version is 10.0, and the deep learning framework used is pytorch3.7.11.

[0120] All the data used are cross-modal image pairs of CBCT and MRI, and the registration experiment is carried out on the intermediate data of the centroid segmentation result.

[0121] The data for segmentation and registration are 2D data, the input image size is 192×320, the batch_size of the multi-organ registration network PoissonNet is set to 2, the initial learning rate is set to 0.0001, the optimizer used is Adam, the number of training epochs is 600, and the loss function is the set total loss function L.

[0122] The performance evaluation metrics used in the simulation include the Dice similarity coefficient DICE, the average surface distance ASD, and the mutual information MI. Their specific calculation formulas are as follows:

[0123]

[0124]

[0125] MI(A,B) = H(A) + H(B) - H(A,B)

[0126] where X and Y are the labels of the segmentation regions of the floating image and the reference image;

[0127] S(A) and S(B) represent the boundaries of the labels respectively;

[0128] represents the information entropy of image A,

[0129] represents the information entropy of image B,

[0130] H(A,B) = -∑ X,Y p XY logp XY , representing the joint entropy of A and B,

[0131] N is the gray level of the image, p is the probability that the gray level i appears, and p XY is the probability that the gray level x in image A and the gray level y in image B appear together.

[0132] II. Simulation content

[0133] Simulation 1: Under the above simulation conditions, the proposed method and the existing 4 registration methods, namely ANTs, demons, Voxelmorph, and SMMC, are used to register the test set after semantic masking, and their respective performance evaluation metrics are calculated. The results are shown in Table 1.

[0134] Table 1 Comparison experimental results of multi-organ registration using the proposed method and existing registration methods

[0135]

[0136] It can be seen from Table 1 that the proposed method has obvious improvements in both the DICE and ASD metrics.

[0137] To reduce the influence of accidental errors, multiple experiments are conducted in this simulation. The DICE metrics of multiple registrations of each method are visualized, and the results are as Figure 4 ; the ASD metrics of multiple registrations of each method are visualized, and the results are as Figure 5 .

[0138] From Figure 4 and Figure 5 results, it can be directly seen that the fluctuation range of the indicators of the present invention is smaller than that of other registration methods, indicating that its standard deviation and stability are good.

[0139] Simulation 2: Under the above simulation conditions, the original image vector field of the multi-organ registration network PoissonNet is fused using the present invention and the deformation field directly for vector addition PoissonNet-V method, and the performance evaluation indicators of the two fusion methods are calculated. The results are shown in Table 2:

[0140] Table 2 Ablation experiment of the fusion module of the multi-organ registration network

[0141]

[0142] As can be seen from Table 2, compared with the fusion method of PoissonNet-V, the present invention has obtained better results in the performance indicators MI, DICE, and ASD, indicating that the fusion module in the present invention plays a role in alleviating the conflict of deformation fields of different organs. The registration results of multiple organs are visualized using the present invention and the existing registration method of directly adding the deformation fields of multiple organs, as Figure 6 shown.

[0143] From Figure 6 it can be seen that the registration effect using the present method is better.

[0144] Simulation 3: Under the above simulation conditions, the present invention is used to perform single-organ registration and multi-organ fusion registration on the bladder, cervix, and rectum respectively, and the DICE and ASD performance indicators are calculated. The results are shown in Table 3:

[0145] Table 3 Comparison results of single-organ and multi-organ registration

[0146]

[0147]

[0148] Figure 7 As can be seen from Table 3, the fusion result of the present invention is significantly better in terms of ASD, and the result of multi-organ fusion is closer to the average result of individual organs in terms of DICE. <000,0412>shown.

[0149] From Figure 7 it can be seen that the fusion effect of the present invention is better.

[0150] The above comparison results show that the present invention can solve the problems of incorrect organ correspondence and vector addition conflict in multi-organ registration. Through the principles of deep learning and the mutual attention module, the VFF vector fusion module effectively solves the problem of extremely large computational amount for solving the inverse of the sparse matrix when Poisson image editing is applied in the vector field. And by introducing a neural network, the process of finding the analytical solution is transformed into a polynomial optimization process, and the multi-organ registration accuracy is improved through the vector fusion deformation field.

[0151] The sources of the four existing methods used in the above experiments are as follows:

[0152] ANTs is an open-source medical image processing toolkit for registration and shape analysis. It adopts advanced technologies, including non-linear transformation and local self-similarity measurement. The characteristic of ANTs is that it can process multi-modal images, such as structural magnetic resonance imaging (MRI) and positron emission tomography (PET), etc.

[0153] Demons is an iterative alignment algorithm published by J P et al. in the journal "Medical Image Analysis". It uses the gradient of the reference image and the gray difference between the reference image and the floating image to calculate the coordinate offset of each point, so as to obtain the coordinate offset between the reference image and the floating image of the whole map, and uses the coordinate offset to resample and interpolate the floating image. The floating image is resampled and interpolated using the coordinate offset, and this process is repeated continuously until the number of iterations reaches the set number or the similarity between the reference image and the floating image reaches the set threshold.

[0154] Voxelmorph is a deep learning-based registration method that uses a convolutional neural network (CNN) to learn the transformation mapping between images. The network architecture of Voxelmorph can learn non-linear transformation, so as to achieve high-quality image registration. By optimizing the network parameters during the registration process, Voxelmorph can automatically learn the correspondence of images and is applicable to various registration tasks.

[0155] SMMC is a deep learning-based registration method. It adopts a multi-level structure. First, it uses Mask-RCNN to segment key organs, then predicts and outputs affine registration parameters through an affine registration network, and finally obtains the registration result through deformation registration. This method is weakly supervised and can better solve the registration problem of large cross-modal deformations.

[0156] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. Obviously, for professionals in this field, after understanding the content and principle of the present invention, various modifications and changes in form and details may be made without departing from the principle and structure of the present invention. However, these corrections and changes based on the idea of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. An abdominal multi-organ registration method based on depth probability map and vector fusion, characterized in that, Including: (1) Using the medical image datasets of three organs with cross-modal large deformations, namely the uterus, bladder, and rectum, as the original data, dividing them into a training set and a test set according to 8:

1. Both the training set and the test set include two parts: labels and images; (2) Inputting the images in the training set into the Mask-RCNN network, calculating its loss value using the labels and the output of the network, and performing iterative training on it using the stochastic gradient descent method to obtain a trained Mask-RCNN network; (3) Inputting the images in the test set and the training set into the trained Mask-RCNN network, and re-dividing the training set and the test set, and merging the re-divided training set and test set to obtain the segmentation results D1, D2, and D3 of the three organs; (4) Using the results of step (3), constructing a multi-organ deformation field and probability map generation network: 4a) Selecting three existing U-net network structures to respectively form three deformation field generation networks Net1, Net2, and Net3. That is, each deformation field generation network includes an encoder and a decoder; 4b) Selecting the encoder in the existing U-net network to form the global probability map network Net4; 4c) Inputting the segmentation results D1, D2, and D3 of the three organs into the three deformation field generation networks Net1, Net2, and Net3 respectively to obtain the deformation fields g1, g2, and g3 of the three organs; 4d) Fuse the segmentation results D1, D2, and D3 of the three organs and input them into the global probability map encoder network Net4 to obtain the global probability map g a ; 4e) Connecting the three deformation field generation networks Net1, Net2, and Net3 and the global probability map network Net4 in parallel to form a multi-organ deformation field and probability map generation network; (5) Constructing a VFF vector field fusion module; 5a) Establish a multi-head attention network branch1 including a linear layer, layer normalization, and a multi-head attention layer, and input the three deformation fields g1, g2, g3 and the global probability map g a into branch1 to obtain the weights W1, W2, W3 of each organ deformation field; 5b) Establish a Laplacian convolutional network branch2 consisting of a Laplacian kernel to obtain the gradients of the three deformation fields g1, g2, and g3 5c) Establish a reverse decoding fusion network U including an MLP layer, layer normalization, and a residual structure N , which is used to input the gradients of the three deformation fields and the weights W1, W2, W3 of each organ deformation field to obtain the deformation field required for registration; 5d) Connect the multi-head attention network branch1 and the Laplacian convolutional network branch2 in parallel, and then cascade them with the reverse decoding fusion network U N to form the VFF vector field fusion module; (6) Cascade the multi-organ deformation field and probability map generation network with the VFF vector field fusion module to form a multi-organ registration network PoissonNet, and define its loss function as: L = λ1(L s1 + L s2 + L s3 ) + λ2L g Among them, L s1 is the local loss function of the uterine deformation field, L s2 is the local loss function of the bladder deformation q field, L s3 is the local loss function of the rectal deformation field, L g = L ncc + λ3L mse + λ4L reg is the fusion loss function, λ1 is the weight coefficient of L s1 , L s2 , L s3 , the weight coefficient of L g is the weight coefficient of L ncc is the normalized cross-correlation loss function, L mse is the mean square error loss function, L reg is the regularization loss function, λ3 is the weight coefficient of L mse , λ4 is the weight coefficient of L reg ; (7) Using the re-divided training set data and the Adam optimizer to perform iterative training on the multi-organ registration network PoissonNet until the loss function converges to obtain a trained multi-organ registration network; (8) Inputting the segmentation results D1, D2, and D3 of the three organs, namely the uterus, bladder, and rectum, into the trained multi-organ registration network to obtain the deformation field required for multi-organ registration; (9) Interpolating each pixel in the target image with the generated deformation field to find the corresponding position in the reference image to obtain the registration result.

2. The method according to claim 1, wherein In step (3), re-dividing the training set and the test set, the implementation steps include the following: 3a) Inputting the images in the test set and the training set into the trained Mask-RCNN network to obtain the segmentation results including multiple organs; 3b) Screening the above segmentation results to obtain data pairs that completely contain the three organs under experiment in each pair of data, and performing semantic mask operations on the data pairs to obtain the re-divided training set and test set.

3. The method according to claim 1, wherein In step 5a), establishing a multi-head attention network branch1 including a linear layer, a normalization layer, and a multi-head attention layer. The implementation steps include the following: 5a1) Concatenate a linear layer with a normalization layer to form the first linear network flow1; 5a2) Cascade three linear layers with three normalization layers respectively to form a second linear network flow2, a third linear network flow3, and a fourth linear network flow4, and then connect these three linear networks flow2, flow3, and flow4 in parallel to form a combined linear network flow a ; 5a3) In the first linear network flow1 and the merged linear network flow a are both input into a multi-head attention layer for multi-head attention interaction operations to form a multi-head attention network branch1. Among them, the first linear network flow1 serves as the key-value and input of the multi-head attention, and the merged linear network flow a serves as the query of the multi-head attention.

4. The method according to claim 1, wherein In step 5b), construct a convolutional kernel to establish a Laplacian convolutional network branch2 composed of Laplacian kernels. The implementation steps are as follows: 5b1) Create a convolutional kernel of size 3×3, that is, a 3×3 matrix; 5b2) Initialize the convolutional kernel parameters, that is, set the values of the first row to 0, 1, 0; the values of the second row to 1, -4, 1; The values of the third row are set to 0, 1, 0 to obtain the Laplacian convolutional network branch2.

5. The method according to claim 1, characterized in that In step 5c), establish a reverse decoding fusion network U that includes a convolutional layer, an MLP layer, a normalization layer, and a residual MLP module M r N , and the implementation steps are as follows:​ 5c1) Establish a convolutional weighted module C1 consisting of a convolutional layer; 5c2) Construct a residual MLP module M consisting of an MLP layer, a cascade of normalization layers, and connecting the input end of the MLP layer to the output end of the normalization layer r ; (5c3) Cascade the convolutional weighting module C1 and the residual MLP module M r to form the reverse decoding fusion network U N .

6. The method according to claim 1, wherein In step (6), define its loss function. The implementation steps are as follows: 6a) Define the normalized cross-correlation loss function L ncc , as shown in the following formula: Where n is the window size, f and t are the reference image and the floating image of the multi-organ registration data respectively, (x, y) are the position coordinates on the image, and σ and μ are the standard deviation and mean of the image respectively; 6b) Define the mean squared loss function L mse , and the formula is as follows: Where N is the total number of pixels, and f(i) and t(i) are the values of the i-th pixel on the reference image and the floating image respectively; 6c) Define the regularization loss function L reg , as shown in the following formula: Among them, θ j is the j-th parameter of the multi-organ registration network PoissonNet, M is the number of parameters of the multi-organ registration network PoissonNet, and λ reg is the weight coefficient; 6d) Define the local loss function L of the uterine deformation field s1 , the local loss function L of the bladder deformation field s2 , the local loss function L of the rectal deformation field s3 , which are respectively expressed as follows: L s1 = L mse + λ5L reg L s2 = L mse + λ5L reg L s3 = L mse + λ5L reg where L mse is the mean squared loss, λ5 is the weight coefficient, and L reg is the regularization loss function; 6e) Define the fusion loss function L g , and the formula is as follows: L g = L ncc + λ3L mse + λ4L reg where λ3 is the weight coefficient of the mean square loss L mse and λ4 is the weight coefficient of the regularization loss L reg ; 6f) Define the total loss function of the multi-organ registration network PoissonNet according to the above function as: L = λ1(L s1 + L s2 + L s3 ) + λ2L g .

7. The method according to claim 1, wherein In step (7), use the training set data and the Adam optimizer to perform iterative training on the multi-organ registration network PoissonNet. The implementation steps are as follows: 7a) Input the images in the training set into the multi-organ registration network PoissonNet, and use the loss function L to calculate the loss values of the reference image and the floating image; 7b) Use the backpropagation algorithm to calculate the parameter gradients of the multi-organ registration network PoissonNet according to the loss values; 7c) Set the learning rate to 0.0001, and use the Adam optimizer to update the network parameters according to the parameter gradients of the multi-organ registration network PoissonNet; 7d) Repeat steps 7a) to 7c). After 600 generations of iterative training, obtain the trained multi-organ registration network.

8. The method according to claim 1, wherein In step (9), interpolate each pixel in the target image with the generated deformation field. The implementation steps are as follows: 9a) For a pixel in the target image, determine its position in the reference image according to the information of the deformation field. Based on the position of the target pixel in the reference image, calculate the weights of the adjacent pixels closest to the target pixel; 9b) According to the calculated weights, use bicubic interpolation to perform weighted summation on the pixel values of the adjacent pixels according to the weights to obtain the estimated pixel value of the current pixel in the target image in the reference image; 9c) Repeat the above steps 9a) to 9b) to process all the pixels in the target image, and fuse all the processed pixels to obtain the registered target image result.

Images