Abdominal multi-organ registration method based on adaptive multi-gated hybrid expert model

Abstract

This invention discloses a method for abdominal multi-organ registration based on an adaptive multi-gated hybrid expert model. This method primarily addresses the existing inability to simultaneously perform deformable cross-modal registration of the bladder, cervix, and rectum in the abdomen. The method comprises the following steps: spatially aligning the bladder, rectum, and cervix using the rigidity of the pelvis; constructing an adaptive multi-gated hybrid expert model; mapping CT and MRI images to a mask space and converting them to the same size, which are then input into a multi-branch deformable registration network; performing multi-objective optimization within the deformable registration network using the expert model to determine the registration results for the three organs; calculating the loss between the final registered image and the original CT image, iteratively updating the network to obtain the optimal registration model; and finally obtaining the final registration results for the three organs using the optimal registration model. This method can simultaneously register the bladder, rectum, and cervix in abdominal CT and MRI images, providing a reference for doctors' medical diagnosis, surgical planning, and radiotherapy.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method for registering multiple organs in the human abdomen. This deformable registration method for multiple organs in the human abdomen can assist doctors in medical diagnosis, surgical planning, radiotherapy, etc., allowing doctors to conduct more accurate analysis and provide a basis for doctors to formulate more precise treatment plans. Background Art

[0002] Diagnosis by observing different images relies on spatial visualization and the doctor's subjective experience. Using the right image registration method can accurately integrate diverse information into a single image, allowing doctors to more easily and accurately observe lesions and structures from all angles. Furthermore, by registering dynamic images acquired at different times, changes in lesions and organs can be quantitatively analyzed, making medical diagnoses, surgical planning, and radiotherapy treatment planning more accurate and reliable. Currently, there are few deformable registration methods for medical images, and deformable registration of multiple organs in the human abdomen has never been attempted in the medical field.

[0003] Abdominal medical images have two major characteristics that significantly impact multi-organ segmentation and registration. First, many abdominal organs have weak boundaries between spatially adjacent structures on CT scans, such as the pancreatic head and duodenum. Furthermore, CT scans may include complex structures that are morphologically and topologically connected, such as the gastrointestinal and vascular structures of the stomach, duodenum, small intestine, and colon. The anatomical boundaries between these structures may not be fully visible on CT scans, especially in 2D slices. This results in experts being limited to analyzing the cause of lesions based on subtle texture and shape changes, which can lead to uncertainty in treatment planning. This also makes it difficult for deep networks to distinguish target organs from complex backgrounds. Second, because most organs are soft tissue, the relative sizes of different target organs vary significantly, such as between the liver and gallbladder. This can impact the registration performance of deep learning networks when applied to multi-organ registration. When registering small structures, lower layers of the network often lack semantic information, which can affect the final registration results.

[0004] However, image registration is a challenging task, especially for multimodal images. Scholars have proposed various non-rigid registration methods for medical images, primarily those based on spatial transformations and those based on physical models. Spatial transformation-based non-rigid registration methods employ spatial transformations of images to fit image deformations, and include polynomial methods, basis functions, and spline functions. Physical model-based non-rigid registration methods assume that differences between images are caused by a physical deformation, and include elastic models, viscous fluid models, and optical flow models. In 1981, Broit proposed an elastic model-based registration algorithm, which views the registration process as a physical process similar to the stretching of elastic materials such as rubber. This process is supported by the internal forces generated by the deformation of the elastic body and the external forces applied to the elastic body. In 1996, Qian Zongcai et al. proposed using a viscous fluid model to simulate the deformation of non-rigid medical images, using fluid particles to model the motion of individual image pixels. In 1998, Thirion proposed the Demons algorithm, based on the optical flow model. This algorithm considers the grayscale variation of each point in an image as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane. It considers the registration process as a flow from the source image to the target image, and uses the optical flow field for image registration. Maes, Wells, and others proposed traditional multimodal image registration methods that rely on maximizing the mutual information between images in 1997 and 1996, respectively.

[0005] In recent years, the focus of registration has shifted to developing unsupervised methods that do not rely on deformation fields, and deep learning has become a powerful tool for medical image registration. Since Ronneberger's U-Net achieved success in the 2015 ISBI cell tracking challenge, methods based on convolutional neural networks have become a major focus in the field of medical image analysis. For example, Guha Balakrishnan proposed the VoxelMorph network in 2019, which uses convolutional neural networks to extract image feature information to quickly calculate the deformation field and complete registration. Jiacheng Shi proposed the XMorpher network in 2022, which uses dual parallel convolutional neural network feature extraction networks to discover multi-level semantic correspondences through cross-attention and information exchange, and gradually extract their respective features for effective registration. These convolutional neural network-based registration methods can significantly improve registration performance in medical image registration compared to traditional methods and run faster after training.

[0006] In the research on multi-task medical image registration, Liang Qiu proposed a joint learning framework by combining a diffeomorphic registration network and a Bayesian segmentation network. Through a reversible deformation field, these two tasks can be seamlessly integrated into a single system. Specifically, segmentation and registration use separate networks, with the results constrained by each other. Yuting He borrowed the Few-Shot approach, sharing the feature extraction component from the segmentation network and adding it to the registration network to complete the registration task.

[0007] Multi-organ registration based on multi-objective optimization needs to consider the optimization problem of the registration deformation field of multiple organs at the same time, and needs to strive to minimize the impact on the accuracy of the deformation of other organs while ensuring the accuracy of the corresponding deformation of each organ. In order to solve the problem of deformation conflicts between adjacent organs, registration-related multi-task learning needs to make breakthroughs in both conditional constraints and training methods.

[0008] Currently, there are relatively few studies on abdominal medical image registration, and even fewer studies specifically on multi-organ registration. Therefore, how to register multi-organ cross-modal images of the abdomen remains a bottleneck in registration.

[0009] When most registration methods are directly applied to multi-organ registration methods, there are two main strategies:

[0010] The first method is to register each organ separately and then fuse the deformation field generated by the registration model. However, when registering one organ, the images of other organs are often not considered. In this case, the other organs, including the abdominal cavity, are treated as the background. When registering other organs, the registered organ is treated as the foreground. Ultimately, each organ will be affected when the deformation field is fused.

[0011] The second method is to place all organs in the same image and uniformly register them. This will form an overall deformation field of the organ, but the registered deformation fields of each organ will interfere with each other, resulting in errors in each organ compared to the individual registration. Summary of the Invention

[0012] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose an abdominal multi-organ registration method based on an adaptive multi-gated hybrid expert model to reduce the mutual interference between the registration deformation fields of each organ, reduce the errors generated during the registration of each organ, and improve the registration effect of each organ.

[0013] The technical concept of this invention is to use a multi-branch network to implement multi-task learning. By returning the loss of each organ, the registration deformation fields of organs such as the bladder, cervix, and rectum are optimized separately. By using a hybrid expert model, the results of each branch, that is, the deformation field of each organ, are jointly optimized. The implementation steps include the following:

[0014] (1) Use the rigidity of the pelvis to perform rigid registration of MRI and CT images:

[0015] (1a) The 3D abdominal image dataset pairs of MRI and CT images of n patients provided by the hospital are used as the original data, and are segmented using the Mask-RCNN network to obtain 3D labeled dataset pairs of CT and MRI of key organs such as the pelvis, bladder, cervix, and rectum. The labeled dataset pairs are then used to perform organ extraction on the original 3D abdominal images to obtain 3D dataset pairs of CT and MRI containing the corresponding key organs;

[0016] (1b) The pelvis in the CT image of each pair of 3D datasets is used as the reference image, and the pelvis in the MRI image is used as the floating image. The pelvic floating image is subjected to a 3D affine transformation to align it with the reference image in spatial position, and the corresponding affine transformation matrix N is obtained;

[0017] (1c) The three key organs of bladder, cervix, and rectum in the CT image are used as reference images, and the three key organs in the MRI image are used as floating images. The affine transformation matrix obtained in (1b) is used to perform matrix transformation on the floating images of the three organs through the imwarp() function in MATLAB. Finally, the MRI key organ 3D image that is spatially aligned with the CT key organ 3D image is obtained, that is, the aligned 3D cross-modal image pair.

[0018] (2) Convert the spatially aligned cross-modal image pairs into binary images and perform padding and cropping:

[0019] (2a) Convert these cross-modal 3D image pairs into 2D images and select all image pairs containing key organs such as bladder, cervix, and rectum in the 2D images;

[0020] (2b) converting all selected images into binary images with grayscale values ​​of 0 and 255;

[0021] (2c) The background of the binary image is cropped and filled with 0 to make it the same size, and the converted reference image P_ref and floating image P_reg data pairs are divided into training set, validation set and test set according to 6:3:1;

[0022] (3) The decoder part of the VoxelMorph network was modified into a three-branch output network model, and a hybrid expert model was added to form an abdominal multi-organ registration model based on an adaptive multi-gated hybrid expert model;

[0023] (4) Training the abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model:

[0024] (4a) The binarized reference image P_ref and the floating image P_reg obtained in (2c) are concatenated and put into the encoder of the abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model to obtain the extracted feature R;

[0025] (4b) Select an existing hybrid expert model, set the three gating networks and three experts needed to be used in the model, input the feature R into the three gating networks of the hybrid expert model, and use linear transformation to map the dimension of feature R to the same dimension as the number of experts to obtain three features R1, R2, and R3;

[0026] (4c) The three mapped features R1, R2, and R3 are respectively combined with the learnable weights W k Multiply them together and calculate the output of each gating network through the softmax function. Then multiply the output of each gating network with the output matrix of its corresponding expert and sum them to get the final output of each expert.

[0027] (4d) Upsample each final output result to obtain the optimized deformation field of each of the three organs Then add these three optimized deformation fields to obtain the final deformation field Φ;

[0028]

[0029] Where O is the number of decoder branches, i∈{1,…,O};

[0030] (4e) Fusing the image to be registered in the training set with the final deformation field to obtain the registered image A_reg;

[0031] (4f) Calculate the training set loss L of the registration network based on the reference image P_ref and the registered image A_reg TDice and validation set loss L VDice , and use the training set loss to perform back propagation to iteratively update the network parameters until the validation set loss converges, and finally obtain the trained abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model;

[0032] (5) Input the test set into the trained registration network model to obtain the multi-organ registration result of the test set.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] 1. This invention uses the shape invariance and position information of the pelvis as prior constraints for rigid registration. It uses the rigidity of the pelvis to perform rigid registration on MRI and CT images, and uses the pelvis to align the spatial positions of the three organs. This solves the problem of low efficiency and easy local optimization of deformable registration caused by the large difference in spatial position between CT and MRI images, thereby improving the final registration effect.

[0035] 2. The present invention maps CT images and MRI images into a mask space, providing feature auxiliary information with more reference value for deformable registration.

[0036] 3. The present invention uses a hybrid expert model in the abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model to perform multi-objective optimization on the deformation fields of the three organs, reduce conflicts during multi-organ registration, and achieve accurate multi-organ registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a flowchart of the implementation process of the present invention;

[0038] Figure 2 This is a diagram of the ablation experiment network framework used in the simulation experiment of the present invention;

[0039] Figure 3 This is a before-and-after comparison of the rigid registration of an MRI image and a CT image using the pelvis of the present invention;

[0040] Figure 4 Comparison of deformable registration results of the bladder, cervix, and rectum using the ablation experiment registration network. DETAILED DESCRIPTION

[0041] The embodiments and effects of the present invention are further described in detail below with reference to the accompanying drawings.

[0042] Reference Figure 1 The implementation steps of this example include the following:

[0043] Step 1: Use the rigidity of the pelvis to perform rigid registration on the MRI and CT images.

[0044] Since the spatial positions of abdominal organs differ significantly between CT and MRI images, it is necessary to perform rigid registration on the CT and MRI images of the abdominal organs to align the spatial positions of the three organs. This can alleviate the problem of low deformable registration efficiency and falling into local optimality caused by the large initial spatial position differences. The implementation steps are as follows:

[0045] 1.1) Using the abdominal multi-organ imaging dataset as the original data, the Mask-RCNN network is used to segment the original abdominal multi-organ imaging 3D dataset to segment the pelvic F of the CT image.fb and MRI images of the pelvis F mb And 3D data of key organs such as bladder, cervix and rectum fbcr and MRI bladder, cervix, rectum key organs 3D data F mbcr right;

[0046] This example uses Mask-RCNN as the segmentation network, which does not require precise pixel-level labels for training. This is essential for images of organ data where labels are difficult to obtain or require extensive expertise. Because registration is difficult to obtain corresponding pixel-level segmentation labels, accurate pixel-level labeling is inherently impossible in cases with large amounts of abdominal soft tissue and complex organ distribution. Mask-RCNN, on the other hand, only requires instance-level labels for training, reducing the intensity of supervision and lowering the training threshold.

[0047] 1.2) The segmented CT and MRI 3D image pairs are converted to the same pixel spacing. A 3D affine transformation is performed on the pelvis to spatially align the resulting affine transformation matrix. This affine transformation matrix is ​​then used to apply the same matrix affine transformation to the key organs, i.e., stretching, shrinking, twisting, and rotating. Ultimately, the resulting MRI 3D images of the key organs are spatially aligned with the CT 3D images of the key organs, i.e., a spatially aligned 3D cross-modal image pair.

[0048] Affine transformation is often used in 3D vision technology. Affine transformation can transform a rectangle into a parallelogram. It can flatten the sides of the rectangle and keep the sides parallel. It can also rotate the rectangle or change it proportionally.

[0049] In this example, the imregtform() function in Matlab is used to calculate the F of the CT pelvic image. fb With MRI pelvic image F mb Align the generated affine transformation matrix N and use the imwarp() function to segment the key organ image F mbcr Perform affine transformation to obtain the transformed image F that is finally aligned with the original CT image a :

[0050] N=imregtform(F mb ,F fb )

[0051] F a =imwarp(F mbcr ,N);

[0052] 1.3) Convert the 3D image into a 2D image and select the data pairs containing the three key organs in the 2D image based on the segmented labels.

[0053] In step 2, the selected cross-modal 2D image pairs are converted into masks and cropped to the same size by padding.

[0054] During abdominal radiotherapy, CT imaging is often used to guide radiotherapy, due to its faster imaging, smaller size, and lower cost. However, CT imaging is less effective for soft tissue imaging. While it clearly shows the positional relationships between organs, it is difficult to discern the location of the tumor. Preoperative MRI images can clearly show soft tissues such as tumors. While both imaging methods have their advantages, their combined use relies heavily on the physician's experience and subjective judgment, making direct fusion and comparison impossible.

[0055] This step is specifically implemented as follows:

[0056] 2.1) Convert the grayscale values ​​of all the images to be matched after the conversion to 255, where the grayscale values ​​are greater than 0, and finally save them as a binary image with grayscale values ​​of 0 and 255:

[0057] The binary image is a special grayscale image with only 2 grayscale levels. Many processing methods are used for subsequent processing and recognition. Image binarization is an important preliminary image processing. Due to its simplicity, it can filter out most texture noise, allowing for better registration.

[0058] 2.2) Crop and fill the binary image with padding function:

[0059] Since all images in the original data are of different sizes, if the resize function is used directly in the code to unify the image size, the grayscale value of the binary image will be changed. Therefore, it is necessary to unify the size of the binary image by using cropping and padding functions to fill it with 0. In this example, the background of the binary image is cropped and filled with 0. The image part with a size larger than 256*256 is cropped and unified to a size of 256*256, and the background edge of the image with a size smaller than 256*256 is filled with a grayscale value of 0. The generated data is divided into training set, validation set, and test set according to the ratio of 6:3:1.

[0060] Step 3: construct a multi-organ registration model based on an adaptive multi-gated hybrid expert model, and obtain the registration deformation field from the feature information of the reference image and the image to be registered.

[0061] If multiple organs are directly registered at the same time to generate a deformation field, although there will be no conflict problem when the deformation field is fused, there will be problems with incorrect edge registration between organs. If the edge pixels of organ A and organ B are close, and their positions in the floating image and the reference image have an intersection relationship, the model may align the edge of A to the edge of B, thereby affecting the registration effect of a single organ.

[0062] 3.1) Modify the decoder part of the VoxelMorph network into a three-branch output network model and add a hybrid expert model to form an abdominal multi-organ registration model based on an adaptive multi-gated hybrid expert model:

[0063] This network is constructed based on the modification of the existing VoxelMorph network. That is, by modifying the decoder part of the existing VoxelMorph network into a three-branch output network, adding experts to each branch, and optimizing it through a gating network, it ultimately forms a multi-organ registration model based on an adaptive multi-gated hybrid expert model. Its overall structure includes four key parts: encoder, decoder, gating network, and expert. Both the encoder and decoder use convolution with a kernel size of 3 and a stride of 2. Each convolution is followed by a LeakyReLU layer with a parameter of 0.2.

[0064] 3.2) Obtain the registered deformation field from the features of the reference image and the image to be registered:

[0065] The network consists of three stages, each with its own functions:

[0066] In the encoding stage, the encoder uses cross-row convolution to reduce the spatial dimension of each layer by half, gradually extracting the feature information of the input image;

[0067] In the optimization phase, the extracted feature information is input into three gating networks and three experts. Each gating network consists of a linear transformation and a softmax function. Each expert is a multi-layer fully connected network. The feature matrix output by each expert is controlled by the gating network.

[0068] In the decoding stage, the decoder is replicated into three copies, and the features learned by each expert are input into each decoder. Through convolution upsampling, the deformation fields of the three key organs, bladder, cervix, and rectum, are finally generated.

[0069] Step 4: Obtain the result image after the multi-organ registration model of the adaptive multi-gated hybrid expert model is registered.

[0070] The deformation fields of the three key organs output in the above steps are added together to obtain the final deformation field. The image to be registered is fused with the final deformation field to obtain the registered result image A_reg. The implementation includes the following:

[0071] 4.1) Increase the input MRI binary image P_reg and CT binary image P_ref by two dimensions, convert them into images of size 1*1*255*255, and perform channel splicing on the second dimension to obtain the spliced ​​matrix J fg, input into the encoder of the multi-organ registration model of the adaptive multi-gated hybrid expert model;

[0072] 4.2) J fg Through the convolution operation, the features are selected through the pooling operation to obtain the foreground feature J from the convolution operation. Cfg Extract the most representative information of each category and obtain the feature information R:

[0073] R = maxpooling(J Cfg )

[0074] Among them, maxpooling() is a pooling operation, and it is only performed in the length and width dimensions of the feature map. The feature channel dimension is not pooled to retain the feature information extracted by the encoder. Since the feature information of the foreground area is supported, J Cfg It contains a lot of information, so it is necessary to extract representative information through pooling operation;

[0075] 4.3) Select an existing hybrid expert model and set the number of gating networks and experts:

[0076] The hybrid expert model intentionally divides the shared layer into multiple experts and introduces a gating network mechanism to enable personalized combination of shared layers. In this example, the model is set to use three gating networks and three experts. The feature R is input into the three gating networks and three experts of the hybrid expert model. In the gating network, a linear transformation is used to map the dimension of feature R to the same dimension as the number of experts, resulting in three features R1, R2, and R3.

[0077] 4.4) The three mapped features R1, R2, and R3 are respectively compared with the learnable weights W k Multiply and calculate the output of each gated network through the softmax function, and define the gate function g k ()for:

[0078] g k (R b )=softmax(W k R b )

[0079] in, Is a learnable matrix, d is the feature dimension, where e∈{0,…,M}, M is the number of experts, R b is the input feature data, b∈{1,2,3}, L is the number of gated networks, k∈{1,…,L}, and softmax() is the activation function;

[0080] 4.5) Each gating network controls the contribution of each expert to each task, and the final output of the expert is gf e ():

[0081] The expert implementation is very flexible and can accept multiple architectures, which is crucial for processing applications with different data types. For example, if processing temporal data, the expert can be LSTMs, GRUs, or RNNs; for non-temporal data, the expert can be a dense layer. The number of experts is defined by the user. The output of each gated network is multiplied by the output matrix of its corresponding expert and the sum is obtained to obtain the final output result f of each expert after learning through the gated network. e ():

[0082]

[0083] Among them, e∈{0,…,M}, M is the number of experts;

[0084] 4.6) The final output gf of the expert after the gated network learning e () is input into the existing decoder to predict the optimized deformation fields of the bladder, cervix, and rectum.

[0085]

[0086] Among them, h i () is the decoder, i∈{1,…,O}, O is the number of decoder branches;

[0087] 4.7) The optimal deformation field of each organ Add together to get the final deformation field Φ:

[0088]

[0089] 4.8) Use the image v to be registered in the training set to fuse with the final deformation field Φ:

[0090] Since the image values ​​are defined only at integer positions, linear interpolation is performed on eight adjacent pixels to obtain the registered image A_reg, which is calculated using the following formula:

[0091]

[0092] Where v is the single-channel image to be registered, p is a pixel, T is all the pixels of the deformation field, v°Φ(p) represents the linear interpolation between each pixel of the deformation field Φ and the image to be registered to obtain the final registered image, p *is a sub-pixel position calculated for each pixel p in v, Z(p * ) is p * The adjacent pixels of Ω are the spatial domain of v, and c is the iteration in the dimension of Ω;

[0093] Step 5: Update the multi-organ registration model of the adaptive multi-gated hybrid expert model by back-propagation through the calculated loss to obtain the trained optimal registration model.

[0094] Calculate the Dice loss between the registration result image A_reg and the bladder, cervix, and rectum organs in the CT image P_ref, update the deformable network by backpropagating the loss, and finally obtain the optimal registration model after training. The specific implementation is as follows:

[0095] 5.1) Calculate the training set loss L of the registration network based on the reference image P_ref and the registered image A_reg Tdice and validation set loss L Vdice :

[0096] 5.1.1) According to the k pixels of the reference image P_ref and k pixels of the registered image A_reg Calculate the Dice coefficient to quantify the volume overlap of structure k:

[0097]

[0098] Where, k∈[1,Y], Y is the number of pixels in the training set image;

[0099] 5.1.2) Evaluate the quality of the trained network model by minimizing the Dice coefficient difference between the registered image A_reg and the reference image P_ref, i.e., the Dice loss, and calculate the overall Dice loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient. Dice :

[0100] During the training process, if the Dice loss is closer to 0, the trained network model is better, and if the Dice loss is closer to 1, the trained network model is worse.

[0101] Overall Dice loss L Dice The calculation formula is as follows:

[0102]

[0103] 5.1.3) Using the Dice loss of the reference and registered images of the bladder, cervix, and rectum, the parameters of each multi-objective optimization network branch are iteratively updated using backpropagation.

[0104] 5.1.3.1) Calculate the bladder organ training set loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient TBladder :

[0105]

[0106] Among them, k B ∈[1,Y B ],Y B is the number of bladder pixels, are the k values ​​of the bladder reference image and the bladder registered image, respectively. B Pixels;

[0107] 5.1.3.2) Calculate the cervical organ training set loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient TCervical :

[0108]

[0109] Among them, k C ∈[1,Y C ],Y C is the number of cervical pixels, are the k values ​​of the cervical reference image and the cervical registered image respectively. C Pixels;

[0110] 5.1.3.3) Calculate the rectal organ training set loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient TRectum :

[0111]

[0112] Among them, k R ∈[1,Y R ],Y R is the number of rectal pixels, are the k values ​​of the rectal reference image and the image after rectal registration, respectively. R Pixels;

[0113] 5.1.4) According to L TBladder , L TCervical , L TRectum Calculate the training set loss L Tdice :

[0114] L Tdice =L TBladder +L TCervical +L TRectum

[0115] 5.1.5) Similarly, based on the k pixels of the reference image in the validation set and k pixels of the registered image obtained through the deformable network Calculate the Dice coefficient:

[0116]

[0117] Where k∈[1,Z], Z is the number of pixels in the validation set image;

[0118] 5.1.6) Based on the Dice coefficient, calculate the bladder organ validation set loss L in the same way as the training set organ loss above. VBladder , cervical organ validation set loss L VCervical , rectal organ validation set loss L VRectum , and get the validation set loss L Vdice :

[0119] L Vdice =L VBladder +L VCervical +L VRectum

[0120] 5.2) Iteratively update the network parameters using back propagation, and iteratively update the network parameters using the training set loss. Repeat steps 4 and 5.1) until the validation set loss converges to obtain the multi-organ registration model of the trained adaptive multi-gated hybrid expert model.

[0121] Step 6: Input the images to be registered in the test set into the multi-organ registration model of the trained adaptive multi-gated hybrid expert model to obtain the multi-organ registration results of the test set.

[0122] The effects of the present invention can be further illustrated by the following simulation.

[0123] 1. Simulation conditions:

[0124] The simulation platform of this experiment is a computer with an Intel Core i7-11800H CPU and 16GB of memory, the operating system is Windows 10, Python 3.8 and Pytorch 1.7 are used to build and train the neural network model, and the NVIDIA 3060 Laptop GPU and CUDA 11.0 are used for acceleration.

[0125] The experimental data used for the simulation consisted of 158 pairs of lower abdominal T2 MRI data from the Department of Radiotherapy and the Department of Radiology at Xijing Hospital, as well as CBCT images of patients from similar periods. Each set included over 60 MRI data sets and over 100 CBCT data sets. After manual screening and alignment, each set contained over ten aligned slices. Because most patients were referred, the data were originally acquired at different hospitals, each with varying conditions, technical staff, and data parameters. To ensure that each data pair fully encompassed the three organs being tested—the cervix, bladder, and rectum—the data underwent further screening, narrowing the original 1,000 image pairs to over 600 pairs. Forty of these pairs served as the test set, and 636 pairs served as the training and validation sets.

[0126] The registration performance evaluation indicators used in the simulation include the Dyess similarity coefficient DSC, mutual information MI and average surface distance ASD. The specific calculation formulas are as follows:

[0127]

[0128] MI(A,B)=H(A)+H(B)-H(A,B)

[0129]

[0130] Among them, A represents the true label, B represents the predicted result, H(A) and H(B) represent the information entropy of image A and image B respectively, H(A,B) is the joint entropy of A and B, S(A) represents the surface pixel of the true label, S(B) represents the surface pixel of the predicted result, d(s A , S(B)) represents the shortest distance from any pixel of the true label to the pixel on the surface of the predicted result, d(s B ,S(A)) represents the shortest distance from any pixel of the prediction result to the pixel on the true label surface.

[0131] The ablation experiment network framework used in the simulation experiment is shown in the figure below: Figure 2 As shown, Figure 2 (a) is the existing VoxelMorph network, Figure 2 (b) is a multi-branch deformable network modified from the VoxelMorph network. Figure 2 (c) The multi-organ registration network of the adaptive multi-gated hybrid expert model constructed by the present invention performs registration tests on the divided test sets.

[0132] 2. Simulation content and results

[0133] Simulation 1: Under the above simulation data, the present invention uses the pelvis to perform rigid registration on the MRI image and the CT image. The results are as follows: Figure 3 As shown, where:

[0134] Figure 3 (a) is the reference image in the original data;

[0135] Figure 3 (b) is the image to be registered in the original data;

[0136] Figure 3 (c) is the organ image registered by the transformation matrix obtained after the pelvic affine transformation After 3DAffine;

[0137] Figure 3 (d) is the fusion image of the original data MRI and CT images;

[0138] Figure 3 (e) is the fusion image of the MRI image and the original CT image after affine transformation registration.

[0139] Depend on Figure 3 It can be seen that compared with the original data, the images of the bladder, cervix, and rectum after pelvic affine transformation registration are basically aligned in space. The quantitative index DSC after alignment is shown in Table 1.

[0140] Table 1 DSC results of multi-organ registration after affine transformation

[0141] method pelvis bladder cervix rectum Three-organ system Original image 0.4762 0.3973 0.2666 0.2329 0.2989 After 3D Affine 0.9660 0.5789 0.3542 0.5819 0.505

[0142] As shown in Table 1, after 3D affine transformation, the DSC index of the pelvis in the original CT and MRI images has reached 0.966, indicating that they are basically aligned in space. The DSC index of the 3D images of the bladder, cervix, and rectum organs in CT and MRI has also been significantly improved compared with the original 3D images, indicating that the three organs are basically aligned in space, laying the foundation for subsequent deformable registration.

[0143] Simulation 2: Using the three networks with different structures, we conduct ablation simulation experiments on the bladder, cervix and rectum in the divided test set. The results are as follows: Figure 4 As shown, where:

[0144] The first row is the original image and its fusion result;

[0145] The second line is Figure 2 (a) The results of registration and fusion of the corresponding network structures;

[0146] The third line is Figure 2 (b) The results of registration and fusion of the corresponding network structures;

[0147] The fourth line is Figure 2(c) The result of registration and fusion of the multi-organ registration model network structure of the adaptive multi-gated hybrid expert model constructed by the present invention.

[0148] Depend on Figure 4 It can be seen that the results of the multi-organ registration model constructed by the adaptive multi-gated hybrid expert model of the present invention show that the deformation of the bladder, cervix and rectum is very obvious compared with the data before registration, and the covered parts are more fitted; Figure 2 (a) The VoxelMorph network registration method has obvious deformation registration effects on the three organs. Figure 2 (b) The multi-branch deformable network registration method has significantly increased the registration fusion coverage of the three organs;

[0149] Simulation 3, for Figure 2 The ablation experiment networks in the

[15] were trained separately, and the trained models were used to perform ablation registration tests on the divided test sets of bladder, cervix, and rectum organs. They were evaluated by the above-mentioned Dyss similarity coefficient DSC, mutual information MI, and average surface distance ASD. The quantitative indicators of the ablation experiment test results are shown in Tables 2, 3, and 4.

[0150] Table 2 DSC results of different methods for multi-organ registration

[0151]

[0152]

[0153] As can be seen from Table 1, in terms of the DSC index, the multi-organ registration model based on the adaptive multi-gated hybrid expert model has improved the DSC index in the overall registration results compared with other methods, and also has obvious DSC index improvements in the three organs.

[0154] Table 3 Mutual information MI results of different methods for multi-organ registration

[0155]

[0156] Table 4 Average surface distance ASD results of different methods for multi-organ registration

[0157]

[0158] As can be seen from Tables 2 and 3, the present invention can achieve the best or close to the best results in both MI and ASD indicators of the multi-organ registration task.

[0159] Because the data comes from cancer patients undergoing radiation therapy, the task targets the cervix, rectum, and bladder in the lower abdomen—important organs used for target volume delineation and dose determination during radiation therapy. Because CBCT scans don't include tumor information, medical scientists typically need to determine tumor locations based on MRI scans and their own knowledge and experience, based on the location of the organs.

[0160] The above experimental results show that the present invention can accurately optimize each organ and obtain good registration results, which can provide a valuable reference for clinical practice.

Claims

1. A method for abdominal multi-organ registration based on an adaptive multi-gated hybrid expert model, characterized in that: include: (1) Use the rigidity of the pelvis to perform rigid registration of MRI and CT images: (1a) The 3D abdominal image dataset pairs of MRI and CT images of n patients provided by the hospital are used as the original data, and are segmented using the Mask-RCNN network to obtain 3D labeled dataset pairs of CT and MRI of key organs such as the pelvis, bladder, cervix, and rectum. The labeled dataset pairs are then used to perform organ extraction on the original 3D abdominal images to obtain 3D dataset pairs of CT and MRI containing the corresponding key organs; (1b) The pelvis in the CT image of each pair of 3D datasets is used as the reference image, and the pelvis in the MRI image is used as the floating image. The pelvic floating image is subjected to a 3D affine transformation to align it with the reference image in spatial position, and the corresponding affine transformation matrix N is obtained; (1c) The three key organs of bladder, cervix, and rectum in the CT image are used as reference images, and the three key organs in the MRI image are used as floating images. The affine transformation matrix obtained in (1b) is used to perform matrix transformation on the floating images of the three organs through the imwarp() function in MATLAB. Finally, the MRI key organ 3D image that is spatially aligned with the CT key organ 3D image is obtained, that is, the aligned 3D cross-modal image pair. (2) Convert the spatially aligned cross-modal image pairs into binary images and perform padding and cropping: (2a) Convert these cross-modal 3D image pairs into 2D images and select all image pairs containing key organs such as bladder, cervix, and rectum in the 2D images; (2b) converting all selected images into binary images with grayscale values ​​of 0 and 255; (2c) The background of the binary image is cropped and filled with 0 to make it the same size, and the converted reference image P_ref and floating image P_reg data pairs are divided into training set, validation set and test set according to 6:3:1; (3) The decoder part of the VoxelMorph network was modified into a three-branch output network model, and a hybrid expert model was added to form an abdominal multi-organ registration model based on an adaptive multi-gated hybrid expert model; (4) Training the abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model: (4a) The binarized reference image P_ref and the floating image P_reg obtained in (2c) are concatenated and put into the encoder of the abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model to obtain the extracted feature R; (4b) Select an existing hybrid expert model, set the three gating networks and three experts needed to be used in the model, input the feature R into the three gating networks of the hybrid expert model, and use linear transformation to map the dimension of feature R to the same dimension as the number of experts to obtain three features R1, R2, and R3; (4c) The three mapped features R1, R2, and R3 are respectively combined with the learnable weights W k Multiply them together and calculate the output of each gating network through the softmax function. Then multiply the output of each gating network with the output matrix of its corresponding expert and sum them to get the final output of each expert. (4d) Upsample each final output result to obtain the optimized deformation field of each of the three organs Then add these three optimized deformation fields to obtain the final deformation field Φ; Where O is the number of decoder branches, i∈{1,…,O}; (4e) Fusing the image to be registered in the training set with the final deformation field to obtain the registered image A_reg; (4f) Calculate the training set loss L of the registration network based on the reference image P_ref and the registered image A_reg TDice and validation set loss L VDice , and use the training set loss to perform back propagation to iteratively update the network parameters until the validation set loss converges, and finally obtain the trained abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model; (5) Input the test set into the trained registration network model to obtain the multi-organ registration result of the test set.

2. The method according to claim 1, characterized in that In step (1b), the 3D affine transformation of the pelvic floating image is performed by sequentially transforming, stretching, shrinking, twisting, and rotating the floating image so that it is aligned with the reference image in space, and the corresponding affine transformation matrix N is obtained: N=imregtform(F mb ,F fb ) Among them, imregtform() is the affine transformation function, F mb is the floating image of the pelvis, F fb This is a reference image of the pelvis.

3. The method according to claim 1, characterized in that In step (1c), the floating images of the three organs are transformed by the imwarp() function in MATLAB. The three key organs are stretched, shrunk, twisted, and rotated in the same way as the floating image of the pelvis. The final transformed image F of the three organs is spatially aligned with the original three-organ CT image. a : F a =imwarp(F bcr ,N) Among them, the imwarp() function is the matrix transformation function, F bcr are floating images of the three key organs, and N is the affine transformation matrix for aligning the spatial position of the pelvis.

4. The method according to claim 1, wherein In step (2c), the background of the binary image is cropped and filled with 0, that is, the image part with a size larger than 256*256 is cropped to a uniform size of 256*256, and the background edge of the image with a size smaller than 256*256 is filled with a grayscale value of 0.

5. The method according to claim 1, wherein The multi-organ registration model based on the adaptive multi-gated hybrid expert model constructed in step (3) has an overall structure consisting of four key parts: encoder, decoder, gating network and expert. The encoder and decoder use convolution with a kernel size of 3 and a stride of 2, and each convolution is followed by a LeakyReLU layer with a parameter of 0.

2. In the encoding stage, the encoder uses cross-row convolution to reduce the spatial dimension of each layer by half, gradually extracting the feature information of the input image; In the optimization phase, the extracted feature information is input into three gating networks and three experts. Each gating network consists of a linear transformation and a softmax function. Each expert is a simple base-level fully connected network. The feature matrix output by each expert is controlled by the gating network. In the decoding stage, the decoder is copied into three, and the features finally learned by each expert are input into each decoder, and the deformation field of each key organ is generated through convolution upsampling.

6. The method according to claim 1, characterized in that In step (4a), the reference image P_ref and the floating image P_reg are channel-joined and put into the encoder of the abdominal multi-organ registration model based on the adaptive multi-gated hybrid expert model to extract their features R. The implementation is as follows: (4a1) Increase the input floating image P_reg and the reference image P_ref by two dimensions, convert them into images of size 1*1*255*255, and perform channel splicing on the second dimension to obtain the spliced ​​matrix J fg , input into the multi-organ registration model based on adaptive multi-gated hybrid expert model; (4a2) J fg Through the convolution operation, and then through the pooling operation to select the features, from the foreground feature J Cfg Extract the most representative information of each category and obtain the feature information R: R=maxpooling(J Cfg ) Among them, maxpooling() is the pooling operation.

7. The method according to claim 1, characterized in that In step (4c), the output of each gating network and the final output of each expert after learning the gating network are calculated. The formulas are as follows: Gating network function g k ()for: g k (R b )=softmax(W k R b ) The final output gf of the expert after gating network learning e ()for: in, is a trainable matrix, M is the number of experts, e∈{1,…,M}, d is the feature dimension, L is the number of gated networks, k∈{1,…,L}, R b is the input feature data, b∈{1,2,3}, softmax() is the activation function, and the expert output is f e ().

8. The method according to claim 1, characterized in that Step (4d) upsamples each final output result to obtain the optimized deformation field of each of the three organs, and converts the final output gf of the expert after the gated network learning into e () is put into the decoder to predict the optimized deformation fields of the three organs Among them, h i () is the decoder, O is the number of decoder branches, i∈{1,…,O}.

9. The method according to claim 1, characterized in that In step (4e), the image to be registered in the training set is transformed with the final deformation field Φ obtained in (4d) to obtain the registered image A_reg, which is performed using the following formula: Where v is the grayscale data of the image to be registered in a single channel, p is a pixel, T is all the pixels of the deformation field, v°Φ(p) represents the linear interpolation between each pixel of the deformation field Φ and the image to be registered to obtain the final registered image, p * is a sub-pixel position calculated for each pixel p in v, Z(p * ) is p * The adjacent pixels of Ω are the spatial domain of v, and c is the iteration in the dimension of Ω.

10. The method according to claim 1, characterized in that In step (4f), the training set loss L of the registration network is calculated based on the reference image P_ref and the registered image A_reg Tdice and validation set loss L Vdice : (4f1) According to the k pixels of the reference image P_ref and k pixels of the registered image A_reg Calculate the Dice coefficient: Where, k∈[1,Y], Y is the number of pixels in the training set image; (4f2) Calculate the overall loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient Dice : (4f3) Calculate the bladder organ training set loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient TBladder : Among them, k B ∈[1,Y B ],Y B is the number of pixels in the bladder image, are the k values ​​of the bladder reference image and the bladder registered image, respectively. B Pixels; (4f4) Calculate the cervical organ training set loss L of the reference image P_ref and the registered image A_reg according to the Dice coefficient TCervical : Among them, k C ∈[1,Y C ],Y C is the number of pixels of the cervical image, are the k values ​​of the cervical reference image and the cervical registered image respectively. C Pixels; (4f5) Calculate the rectal organ training set loss L of the reference image P_ref and the registered image A_reg based on the Dice coefficient TRectum : Among them, k R ∈[1,Y R ],Y R is the number of pixels in the rectal image, are the k values ​​of the rectal reference image and the image after rectal registration, respectively. R Pixels; (4f6) According to L TBladder , L TCervical , L TRectum Calculate the training set loss L Tdice : L Tdice =L TBladder +L TCervical +L TRectum (4f7) According to the k pixels of the reference image in the validation set and k pixels of the registered image obtained through the deformable network Calculate the Dice coefficient: Where k∈[1,Z], Z is the number of pixels in the validation set image; (4f8) Based on the Dice coefficient, calculate the bladder organ validation set loss L in the same way as steps (4f3), (4f4), and (4f5). VBladder , cervical organ validation set loss L VCervical , rectal organ validation set loss L VRectum , get the validation set loss L Vdice : L Vdice =L VBladder +L VCervical +L VRectum 。

Images