A joint optimization method for medical image registration and segmentation

Abstract

The present invention discloses a joint optimization method for medical image registration and segmentation. The present invention combines the medical image segmentation task in the process of medical image registration. In order to reduce the influence of registration error and segmentation error on each other's performance, in the process of generating deformable pseudo-labels by a weakly supervised registration network RegNet, the difference between the deformable pseudo-labels and the true segmentation labels is reduced by two continuous deformations of registration. In the process of generating true segmentation labels by a semi-supervised segmentation network SegNet, the difference between the deformed image generated by the weakly supervised registration network RegNet and the fixed image is used as feature perturbation, which is randomly sampled and added to the semi-supervised segmentation network SegNet, and the feature difference in the fixed image domain is learned, thereby increasing the robustness of the semi-supervised segmentation network SegNet.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image registration and medical image segmentation, and in particular relates to a medical image registration and segmentation joint optimization method, which is suitable for simultaneously registering and segmenting medical images. Background Art

[0002] Image registration refers to solving the spatial transformation of two images (or multiple images) of the same object. Make the corresponding points in different images have spatial consistency. Usually, the image to be registered is called a floating image, and the other image that remains unchanged is called a fixed image. However, in the existing medical image registration tasks based on deep learning, there is a lack of a true deformation field for registration. The current supervised registration method uses the registration results of traditional methods as the true deformation field [Cao X, Yang J, Zhang J. IEEE. Trans. Biomed. Eng., 2018, 45: 1900.], or generates a true deformation field by artificial synthesis [Sokooti H, de Vos B, Berendsen F. MICCAI, 2017, 232-239.], but the registration accuracy of the former is limited by the error of the traditional method, and the artificially generated pseudo-labels of the latter cannot reflect the true deformation process. The unsupervised registration method completes the spatial transformation of the floating image according to the learned parameters through the Spatial Transformer Network (STN), thereby realizing the training of the network with similarity measurement as the loss function, avoiding the influence of the lack of real deformation field on the registration task.

[0003] In recent years, with the rapid development of deep learning technology, convolutional neural networks have been successfully applied to medical image segmentation [Lequan Y, Shujun W, Xiaomeng L. MICCAI, 2019, 605-613.]. Among them, supervised learning can be trained based on labeled data, and the model performance is better, but obtaining labeled data is time-consuming and labor-intensive; unsupervised learning does not require labeled data, but increases the difficulty of model learning; and semi-supervised learning is between the above two, and only a small part of labeled data is needed to train a better model. Therefore, using semi-supervised learning methods to directly perform segmentation learning on limited labeled data and a large amount of unlabeled data has important research significance.

[0004] Since medical image registration and medical image segmentation are two important research tasks in the field of medical image analysis, the two are complementary and closely related. By organically combining these two tasks, we can use the additional auxiliary information provided by each other to learn better registration or better segmentation, thereby performing medical image analysis more efficiently [Xu Z, Niethammer M. MICCAI, 2019, 420-429.]. However, the existing joint optimization methods for medical image registration and segmentation have the following problems:

[0005] (1) Registration errors will lead to incorrect labeled data. Ineffectively aligned areas will convey incorrect supervision information, and inaccurate large displacement deformations will destroy the true anatomical structure of the image. These registration errors will cause the segmentation network to learn incorrect representations, affecting segmentation accuracy.

[0006] (2) Segmentation errors will produce incorrect anatomical correspondences. Similarly, this will also cause the registration network to learn incorrect representations, affecting the registration accuracy. Summary of the invention

[0007] The purpose of the present invention is to provide a joint optimization method for medical image registration and segmentation in view of the above problems existing in the prior art.

[0008] The above-mentioned purpose of the present invention is achieved by the following technical means:

[0009] A joint optimization method for medical image registration and segmentation comprises the following steps:

[0010] Step 1: Preprocess the original medical image and the corresponding image label to obtain the preprocessed image and the corresponding true segmentation label, then divide the preprocessed image into the training set and the test set in proportion, and then divide the preprocessed image in the training set into the labeled data set and the unlabeled data set in proportion, and use the preprocessed image in the labeled data set as the floating image I M , using the pre-processed image in the unlabeled dataset as the fixed image I F ;

[0011] Step 2: Build a joint optimization network model for medical image registration and segmentation based on deep learning, including the weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet;

[0012] Step 3: Floating Image I M With fixed image I F Input to the weakly supervised registration network RegNet to obtain the registration result and the feature perturbation of the fixed image domain. The registration result includes the deformed image I after registration W , Registration deformation field and a fixed image I FThe deformation pseudo label of the floating image I M With fixed image I F Input into the semi-supervised segmentation network SegNet to obtain the segmentation result;

[0013] Step 4: Calculate the loss function of the weakly supervised registration network RegNet And the loss function of the semi-supervised segmentation network SegNet

[0014] Step 5: Loss function of the weakly supervised registration network RegNet calculated in step 4 And the loss function of the semi-supervised segmentation network SegNet Using the floating image I from step 1 M and a fixed image I F The weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet are trained alternately through the ADAM optimizer and the stochastic gradient descent method, and the parameter model is saved after the training is completed.

[0015] The preprocessing method as described above is: normalize the original medical image to obtain a normalized image, classify each voxel of the image label corresponding to the original medical image, replace each voxel of the image label with the classification value corresponding to the voxel, and obtain the original segmentation label corresponding to the normalized image; then crop the pixel size of the normalized image and the pixel size of the corresponding original segmentation label to the same, and obtain the preprocessed image and the corresponding true segmentation label.

[0016] As mentioned above, step 2 specifically includes the following steps:

[0017] Step 2.1, build a weakly supervised registration network RegNet, which includes an encoder, a decoder, and an additional convolution module;

[0018] The encoder includes multiple convolution modules, each of which includes 1 convolution layer and 1 LeakyReLU activation layer. The convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer are set.

[0019] The number of convolution modules in the decoder is the same as that in the encoder. Each convolution module includes 1 convolution layer and 1 LeakyReLU activation layer. The convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer are set. The feature map output by each convolution module in the decoder is jump-connected with the feature map output by the corresponding convolution module in the encoder. An upsampling layer is also set after each convolution module in the decoder.

[0020] The additional convolution module includes one convolution layer and one LeakyReLU activation layer. The convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer are consistent with those of the convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer in the convolution module of the decoder. The additional convolution module also includes three output channels.

[0021] Step 2.2, build a semi-supervised segmentation network SegNet. The semi-supervised segmentation network SegNet includes a teacher model and a student model; the student model includes an encoder, a decoder, and an additional convolution module. The encoder and decoder of the student model are the same as the encoder and decoder of the weakly supervised registration network RegNet. The input of the encoder of the student model is the single-channel image before splicing, that is, the input is the floating image I M or fixed image I F ; Except for the output channel of 1, the additional convolution module of the student model is the same as that of the weakly supervised registration network RegNet; the network structure of the teacher model is the same as that of the student model.

[0022] As mentioned above, in step 3, the floating image I M With fixed image I F Inputting into the weakly supervised registration network RegNet to obtain the registration result specifically includes the following steps:

[0023] Step 3.1: Floating image I M With fixed image I F After splicing in the channel dimension, the input is sent to the weakly supervised registration network RegNet for the first registration to obtain the first-level deformation field And use the first order deformation field For floating images I M Perform spatial transformation to obtain the deformed image I W ;

[0024] Step 3.2: Deformed image I W and a fixed image I F After splicing, it is re-input into the weakly supervised registration network RegNet for a second registration to obtain the second-level deformation field

[0025] Step 3.3: Set the first level deformation field Acts on floating images I M The true segmentation label Y M Above, floating image I M The true segmentation label Y M After a spatial transformation, the first-level deformation label Y is obtained 1 , and then the second level deformation field Acts on the first level deformation label Y 1 On the top, the first level deformation label Y 1 After a spatial transformation, the second-level deformation label Y is obtained 2 , with the second level deformation label Y 2 As a fixed image I F Deformed pseudo labels;

[0026] Step 3.4: Get the random feature perturbation T based on the following formula, and use the random feature perturbation T as the feature perturbation of the fixed image domain:

[0027] T=α 3 *(I W -I F )

[0028] α 3 is the coefficient of random sampling in the β distribution, (I W -I F ) is the feature perturbation, i.e., the deformed image I W With fixed image I F The difference between.

[0029] As mentioned above, in step 3, the floating image I M With fixed image I F Inputting into the semi-supervised segmentation network SegNet to obtain the segmentation result specifically includes the following steps:

[0030] Step 3.5: Floating image I M Input to the student model to obtain floating image I M Segmentation results of the student model Fix the image I F Input into the student model to obtain a fixed image I F Segmentation results of the student model

[0031] Step 3.6: In the floating image I M After adding random noise N and the random feature perturbation T obtained in step 3.4, it is input into the teacher model to obtain the floating image I M Segmentation result diagram of the teacher model In the fixed image I F After adding random noise N and the random feature perturbation T obtained in step 3.4, it is input into the teacher model to obtain a fixed image I F Segmentation result diagram of the teacher model

[0032] As mentioned above, the loss function of the weakly supervised registration network RegNet Including deformation image I W With fixed image IF The similarity measure loss L between sim , the first-order deformation field The smoothness constraint loss L smooth , and fixed image I F Segmentation results of the student model With the first level deformation label Y 1 The segmentation loss between

[0033] Loss function of weakly supervised registration network RegNet Calculated based on the following formula:

[0034]

[0035] In the formula, α 1 is the segmentation loss Weight;

[0036] Deformed Image I W With fixed image I F The similarity measure loss L between sim is the deformed image I W With fixed image I F The negative local correlation coefficient between them is used as the loss function;

[0037] The first deformation field The smoothness constraint loss L smooth is the first-order deformation field The diffusion regularization loss of spatial gradients in all directions is used as the loss function;

[0038] Fixed Image I F Segmentation results of the student model With the first level deformation label Y 1 The Dice similarity coefficient between

[0039] As mentioned above, the loss function of the semi-supervised segmentation network SegNet Include floating images I M The true segmentation label Y M The supervision loss L sup , fixed image I F The segmentation loss L of the deformable pseudo-label pse , and the consistency loss L between the teacher and student models cons ;

[0040] Loss function of the semi-supervised segmentation network SegNet Calculated based on the following formula:

[0041]

[0042] α 2 is a fixed image I F The segmentation loss L of the deformable pseudo-label pse Weight; λ is a gradient weighting coefficient t is the current training epoch; t max is set to the maximum training epoch; T is the random feature perturbation obtained in step 3.4, The floating image I obtained in step 3.5 M The segmentation result diagram of the student model, is the fixed image I obtained in step 3.5 F The segmentation result diagram of the student model, The fixed image I obtained in step 3.6 F The segmentation result diagram of the teacher model, Y 2 is the second-level deformation label obtained in step 3.2, also the fixed image I F Deformed pseudo labels;

[0043] Floating Image I M The true segmentation label Y M The supervision loss L sup and a fixed image I F The Dice similarity coefficient of the deformed pseudo-label is used as the segmentation loss L pse .

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The medical image registration and segmentation joint optimization method of the present invention combines the medical image segmentation task in the medical image registration process, uses the real segmentation label generated by the semi-supervised segmentation network SegNet to provide auxiliary anatomical structure information, and constrains the weakly supervised registration network RegNet to generate a deformation field that is more in line with the real anatomical structure. In turn, the weakly supervised registration network RegNet can also provide deformation pseudo labels for the semi-supervised segmentation network SegNet for supervised training. In order to reduce the impact of registration error and segmentation error on each other's performance, in the process of generating deformation pseudo labels by the weakly supervised registration network RegNet, the difference between the deformation pseudo label and the real segmentation label is reduced by two continuous deformations of registration; in the process of generating real segmentation labels by the semi-supervised segmentation network SegNet, the difference between the deformed image generated by the weakly supervised registration network RegNet and the fixed image is used as a feature perturbation, randomly sampled and added to the semi-supervised segmentation network SegNet, and the feature differences in the fixed image domain are learned, thereby increasing the robustness of the semi-supervised segmentation network SegNet. The semi-supervised segmentation network SegNet and the weakly supervised registration network RegNet can use complementary information in training to improve each other's performance. In view of the related problems of existing medical image registration, the method of the present invention can quickly calculate the registration result, improve the registration accuracy, and also obtain the corresponding real segmentation labels of the image pair to be registered. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the present invention;

[0047] Figure 2 A schematic diagram of the process of the joint optimization network model of medical image registration and segmentation based on deep learning of the present invention;

[0048] Figure 3 Schematic diagram of the structure of the weakly supervised registration network RegNet in Example 1 of the present invention. The numbers in the image blocks refer to the number of channels of the input image or the feature map output by the convolution module, and the numbers below represent the ratio of the image resolution to the input image;

[0049] Figure 4 It is a schematic diagram of the result of registering and segmenting the test set using the trained weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet in Example 1 of the present invention;

[0050] Among them, the first column is a fixed image I F , the second column is the floating image I M , the third column is the fixed image I F The fourth column is the floating image I M The actual segmentation label of the fifth column is fixed image I F The segmentation result diagram, the sixth column is the fixed image IF The deformable pseudo-labels. DETAILED DESCRIPTION

[0051] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below with reference to examples. The implementation examples described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0052] Embodiment 1:

[0053] A joint optimization method for medical image registration and segmentation comprises the following steps:

[0054] Step 1: Preprocess the original medical image and the corresponding image label to obtain the preprocessed image and the corresponding true segmentation label, then divide the preprocessed image into a training set and a test set in proportion, and then divide the preprocessed image in the training set into a labeled data set and an unlabeled data set in proportion. The preprocessed image in the labeled data set provides the true segmentation label during training, while the preprocessed image in the unlabeled data set does not provide the true segmentation label during training. The preprocessed image in the labeled data set is used as the floating image I M , using the pre-processed image in the unlabeled dataset as the fixed image I F .

[0055] The preprocessing method is: normalize the original medical image to obtain a normalized image, classify each voxel of the image label corresponding to the original medical image, replace each voxel of the image label with the classification value corresponding to the voxel, and obtain the original segmentation label corresponding to the normalized image; then crop the pixel size of the normalized image and the pixel size of the corresponding original segmentation label to the same, reduce the proportion of background voxels, and obtain the preprocessed image and the corresponding true segmentation label.

[0056] In this embodiment, 60 original cardiac CT images in the MM-WHS cardiac CT data set are selected as original medical images, and the 60 original cardiac CT images are normalized so that the image intensity is between [0, 1] to obtain normalized CT images. The classification value of image segmentation is set to 0-7, where 0 is the background voxel and 1-7 is the different cardiac structures. Each voxel of the image label of the original cardiac CT image is replaced with the classification value corresponding to the voxel to obtain the original segmentation label corresponding to the normalized CT image; then the pixel size of the normalized CT image and the pixel size of the corresponding original segmentation label are cropped to 128×144×144 to obtain the preprocessed CT image and the corresponding true segmentation label. After obtaining the preprocessed CT image and the corresponding true segmentation label, 50 cases are selected as the training set and the remaining 10 cases are used as the test set. The preprocessed CT images in the training set are then divided into a labeled data set and an unlabeled data set at a ratio of 1:4, and the images in the 10 labeled data sets are used as floating images I. M , using 40 images from the unlabeled dataset as fixed images I F .

[0057] Step 2: Build a joint optimization network model for medical image registration and segmentation based on deep learning. The joint optimization network model for medical image registration and segmentation based on deep learning includes a weakly supervised registration network RegNet and a semi-supervised segmentation network SegNet. The weakly supervised registration network RegNet is used to output the registration result (i.e., the deformed image I after registration). W , Registration deformation field And fixed image I F The semi-supervised segmentation network SegNet is mainly a teacher-student segmentation model used to output segmentation results (i.e., the segmentation result map corresponding to the image).

[0058] Preferably, since there is a certain registration error between the deformed pseudo-labels generated by registration and the real segmentation labels, in order to avoid the influence of the registration error on the semi-supervised segmentation network SegNet, the continuous deformation idea is introduced to reduce the difference between the deformed pseudo-labels and the real segmentation labels through two registrations. When training the weakly supervised registration network RegNet, only the deformed image, the first-level deformation field and the first-level deformation label are used to calculate the loss function of the weakly supervised registration network RegNet. The second-level deformation field and second-level deformation label are only used when training the semi-supervised segmentation network SegNet.

[0059] Step 2.1: Build the weakly supervised registration network RegNet. The weakly supervised registration network RegNet includes an encoder, a decoder, and an additional convolutional module.

[0060] The encoder includes multiple convolution modules, each of which includes 1 convolution layer and 1 LeakyReLU activation layer. The convolution kernel size of the convolution layer is set to 3×3×3, the convolution step size is set to 2, and the LeakyReLU activation layer parameter is set to 0.2. The image resolution is halved after each convolution. This embodiment uses 4 convolution modules for downsampling.

[0061] The decoder includes multiple convolution modules, and the number of convolution modules is the same as that of the encoder. Each convolution module includes 1 convolution layer and 1 LeakyReLU activation layer. The convolution kernel size of the convolution layer is 3×3×3, the convolution step size is 1, and the LeakyReLU activation layer parameter is set to 0.2. The feature map output by each convolution module in the decoder is jump-connected with the feature map output by the corresponding convolution module in the encoder. There is also an upsampling layer after each convolution module in the decoder. The upsampling layer uses a bilinear interpolation method to double the image resolution and restore it to the original image size layer by layer.

[0062] The decoder of this embodiment uses 4 convolution modules and 4 upsampling layers, with 4 jump connections.

[0063] The feature map output by each convolution module in the decoder is skipped with the feature map output by the corresponding convolution module in the encoder to maintain the reusability of the features and solve the problem of model degradation.

[0064] The structure of the additional convolution module is consistent with that of the convolution module in the decoder, that is, the additional convolution module includes one convolution layer and one LeakyReLU activation layer. The convolution kernel size of the convolution layer is 3×3×3, the convolution step size is 1, and the LeakyReLU activation layer parameter is set to 0.2. In addition, the additional convolution module also includes three output channels, which are used to output as floating images I M The displacement of each voxel point in the preprocessed image in the three coordinate directions is calculated to obtain the registration deformation field. The additional convolution module further fuses features and adjusts the number of output channels based on the feature map output by the decoder.

[0065] Step 2.2, build a semi-supervised segmentation network SegNet. The semi-supervised segmentation network SegNet includes a teacher model and a student model. The network structure of the student model is similar to that of the weakly supervised registration network RegNet. The student model includes an encoder, a decoder, and an additional convolution module. The encoder and decoder of the student model are the same as those of the weakly supervised registration network RegNet. The input of the encoder of the student model is the single-channel image before splicing, that is, the input is the floating image I M or fixed image I FThe additional convolution module of the student model is the same as that of the weakly supervised registration network RegNet, except that the output channel is 1, and is used to output the segmentation result map of the student model; the network structure of the teacher model is the same as that of the student model, and the output channel of the teacher model outputs the segmentation result map of the teacher model.

[0066] The student model minimizes the floating image I M The true segmentation label Y M The supervised loss, fixed image I F The segmentation loss of the deformable pseudo-label and the consistency loss between the teacher and student models are optimized. The student model uses the uncertainty information of the teacher model to gradually learn from meaningful and reliable segmentation results, which increases the robustness of the network. The teacher model updates the teacher model parameters θ′ through the exponential moving average (EMA) of the student model parameters θ. Specifically, the teacher model parameters θ′ at time t are t Updated to: θ′ t =αθ′ t-1 +(1-α)θ t , where α is the exponential moving average update rate, α is set to 0.99 based on experience, and θ′ t-1 is the teacher model parameter at time t-1, θ t is the student model parameter at time t.

[0067] The teacher model generates segmentation results that the student model needs to learn, and also adds random noise through random feature perturbation T and Monte Carlo Dropout to estimate the uncertainty of each segmentation result. Under the guidance of estimated uncertainty, unreliable segmentation results are filtered out when calculating consistency loss, and only reliable segmentation results (low uncertainty) are retained. Therefore, the student model is more reliably supervised during the optimization process, and in turn encourages the teacher model to generate higher quality segmentation results.

[0068] Step 3: Floating Image I M With fixed image I F Input to the weakly supervised registration network RegNet to obtain the registration result (i.e. the deformed image I after registration) W , Registration deformation field and a fixed image I F The deformation pseudo label of the floating image I M With fixed image I F Input into the semi-supervised segmentation network SegNet to obtain the segmentation result, which specifically includes the following steps:

[0069] Step 3.1: Floating image I M With fixed image I FAfter splicing in the channel dimension, the input is sent to the weakly supervised registration network RegNet for the first registration to obtain the first-level deformation field And use the first order deformation field For floating images I M Perform spatial transformation to obtain the deformed image I W ;

[0070] Step 3.2: Deformed image I W and a fixed image I F After splicing, it is re-input into the weakly supervised registration network RegNet for a second registration to obtain the second-level deformation field

[0071] Step 3.3: Set the first level deformation field Acts on floating images I M The true segmentation label Y M Above, floating image I M The true segmentation label Y M After a spatial transformation, the first-level deformation label Y is obtained 1 , and then the second level deformation field Acts on the first level deformation label Y 1 On the top, the first level deformation label Y 1 After a spatial transformation, the second-level deformation label Y is obtained 2 , with the second level deformation label Y 2 As a fixed image I F Deformed pseudo labels;

[0072] Step 3.4: Get the random feature perturbation T based on the following formula, and use the random feature perturbation T as the feature perturbation of the fixed image domain:

[0073] T=α 3 *(I W -I F )

[0074] α 3 is the coefficient of random sampling in the β distribution, (I W -I F ) is the feature perturbation, i.e., the deformed image I W With fixed image I F The difference between.

[0075] Step 3.5: Floating image I M Input to the student model to obtain floating image I M Segmentation results of the student model Fix the image I F Input into the student model to obtain a fixed image I F Segmentation results of the student model

[0076] Step 3.6: In the floating image I M After adding random noise N and the random feature perturbation T obtained in step 3.4, it is input into the teacher model to obtain the floating image I M Segmentation result diagram of the teacher model In the fixed image I F After adding random noise N and the random feature perturbation T obtained in step 3.4, it is input into the teacher model to obtain a fixed image I F Segmentation result diagram of the teacher model

[0077] Step 4: Calculate the loss function of the weakly supervised registration network RegNet And the loss function of the semi-supervised segmentation network SegNet

[0078] Loss function of weakly supervised registration network RegNet Including deformation image I W With fixed image I F The similarity measure loss L between sim , the first-order deformation field The smoothness constraint loss L smooth , and fixed image I F Segmentation results of the student model With the first level deformation label Y 1 The segmentation loss between

[0079] Loss function of the semi-supervised segmentation network SegNet Include floating images I M The true segmentation label Y M The supervision loss L sup , fixed image I F The segmentation loss L of the deformable pseudo-label pse , and the consistency loss L between the teacher and student models cons .

[0080] Loss function of weakly supervised registration network RegNet And the loss function of the semi-supervised segmentation network SegNet The specific calculation formula is as follows:

[0081]

[0082] Among them, α 1 , α 2 The segmentation loss is Weights and fixed images I F The segmentation loss L of the deformable pseudo-label pse The weights are set to 100 and 1 in this embodiment; λ is a gradient weighting coefficient used to balance the weight between the supervision loss and the consistency loss, and is defined as a Gaussian heating function in this embodiment, that is: t is the current training epoch, t max is the maximum training epoch set; T is the random feature perturbation obtained in step 3.1.

[0083] Similarity measure loss L sim Select the deformation image I W With fixed image I F The negative local cross-correlation coefficient between the two images is used as the loss function, which is robust to the intensity changes of the two images.

[0084] Smoothness constraint loss L smooth Select the first level deformation field The diffusion regularization loss of spatial gradients in all directions is used as the loss function to make the registration deformation field have good smoothness and avoid deformation that does not conform to the actual topological structure.

[0085] Segmentation loss Select Fixed Image I F Segmentation results of the student model With the first level deformation label Y 1 The Dice similarity coefficient between them.

[0086] Floating Image I M The true segmentation label Y M The supervision loss L sup and a fixed image I F The segmentation loss L of the deformable pseudo-label pse Both are Dice similarity coefficients, which can measure the overlap of the segmentation result maps output by the semi-supervised segmentation SegNet network.

[0087] The consistency loss L between the teacher and student models cons Random noise and random feature perturbation T are added through the Monte Carlo Dropout layer to estimate the uncertainty of each segmentation result. Unreliable segmentation results are filtered out when calculating the consistency loss, and only reliable segmentation results (low uncertainty) are retained.

[0088] Step 5: Loss function of the weakly supervised registration network RegNet calculated in step 4 And the loss function of the semi-supervised segmentation network SegNet The weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet are alternately trained using the training set in step 1 through the ADAM optimizer and the stochastic gradient descent method. The batch size is set to 1, the epoch is set to 100, and each epoch is iterated 100 times. The learning rate of the weakly supervised registration network RegNet is set to 0.0001, and the initial learning rate of the semi-supervised segmentation network SegNet is set to 0.01, which is reduced to 1 / 10 every 20 epochs. The ADAM optimizer is used for optimization, and the parameter model is saved after training.

[0089] When testing the weakly supervised registration network RegNet, the two preprocessed images in the test set are used as fixed images I and F and floating image I M Input it to the weakly supervised registration network RegNet trained in step 5, and then output the registration result (i.e. the deformed image I after registration) W , Registration deformation field and a fixed image I F When testing the semi-supervised segmentation network SegNet, an image to be segmented in the test set is input into the semi-supervised segmentation network SegNet trained in step 5, and then the segmentation result (i.e., the segmentation result map output by the student model) is output. The registration results and segmentation results are quantitatively evaluated, and the fixed image I is used to F The real segmentation label and the fixed image I in the registration result F The deformation pseudo-labels are used to calculate the registration Dice similarity coefficient; the segmentation Dice similarity coefficient is calculated using the segmentation results and the corresponding true segmentation labels, so as to quantitatively analyze the performance of the weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet.

[0090] Table 1 shows the quantitative analysis results of the registration and segmentation of the test set using the trained weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet.

[0091]

[0092] From Table 1 and Figure 4 It can be seen from the figure that the fixed image I of the weakly supervised registration network RegNet finally obtained by the present invention F The deformation pseudo-labels have good effects on visual and similarity indicators, which are consistent with the actual deformation process. The segmentation results generated by the semi-supervised segmentation network SegNet are equally good as the real segmentation labels in terms of visual and similarity indicators, indicating that the method of the present invention can quickly, accurately and simultaneously realize medical image registration and segmentation.

[0093] A medical image registration and segmentation joint optimization device, comprising a preprocessing module, a RegNet network and a SegNet network building module, a RegNet network and a SegNet network execution module, a loss function calculation module, and a training module;

[0094] A preprocessing module storing a program for implementing step 1;

[0095] The RegNet network and SegNet network building modules store the program for implementing step 2;

[0096] RegNet network and SegNet network execution modules, storing a program for implementing step 3;

[0097] A loss function calculation module storing a program for implementing step 4;

[0098] A training module storing a program for implementing step 5;

[0099] A computer medium stores a program for implementing the method of steps 1 to 5.

[0100] It should be noted that the specific embodiments described in the present invention are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.

Claims

1. A joint optimization method for medical image registration and segmentation, It is characterized in that The following steps are involved: Step 1: Preprocess the original medical image and the corresponding image label to obtain the preprocessed image and the corresponding true segmentation label, then divide the preprocessed image into the training set and the test set in proportion, and then divide the preprocessed image in the training set into the labeled data set and the unlabeled data set in proportion, and use the preprocessed image in the labeled data set as the floating image I M , using the pre-processed image in the unlabeled dataset as the fixed image I F ; Step 2: Build a joint optimization network model for medical image registration and segmentation based on deep learning, including the weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet; Step 3: Floating Image I M With fixed image I F Input to the weakly supervised registration network RegNet to obtain the registration result and the feature perturbation of the fixed image domain. The registration result includes the deformed image I after registration W , Registration deformation field and a fixed image I F The deformation pseudo label of the floating image I M With fixed image I F Input into the semi-supervised segmentation network SegNet to obtain the segmentation result; Step 4: Calculate the loss function of the weakly supervised registration network RegNet And the loss function of the semi-supervised segmentation network SegNet is the first-order deformation field, For a fixed image I F Segmentation result diagram of the student model, Y 1 is the first-level deformation label, For floating image I M The segmentation result diagram of the student model, For a fixed image I F The segmentation result diagram of the student model, For a fixed image I F The segmentation result diagram of the teacher model, Y M For floating image I M The true segmentation label, Y 2 is the second-level deformation label, T is the random feature perturbation; Step 5: Loss function of the weakly supervised registration network RegNet calculated in step 4 And the loss function of the semi-supervised segmentation network SegNet Using the floating image I from step 1 M and a fixed image I F The weakly supervised registration network RegNet and the semi-supervised segmentation network SegNet are trained alternately through the ADAM optimizer and the stochastic gradient descent method, and the parameter model is saved after the training is completed.

2. According to claim 1, a joint optimization method for medical image registration and segmentation, It is characterized in that The preprocessing method comprises: performing normalization processing on the original medical image to obtain a normalized image, classifying each voxel of the image label corresponding to the original medical image, replacing each voxel of the image label with a classification value corresponding to the voxel, and obtaining an original segmentation label corresponding to the normalized image; Then the pixel size of the normalized image and the pixel size of the corresponding original segmentation label are cropped to the same size to obtain the preprocessed image and the corresponding true segmentation label.

3. According to claim 2, a joint optimization method for medical image registration and segmentation, It is characterized in that The step 2 specifically includes the following steps: Step 2.1, build a weakly supervised registration network RegNet, which includes an encoder, a decoder, and an additional convolution module; The encoder includes multiple convolution modules, each of which includes 1 convolution layer and 1 LeakyReLU activation layer. The convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer are set. The number of convolution modules in the decoder is the same as that in the encoder. Each convolution module includes one convolution layer and one LeakyReLU activation layer. The convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer are set. The feature map output by each convolution module in the decoder is jump-connected with the feature map output by the corresponding convolution module in the encoder. An upsampling layer is also set after each convolution module in the decoder. The additional convolution module includes one convolution layer and one LeakyReLU activation layer. The convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer are consistent with those of the convolution kernel size, convolution step size, and LeakyReLU activation layer parameters of the convolution layer in the convolution module of the decoder. The additional convolution module also includes three output channels. Step 2.2, build a semi-supervised segmentation network SegNet. The semi-supervised segmentation network SegNet includes a teacher model and a student model; the student model includes an encoder, a decoder, and an additional convolution module. The encoder and decoder of the student model are the same as the encoder and decoder of the weakly supervised registration network RegNet. The input of the encoder of the student model is the single-channel image before splicing, that is, the input is the floating image I M or fixed image I F ; Except for the output channel of 1, the additional convolution module of the student model is the same as that of the weakly supervised registration network RegNet; the network structure of the teacher model is the same as that of the student model.

4. According to claim 3, a joint optimization method for medical image registration and segmentation, It is characterized in that In step 3, the floating image I M With fixed image I F Inputting into the weakly supervised registration network RegNet to obtain the registration result specifically includes the following steps: Step 3.1: Floating image I M With fixed image I F After splicing in the channel dimension, the input is sent to the weakly supervised registration network RegNet for the first registration to obtain the first-level deformation field And use the first order deformation field For floating images I M Perform spatial transformation to obtain the deformed image I W ; Step 3.2: Deformed image I W and a fixed image I F After splicing, it is re-input into the weakly supervised registration network RegNet for a second registration to obtain the second-level deformation field Step 3.3: Set the first level deformation field Acts on floating images I M The true segmentation label Y M Above, floating image I M The true segmentation label Y M After a spatial transformation, the first-level deformation label Y is obtained 1 , and then the second level deformation field Acts on the first level deformation label Y 1 On the top, the first level deformation label Y 1 After a spatial transformation, the second-level deformation label Y is obtained 2 , with the second level deformation label Y 2 As a fixed image I F Deformed pseudo labels; Step 3.4: Get the random feature perturbation T based on the following formula, and use the random feature perturbation T as the feature perturbation of the fixed image domain: T=α 3 *(I W -I F ) α 3 is a coefficient randomly sampled from the β distribution, (I W - I F ) is the feature perturbation, that is, the difference between the deformed image I W and the fixed image I F .

5. According to claim 4, a joint optimization method for medical image registration and segmentation, It is characterized in that In step 3, the floating image I M With fixed image I F Inputting into the semi-supervised segmentation network SegNet to obtain the segmentation result specifically includes the following steps: Step 3.5: Floating image I M Input to the student model to obtain floating image I M Segmentation results of the student model Fix the image I F Input into the student model to obtain a fixed image I F Segmentation results of the student model Step 3.6: In the floating image I M After adding random noise N and the random feature perturbation T obtained in step 3.4, it is input into the teacher model to obtain the floating image I M Segmentation result diagram of the teacher model In the fixed image I F After adding random noise N and the random feature perturbation T obtained in step 3.4, it is input into the teacher model to obtain a fixed image I F Segmentation result diagram of the teacher model 6. A medical image registration and segmentation joint optimization method according to claim 5, It is characterized in that The loss function of the weakly supervised registration network RegNet Including deformation image I W With fixed image I F The similarity measure loss L between sim , the first-order deformation field The smoothness constraint loss L smooth , and fixed image I F Segmentation results of the student model With the first level deformation label Y 1 The segmentation loss between Loss function of weakly supervised registration network RegNet Calculated based on the following formula: In the formula, α 1 is the segmentation loss Weight; Deformed Image I W With fixed image I F The similarity measure loss L between sim is the deformed image I W With fixed image I F The negative local correlation coefficient between them is used as the loss function; First level deformation field The smoothness constraint loss L smooth is the first-order deformation field The diffusion regularization loss of spatial gradients in all directions is used as the loss function; Fixed Image I F Segmentation results of the student model With the first level deformation label Y 1 The Dice similarity coefficient between 7. A medical image registration and segmentation joint optimization method according to claim 6, It is characterized in that The loss function of the semi-supervised segmentation network SegNet Include floating images I M The true segmentation label Y M The supervision loss L sup , fixed image I F The segmentation loss L of the deformable pseudo-label pse , and the consistency loss L between the teacher model and the student model cons ; Loss function of the semi-supervised segmentation network SegNet Calculated based on the following formula: α 2 is a fixed image I F The segmentation loss L of the deformable pseudo-label pse Weight; λ is a gradient weighting coefficient t is the current training epoch; t max Set to the maximum training epoch; T is the random feature perturbation obtained in step 3.4, The floating image I obtained in step 3.5 M The segmentation result diagram of the student model, is the fixed image I obtained in step 3.5 F The segmentation result diagram of the student model, is the fixed image I obtained in step 3.6 F The segmentation result diagram of the teacher model, Y 2 is the second-level deformation label obtained in step 3.2, also the fixed image I F Deformed pseudo labels; Floating Image I M The true segmentation label Y M The supervision loss L sup and a fixed image I F The segmentation loss L of the deformable pseudo-label pse Both are Dice similarity coefficients.

Images