A weakly supervised medical image segmentation and registration collaborative processing method

Abstract

The present invention relates to a weakly supervised medical image segmentation and registration collaborative processing method, comprising: obtaining medical image data and annotations, and dividing the medical image into a floating image and a fixed image; constructing a registration model from the medical image data, and updating the weakly supervised segmentation data based on the registration model; constructing a segmentation model from the weakly supervised segmentation data, and updating the weakly supervised registration data based on the segmentation model; training the registration model from the weakly supervised registration data, and updating the weakly supervised segmentation data based on the registration model; repeating steps (3) to (4) until the segmentation model and the registration model converge; and completing the image segmentation and registration tasks based on the converged segmentation model and registration model. The present invention greatly improves the performance of image segmentation tasks and image registration tasks in small sample scenarios, improves both the segmentation accuracy of brain images and the registration accuracy of brain images, and provides a new solution for medical image segmentation and registration tasks in small sample scenarios.

Description

Technical Field

[0001] The invention relates to the technical field of image segmentation and image registration research, and in particular to a weakly supervised medical image segmentation and registration collaborative processing method. Background Art

[0002] Image segmentation and image registration are basic tasks in the biomedical field, and are widely used in lesion identification, 3D image reconstruction, disease prediction, and biological research. In the biomedical field, segmentation can be achieved based on registration, and registration can also be achieved based on segmentation; registration can provide data augmentation for segmentation, and segmentation can provide regional constraints for registration. This fully reflects the high correlation and complementarity between the two tasks.

[0003] In recent years, with the rise of deep learning technology, image segmentation and image registration technology has developed rapidly again. However, unlike other computer vision fields where data volumes are often tens of thousands, data in the biomedical field is difficult to obtain. At the same time, data annotation is time-consuming and labor-intensive and requires expert knowledge, and the amount of data is usually in the hundreds. Therefore, deep learning technology in the biomedical field often needs to be implemented in scenarios with small samples and a small amount of labeled data. In this scenario, how to make full use of existing data, while mining the correlation and complementarity between multiple tasks, improving the generalization ability of the model, and ultimately improving the performance of the two tasks is a challenging and practical topic. Summary of the invention

[0004] The purpose of the present invention is to provide a weakly supervised medical image segmentation and registration collaborative processing method that greatly improves the performance of image segmentation tasks and image registration tasks in small sample scenarios, improves both the image segmentation accuracy and the image registration accuracy, and provides a new solution for medical image segmentation and registration tasks in small sample scenarios.

[0005] To achieve the above object, the present invention adopts the following technical solution: a weakly supervised medical image segmentation and registration collaborative processing method, the method comprising the following steps in order:

[0006] (1) Obtain medical image data and annotations, and divide medical images into floating images and fixed images;

[0007] (2) Building a registration model from medical image data and updating weakly supervised segmentation data based on the registration model;

[0008] (3) construct a segmentation model based on the weakly supervised segmentation data, and update the weakly supervised registration data based on the segmentation model;

[0009] (4) training the registration model with the weakly supervised registration data and updating the weakly supervised segmentation data based on the registration model;

[0010] (5) Repeat steps (3) to (4) until the segmentation model and the registration model converge;

[0011] (6) Based on the converged segmentation model and registration model, the image segmentation and registration tasks are completed respectively: the trained segmentation model is called to independently complete the segmentation task; the trained registration model is called to independently complete the registration task.

[0012] The step (1) specifically refers to: assuming that the number of images in the medical image dataset is n, among which only one image contains annotations, which is called a floating image, and the remaining n-1 images have no annotations, which are called fixed images.

[0013] The step (2) specifically comprises the following steps:

[0014] (2a) The registration model adopts a Voxel-Morph network architecture and is based on medical image data to train a registration model for registering a floating image to a fixed image;

[0015] (2b) Based on the trained registration model, the deformation field and inverse deformation field of the floating image registered to the fixed image are obtained, as shown in the following formula:

[0016]

[0017] Among them, atlas represents a floating image, fix represents a fixed image, Φ represents a deformation field, represents the inverse deformation field, the operator Indicates that a deformation field is applied to the image to transform it;

[0018] (2c) Applying the inverse deformation field to the floating image, an inverse deformation image with an anatomical structure close to the floating image and a brightness distribution close to the fixed image is obtained, as shown in the following formula:

[0019]

[0020] Among them, aug represents the inverse deformation image;

[0021] (2d) The brightness field of the real data is fitted based on the inverse deformation image and the floating image, and a brightness sampling image is obtained by sampling, as shown in the following formula:

[0022]

[0023] in, represents the brightness sampling image, λ represents the sampling coefficient of the floating image, and λ is obtained by uniform sampling from 0 to 1;

[0024] (2e) The deformation field generated by the registration model is used to fit the deformation field of the real data, and the sampled deformation field is applied to the annotations corresponding to the inverse deformed image and the floating image, respectively, to obtain the double-sampled augmented image and the corresponding pseudo annotations, as shown in the following formula:

[0025]

[0026] in, represents the double-sampled augmented image, manual represents the manual annotation corresponding to the floating image, represents the pseudo annotation corresponding to the double-sampled augmented image, is the deformation field after sampling, The deformation field Φ is obtained by uniform sampling from 0 to 1;

[0027] (2f) The deformation field is applied to the manual annotation corresponding to the floating image to obtain the pseudo annotation corresponding to the fixed image, as shown in the following formula:

[0028]

[0029] Among them, label represents the pseudo-label corresponding to the fixed image;

[0030] (2g) The fixed image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, the double-sampled augmented image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, and the two sets of new weakly supervised data are taken as new weakly supervised segmentation data, wherein the weakly supervised segmentation data refers to weakly supervised data used as input of the segmentation model.

[0031] The step (3) specifically comprises the following steps:

[0032] (3a) The segmentation model adopts a U-Net network architecture, and the weakly supervised segmentation data is sent to the segmentation model for training;

[0033] (3b) predicting the floating image using the trained segmentation model to obtain the predicted label of the floating image;

[0034] (3c) The floating image and its corresponding predicted label are used as new weakly supervised data, and the new weakly supervised data are used as new weakly supervised registration data, where the weakly supervised registration data refers to weakly supervised data used as input of the registration model.

[0035] The step (4) specifically comprises the following steps:

[0036] (4a) Send the weakly supervised registration data to the registration model and train it again;

[0037] (4b) Based on the trained registration model, the deformation field and inverse deformation field of the floating image registered to the fixed image are obtained, as shown in the following formula:

[0038]

[0039] Among them, atlas represents a floating image, fix represents a fixed image, Φ represents a deformation field, represents the inverse deformation field, the operator Indicates that a deformation field is applied to the image to transform it;

[0040] (4c) Applying the inverse deformation field to the floating image, an inverse deformation image with an anatomical structure close to the floating image and a brightness distribution close to the fixed image is obtained, as shown in the following formula:

[0041]

[0042] Among them, aug represents the inverse deformation image;

[0043] (4d) The brightness field of the real data is fitted based on the inverse deformation image and the floating image, and a brightness sampling image is obtained by sampling, as shown in the following formula:

[0044]

[0045] in, represents the brightness sampling image, λ represents the sampling coefficient of the floating image, and λ is obtained by uniform sampling from 0 to 1;

[0046] (4e) The deformation field generated by the registration model is used to fit the deformation field of the real data, and the sampled deformation field is applied to the annotations corresponding to the inverse deformed image and the floating image, respectively, to obtain the double-sampled augmented image and the corresponding pseudo annotations, as shown in the following formula:

[0047]

[0048] in, represents the double-sampled augmented image, manual represents the manual annotation corresponding to the floating image, represents the pseudo annotation corresponding to the double-sampled augmented image, The deformation field Φ is obtained by uniform sampling from 0 to 1;

[0049] (4f) The deformation field is applied to the manual annotation corresponding to the floating image to obtain the pseudo annotation corresponding to the fixed image, as shown in the following formula:

[0050]

[0051] Among them, label represents the pseudo-label corresponding to the fixed image;

[0052] (4g) The fixed image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, the double-sampled augmented image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, and the two sets of new weakly supervised data are taken as new weakly supervised segmentation data.

[0053] It can be seen from the above technical scheme that the beneficial effects of the present invention are: compared with the prior art, the present invention closely combines the data augmentation technology based on double sampling of deformation field and brightness field with the image segmentation-registration collaborative technology, thereby greatly improving the performance of image segmentation tasks and image registration tasks in small sample scenarios, which not only improves the segmentation accuracy of brain images, but also improves the registration accuracy of brain images, providing a new solution for medical image segmentation and registration tasks in small sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram for updating weakly supervised segmentation data based on the registration model;

[0055] Figure 2 Schematic diagram of updating weakly supervised registration data based on the segmentation model. DETAILED DESCRIPTION

[0056] like Figure 1 , 2 As shown, a weakly supervised medical image segmentation and registration collaborative processing method comprises the following steps in order:

[0057] (1) Obtain medical image data and annotations, and divide medical images into floating images and fixed images;

[0058] (2) Building a registration model from medical image data and updating weakly supervised segmentation data based on the registration model;

[0059] (3) construct a segmentation model based on the weakly supervised segmentation data, and update the weakly supervised registration data based on the segmentation model;

[0060] (4) training the registration model with the weakly supervised registration data and updating the weakly supervised segmentation data based on the registration model;

[0061] (5) Repeat steps (3) to (4) until the segmentation model and the registration model converge;

[0062] (6) Based on the converged segmentation model and registration model, the image segmentation and registration tasks are completed respectively: the trained segmentation model is called to independently complete the segmentation task; the trained registration model is called to independently complete the registration task.

[0063] The step (1) specifically refers to: assuming that the number of images in the medical image dataset is n, among which only one image contains annotations, which is called a floating image, and the remaining n-1 images have no annotations, which are called fixed images.

[0064] like Figure 1 As shown, the step (2) specifically includes the following steps:

[0065] (2a) The registration model adopts a Voxel-Morph network architecture and is based on medical image data to train a registration model for registering a floating image to a fixed image;

[0066] (2b) Based on the trained registration model, the deformation field and inverse deformation field of the floating image registered to the fixed image are obtained, as shown in the following formula:

[0067]

[0068] Among them, atlas represents a floating image, fix represents a fixed image, Φ represents a deformation field, represents the inverse deformation field, the operator Indicates that a deformation field is applied to the image to transform it;

[0069] (2c) Applying the inverse deformation field to the floating image, an inverse deformation image with an anatomical structure close to the floating image and a brightness distribution close to the fixed image is obtained, as shown in the following formula:

[0070]

[0071] Among them, aug represents the inverse deformation image;

[0072] (2d) The brightness field of the real data is fitted based on the inverse deformation image and the floating image, and a brightness sampling image is obtained by sampling, as shown in the following formula:

[0073]

[0074] in, represents the brightness sampling image, λ represents the sampling coefficient of the floating image, and λ is obtained by uniform sampling from 0 to 1;

[0075] (2e) The deformation field generated by the registration model is used to fit the deformation field of the real data, and the sampled deformation field is applied to the annotations corresponding to the inverse deformed image and the floating image, respectively, to obtain the double-sampled augmented image and the corresponding pseudo annotations, as shown in the following formula:

[0076]

[0077] in, represents the double-sampled augmented image, manual represents the manual annotation corresponding to the floating image, represents the pseudo annotation corresponding to the double-sampled augmented image, is the deformation field after sampling, The deformation field Φ is obtained by uniform sampling from 0 to 1;

[0078] (2f) The deformation field is applied to the manual annotation corresponding to the floating image to obtain the pseudo annotation corresponding to the fixed image, as shown in the following formula:

[0079]

[0080] Among them, label represents the pseudo-label corresponding to the fixed image;

[0081] (2g) The fixed image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, the double-sampled augmented image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, and the two sets of new weakly supervised data are taken as new weakly supervised segmentation data, wherein the weakly supervised segmentation data refers to weakly supervised data used as input of the segmentation model.

[0082] like Figure 2 As shown, the step (3) specifically includes the following steps:

[0083] (3a) The segmentation model adopts a U-Net network architecture, and the weakly supervised segmentation data is sent to the segmentation model for training;

[0084] (3b) predicting the floating image using the trained segmentation model to obtain the predicted label of the floating image;

[0085] (3c) The floating image and its corresponding predicted label are used as new weakly supervised data, and the new weakly supervised data are used as new weakly supervised registration data, where the weakly supervised registration data refers to weakly supervised data used as input of the registration model.

[0086] The step (4) specifically comprises the following steps:

[0087] (4a) Send the weakly supervised registration data to the registration model and train it again;

[0088] (4b) Based on the trained registration model, the deformation field and inverse deformation field of the floating image registered to the fixed image are obtained, as shown in the following formula:

[0089]

[0090] Among them, atlas represents a floating image, fix represents a fixed image, Φ represents a deformation field, represents the inverse deformation field, the operator Indicates that a deformation field is applied to the image to transform it;

[0091] (4c) Applying the inverse deformation field to the floating image, an inverse deformation image with an anatomical structure close to the floating image and a brightness distribution close to the fixed image is obtained, as shown in the following formula:

[0092]

[0093] Among them, aug represents the inverse deformation image;

[0094] (4d) The brightness field of the real data is fitted based on the inverse deformation image and the floating image, and a brightness sampling image is obtained by sampling, as shown in the following formula:

[0095]

[0096] in, represents the brightness sampling image, λ represents the sampling coefficient of the floating image, and λ is obtained by uniform sampling from 0 to 1;

[0097] (4e) The deformation field generated by the registration model is used to fit the deformation field of the real data, and the sampled deformation field is applied to the annotations corresponding to the inverse deformed image and the floating image, respectively, to obtain the double-sampled augmented image and the corresponding pseudo annotations, as shown in the following formula:

[0098]

[0099] in, represents the double-sampled augmented image, manual represents the manual annotation corresponding to the floating image, represents the pseudo annotation corresponding to the double-sampled augmented image, The deformation field Φ is obtained by uniform sampling from 0 to 1;

[0100] (4f) The deformation field is applied to the manual annotation corresponding to the floating image to obtain the pseudo annotation corresponding to the fixed image, as shown in the following formula:

[0101]

[0102] Among them, label represents the pseudo-label corresponding to the fixed image;

[0103] (4g) The fixed image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, the double-sampled augmented image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, and the two sets of new weakly supervised data are taken as new weakly supervised segmentation data.

[0104] In order to verify the effect of the above method, this embodiment uses this method to generate augmented data and does not use this method to generate augmented data, and trains the registration network and the segmentation network under two conditions, as shown in Table 1 below:

[0105] Table 1 Comparison of the results of the segmentation-registration collaborative model with and without this method

[0106]

[0107] It can be seen that after using this method, the Dice score of the registration model increased from 74.82% to 76.02%, and the Dice score of the segmentation model increased from 72.69% to 78.80%, indicating that this method improves the performance of the registration-segmentation collaborative method in scenarios with a small amount of manual annotation.

[0108] In order to verify the effectiveness of the dual sampling augmentation strategy, this embodiment changes the augmentation method to deformation field sampling only and brightness field sampling only, and compares it with the dual sampling strategy. The segmentation network is trained under three conditions, as shown in Table 2 below:

[0109] Table 2 Comparison of results of different augmentation strategies for segmentation models

[0110]

[0111] It can be seen that the data augmentation strategy based on double sampling has the highest Dice score of the segmentation model, reaching 78.80, which is significantly better than the single sampling strategy, indicating the necessity and superiority of double sampling in this method.

[0112] In summary, compared with the prior art, the present invention closely combines the data augmentation technology based on double sampling of deformation field and brightness field with the image segmentation-registration collaborative technology, thereby greatly improving the performance of image segmentation tasks and image registration tasks in small sample scenarios, which not only improves the segmentation accuracy of brain images, but also improves the registration accuracy of brain images, providing a new solution for medical image segmentation and registration tasks in small sample scenarios.

Claims

1. A weakly supervised medical image segmentation and registration collaborative processing method, characterized by: The method comprises the following steps in order: (1) Obtain medical image data and annotations, and divide medical images into floating images and fixed images; (2) Building a registration model from medical image data and updating weakly supervised segmentation data based on the registration model; (3) construct a segmentation model based on the weakly supervised segmentation data, and update the weakly supervised registration data based on the segmentation model; (4) training the registration model with the weakly supervised registration data and updating the weakly supervised segmentation data based on the registration model; (5) Repeat steps (3) to (4) until the segmentation model and the registration model converge; (6) Based on the converged segmentation model and registration model, the image segmentation and registration tasks are completed respectively: the trained segmentation model is called to independently complete the segmentation task; the trained registration model is called to independently complete the registration task; The step (2) specifically comprises the following steps: (2a) The registration model adopts a Voxel-Morph network architecture and is based on medical image data to train a registration model for registering a floating image to a fixed image; (2b) Based on the trained registration model, the deformation field and inverse deformation field of the floating image registered to the fixed image are obtained, as shown in the following formula: Among them, atlas represents a floating image, fix represents a fixed image, Φ represents a deformation field, represents the inverse deformation field, and the operator ° represents the transformation of the image by applying the deformation field; (2c) Applying the inverse deformation field to the floating image, an inverse deformation image with an anatomical structure close to the floating image and a brightness distribution close to the fixed image is obtained, as shown in the following formula: Among them, aug represents the inverse deformation image; (2d) The brightness field of the real data is fitted based on the inverse deformation image and the floating image, and a brightness sampling image is obtained by sampling, as shown in the following formula: in, represents the brightness sampling image, λ represents the sampling coefficient of the floating image, and λ is obtained by uniform sampling from 0 to 1; (2e) The deformation field generated by the registration model is used to fit the deformation field of the real data, and the sampled deformation field is applied to the annotations corresponding to the inverse deformed image and the floating image, respectively, to obtain the double-sampled augmented image and the corresponding pseudo annotations, as shown in the following formula: in, represents the double-sampled augmented image, manual represents the manual annotation corresponding to the floating image, represents the pseudo annotation corresponding to the double-sampled augmented image, is the deformation field after sampling, The deformation field Φ is obtained by uniform sampling from 0 to 1; (2f) The deformation field is applied to the manual annotation corresponding to the floating image to obtain the pseudo annotation corresponding to the fixed image, as shown in the following formula: Among them, label represents the pseudo-label corresponding to the fixed image; (2g) taking the fixed image and its corresponding pseudo-annotation as a new set of weakly supervised data, taking the double-sampled augmented image and its corresponding pseudo-annotation as a new set of weakly supervised data, and using the two sets of new weakly supervised data as new weakly supervised segmentation data, wherein the weakly supervised segmentation data refers to weakly supervised data used as input of a segmentation model; The step (4) specifically comprises the following steps: (4a) Send the weakly supervised registration data to the registration model and train it again; (4b) Based on the trained registration model, the deformation field and inverse deformation field of the floating image registered to the fixed image are obtained, as shown in the following formula: Among them, atlas represents a floating image, fix represents a fixed image, Φ represents a deformation field, represents the inverse deformation field, and the operator ° represents the transformation of the image by applying the deformation field; (4c) Applying the inverse deformation field to the floating image, an inverse deformation image with an anatomical structure close to the floating image and a brightness distribution close to the fixed image is obtained, as shown in the following formula: Among them, aug represents the inverse deformation image; (4d) The brightness field of the real data is fitted based on the inverse deformation image and the floating image, and a brightness sampling image is obtained by sampling, as shown in the following formula: in, represents the brightness sampling image, λ represents the sampling coefficient of the floating image, and λ is obtained by uniform sampling from 0 to 1; (4e) The deformation field generated by the registration model is used to fit the deformation field of the real data, and the sampled deformation field is applied to the annotations corresponding to the inverse deformed image and the floating image, respectively, to obtain the double-sampled augmented image and the corresponding pseudo annotations, as shown in the following formula: in, represents the double-sampled augmented image, manual represents the manual annotation corresponding to the floating image, represents the pseudo annotation corresponding to the double-sampled augmented image, The deformation field Φ is obtained by uniform sampling from 0 to 1; (4f) The deformation field is applied to the manual annotation corresponding to the floating image to obtain the pseudo annotation corresponding to the fixed image, as shown in the following formula: Among them, label represents the pseudo-label corresponding to the fixed image; (4g) The fixed image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, the double-sampled augmented image and its corresponding pseudo-annotation are taken as a new set of weakly supervised data, and the two sets of new weakly supervised data are taken as new weakly supervised segmentation data.

2. The weakly supervised medical image segmentation and registration collaborative processing method according to claim 1, characterized in that: The step (1) specifically refers to: assuming that the number of images in the medical image dataset is n, among which only one image contains annotations, which is called a floating image, and the remaining n-1 images have no annotations, which are called fixed images.

3. The weakly supervised medical image segmentation and registration collaborative processing method according to claim 1, characterized in that: The step (3) specifically comprises the following steps: (3a) The segmentation model adopts a U-Net network architecture, and the weakly supervised segmentation data is sent to the segmentation model for training; (3b) predicting the floating image using the trained segmentation model to obtain the predicted label of the floating image; (3c) The floating image and its corresponding predicted label are used as new weakly supervised data, and the new weakly supervised data are used as new weakly supervised registration data, where the weakly supervised registration data refers to weakly supervised data used as input of the registration model.

Images