Weak supervision image segmentation method and system based on incomplete multi-modal medical image
By constructing a weakly supervised method with pairwise pixel constraints and iterative fine-tuning, the problems of modality loss and limited annotation in multimodal medical image segmentation are solved, achieving efficient image segmentation under incomplete data, which is suitable for auxiliary diagnostic systems.
Patent Information
- Application Number
- CN202411278460.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-12
AI Technical Summary
Existing multimodal medical image segmentation methods rely on a large amount of labeled data and computational resources, which cannot effectively cope with incomplete multimodal data scenarios, especially when modalities are missing, resulting in insufficient generalization ability.
We design an image segmentation method based on weakly supervised annotation. By constructing pairwise pixel constraints, iterative fine-tuning, and cluster analysis, we generate segmentation results for incomplete multimodal medical images. We then use a denoising neural network and a fusion function to handle the negative impact of missing modalities.
It achieves effective image segmentation under conditions of modality loss and limited annotation, reduces complexity, and can be integrated into auxiliary diagnostic systems.
Smart Images

Figure CN118982673A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of auxiliary diagnosis of multimodal medical images, and in particular to a weakly supervised image segmentation method and system based on incomplete multimodal medical images. Background Art
[0002] Multimodal medical image segmentation aims to organically integrate medical image data from different imaging technologies, such as MRI (magnetic resonance imaging), CT (computed tomography), ultrasound, etc., into a unified comprehensive representation, so that it can contain more intuitive, detailed, and accurate information as much as possible, so as to improve the accuracy and efficiency of medical image segmentation and assist clinicians in making medical diagnoses. By using the information of different modal images to complement each other organically, the accuracy of image segmentation can be improved. For example, in brain tumor segmentation, images of multiple MRI modalities such as T1-weighted, T2-weighted, and FLAIR (fluid attenuated inversion recovery) can be combined for fusion to more accurately segment the lesion area.
[0003] The technical challenges faced by multimodal medical image segmentation include dataset annotation, computing resource limitations, and algorithm generalization capabilities. Specifically, (1) Most existing methods rely on a large amount of image data annotation, but in real-world scenarios, obtaining image data annotations requires huge manpower costs and financial investment; (2) Most existing methods are usually limited by their high complexity, which makes them unable to cope with large-scale data scenarios well; (3) Most existing methods can only cope with complete multimodal data (i.e., all modalities are not missing). When modalities are missing, they cannot generalize well to such incomplete multimodal data scenarios. Summary of the invention
[0004] In order to solve the above technical problems, the present invention proposes a weakly supervised image segmentation method and system based on incomplete multimodal medical images, constructs paired pixel constraints for each image sample in the incomplete multimodal medical image data based on some given weakly supervised annotation information, designs an optimization algorithm to iteratively fine-tune each modality of the denoised image samples, and ensures that the two constraints are met during the iteration process, designs a fusion function to organically fuse multiple modalities of the denoised image samples, designs a base learning objective, performs cluster analysis on the fused image samples, generates clustering results of its pixel points, and then obtains its segmentation result, jointly trains all learning objectives, and outputs the final image segmentation result.
[0005] In order to achieve the above object, the technical solution of the present invention is as follows: The weakly supervised image segmentation method based on incomplete multimodal medical images includes the following steps: T1. Collection of incomplete multimodal medical imaging data; T2. Based on the given weakly supervised annotation information, pairwise pixel constraints are constructed for each image sample in the incomplete multimodal medical image data. Pairwise pixel constraints are divided into must-connect constraints ML and no-connect constraints CL. When the must-connect constraints ML are satisfied, the two pixels belong to the same cluster. When the no-connect constraints CL are satisfied, the two pixels belong to different clusters. T3. Clean multimodal medical imaging data; T4. Construct a multimodal medical imaging sample dataset; T5. Input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; T6. Iteratively fine-tune each modality of the denoised multimodal medical image samples, and ensure that the must-connect constraint ML and the must-connect constraint CL are satisfied during the iteration process; T7. Design a fusion function to organically fuse multiple modalities of denoised multimodal medical image samples while avoiding the negative impact of missing modalities; T8. Design a loss function to perform cluster analysis on the fused multimodal medical image samples, generate clustering results of its pixels, and then obtain its segmentation results; T9. Jointly optimize the loss function until the convergence criterion is met and output the final image segmentation result.
[0006] The present invention also provides a weakly supervised image segmentation system based on incomplete multimodal medical images, comprising the following modules: Data acquisition module, used to collect incomplete multimodal medical imaging data; Construct a constraint module to construct paired pixel constraints for each image sample in the incomplete multimodal medical image data based on some given weakly supervised annotation information. The constraints are divided into the must-connect constraint ML and the no-connect constraint CL. When the must-connect constraint ML is satisfied, the two pixels belong to the same cluster. When the no-connect constraint CL is satisfied, the two pixels belong to different clusters. Data cleaning module, used to clean multimodal medical imaging data; Dataset construction module, used to construct multimodal medical imaging sample dataset; The denoising module is used to input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; An iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised multimodal medical image samples, and ensure that the must-connect constraint ML and the no-connect constraint CL are satisfied during the iteration process; The fusion module is used to design a fusion function to organically fuse multiple modalities of denoised multimodal medical image samples while avoiding the negative impact of missing modalities; The segmentation module is used to design the loss function, perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain its segmentation results; The output module is used to jointly optimize the loss function until the convergence criterion is met and output the final image segmentation result.
[0007] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned weakly supervised image segmentation method based on incomplete multimodal medical images are implemented.
[0008] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-mentioned weakly supervised image segmentation method based on incomplete multimodal medical images.
[0009] Beneficial effects: The present invention can be effectively generalized to scenarios with missing modalities and application scenarios with limited annotations, and thanks to its low complexity (positively correlated with the sample size), it can be effectively integrated into auxiliary diagnosis systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 It is a flow chart of the weakly supervised image segmentation method based on incomplete multimodal medical images of the present invention; Figure 2 It is a schematic diagram of the weakly supervised image segmentation method based on incomplete multimodal medical images of the present invention; Figure 3a This is a diagram showing the effect achieved on the original images of the brain tumor dataset BRATS 2017; Figure 3b A schematic diagram of a segmented image achieved by applying the method of the present invention to the brain tumor dataset BRATS 2017. DETAILED DESCRIPTION
[0011] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0012] like Figure 1 As shown, the weakly supervised image segmentation method based on incomplete multimodal medical images of the present invention comprises the following steps: T1. Collection of incomplete multimodal medical imaging data; T2. Based on the given weakly supervised annotation information, pairwise pixel constraints are constructed for each image sample in the incomplete multimodal medical image data. Pairwise pixel constraints are divided into must-connect constraints ML and no-connect constraints CL. When the must-connect constraints ML are satisfied, the two pixels belong to the same cluster. When the no-connect constraints CL are satisfied, the two pixels belong to different clusters. T3. Clean multimodal medical imaging data; T4. Construct a multimodal medical imaging sample dataset; T5. Input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; T6. Iteratively fine-tune each modality of the denoised multimodal medical image samples, and ensure that the must-connect constraint ML and the must-connect constraint CL are satisfied during the iteration process; T7. Design a fusion function to organically fuse multiple modalities of denoised multimodal medical image samples while avoiding the negative impact of missing modalities; T8. Design a loss function to perform cluster analysis on the fused multimodal medical image samples, generate clustering results of its pixels, and then obtain its segmentation results; T9. Jointly optimize the loss function until the convergence criterion is met and output the final image segmentation result.
[0013] Preferably, in said T3, data cleaning includes maximum normalization, minimum normalization, Z-score normalization and L2 norm normalization; Preferably, if Figure 2 As shown, in T5, the training process of the denoising neural network is as follows: The multimodal medical image sample data is input into the denoising neural network, and its encoder converts the input data into low-dimensional features. Then the decoder reconstructs the input data based on the low-dimensional features to achieve the denoising effect. Iterative fine-tuning is performed by minimizing the error between the original input multimodal medical image sample data and the reconstructed input data. The specific formula is as follows: ; in, To reconstruct the loss function, Indicates whether the vth modality of the ith image sample is missing, if not missing it is 1, if missing it is 0. represents the L2 norm of the vector, represents the vth mode of the i-th image sample, express The denoising result is and Respectively represent the width and height of the image; Represents the coordinates of the pixel points of the image; represents the real number domain, which means that all pixel values are real numbers; V represents the number of modalities, and N represents the number of image samples. Here, it is assumed that the image samples are single-channel data, and the multi-channel data also has the same derivation paradigm as the single-channel data.
[0014] Preferably, in T6, the formulas of the two constraints (ML and CL) are as follows: ; , ; in, represents the weakly supervised learning loss function, is the coordinate of another pixel point in the image, Represents pixel and Whether there is a must-connect constraint ML and a must-connect constraint CL between them, if yes, it is equal to 1, if not, it is equal to 0, Represents pixel and The subordinate relationship between them, that is, whether the two pixels belong to the same category, that is, if the must-connect constraint ML is equal to 1, and if the do not-connect constraint CL is equal to 0; Indicates other; Represents a constraint.
[0015] Preferably, in T7, the fusion function organically fuses different modalities of the same image sample and can effectively avoid the impact of missing modalities. The specific formula is as follows: ; in, It represents a fusion based on denoised image samples, which encodes different modalities of the same multimodal medical image sample into an image sample of a unified modality.
[0016] Preferably, in T8, K-means is used as the base learning target for each fusion body based on the denoised image sample. Perform cluster analysis to obtain image segmentation results. The specific formula is as follows: ; ; in, Represents the objective function of the K-means clustering algorithm, which is used to segment the image at the pixel level. represents the pixel cluster centroid of the i-th image sample; represents the pixel clustering label of the i-th image sample; K represents the number of target clusters; and the superscript T represents the transpose of the matrix.
[0017] Preferably, if Figure 2 As shown, in T9, an optimization algorithm is designed to optimize and solve the joint learning objective until the convergence criterion is met, and the final image segmentation result is output. The formula of the joint learning objective is as follows: .
[0018] in, is the weighted joint loss function, and Represents the balance factor (i.e., hyperparameter), which is used to adjust the loss value range of the joint learning objective L.
[0019] The present invention also provides a weakly supervised image segmentation system based on incomplete multimodal medical images, comprising the following modules: Data acquisition module, used to collect incomplete multimodal medical imaging data; Construct a constraint module to construct paired pixel constraints for each image sample in incomplete multimodal medical image data based on some given weakly supervised annotation information. The constraints are divided into must-connect constraints (ML) and must-connect constraints (CL). When the ML constraint is satisfied, the two pixels belong to the same cluster. When the CL constraint is satisfied, the two pixels belong to different clusters. Data cleaning module, used to clean multimodal medical imaging data; Dataset construction module, used to construct multimodal medical imaging sample dataset; The denoising module is used to input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; Iterative fine-tuning module, used to iteratively fine-tune each modality of the denoised image samples and ensure that two constraints (ML and CL) are satisfied during the iteration process; The fusion module is used to design the fusion function to organically fuse multiple modalities of the denoised image samples while avoiding the negative impact of missing modalities; The segmentation module is used to design the base learning target, perform cluster analysis on the fused image samples, generate the clustering results of its pixels, and then obtain its segmentation results; The output module is used to jointly train all learning objectives until the convergence criteria are met and output the final image segmentation result.
[0020] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned weakly supervised image segmentation method based on incomplete multimodal medical images are implemented.
[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned weakly supervised image segmentation method based on incomplete multimodal medical images.
[0022] The present invention also provides a weakly supervised image segmentation system based on incomplete multimodal medical images, comprising the following modules: Data acquisition module, used to collect incomplete multimodal medical imaging data; Construct a constraint module to construct paired pixel constraints for each image sample in the incomplete multimodal medical image data based on some given weakly supervised annotation information. The constraints are divided into the must-connect constraint ML and the no-connect constraint CL. When the must-connect constraint ML is satisfied, the two pixels belong to the same cluster. When the no-connect constraint CL is satisfied, the two pixels belong to different clusters. Data cleaning module, used to clean multimodal medical imaging data; Dataset construction module, used to construct multimodal medical imaging sample dataset; The denoising module is used to input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; An iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised multimodal medical image samples, and ensure that the must-connect constraint ML and the no-connect constraint CL are satisfied during the iteration process; The fusion module is used to design a fusion function to organically fuse multiple modalities of denoised multimodal medical image samples while avoiding the negative impact of missing modalities; The segmentation module is used to design the loss function, perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain its segmentation results; The output module is used to jointly optimize the loss function until the convergence criterion is met and output the final image segmentation result.
[0023] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned weakly supervised image segmentation method based on incomplete multimodal medical images are implemented.
[0024] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-mentioned weakly supervised image segmentation method based on incomplete multimodal medical images.
[0025] Example: To verify the effect of the present invention, a simulation experiment was conducted. The two hyperparameters were set to and The number of iterations is set to 100, the number of clusters is set to 8, and the experiment is based on the public brain tumor dataset BRATS 2017, which contains 285 brain tumor MRI scans, each scan has four MRI modes, namely T1, T1ce, T2 and Flair. The experimental results are shown in Figure 3a , Figure 3b , Figure 3a is the original image, Figure 3b The segmented image after executing the algorithm. The experimental results show that the proposed algorithm has a good medical image segmentation effect.
[0026] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.
[0027] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0028] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0029] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0030] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0031] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A weakly supervised image segmentation method based on incomplete multimodal medical images, characterized in that: The steps include: T1. Collection of incomplete multimodal medical imaging data; T2. Based on the given weakly supervised annotation information, pairwise pixel constraints are constructed for each image sample in the incomplete multimodal medical image data. Pairwise pixel constraints are divided into must-connect constraints ML and no-connect constraints CL. When the must-connect constraints ML are satisfied, the two pixels belong to the same cluster. When the no-connect constraints CL are satisfied, the two pixels belong to different clusters. T3. Clean multimodal medical imaging data; T4. Construct a multimodal medical imaging sample dataset; T5. Input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; T6. Iteratively fine-tune each modality of the denoised multimodal medical image samples, and ensure that the must-connect constraint ML and the must-connect constraint CL are satisfied during the iteration process; T7. Design a fusion function to organically fuse multiple modalities of denoised multimodal medical image samples while avoiding the negative impact of missing modalities; T8. Design a loss function to perform cluster analysis on the fused multimodal medical image samples, generate clustering results of its pixels, and then obtain its segmentation results; T9. Jointly optimize the loss function until the convergence criterion is met and output the final image segmentation result.
2. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 1, characterized in that: In T3, data cleaning includes maximum normalization, minimum normalization, Z-score normalization and L2 norm normalization.
3. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 1, characterized in that: In T5, the training process of the denoising neural network includes: The multimodal medical image sample data is input into the denoising neural network, and its encoder converts the input data into low-dimensional features. Then the decoder reconstructs the input data based on the low-dimensional features to achieve the denoising effect. It is iteratively fine-tuned by minimizing the loss between the original input multimodal medical image sample data and the reconstructed input data. The specific formula is as follows: ; In the formula, To reconstruct the loss function, Indicates whether the vth modality of the ith image sample is missing, if not missing it is 1, if missing it is 0. represents the L2 norm of the vector, represents the vth mode of the i-th image sample, express The denoising result is and Respectively represent the width and height of the image; Represents the coordinates of the pixel points of the image; It represents the real number domain, which means that all pixel values are real numbers; V represents the number of modalities, and N represents the number of image samples.
4. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 3, characterized in that: In T6, the formulas of the must-connect constraint ML and the must-connect constraint CL are as follows: ; , ; in, represents the weakly supervised learning loss function, is the coordinate of another pixel point in the image, Represents pixel and Whether there is a must-connect constraint ML and a must-connect constraint CL between them, if yes, it is equal to 1, if not, it is equal to 0, Represents pixel and The subordinate relationship between them, that is, whether the two pixels belong to the same category, that is, if the must-connect constraint ML is equal to 1, and if the do not-connect constraint CL is equal to 0; Indicates other; Represents a constraint.
5. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 4, characterized in that: In T7, the fusion function organically fuses different modalities of the same multimodal medical image sample and avoids the impact of missing modalities. The specific formula is as follows: ; in, It represents a fusion based on denoised image samples, which encodes different modalities of the same multimodal medical image sample into an image sample of a unified modality.
6. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 5, characterized in that: In T8, K-means is used as the base learning target for each fusion of multimodal medical image samples after denoising. Perform cluster analysis to obtain image segmentation results. The specific formula is as follows: ; ; in, Represents the objective function of the K-means clustering algorithm, which is used to segment the image at the pixel level. represents the pixel cluster centroid of the i-th image sample; represents the pixel clustering label of the i-th image sample; K represents the number of target clusters; and the superscript T represents the transpose of the matrix.
7. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 6, characterized in that: In T9, an optimization algorithm is designed to optimize and solve the joint learning objective until the convergence criterion is met, and the final image segmentation result is output. The formula of the joint learning objective is as follows: ; in, is the weighted joint loss function, and Represents the balance factor, used to regulate the joint The numerical range of the loss value.
8. A weakly supervised image segmentation system based on incomplete multimodal medical images, characterized in that: Includes the following modules: Data acquisition module, used to collect incomplete multimodal medical imaging data; Construct a constraint module to construct paired pixel constraints for each image sample in the incomplete multimodal medical image data based on some given weakly supervised annotation information. The constraints are divided into the must-connect constraint ML and the no-connect constraint CL. When the must-connect constraint ML is satisfied, the two pixels belong to the same cluster. When the no-connect constraint CL is satisfied, the two pixels belong to different clusters. Data cleaning module, used to clean multimodal medical imaging data; Dataset construction module, used to construct multimodal medical imaging sample dataset; The denoising module is used to input the multimodal medical image sample data set into the denoising neural network to preliminarily filter out the noise therein; An iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised multimodal medical image samples, and ensure that the must-connect constraint ML and the no-connect constraint CL are satisfied during the iteration process; The fusion module is used to design a fusion function to organically fuse multiple modalities of denoised multimodal medical image samples while avoiding the negative impact of missing modalities; The segmentation module is used to design the loss function, perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain its segmentation results; The output module is used to jointly optimize the loss function until the convergence criterion is met and output the final image segmentation result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described in one of claims 1-7 are implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Three-dimensional scene segmentation method and device under weak supervision
CN112927244A
Medical image segmentation method based on weak supervised learning and training method of medical image segmentation model based on weak supervised learning
CN114445423A
Incomplete multi-modal medical image learning method
CN117218453A
Medical imaging omics multi-view semi-supervised representation learning method
CN117523356A
