Weakly supervised image segmentation method and system based on incomplete multi-modal medical images

Abstract

The application provides a weakly supervised image segmentation method and system based on incomplete multi-modal medical images, relates to the auxiliary diagnosis technical field of multi-modal medical images, and comprises the following steps: collecting incomplete multi-modal medical image data; constructing a pair of pixel point constraint of each image sample in the incomplete multi-modal medical image data; constructing a multi-modal medical image sample data set; inputting the multi-modal medical image sample data set into a denoising neural network to preliminarily filter out the noise existing in the multi-modal medical image sample data set; designing an optimization algorithm to iteratively fine-tune each mode of the denoised image sample; designing a fusion function to organically fuse the multiple modes of the denoised image sample; designing a base learning target to obtain a segmentation result; jointly training all the learning targets to output a final image segmentation result. The application can effectively generalize to a large-scale incomplete multi-modal medical image data scene under the condition that the labeling is limited.

Description

Technical Field

[0001] This invention relates to the field of auxiliary diagnostic technology for multimodal medical images, and in particular to a weakly supervised image segmentation method and system based on incomplete multimodal medical images. Background Technology

[0002] Multimodal medical image segmentation aims to organically fuse medical image data from different imaging technologies, such as MRI (Magnetic Resonance Imaging), CT (Computed Tomography), and ultrasound, into a unified comprehensive representation. This representation aims to include as much intuitive, detailed, and accurate information as possible, thereby improving the accuracy and efficiency of medical image segmentation and assisting clinicians in making medical diagnoses. By organically complementing information from different image modalities, the segmentation accuracy can be improved. For example, in brain tumor segmentation, images from multiple MRI modalities, such as T1-weighted, T2-weighted, and FLAIR (Fluorescent Attenuation Inversion Recovery) images, can be fused to more accurately segment the lesion area.

[0003] The technical challenges faced by multimodal medical image segmentation include dataset annotation, limitations in computational resources, and the generalization ability of algorithms. Specifically: (1) Most existing methods rely on a large amount of image data annotation, but in real-world scenarios, obtaining image data annotation requires significant human and financial investment; (2) Most existing methods are usually limited by excessive complexity, making them unable to cope well with large-scale data scenarios; (3) Most existing methods can only cope with complete multimodal data (i.e., all modalities are complete). When modalities are missing, they cannot generalize well to such incomplete multimodal data scenarios. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a weakly supervised image segmentation method and system based on incomplete multimodal medical images. Based on given weakly supervised annotation information, pairwise pixel constraints are constructed for each image sample in the incomplete multimodal medical image data. An optimization algorithm is designed to iteratively fine-tune each modality of the denoised image sample, ensuring that both constraints are satisfied during the iteration process. A fusion function is designed to organically fuse multiple modalities of the denoised image sample. A base learning objective is designed, and cluster analysis is performed on the fused image sample to generate its pixel clustering results, thereby obtaining the segmentation result. All learning objectives are jointly trained to output the final image segmentation result.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A weakly supervised image segmentation method based on incomplete multimodal medical images includes the following steps: T1. Incomplete acquisition of multimodal medical imaging data; T2. Based on the given weakly supervised annotation information, construct pairwise pixel constraints for each image sample in the incomplete multimodal medical image data. Pairwise pixel constraints are divided into mandatory connection constraints (ML) and non-mandatory connection constraints (CL). When the mandatory connection constraint (ML) is satisfied, the two pixels belong to the same cluster. When the non-mandatory connection constraint (CL) is satisfied, the two pixels belong to different clusters. T3. Cleaning of multimodal medical image data; T4. Construct a multimodal medical image sample dataset; T5. Input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; T6. Iteratively fine-tune each modality of the denoised multimodal medical image sample, and ensure that the mandatory connection constraint ML and the non-mandatory connection constraint CL are satisfied during the iteration process; T7. Design a fusion function to organically fuse multiple modalities of the denoised multimodal medical image samples, while avoiding the negative impact of missing modalities. T8. Design a loss function to perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain its segmentation results; T9. Perform joint optimization of the loss function until the convergence criterion is met, and output the final image segmentation result.

[0006] This invention also provides a weakly supervised image segmentation system based on incomplete multimodal medical images, comprising the following modules: The data acquisition module is used to acquire incomplete multimodal medical image data; The constraint module is constructed based on some given weakly supervised annotation information to construct pairwise pixel constraints for each image sample in incomplete multimodal medical image data. It is divided into mandatory connection constraint (ML) and non-connection constraint (CL). When the mandatory connection constraint (ML) is satisfied, the two pixels belong to the same cluster. When the non-connection constraint (CL) is satisfied, the two pixels belong to different clusters. The data cleaning module is used to clean multimodal medical image data; The dataset building module is used to construct multimodal medical image sample datasets; The denoising module is used to input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; The iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised multimodal medical image sample, and ensure that the mandatory connection constraint ML and the non-connection constraint CL are satisfied during the iteration process; The fusion module is used to design fusion functions to organically fuse multiple modalities of denoised multimodal medical image samples, while avoiding the negative impact of missing modalities. The segmentation module is used to design the loss function, perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain the segmentation results. The output module is used to jointly optimize the loss function until the convergence criterion is met, and output the final image segmentation result.

[0007] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described above.

[0008] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described above.

[0009] Beneficial effects: This invention can be effectively generalized to scenarios with missing modalities and application scenarios with limited annotation, and thanks to its low complexity (positively correlated with sample size), it can be effectively integrated into auxiliary diagnostic systems. Attached Figure Description

[0010] Figure 1 This is a flowchart of the weakly supervised image segmentation method based on incomplete multimodal medical images according to the present invention; Figure 2 This is a schematic diagram of the weakly supervised image segmentation method based on incomplete multimodal medical images according to the present invention. Figure 3a This is a demonstration of the results achieved on the original images from the BRATS 2017 brain tumor dataset; Figure 3b This is a schematic diagram of the segmentation image achieved using the method of the present invention on the BRATS 2017 brain tumor dataset. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0012] like Figure 1 As shown, the weakly supervised image segmentation method based on incomplete multimodal medical images of the present invention includes the following steps: T1. Incomplete acquisition of multimodal medical imaging data; T2. Based on the given weakly supervised annotation information, construct pairwise pixel constraints for each image sample in the incomplete multimodal medical image data. Pairwise pixel constraints are divided into mandatory connection constraints (ML) and non-mandatory connection constraints (CL). When the mandatory connection constraint (ML) is satisfied, the two pixels belong to the same cluster. When the non-mandatory connection constraint (CL) is satisfied, the two pixels belong to different clusters. T3. Cleaning of multimodal medical image data; T4. Construct a multimodal medical image sample dataset; T5. Input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; T6. Iteratively fine-tune each modality of the denoised multimodal medical image sample, and ensure that the mandatory connection constraint ML and the non-mandatory connection constraint CL are satisfied during the iteration process; T7. Design a fusion function to organically fuse multiple modalities of the denoised multimodal medical image samples, while avoiding the negative impact of missing modalities. T8. Design a loss function to perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain its segmentation results; T9. Perform joint optimization of the loss function until the convergence criterion is met, and output the final image segmentation result.

[0013] Preferably, in T3, data cleaning includes maximum standardization, minimum standardization, Z-score normalization, and L2 norm normalization; Preferably, such as Figure 2 As shown, the training process of the denoising neural network in T5 is as follows: Multimodal medical image sample data is input into a denoising neural network. The encoder converts the input data into low-dimensional features, and the decoder reconstructs the input data based on these low-dimensional features, achieving denoising. Iterative fine-tuning is performed by minimizing the error between the original multimodal medical image sample data and the reconstructed input data, as shown in the following formula: ; in, To reconstruct the loss function, This indicates whether the v-th modality of the i-th image sample is missing; 1 indicates no missing, and 0 indicates missing. Describes the L2 norm of a vector. This represents the v-th modality of the i-th image sample. express The denoising results and These represent the width and height of the image, respectively. Represents the coordinates of pixels in the image; denoted by , V represents the real number field, indicating that all pixel values ​​are real numbers; V represents the number of modalities, and N represents the number of image samples. Here, it is assumed that the image samples are single-channel data; the derivation paradigm for multi-channel data is the same as for single-channel data.

[0014] Preferably, in T6, the formulas for the two constraints (ML and CL) are as follows: ; , ; in, This represents the loss function for weakly supervised learning. The coordinates of another pixel in the image. Represents pixels and Do there exist mandatory connection constraints (ML) and non-mandatory connection constraints (CL) between them? If they exist, the value is 1; otherwise, it is 0. Represents pixels and The subordinate relationship between two pixels, that is, whether two pixels belong to the same category, that is, satisfying the mandatory connection constraint ML = 1 and the non-connection constraint CL = 0. Indicates other; This indicates a constraint.

[0015] Preferably, in T7, the fusion function organically fuses different modalities of the same image sample, while effectively avoiding the influence of missing modalities. The specific formula is as follows: ; in, It represents a fusion based on denoised image samples, which encodes different modalities of the same multimodal medical image sample into a unified modal image sample.

[0016] Preferably, in T8, K-means is used as the basis learning objective for each fusion based on the denoised image samples. Cluster analysis is performed to obtain image segmentation results. The specific formula is shown below: ; ; in, Let represent the objective function of the K-means clustering algorithm, used for pixel-level segmentation of images. This represents the cluster centroid of the pixels in the i-th image sample; The pixel cluster label represents the i-th image sample; K represents the number of target clusters; the superscript T represents the transpose of the matrix.

[0017] Preferably, such as Figure 2 As shown, in T9, an optimization algorithm is designed to optimize the joint learning objective until the convergence criterion is met, and the final image segmentation result is output. The formula for the joint learning objective is as follows: .

[0018] in, For the weighted joint loss function, and This represents the balancing factor (i.e., hyperparameter), used to adjust the numerical range of the loss value of the joint learning objective L.

[0019] This invention also provides a weakly supervised image segmentation system based on incomplete multimodal medical images, comprising the following modules: The data acquisition module is used to acquire incomplete multimodal medical image data; The constraint module constructs pairwise pixel constraints for each image sample in incomplete multimodal medical image data based on some given weakly supervised annotation information. It is divided into mandatory connection constraint (ML) and non-continuous connection constraint (CL). When the ML constraint is satisfied, the two pixels belong to the same cluster. When the CL constraint is satisfied, the two pixels belong to different clusters. The data cleaning module is used to clean multimodal medical image data; The dataset building module is used to construct multimodal medical image sample datasets; The denoising module is used to input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; The iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised image sample and ensure that two constraints (ML and CL) are satisfied during the iteration process; The fusion module is used to design fusion functions to organically fuse multiple modalities of the denoised image samples, while avoiding the negative impact of missing modalities. The segmentation module is used to design the basis learning objective, perform cluster analysis on the fused image samples, generate the clustering results of its pixels, and then obtain the segmentation results. The output module is used to jointly train all learning objectives until the convergence criterion is met, and output the final image segmentation result.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described above.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described above.

[0022] This invention also provides a weakly supervised image segmentation system based on incomplete multimodal medical images, comprising the following modules: The data acquisition module is used to acquire incomplete multimodal medical image data; The constraint module is constructed based on some given weakly supervised annotation information to construct pairwise pixel constraints for each image sample in incomplete multimodal medical image data. It is divided into mandatory connection constraint (ML) and non-connection constraint (CL). When the mandatory connection constraint (ML) is satisfied, the two pixels belong to the same cluster. When the non-connection constraint (CL) is satisfied, the two pixels belong to different clusters. The data cleaning module is used to clean multimodal medical image data; The dataset building module is used to construct multimodal medical image sample datasets; The denoising module is used to input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; The iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised multimodal medical image sample, and ensure that the mandatory connection constraint ML and the non-connection constraint CL are satisfied during the iteration process; The fusion module is used to design fusion functions to organically fuse multiple modalities of denoised multimodal medical image samples, while avoiding the negative impact of missing modalities. The segmentation module is used to design the loss function, perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain the segmentation results. The output module is used to jointly optimize the loss function until the convergence criterion is met, and output the final image segmentation result.

[0023] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described above.

[0024] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images described above.

[0025] Example: To verify the effectiveness of the invention, a simulation experiment was conducted. The two hyperparameters were set to... and The number of iterations was set to 100, and the number of clusters was set to 8. Experiments were conducted using the publicly available brain tumor dataset BRATS 2017, which contains 285 brain tumor MRI scans, each with four MRI modes: T1, T1ce, T2, and FLAIR. Experimental results are presented in [link to results]. Figure 3a , Figure 3b , Figure 3a For the original image, Figure 3b The image shown is the segmented image after the algorithm has been applied. Experimental results demonstrate that the proposed algorithm achieves good medical image segmentation performance.

[0026] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0027] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0028] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0029] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0030] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0031] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A weakly supervised image segmentation method based on incomplete multimodal medical images, characterized in that, Includes the following steps: T1. Incomplete acquisition of multimodal medical imaging data; T2. Based on the given weakly supervised annotation information, construct pairwise pixel constraints for each image sample in the incomplete multimodal medical image data. Pairwise pixel constraints are divided into mandatory connection constraints (ML) and non-mandatory connection constraints (CL). When the mandatory connection constraint (ML) is satisfied, the two pixels belong to the same cluster. When the non-mandatory connection constraint (CL) is satisfied, the two pixels belong to different clusters. T3. Cleaning of multimodal medical image data; T4. Construct a multimodal medical image sample dataset; T5. Input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; T6. Iteratively fine-tune each modality of the denoised multimodal medical image sample, and ensure that the mandatory connection constraint ML and the non-mandatory connection constraint CL are satisfied during the iteration process; T7. Design a fusion function to organically fuse multiple modalities of the denoised multimodal medical image samples, while avoiding the negative impact of missing modalities. T8. Design a loss function to perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain its segmentation results; T9. Perform joint optimization of the loss function until the convergence criterion is met, and output the final image segmentation result.

2. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 1, characterized in that, In T3, data cleaning includes maximum standardization, minimum standardization, Z-score normalization, and L2 norm normalization.

3. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 1, characterized in that, In T5, the training process of the denoising neural network includes: Multimodal medical image sample data is input into a denoising neural network. The encoder converts the input data into low-dimensional features, and the decoder reconstructs the input data based on these low-dimensional features to achieve denoising. Iterative fine-tuning is performed by minimizing the loss between the original input multimodal medical image sample data and the reconstructed input data. The specific formula is as follows: ； In the formula, To reconstruct the loss function, This indicates whether the v-th modality of the i-th image sample is missing; 1 indicates no missing, and 0 indicates missing. Describes the L2 norm of a vector. This represents the v-th modality of the i-th image sample. express The denoising results and These represent the width and height of the image, respectively. Represents the coordinates of pixels in the image; V represents the real number field, indicating that all pixel values ​​are real numbers; V represents the number of modalities, and N represents the number of image samples.

4. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 3, characterized in that, In T6, the formulas for the mandatory connection constraint ML and the non-mandatory connection constraint CL are as follows: ； ， ； in, This represents the loss function for weakly supervised learning. The coordinates of another pixel in the image. Represents pixels and Do there exist mandatory connection constraints (ML) and non-mandatory connection constraints (CL) between them? If they exist, the value is 1; otherwise, it is 0. Represents pixels and The subordinate relationship between two pixels, that is, whether two pixels belong to the same category, that is, satisfying the mandatory connection constraint ML = 1 and the non-connection constraint CL = 0. Indicates other; This indicates a constraint condition.

5. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 4, characterized in that, In T7, the fusion function organically fuses different modalities of the same multimodal medical image sample while avoiding the influence of missing modalities. The specific formula is as follows: ； in, It represents a fusion based on denoised image samples, which encodes different modalities of the same multimodal medical image sample into a unified modal image sample.

6. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 5, characterized in that, In T8, K-means is used as the basis learning objective for each fusion based on denoised multimodal medical image samples. Cluster analysis is performed to obtain image segmentation results. The specific formula is shown below: ； ； in, Let represent the objective function of the K-means clustering algorithm, used for pixel-level segmentation of images. This represents the cluster centroid of the pixels in the i-th image sample; The pixel cluster label represents the i-th image sample; K represents the number of target clusters; the superscript T represents the transpose of the matrix.

7. The weakly supervised image segmentation method based on incomplete multimodal medical images according to claim 6, characterized in that, In step T9, an optimization algorithm is designed to optimize the joint learning objective until the convergence criterion is met, and the final image segmentation result is output. The formula for the joint learning objective is as follows: ； in, For the weighted joint loss function, and This represents the balance factor, used to regulate the combined... The numerical range of the loss value.

8. A weakly supervised image segmentation system based on incomplete multimodal medical images, characterized in that, Includes the following modules: The data acquisition module is used to acquire incomplete multimodal medical image data; The constraint module is constructed based on some given weakly supervised annotation information to construct pairwise pixel constraints for each image sample in incomplete multimodal medical image data. It is divided into mandatory connection constraint (ML) and non-connection constraint (CL). When the mandatory connection constraint (ML) is satisfied, the two pixels belong to the same cluster. When the non-connection constraint (CL) is satisfied, the two pixels belong to different clusters. The data cleaning module is used to clean multimodal medical image data; The dataset building module is used to construct multimodal medical image sample datasets; The denoising module is used to input the multimodal medical image sample dataset into the denoising neural network to initially filter out the noise present in it; The iterative fine-tuning module is used to iteratively fine-tune each modality of the denoised multimodal medical image sample, and ensure that the mandatory connection constraint ML and the non-connection constraint CL are satisfied during the iteration process; The fusion module is used to design fusion functions to organically fuse multiple modalities of denoised multimodal medical image samples, while avoiding the negative impact of missing modalities. The segmentation module is used to design the loss function, perform cluster analysis on the fused multimodal medical image samples, generate the clustering results of its pixels, and then obtain the segmentation results. The output module is used to jointly optimize the loss function until the convergence criterion is met, and output the final image segmentation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the weakly supervised image segmentation method based on incomplete multimodal medical images as described in any one of claims 1-7.

Images