Multimodal Weakly Supervised Medical Image Segmentation Method and System

Abstract

This invention provides a multimodal weakly supervised medical image segmentation method, belonging to the field of medical image segmentation. The method includes: extracting features from CT and MR images separately to obtain CT and MR modal feature representations; fusing features from the same layer of CT and MR modal feature representations at each scale; performing downsampling and multi-layer convolution processing on the fused feature representations; decoding the enhanced feature representations, CT and MR modal feature representations separately to obtain multimodal prediction maps, CT and MR prediction maps, respectively; calculating cross-entropy loss; calculating multi-view CRF loss for CT and MR images separately to obtain intramodal regularization loss; calculating intermodal consistency loss; summing the cross-entropy, intramodal regularization, and intermodal consistency losses; and obtaining the trained image segmentation model when the total loss is minimized. The method inputs the CT and MR images to be segmented into the trained image segmentation model and outputs segmented images. It also provides an image segmentation system; effectively fusing multimodal image information and suppressing overfitting under weak supervision.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation technology, and in particular to a multimodal weakly supervised medical image segmentation method and system. Background Technology

[0002] Medical image segmentation is a crucial task in computer vision and medical image analysis, widely applied in clinical fields such as disease diagnosis, surgical planning, and radiotherapy target delineation. In recent years, deep learning methods have made significant progress in medical image segmentation, but their performance is highly dependent on large amounts of high-quality, pixel-level manually annotated data. Medical image annotation typically requires professional doctors to manually delineate each frame, which is time-consuming and labor-intensive, especially with three-dimensional volumetric data. Therefore, in large-scale clinical applications, obtaining dense annotations is costly and difficult to scale, severely limiting the practicality of fully supervised learning methods. To alleviate the dependence on dense annotations, weakly supervised learning methods have gradually gained attention, especially weakly supervised frameworks based on graffiti annotations, which have become one of the mainstream directions due to their efficiency and flexibility. Current research mainly focuses on how to improve segmentation accuracy and model robustness under sparse supervision, with related methods mainly divided into two categories: pseudo-label-based methods and multimodal fusion-based methods.

[0003] Pseudo-label-based methods use weak annotations to infer semantic labels for unlabeled regions. For example, the paper "Scribble-based hierarchical weakly supervised learning for brain tumor segmentation" (ZJi et al., MICCAI, 2019) proposes a scribble-based hierarchical weakly supervised learning framework. By introducing a phased hierarchical learning strategy, it guides the model to gradually complete the structural segmentation task from the whole to the local using only scribble annotations of normal tissue and global labels of tumor substructures, thus significantly reducing the dependence on pixel-level fine annotations. This method cleverly uses clustering to generate pseudo-labels and combines conditional random field loss to optimize the boundary, enabling the model to maintain high-precision substructure segmentation ability under weak supervision. It performs well in structural modeling and label generation strategies, and is particularly suitable for medical image segmentation tasks with complex hierarchical structures such as brain tumors. Although this method can alleviate the label scarcity problem to some extent through pseudo-labels and self-supervised regularization, the sparse scribble annotations make the model prone to overfitting during training. Furthermore, it cannot effectively capture detailed changes in the target structure in low-contrast, blurred-boundary regions, ultimately leading to coarse and incomplete segmentation results.

[0004] Multimodal fusion-based methods focus on feature collaboration and fusion between multimodal images. For example, the paper "Robust Semi-supervised Multimodal Medical Image Segmentation via CrossModality Collaboration" (Zhou X et al., MICCAI, 2024) proposes a robust semi-supervised multimodal segmentation framework. By introducing a cross-modal collaboration strategy, it extracts shareable modality-independent knowledge from each modality and performs feature alignment and integration in the fusion layer, effectively alleviating the problem of anatomical misalignment between modalities. This method further combines channel semantic consistency loss and contrast consistency learning modules, achieving high-precision multi-organ and multi-structure segmentation even with limited labeled data. It performs excellently in multimodal feature fusion and alignment mechanisms, and is particularly suitable for real-world clinical scenarios with scarce labels and non-registered modalities. Multimodal fusion methods attempt to jointly model modalities such as CT and MR to improve segmentation accuracy. However, most methods use simple input splicing or multi-level feature fusion, often ignoring the structural differences and specific expressions between modalities, easily leading to modal conflicts, redundant fusion information, and even misleading the model's learning direction. While some methods achieve hierarchical fusion by designing complex network structures, they introduce additional computational overhead, making it difficult to balance performance and efficiency. Summary of the Invention

[0005] The technical problem to be solved by this invention is: how to effectively fuse multimodal image information and suppress overfitting under weak supervision.

[0006] This invention solves the above-mentioned technical problems through the following technical solution: a multimodal weakly supervised medical image segmentation method, the method comprising: Feature extraction was performed on CT images and MR images respectively to obtain CT modal feature representations and MR modal feature representations; At each scale, the same-layer features of CT modal feature representation and MR modal feature representation are fused to obtain the fused feature representation; The fused feature representation is downsampled and processed by multiple convolutions to obtain the enhanced feature representation. The enhanced feature representation, CT modal feature representation, and MR modal feature representation are decoded to obtain multimodal prediction maps, CT prediction maps, and MR prediction maps. The cross-entropy loss of the multimodal prediction maps, CT prediction maps, MR prediction maps, and graffiti annotations is calculated. The multi-view CRF loss of the CT image and MR image is calculated separately to obtain the intramodal regularization loss. The intermodal consistency loss of the multimodal prediction map and the CT prediction map and MR prediction map is calculated. The cross-entropy loss, intramodal regularization loss, and intermodal consistency loss are added together to obtain the total loss. When the total loss is minimized, the trained image segmentation model is obtained. Input the CT and MR images to be segmented into the trained image segmentation model, and output the segmented images.

[0007] This invention employs a three-branch output structure: two modality-specific feature branches and one cross-modal shared branch. The two modality-specific feature branches effectively preserve the structural advantages of both CT and MR. In the cross-modal shared branch, multimodal feature interaction enables semantic alignment and fusion of multi-scale modal features. Multimodal feature enhancement improves boundary details and structural consistency, mitigating intermodal alignment errors and information redundancy. Relying solely on doodle-style weak annotation, it achieves accurate segmentation of complex anatomical structures with low annotation costs, significantly improving segmentation accuracy and model stability. The three branch outputs are trained end-to-end under a hybrid supervision strategy. Doodle supervision calculates cross-entropy loss only for labeled pixels, intramodal regularization introduces three-view CRF loss to improve spatial consistency within branches, and intermodal consistency enhances structural consistency by minimizing prediction differences between the three branches, preventing overfitting and achieving stable learning and robustness improvement of the segmentation model to weak annotations. This addresses the problem of prediction instability that easily arises under sparse supervision.

[0008] Preferably, the process of fusing the same-layer features of CT modal feature representation and MR modal feature representation at each scale includes: The same-layer features represented by CT modality features and MR modality features are respectively concatenated along the channel dimension to obtain the concatenated features of each layer; Convolution is performed on the concatenated features of each layer to obtain the processed features. The processed features are then input into the AReLU activation function to obtain the fused feature representation of each layer.

[0009] Preferably, the fusion feature representation of each layer for:

[0010] in, This indicates that the CT image has passed through the CT encoder. i Modal feature representation after layer downsampling This indicates that the MR image has passed through the MR encoder. i Modal feature representation after layer downsampling , This is a channel-level splicing operation. This is a 3×3×3 convolution operation. This is the AReLU activation function.

[0011] This invention concatenates the same-layer features of CT modal feature representation and MR modal feature representation in the channel dimension, and uses convolution operation to uniformly represent spatial scale and channel semantics. This enables the fusion of the same-layer features of CT modal feature representation and MR modal feature representation at each scale, thereby achieving cross-modal semantic alignment and multi-scale context fusion, and thus improving the efficiency of cross-modal information utilization.

[0012] Preferably, the process of downsampling and multi-layer convolution processing on the fused feature representation includes: fusion feature representation Perform a 1 / 2 resampling operation to obtain the feature representation. ; Feature representation Inputting the AReLU activation function yields the feature representation. ; For feature representation Three convolutional operations are performed. The features after each convolutional operation are input into an AReLU activation function. The features after the third convolutional operation are then compared with the feature representation. The features are added together to obtain the enhanced feature representation.

[0013] Preferred, enhanced feature representation for:

[0014] in, This represents the nth convolution operation. , , , This indicates a 1 / 2 resampling operation. The AReLU activation function is used. This indicates the addition of features.

[0015] This invention improves the boundary modeling capability and structural alignment accuracy of CT and MR images by downsampling and multi-layer convolution operations on the fused feature representation, further enhancing modal alignment consistency and feature representation capabilities. A cross-modal shared branch is constructed through the fusion-enhancement process, effectively integrating the information representation capabilities of both CT and MR modalities.

[0016] Preferably, the cross-entropy loss between the multimodal prediction map, CT prediction map, MR prediction map, and graffiti annotation is:

[0017] in, Mark the graffiti. This could be a multimodal prediction map, a CT prediction map, or an MR prediction map. It is the probability that pixel i belongs to category c. for The set of labeled pixels in the image.

[0018] Preferably, the multi-view CRF loss for CT and MR images is:

[0019]

[0020] in, , , Two-dimensional CRF losses were obtained from axial, sagittal, and coronal slices, respectively. For CT prediction maps or MR prediction maps, It can be A, S, or C. and They are pixels and The predicted value, It is a collection of pixels; , Indicates based on The cost of discontinuity obtained by mixing kernel functions For the first individual core The weight, This is the corresponding feature vector.

[0021] Preferably, intramodal regularized loss for:

[0022] in, For multi-view CRF loss of CT images, For multi-view CRF loss of MR images, , These are the weighting coefficients.

[0023] Preferably, the intermodal consistency loss between the multimodal prediction map and the CT and MR prediction maps for:

[0024] in, , , These are multimodal prediction maps, CT prediction maps, and MR prediction maps, respectively. , These are the weighting coefficients.

[0025] This invention also provides a multimodal weakly supervised medical image segmentation system, the system comprising: The feature extraction module is used to extract features from CT images and MR images respectively, to obtain CT modal feature representations and MR modal feature representations; The multimodal feature interaction module is used to fuse the same-layer features of CT modal feature representation and MR modal feature representation at each scale to obtain the fused feature representation; The multimodal feature enhancement module is used to downsample and perform multi-layer convolution processing on the fused feature representation to obtain the enhanced feature representation; The training module decodes the enhanced feature representation, CT modal feature representation, and MR modal feature representation to obtain multimodal prediction maps, CT prediction maps, and MR prediction maps. It calculates the cross-entropy loss between the multimodal prediction maps, CT prediction maps, MR prediction maps, and graffiti annotations. It also calculates the multi-view CRF loss for CT images and MR images to obtain the intramodal regularization loss. Finally, it calculates the intermodal consistency loss between the multimodal prediction maps and the CT and MR prediction maps. The cross-entropy loss, intramodal regularization loss, and intermodal consistency loss are added together to obtain the total loss. When the total loss is minimized, the trained image segmentation model is obtained. The inference module is used to input the CT and MR images to be segmented into the trained image segmentation model and output the segmented images. Attached Figure Description

[0026] Figure 1 This is a flowchart of the multimodal weakly supervised medical image segmentation method provided in Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of the multimodal weakly supervised medical image segmentation system provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the multimodal feature interaction module in the multimodal weakly supervised medical image segmentation system provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the multimodal feature enhancement module in the multimodal weakly supervised medical image segmentation system provided in Embodiment 2 of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Example 1 like Figure 1 As shown, this embodiment provides a multimodal weakly supervised medical image segmentation method, including the following steps: Step 1: Input CT images into a CT encoder for feature extraction to obtain CT modal feature representations. Input MR images into an MR encoder for feature extraction to obtain MR modal feature representations. Both the CT and MR encoders use the V-Net model, a deep learning model currently used for medical image segmentation. The CT encoder employs a three-layer downsampling structure to retain more anatomical details, while the MR encoder uses a four-layer downsampling structure to enhance its context awareness. Each encoder consists of multiple convolutional layers, extracting feature maps at multiple scales to form modality-specific feature representations. Therefore, the CT modal feature representation... It can be represented as:

[0029] MR modal feature representation It can be represented as:

[0030] This invention designs two modality-specific feature extraction branches: CT modality and MR modality. The CT modality feature extraction branch adopts a three-layer downsampling structure to preserve detail clarity, while the MR modality feature extraction branch adopts a four-layer downsampling structure to enhance contextual modeling capabilities. The two branches learn modality-specific features respectively, which can more effectively extract the differential semantic structural information in images of different modalities (CT modality and MR modality), thereby forming a multi-scale feature representation and laying the foundation for subsequent cross-modal fusion.

[0031] Step 2: Fuse the same-layer features of CT modal feature representation and MR modal feature representation at each scale to obtain the fused feature representation; the specific process includes: The same-layer features represented by CT modality features and MR modality features are respectively concatenated along the channel dimension to obtain the concatenated features of each layer; Convolution is performed on the concatenated features of each layer to obtain processed features. These processed features are then input into the AReLU activation function to obtain the fused feature representation of each layer. for:

[0032] in, This indicates that the CT image has passed through the CT encoder. i Modal feature representation after layer downsampling This indicates that the MR image has passed through the MR encoder. i Modal feature representation after layer downsampling , This is a channel-level splicing operation. This is a 3×3×3 convolution operation. This is the AReLU activation function.

[0033] This invention concatenates the same-layer features of CT modal feature representation and MR modal feature representation in the channel dimension, and uses convolution operation to uniformly represent spatial scale and channel semantics. This enables the fusion of the same-layer features of CT modal feature representation and MR modal feature representation at each scale, thereby achieving cross-modal semantic alignment and multi-scale context fusion, and thus improving the efficiency of cross-modal information utilization.

[0034] Step 3: Perform downsampling and multi-layer convolution on the fused feature representation to obtain the enhanced feature representation; The process of downsampling and multi-layer convolution for the fused feature representation includes: fusion feature representation Perform a 1 / 2 resampling operation to obtain the feature representation. ; Feature representation Inputting the AReLU activation function yields the feature representation. ; For feature representation Three convolutional operations are performed. The features after each convolutional operation are input into an AReLU activation function. The features after the third convolutional operation are then compared with the feature representation. By adding the features, we obtain the enhanced feature representation. for:

[0035] in, This represents the nth convolution operation. , , , This indicates a 1 / 2 resampling operation. The AReLU activation function is used. This indicates the addition of features.

[0036] This invention improves the boundary modeling capability and structural alignment accuracy of CT and MR images by downsampling and multi-layer convolution operations on the fused feature representation, further enhancing modal alignment consistency and feature representation capabilities. A cross-modal shared branch is constructed through the fusion-enhancement process, effectively integrating the information representation capabilities of both CT and MR modalities.

[0037] This invention designs a multimodal-specific and shared feature collaborative modeling framework by constructing two modality-specific feature branches and one cross-modal shared branch. This framework effectively preserves the structural advantages of CT and MR, while extracting shared information between modalities, significantly improving segmentation accuracy and model generalization ability. Multimodal feature interaction enables semantic alignment and fusion of multi-scale modal features, and multimodal feature enhancement is introduced to improve boundary details and structural consistency, thereby alleviating the problems of intermodal alignment errors and information redundancy.

[0038] Step 4: Through the processing in Steps 2 and 3, the CT modal features are represented. and MR modal feature representation Cross-modal fusion involves inputting the enhanced feature representation into a multimodal decoder to obtain a multimodal prediction map; inputting the CT modal feature representation into a CT image decoder to obtain a CT prediction map; and inputting the MR modal feature representation into an MR image decoder to obtain an MR prediction map, thus achieving the processing of CT modal feature representations. and MR modal feature representation Individual predictions are performed; the multimodal decoder, CT image decoder, and MR image decoder all adopt the V-Net model architecture. All three decoding paths use upsampling and skip connections to recover spatial details. Skip connections are achieved by copying features from corresponding layers of the encoder and concatenating them with features from corresponding layers of the decoder, thus increasing the spatial detail of the features in the decoder. Upsampling refers to using upsampling (upsample) on features from each layer in the decoder to recover spatial details.

[0039] The cross-entropy loss of the multimodal prediction map, CT prediction map, MR prediction map and graffiti annotation is calculated. The multi-view CRF loss of CT image and MR image is calculated separately to obtain the intramodal regularization loss. The intermodal consistency loss of the multimodal prediction map and CT prediction map and MR prediction map is calculated. The cross-entropy loss, intramodal regularization loss and intermodal consistency loss are added to obtain the total loss. The total loss is combined with three loss constraints: graffiti supervision (SSL), intramodal regularization (IMR) and intermodal consistency (IMC). Under the condition of sparse annotation, it can still stably learn the structurally accurate segmentation results and effectively suppress the overfitting phenomenon.

[0040] When the total loss is minimized, the trained image segmentation model is obtained. The trained image segmentation model includes the trained CT encoder, MR encoder, multimodal feature interaction module, multimodal feature enhancement module, CT decoder, MR decoder, and multimodal decoder.

[0041] Calculate the cross-entropy loss for the multimodal prediction map and the graffiti annotation separately. Cross-entropy loss between CT prediction images and graffiti annotations Cross-entropy loss between MR prediction map and graffiti annotation Cross-entropy loss of the three branches The calculation method is the same, as follows:

[0042] in, Mark the graffiti. This could be a multimodal prediction map, a CT prediction map, or an MR prediction map. It is the probability that pixel i belongs to category c. for The set of labeled pixels in the image.

[0043] The cross-entropy loss is obtained by summing the cross-entropy losses of the three branches. for:

[0044] in, These are prediction maps for the CT branch, MR branch, and multimodal branch, respectively.

[0045] The Conditional Random Field (CRF) is introduced, which encourages pixels with similar characteristics to be assigned consistent labels by calculating the energy between pixel pairs:

[0046] CT prediction map or MR prediction map .

[0047] The multi-view CRF loss of CT and MR images is:

[0048]

[0049] in, , , Two-dimensional CRF losses were obtained from axial, sagittal, and coronal slices, respectively. It can be A, S, or C. and They are pixels and The predicted value, It is a collection of pixels; , Indicates based on The cost of discontinuity obtained by mixing kernel functions For the first individual core The weight, This is the corresponding feature vector.

[0050] Intramodal regularization loss for:

[0051] in, For multi-view CRF loss of CT images, For multi-view CRF loss of MR images, , These are the weighting coefficients.

[0052] Intermodal consistency loss between multimodal prediction maps and CT / MR prediction maps for:

[0053] in, , , These are multimodal prediction maps, CT prediction maps, and MR prediction maps, respectively. , These are the weighting coefficients.

[0054] Total loss for:

[0055] This invention designs a three-branch output structure, namely the prediction results of the CT branch, the prediction results of the MR branch, and the prediction results of the fusion branch. The three branch outputs are trained end-to-end under the guidance of a hybrid supervision strategy. Among them, the graffiti supervision only calculates the cross-entropy loss for the labeled pixels, the intramodal regularization introduces the three-view CRF loss to improve the spatial consistency within the branch, and the intermodal consistency enhances the structural consistency by minimizing the prediction difference between the three branches, prevents overfitting, and achieves stable learning and robustness improvement of the segmentation model for weak labels, thereby solving the problem of prediction instability that is easy to occur under sparse supervision.

[0056] This invention proposes a medical image segmentation method based on multimodal collaborative modeling and hybrid supervision, which can be implemented using the PyTorch framework. By designing a three-branch segmentation framework suitable for doodle-style weak annotation scenarios, and combining a multimodal feature interaction module and feature enhancement strategies, it can achieve accurate segmentation of complex anatomical structures with low annotation costs, relying solely on doodle-style weak annotation. This significantly improves segmentation accuracy and model stability, and is particularly suitable for medical image segmentation tasks involving complementary fusion of multimodal images such as CT and MR, such as the automatic 3D segmentation of lesions with complex structural morphology, such as nasopharyngeal carcinoma, and common abdominal organs. It achieves segmentation performance superior to existing weakly supervised methods on multiple real medical datasets. Step 5: Input the CT image and MR image to be segmented into the trained image segmentation model. Input the CT image into the trained CT encoder for feature extraction to obtain CT modal features. Input the MR image into the trained MR encoder for feature extraction to obtain MR modal features. Input the CT modal features and MR modal features into the multimodal feature interaction module for fusion. The fusion process is the same as in Step 2. Input the fused features into the multimodal feature enhancement module to obtain enhanced features. The enhancement process is the same as in Step 3. Input the enhanced features into the trained multimodal decoder to output the segmented image.

[0057] To address the challenge of effectively fusing multimodal image information and suppressing overfitting under weak supervision, this invention proposes a three-branch multimodal weakly supervised segmentation framework. This method tackles the difficulty of co-modeling modality-specific and shared information by designing two modality-specific feature learning branches (CT branch and MR branch) and one cross-modal sharing branch, extracting differential features and shared structural features respectively, effectively avoiding information redundancy and conflicts. Furthermore, to improve the stability and boundary consistency of the fused feature representation, this invention further designs a multimodal feature interaction module and a multimodal feature enhancement module. The former achieves modality-level semantic alignment and channel integration, while the latter refines the fused features, improving the ability to recover segmentation details. Finally, to overcome the sparsity of supervision information caused by weak annotation, this invention introduces a hybrid supervision mechanism that integrates graffiti supervision, intramodal regularization constraints, and intermodal consistency loss. This mechanism strengthens the model's perception of structural boundaries and suppresses overfitting, thereby constructing a multimodal weakly supervised segmentation system that balances accuracy and robustness.

[0058] Example 2 like Figure 2 As shown, this embodiment provides a multimodal weakly supervised medical image segmentation system, including: The feature extraction module is used to extract features from CT images and MR images respectively, to obtain CT modal feature representations and MR modal feature representations.

[0059] The multimodal feature interaction module is used to fuse the same-layer features of CT modal feature representation and MR modal feature representation at each scale to obtain the fused feature representation.

[0060] See Figure 3 The process of fusing the same-layer features from CT modal feature representations and MR modal feature representations at each scale includes: The same-layer features represented by CT modality features and MR modality features are respectively concatenated along the channel dimension to obtain the concatenated features of each layer; Convolution is performed on the concatenated features of each layer to obtain the processed features. The processed features are then input into the AReLU activation function to obtain the fused feature representation of each layer.

[0061] The fusion feature representation of each layer for:

[0062] in, This indicates that the CT image has passed through the CT encoder. i Modal feature representation after layer downsampling This indicates that the MR image has passed through the MR encoder. i Modal feature representation after layer downsampling , This is a channel-level splicing operation. This is a 3×3×3 convolution operation. This is the AReLU activation function.

[0063] The multimodal feature enhancement module is used to downsample and perform multi-layer convolution processing on the fused feature representation to obtain the enhanced feature representation.

[0064] See Figure 4 The process of downsampling and multi-layer convolution for the fused feature representation includes: fusion feature representation Perform a 1 / 2 resampling operation to obtain the feature representation. ; Feature representation Inputting the AReLU activation function yields the feature representation. ; For feature representation Three convolutional operations are performed. The features after each convolutional operation are input into an AReLU activation function. The features after the third convolutional operation are then compared with the feature representation. The features are added together to obtain the enhanced feature representation.

[0065] Enhanced feature representation for:

[0066] in, This represents the nth convolution operation. , , , This indicates a 1 / 2 resampling operation. The AReLU activation function is used. This indicates the addition of features.

[0067] The multimodal feature interaction module fuses features from the same layer across different modalities at multiple scales. Through channel concatenation and convolution, it achieves semantic alignment and multi-scale context fusion, effectively mitigating the problems of information misalignment and inconsistent feature representation between modalities. The multimodal feature enhancement module downsamples and performs multi-layer convolution on the fused mid-to-high-level features, improving the recovery capability of boundary structures and the accuracy of feature representation, further enhancing the segmentation effect.

[0068] The multimodal feature interaction module and the multimodal feature enhancement module have good versatility and portability, and can be easily embedded into any existing segmentation network structure, providing general structural enhancement capabilities for multimodal medical image segmentation tasks.

[0069] The training module decodes the enhanced feature representation, CT modality feature representation, and MR modality feature representation to obtain multimodal prediction maps, CT prediction maps, and MR prediction maps. It calculates the cross-entropy loss between the multimodal prediction maps, CT prediction maps, MR prediction maps, and graffiti annotations. It also calculates the multi-view CRF loss for the CT and MR images to obtain the intramodal regularization loss, and calculates the intermodal consistency loss between the multimodal prediction maps and the CT and MR prediction maps. The total loss is obtained by summing the cross-entropy loss, intramodal regularization loss, and intermodal consistency loss. The trained image segmentation model is obtained when the total loss is minimized. By constructing CT modality-specific branches, MR modality-specific branches, and cross-modal shared branches, joint modeling of modality-discrepancy features and modality-shared features is achieved. Even under weak annotation conditions, it can fully utilize multimodal complementary information, significantly improving segmentation accuracy and robustness.

[0070] The cross-entropy loss between the multimodal prediction map, CT prediction map, MR prediction map, and graffiti annotation is:

[0071] in, Mark the graffiti. This could be a multimodal prediction map, a CT prediction map, or an MR prediction map. It is the probability that pixel i belongs to category c. for The set of labeled pixels in the image.

[0072] The multi-view CRF loss of CT and MR images is:

[0073]

[0074] in, , , Two-dimensional CRF losses were obtained from axial, sagittal, and coronal slices, respectively. For CT prediction maps or MR prediction maps, It can be A, S, or C. and They are pixels and The predicted value, It is a collection of pixels; , Indicates based on The cost of discontinuity obtained by mixing kernel functions For the first individual core The weight, This is the corresponding feature vector.

[0075] Intramodal regularization loss for:

[0076] in, For multi-view CRF loss of CT images, For multi-view CRF loss of MR images, , These are the weighting coefficients.

[0077] Intermodal consistency loss between multimodal prediction maps and CT and MR prediction maps for:

[0078] in, , , These are multimodal prediction maps, CT prediction maps, and MR prediction maps, respectively. , These are the weighting coefficients.

[0079] Total loss for:

[0080] Among them, cross-entropy loss for:

[0081] in, These are prediction maps for the CT branch, MR branch, and multimodal branch, respectively. The cross-entropy loss is calculated between the multimodal prediction map and the graffiti annotation. The cross-entropy loss is calculated between the CT prediction image and the graffiti annotation. The cross-entropy loss is calculated between the MR prediction map and the graffiti annotation.

[0082] The training module combines three loss constraints: scribble supervision (SSL), intra-modal regularization (IMR), and inter-modal consistency (IMC), which effectively suppress overfitting under weak labeling conditions and improve the consistency and stability of multimodal prediction results.

[0083] The inference module is used to input the CT and MR images to be segmented into the trained image segmentation model and output the segmented images.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal weakly supervised medical image segmentation method, characterized by: The methods include: Feature extraction was performed on CT images and MR images respectively to obtain CT modal feature representations and MR modal feature representations; At each scale, the same-layer features of CT modal feature representation and MR modal feature representation are fused to obtain the fused feature representation; The fused feature representation is downsampled and processed by multiple convolutions to obtain the enhanced feature representation. The enhanced feature representation, CT modal feature representation, and MR modal feature representation are decoded separately to obtain multimodal prediction maps, CT prediction maps, and MR prediction maps. The cross-entropy loss between the multimodal prediction maps, CT prediction maps, MR prediction maps, and graffiti annotations is calculated. The multi-view CRF loss for the CT image and MR image is calculated separately to obtain the intramodal regularization loss. The intermodal consistency loss between the multimodal prediction map and the CT and MR prediction maps is calculated. The cross-entropy loss, intramodal regularization loss, and intermodal consistency loss are added together to obtain the total loss. The image segmentation model is obtained when the total loss is minimized. The intermodal consistency loss between the multimodal prediction map and the CT and MR prediction maps is calculated. for: ； in, , , These are multimodal prediction maps, CT prediction maps, and MR prediction maps, respectively. , These are the weighting coefficients; Input the CT and MR images to be segmented into the trained image segmentation model, and output the segmented images.

2. The multimodal weakly supervised medical image segmentation method according to claim 1, characterized in that: The process of fusing the same-layer features of CT modal feature representation and MR modal feature representation at each scale includes: The same-layer features represented by CT modality features and MR modality features are respectively concatenated along the channel dimension to obtain the concatenated features of each layer; Convolution is performed on the concatenated features of each layer to obtain the processed features. The processed features are then input into the AReLU activation function to obtain the fused feature representation of each layer.

3. The multimodal weakly supervised medical image segmentation method according to claim 2, characterized in that: The fusion feature representation of each layer for: ； in, This indicates that the CT image has passed through the CT encoder. i Modal feature representation after layer downsampling This indicates that the MR image has passed through the MR encoder. i Modal feature representation after layer downsampling , This is a channel-level splicing operation. This is a 3×3×3 convolution operation. This is the AReLU activation function.

4. The multimodal weakly supervised medical image segmentation method according to claim 3, characterized in that: The process of downsampling and multi-layer convolution for the fused feature representation includes: fusion feature representation Perform a 1 / 2 resampling operation to obtain the feature representation. ; Feature representation Inputting the AReLU activation function yields the feature representation. ; Feature representation Three convolutional operations are performed. The features after each convolutional operation are input into an AReLU activation function. The features after the third convolutional operation are then compared with the feature representation. The features are added together to obtain the enhanced feature representation.

5. The multimodal weakly supervised medical image segmentation method according to claim 4, characterized in that: Enhanced feature representation for: ； in, This represents the nth convolution operation. , , , This indicates a 1 / 2 resampling operation. The AReLU activation function is used. This indicates the addition of features.

6. The multimodal weakly supervised medical image segmentation method according to claim 1, characterized in that: The cross-entropy loss between the multimodal prediction map, CT prediction map, MR prediction map, and graffiti annotation is: ； in, Label the graffiti. This could be a multimodal prediction map, a CT prediction map, or an MR prediction map. It is the probability that pixel i belongs to category c. for The set of labeled pixels in the image.

7. The multimodal weakly supervised medical image segmentation method according to claim 1, characterized in that: The multi-view CRF loss of CT and MR images is: ； ； in, , , Two-dimensional CRF losses were obtained from axial, sagittal, and coronal slices, respectively. For CT prediction maps or MR prediction maps, It can be A, S, or C. and They are pixels and The predicted value, It is a collection of pixels; , Indicates based on The cost of discontinuity obtained by mixing kernel functions For the first individual core The weight, This is the corresponding feature vector.

8. The multimodal weakly supervised medical image segmentation method according to claim 1, characterized in that: Intramodal regularization loss for: ； in, For multi-view CRF loss of CT images, For multi-view CRF loss of MR images, , These are the weighting coefficients.

9. A multimodal weakly supervised medical image segmentation system, characterized in that: The system includes: The feature extraction module is used to extract features from CT images and MR images respectively, to obtain CT modal feature representations and MR modal feature representations; The multimodal feature interaction module is used to fuse the same-layer features of CT modal feature representation and MR modal feature representation at each scale to obtain the fused feature representation; The multimodal feature enhancement module is used to downsample and perform multi-layer convolution processing on the fused feature representation to obtain the enhanced feature representation; The training module decodes the enhanced feature representation, CT modal feature representation, and MR modal feature representation to obtain multimodal prediction maps, CT prediction maps, and MR prediction maps. It calculates the cross-entropy loss between the multimodal prediction maps, CT prediction maps, MR prediction maps, and graffiti annotations. It also calculates the multi-view CRF loss for the CT and MR images to obtain the intramodal regularization loss, and calculates the intermodal consistency loss between the multimodal prediction maps and the CT and MR prediction maps. The cross-entropy loss, intramodal regularization loss, and intermodal consistency loss are summed to obtain the total loss. The trained image segmentation model is obtained when the total loss is minimized. The intermodal consistency loss between the multimodal prediction maps and the CT and MR prediction maps is also considered. for: ； in, , , These are multimodal prediction maps, CT prediction maps, and MR prediction maps, respectively. , These are the weighting coefficients; The inference module is used to input the CT and MR images to be segmented into the trained image segmentation model and output the segmented images.

10. The multimodal weakly supervised medical image segmentation system according to claim 9, characterized in that: The cross-entropy loss between the multimodal prediction map, CT prediction map, MR prediction map, and graffiti annotation is: ； in, Label the graffiti. This could be a multimodal prediction map, a CT prediction map, or an MR prediction map. It is the probability that pixel i belongs to category c. for The set of labeled pixels in the image.

Images