Layered semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion

Abstract

The invention discloses a PET / CT (positron emission tomography / computed tomography) dynamic cross-modal fusion-based layered semi-supervised tumor segmentation system, and provides a multi-modal fusion semi-supervised segmentation framework which can realize accurate medical image segmentation by performing consistency learning by using unmarked multi-modal data. Specifically, multi-modal feature information is fully fused by designing CMAF-Net, and multi-scale pyramid depth supervision-uncertainty perception semi-supervised segmentation is realized by using an MUP-SSL method. A large number of experiments are carried out on a plurality of data sets, and results show that the performance of the method (only using a small part of marked data) provided by the invention is close to or even sometimes superior to that of a completely supervised method with 100% marked data, and the method provided by the invention is generally superior to the most advanced single-mode semi-supervised solution. In addition, the method is high in universality and can be widely applied to other segmentation tasks.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion. Background Technology

[0002] Medical imaging uses non-invasive methods to obtain images of internal human tissues for disease detection and analysis. Lesion segmentation is one of the important applications of medical image analysis (Qureshi I, Yan J, Abbas Q, et al. Medical imagesegmentation using deep semantic-based methods: A review of techniques, applications and emerging trends[J]. Information Fusion, 2023, 90: 316-352.). A precisely segmented lesion can effectively mark the location and structure of the disease, and can also provide significant assistance to doctors in future diagnosis and treatment. Therefore, researching fast and accurate lesion image segmentation methods has important clinical significance and potential value.

[0003] Most existing medical image segmentation techniques focus on segmentation performance on fully supervised datasets. For example, supervised learning-based encoder-decoder networks such as U-Net (Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation[C] / / Medical imagecomputing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, procedures, part III 18. Springer International Publishing, 2015: 234-241.) and their variants can fuse shallow and deep features to achieve relatively good segmentation results. For example, VNet (Abdollahi A, Pradhan B, Alamri A. VNet: An end-to-end fully convolutional neural network for road extraction from high-resolution remote sensing data[J]. IEEE Access, 2020, 8: 179424-179436.) extends UNet (Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation[C] / / Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, procedures, part III 18. Springer international publishing, 2015: 234-241.) to improve residual connections for 3D volumetric medical image segmentation.UNet++ (Zhou Z, Rahman SiddiqueeM, Tajbakhsh N, et al. Unet++: A nested u-net architecture for medical imagesegmentation[C] / / Deep learning in medical image analysis and multimodallearning for clinical decision support: 4th international workshop, DLMIA 2018, and 8th international workshop, ML-CDS2018, held in conjunction with MICCAI2018, Granada, Spain, September 20, 2018, proceeding 4. Springer International Publishing, 2018: 3-11.) and UNet3+ (Huang H, Lin L, Tong R, et al. Unet 3+: A full-scale connected unet for medical image segmentation[C] / / ICASSP 2020-2020IEEEinternational conference on acoustics, speech and signal The skip connections were redesigned to aggregate features with different stages and scales, further improving model performance. Subsequently, an attention mechanism was introduced to calibrate and enhance features in the channel and spatial dimensions to obtain better feature representation capabilities. Attention UNet (Oktay O, Schlemper J, Folgoc LL, et al. Attention u-net: Learning where to look for the pancreas[J]. arXiv preprint arXiv: 1804.03999, 2018.) integrates attention gates into UNet to improve low-level features of skip connections.Hu et al. (Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.) proposed a spatial and channel module to enhance the performance of segmentation networks. CANet (Gu R, Wang G, Song T. et al. CA-Net: Comprehensive attention convolutional neural networks for explainable medical image segmentation[J]. IEEE transactions on medical imaging, 2020, 40(2): 699-711.) combines channel, spatial, and scale attention with segmentation networks for explainable medical image segmentation. Recently, Transformer (Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.) has been used to explicitly model long-range dependencies to capture relationships between multiple organs and improve segmentation results (Chen J, Lu Y, Yu Q, et al. Transunet: Transformers make strong encoders for medical image segmentation[J]. arXiv preprint arXiv: 2102.04306, 2021.). The loss function aims to minimize the difference between network predictions and true labels, playing an irreplaceable role in model training (Ma J, Chen J, Ng M, et al. Loss odyssey in medical image segmentation[J]. Medical image analysis, 2021, 71: 102035.). These unimodal U-Net variants have demonstrated broad application potential and excellent performance in various application domains.

[0004] Meanwhile, thanks to the rapid development of multimodal medical imaging technology, multimodal medical image fusion methods have been extensively studied. Compared to single-modal medical images, multimodal medical images can provide doctors with rich complementary information. Currently, there are various high-performance fully supervised multimodal fusion segmentation networks, the success of which largely depends on a large amount of pixel-level densely labeled data. However, medical images typically have only a small amount of labeled data and a large amount of unlabeled data. In practice, due to the limited number of radiologists, their time, and the effectiveness of annotation efficiency, obtaining large datasets of accurately annotated medical images is very difficult. At the same time, commonly used medical imaging methods such as CT and PET are 3D imaging, which further increases the workload of manual annotation compared to 2D images that require experts to annotate volumes layer by layer (Y. Zhang, Q. Liao, L. Ding, and J. Zhang, “Bridging 2d and 3d segmentation networks for computation-efficient volumetric medical image segmentation: An empirical study of 2.5 d solutions,” Computerized Medical Imaging and Graphics, 2022.).

[0005] To alleviate the burden of manual labeling and address the difficulties of lesion annotation, thereby promoting intelligent medical diagnosis, an increasing number of experts are devoting significant effort to researching efficient deep learning for medical image segmentation and annotation (Y. Zhang, Q. Liao, L. Yuan, H. Zhu, J. Xing, and J. Zhang, “Exploiting shared knowledge from non-covid lesions for annotation-efficient covid-19 ct lung infections segmentation,” IEEE journal of biomedical and health informatics, vol. 25, no. 11, pp. 4152-4162, 2021; X. Zhao, Z. Shen, D. Chen, S. Wang, Z. Zhuang, Q. Wang, and L. Zhang, “One-shot traumatic brain segmentation with adversarial training and uncertainty rectification,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 120-129; L. Zhu, K. Yang, M. Zhang, LLChan, TKNg, and BCOoi, “Semi-supervised unpaired multi-modal learning for label-efficient medical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 394-404.). These methods combine the accuracy of fully supervised learning with the flexibility of unsupervised learning, allowing for improved accuracy in segmentation tasks with limited labeled data. Researchers have developed various semi-supervised learning techniques, such as pseudo-label generation, consistency regularization, adversarial learning, contrastive learning, and hybrid learning methods, which have brought new progress to the field of medical image segmentation.Self-training algorithms are considered the basic prototype of pseudo-label methods. They first pre-train a model on a labeled dataset and then iteratively retrain or fine-tune the model itself by predicting unlabeled data. To expand labeled datasets, Antti et al. proposed a new semi-supervised learning strategy: the Teacher-Student Model (Mean Teacher) (Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[J]. Advances in neural information processing systems, 2017, 30.), which learns from unlabeled data by encouraging the student model output to be consistent with the teacher model. Although URPC (Luo X, Wang G, Liao W, et al. Semi-supervised medical image segmentation via uncertainty rectified pyramid consistency[J]. Medical Image Analysis, 2022, 80: 102517.) introduces an uncertainty estimation mechanism based on multi-scale prediction differences, achieving computational efficiency optimization for a single forward propagation, this simple computational method struggles to accurately capture the reliability of modality-specific regions and cannot achieve modality difference perception in dynamic pseudo-label cleanup. The pseudo-label method refers to training an initial segmentation model using limited labeled data and then using this model to predict unlabeled data, generating pseudo-labels. These pseudo-labels, along with the original labeled data, are then used to further train the model. By iteratively updating the model and pseudo-labels, the segmentation accuracy is gradually improved. David et al. (Berthelot D, Carlini N, Goodfellow I, et al. Mixmatch: A holistic approach to semi-supervised learning[J]. Advances in neural information processing systems, 2019, 32.) proposed the MixMatch method, which generates pseudo-labels by mixing and augmenting labeled and unlabeled data, and uses the pseudo-labels for model training.Kihyuk et al. proposed FixMatch based on MixMatch (Sohn K, Berthelot D, Carlini N, et al. Fixmatch: Simplifying semi-supervised learning with consistency and confidence[J]. Advances in neural information processing systems, 2020, 33: 596-608.). FixMatch applies two different data augmentations to unlabeled images: weak augmentation and strong augmentation. It then uses the weakly augmented image prediction to generate pseudo-labels to supervise the strongly augmented image prediction. Consistency regularization is an important technique in image segmentation. By adding different perturbations to the same input image at the input layer or feature layer, it requires the model to output consistent prediction results. Recent trends (D. Chen, Y. Bai, W. Shen, Q. Li, LYu, and Y. Wang, “MagicNet: Semi-supervised multi-organ segmentation via magic-cube partition and recovery,” in CVPR, 2023, pp. 23 869-23 878.; Xu Z, Wang Y, Lu D, et al. Ambiguity-selective consistency regularization for mean-teacher semi-supervised medical image segmentation[J]. Medical Image Analysis, 2023, 88: 102880.; Y. Bai, D. Chen, Q. Li, W. Shen, and Y. Wang, “Bidirectional copy-paste for semi-supervised medical image segmentation,” in CVPR, 2023, pp. 11514-11524.) indicate that, in some cases, the performance of semi-supervised methods is almost comparable to that of fully supervised methods.For example, (D. Chen, Y. Bai, W. Shen, Q. Li, L. Yu, and Y. Wang, “MagicNet: Semi-supervised multi-organsegmentation via magic-cube partition and recovery,” in CVPR, 2023, pp. 23869-23878.) outperforms fully supervised methods on the BTCV dataset with 40% annotation.

[0006] Research indicates a natural synergistic relationship between semi-supervised learning and multimodal fusion: the complementarity of multimodal information can provide a basis for generating more reliable pseudo-labels in semi-supervised learning; and the consistency constraints of semi-supervised learning can guide feature fusion in PET / CT, enabling the model to learn more robust representations under limited labeling conditions. This prompts us to consider how to design a novel medical image segmentation framework that can address both the label scarcity problem and the difficulty of effectively coordinating the complementarity between heterogeneous modalities under low-labeled data.

[0007] The research leads to the conclusion that most current semi-supervised segmentation methods are based on single-modal medical images. Unfortunately, due to limitations in effective labels, SSL-based medical image segmentation methods cannot fully exploit the learnable features within medical images. In contrast, multimodal images can provide rich complementary information. With limited labeled data, building a segmentation model that can deeply integrate bimodal information while resisting interference from insufficient labeling requires a return to fully supervised frameworks to further explore multimodal fusion mechanisms and seek more robust multimodal interaction mechanisms.

[0008] Existing fully supervised methods (such as A2FSeg (Wang Z, Hong Y. A2FSeg: Adaptive multi-modal fusion network for medical image segmentation [C] / / International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023: 673-681.)) have made progress in improving fusion accuracy by designing modality-specific encoders or dynamic weight allocation strategies. However, these methods heavily rely on a large amount of labeled data and are difficult to directly transfer to semi-supervised scenarios. The fundamental reason is that the fusion priors learned by fully supervised models (such as the feature association patterns between PET and CT) are often based on the assumption of complete annotation, while the intermodal relationships of unlabeled data in semi-supervised tasks may be shifted due to pseudo-label noise or missing annotations. Therefore, how to adapt the modal interaction prior knowledge under the fully supervised framework to the semi-supervised learning paradigm has become the key to unlocking the potential of multimodal data.

[0009] Meanwhile, semi-supervised learning offers an effective solution to alleviate the scarcity of labeled medical image data. However, when directly applied to multimodal PET / CT segmentation tasks, it still faces the complex challenge of coexisting heterogeneity and synergy between modalities. PET and CT differ fundamentally in their imaging principles, resolution, and information representation: PET highlights the functional characteristics of tumors through metabolic activity, while CT delineates anatomical structures through spatial details. Using simple modal cascading or shallow feature fusion may lead to problems such as feature space chaos and noise coupling (this phenomenon will be further verified in the experimental section). This coarse-grained fusion strategy not only fails to fully exploit the synergistic potential of the two modalities but may also interfere with pseudo-label generation and consistency constraints due to modal heterogeneity, ultimately resulting in performance bottlenecks for the model on unlabeled data.

[0010] Existing research has attempted to incorporate multimodal learning strategies into semi-supervised frameworks. These methods typically address the issue of missing modalities, employing simple modality concatenation or feature fusion, treating different modalities as isolated information sources. They lack effective deep interaction mechanisms and fail to establish cross-modal collaborative learning capabilities, significantly limiting the utilization of unlabeled multimodal data. Furthermore, existing semi-supervised segmentation methods often neglect the importance of multi-scale features during knowledge transfer. How to dynamically adjust label generation strategies based on the reliability of different modalities in different regions is also a pressing issue.

[0011] Currently, research on semi-supervised multimodal tasks mainly focuses on classification or clustering tasks. Research on semi-supervised segmentation tasks for multimodal data is scarce. DAFNet (Xu J, He X. DAF-Net: A Dual-Branch Feature Decomposition Fusion Network with Domain Adaptive for Infrared and Visible Image Fusion[J]. arXiv preprint arXiv: 2409.11642, 2024.) uses incompletely labeled multimodal data for image segmentation; this work constructs a complex network using Disentanglement, Alignment, and Fusion to fuse multimodal data, thereby improving the performance of the target modality. CMC (Zhou X, Sun Y, Deng M, et al. Robust Semi-supervised Multimodal Medical Image Segmentation via CrossModality Collaboration [C] / / International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024: 57-67) uses unimodal images as input to a traditional consistency learning network, employing channel attention in intermediate layers to extract irrelevant knowledge for information fusion. This combination allows semi-supervised segmentation methods to be transferred to the multimodal semi-supervised domain. However, unfortunately, non-customized networks do not maximize the potential correlations and rich learnable features between the two modalities, leading to significant limitations in multimodal semi-supervised segmentation. Existing methods often ignore intermodal complementarity, performing only simple fusion without fully exploring cross-modal information. Future research should focus on designing specialized architectures to capture intermodal dependencies while addressing the imbalance problem in multimodal data. By developing more refined fusion strategies and introducing cross-modal learning techniques, this invention is expected to significantly improve the performance of multimodal semi-supervised segmentation. Summary of the Invention

[0012] The purpose of this invention is to address the shortcomings of existing technologies by proposing a hierarchical semi-supervised tumor segmentation system based on dynamic cross-modal fusion of PET / CT.

[0013] The objective of this invention is achieved through the following technical solution: a hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion, the system comprising:

[0014] The data acquisition module is used to acquire dual-modal data of PET and CT images and divide them into labeled and unlabeled data.

[0015] A cross-modal adaptive fusion network module for tumor segmentation is used to independently extract features from CT and PET images during the encoding stage and perform dynamic feature fusion through a cross-modal attention mechanism. Specifically, cross-modal attention calculation is performed based on the cross-resolution attention mechanism (CRA). The CRA output is upsampled and combined with the residual connection, and then input into the region-limited attention mechanism (RCA) for further processing, transforming the original input data into an output with enhanced multimodal features. During the decoding stage, a lesion-focusing adaptive weighted network is used to enhance tumor-related features from multimodal features to achieve accurate segmentation. Specifically, the features generated by the k-th encoder are concatenated with the upsampled features of the k+1-th decoder to obtain a comprehensive CT and PET feature representation. The prediction map obtained by upsampling from the k+1-th decoder is used to separate foreground and background features, and a refined feature representation is obtained based on the importance weights of foreground and background features.

[0016] The multi-scale pyramid deep supervision module is used to add an auxiliary layer after each decoder in the decoding stage to form a hierarchical supervision component. It considers multi-scale prediction as a model optimization for deep supervised learning, upsamples the decoder's prediction map to be the same as the real label, and calculates the cross-entropy loss (Diceloss) between the prediction and the real label at different scales to obtain multi-scale deep supervision.

[0017] The multi-scale uncertainty-aware pyramid module for semi-supervised segmentation is used for the synergy of pyramid-based supervision and uncertainty-guided learning. The student model obtains increasingly reliable supervision at multiple scales, while the teacher model can generate increasingly high-quality targets through an exponential moving average update mechanism. Both the student model and the teacher model adopt the network architecture of a cross-modal adaptive fusion network module for tumor segmentation. Labeled data is trained by the student model through pyramid-style hierarchical supervision, while unlabeled data is trained by the teacher model to generate pseudo-labels after uncertainty correction, forming a hierarchical consistency loss. Hierarchical feature representation is used for semi-supervised tumor segmentation.

[0018] Furthermore, the Cross-Resolution Attention (CRA) mechanism precisely modulates the information flow through learnable global and local gate control units in the bidirectional reciprocal gated attention (MGFM) mechanism, capturing the long-distance dependencies between CT and PET images at downsampled resolution. Subsequently, the region-restricted attention (RCA) mechanism divides the features into local window regions at the original resolution, and captures fine spatial details within the window by reapplying MGFM.

[0019] Furthermore, the Cross-Resolution Attention (CRA) mechanism also includes a global context-aware gate mechanism to adjust the overall information exchange rate and achieve dynamic adjustment of multimodal information exchange. Specifically, it first performs channel-dimensional concatenation on the CT modality downsampling features and PET modality downsampling features to form joint features; then, it extracts the global context information of the joint features through global average pooling, and then processes it through linear transformation of the learnable weight matrix and activation function, enabling the network to adaptively determine the retention ratio of global information of each modality according to different combinations of imaging features.

[0020] Furthermore, Region-Restricted Attention (RCA) specifically involves recombining the output features from CRA into a fixed-size 3D window. Within each window, CT and PET features utilize the same MGFM bidirectional reciprocal gating attention mechanism as RCA, performing granular detail relationships within a smaller spatial region to obtain the final output.

[0021] Furthermore, in the multi-scale uncertainty-aware pyramid module of semi-supervised segmentation, for unlabeled data, a multi-scale pyramid uncertainty-aware consistency loss is used to minimize the difference between predictions at different scales. Comprehensive pyramid supervision is implemented throughout the architecture of the student model. The consistency loss is calculated for each layer of feature maps of the student network through the teacher network. This multi-level consistency calculation ensures that the network maintains consistency with the teacher model from shallow local texture features to deep global semantic features.

[0022] Furthermore, in the multi-scale uncertainty perception pyramid module of semi-supervised segmentation, uncertainty quantification is integrated with pyramid supervision, and consistency constraints are applied only to high-confidence regions; for each unlabeled sample, the teacher model not only generates target predictions, but also estimates the uncertainty of the predictions through Monte Carlo Dropout technology.

[0023] Furthermore, in the semi-supervised segmentation multi-scale uncertainty perception pyramid module, the uncertainty perception mechanism is integrated with a multi-layer supervision strategy to create a dynamic self-correcting knowledge transfer framework. At each network level, the uncertainty graph with high confidence of the teacher network is used to guide the consistency training of the corresponding level. This hierarchical uncertainty perception mechanism ensures that the knowledge transfer process can guide the reliability constraints of the features of different abstraction levels of the student network in real time.

[0024] Furthermore, in the semi-supervised segmentation multi-scale uncertainty perception pyramid module, as training progresses, the teacher model is updated through the exponential moving average (EMA) mechanism: the student network and the teacher network share the same network backbone, and the teacher network weights are updated through the exponential moving average (EMA) parameter of the student network weights.

[0025] Furthermore, the multi-scale uncertainty perception pyramid module for semi-supervised segmentation combines unsupervised regularization loss and deep supervised loss to jointly optimize the network; and uses a time-dependent Gaussian warm-up function to control the balance between supervised loss and unsupervised consistency loss.

[0026] The beneficial effects of this invention are as follows: This invention proposes a multimodal fusion semi-supervised segmentation framework, DyCross-HSeg, which can achieve accurate medical image segmentation by utilizing unlabeled multimodal data for consistency learning. Specifically, by designing CMAF-Net to fully fuse multimodal feature information, the MUP-SSL method is used to achieve multi-scale pyramid depth-supervised, uncertainty-aware semi-supervised segmentation. Extensive experiments were conducted on multiple datasets, and the results show that the proposed method (using only a small portion of labeled data) performs at a level close to or sometimes even better than fully supervised methods with 100% labeled data. Furthermore, the proposed method generally outperforms state-of-the-art single-modal semi-supervised solutions. In addition, the method of this invention is highly versatile and can be widely applied to other segmentation tasks. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of a semi-supervised segmentation framework for multimodal fusion teacher-student collaboration;

[0029] Figure 2 This is a schematic diagram of the CMAF-Net network structure;

[0030] Figure 3 A schematic diagram of the two-way reciprocal gating attention mechanism MGFM;

[0031] Figure 4 A comparative diagram of DICE box plots for tumor segmentation methods;

[0032] Figure 5 This is a schematic diagram illustrating a quantitative comparison between the multimodal fusion semi-supervised segmentation scheme of the present invention and various multimodal fusion fully supervised segmentation methods;

[0033] Figure 6 This is a schematic diagram of the visual segmentation results of the tumor by the semi-supervised network of the present invention;

[0034] Figure 7This is a visual illustration of the feature maps generated at different encoder stages according to the present invention.

[0035] Figure 8 The present invention is visualized in thermal images on PET and CT. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the invention.

[0037] This invention provides a hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion, and designs a multi-modal fusion teacher-student collaborative semi-supervised segmentation model, such as... Figure 1 As shown. This model processes two types of data simultaneously: labeled data ( Figure 1 (Green box on the left) and unlabeled data ( Figure 1 (Yellow box on the left). The core structure includes the teacher model ( Figure 1 (blue network) and student model ( Figure 1 Both (the pink network in the middle) and (the two networks) adopt the CMAF-Net (Cross-modal Adaptive Fusion Network for Tumor Segmentation) architecture, such as Figure 2 As shown, the teacher model updates parameters and generates pseudo-labels using exponential moving average (EMA), providing additional supervision signals to the student model. In terms of training strategy, labeled data is trained on the student network through a pyramid-style hierarchical supervision, while unlabeled data is trained on pseudo-labels generated by the teacher network after uncertainty correction, forming a hierarchical consistency loss. This structured design effectively overcomes the challenge of scarce medical data annotation, constructing a complete inference path from raw multimodal images to high-precision tumor segmentation results.

[0038] The innovative internal design of the CMAF-Net cross-modal adaptive fusion network for tumor segmentation lies in the following: First, during the encoding phase, complementary information from CT and PET is integrated through Cross-Modal Dual-Stage Gated Fusion (CDSGF). This module dynamically fuses heterogeneous modal features using two gating mechanisms: Cross-Resolution Attention (CRA) and Region-Confined Attention (RCA). The specific fusion strategy, MGFM (Mutual Gate Flow Module), is as follows: Figure 3 As shown; secondly, in the decoding stage, a Lesion-Focused Adaptive Weighting Network (LFAWN) is used to enhance the features of the lesion region, and the segmentation accuracy of the target region is improved through multi-level prediction.

[0039] The Cross-Modal Dual-Stage Gated Fusion (CDSGF) is detailed below:

[0040] CT and PET images are input into separate encoder branches for feature extraction, and dynamic feature fusion is performed through a cross-modal attention mechanism. th The output characteristics of each encoder module are: CDSGF considers them as two groups with the same number of channels, C. k And the window size is The CDSGF first performs a cross-resolution attention (CRA) mechanism to achieve cross-modal attention computation. This mechanism precisely modulates the information flow through learnable global and local gate control units in bidirectional reciprocal gated attention (MGFM), capturing long-distance dependencies between CT and PET at downsampling resolution. Subsequently, a region-restricted attention (RCA) mechanism divides the features into local window regions at the original resolution, and captures fine spatial details within the window by reapplying MGFM. The CRA output is upsampled and combined with the residual connection, then input into RCA for further processing. After this series of operations, the original input data is transformed into an output with enhanced features, where each voxel contains rich contextual information within and between different modalities. This increases the amount of information it carries without changing the original spatial resolution, providing multi-level precise control over the transmission of information flow, enabling the network to adaptively determine the degree and manner of intermodal interactions based on the contextual information of the input data.

[0041] The Cross-Resolution Attention (CRA) mechanism described above is as follows:

[0042] Input feature maps for CT and PET modalities CRA first downsamples the features using window convolution, where the size of the convolution sum and the stride are both equal to the window size.

[0043]

[0044] The feature dimension after downsampling is . Next, the intermodal bidirectional reciprocal gating (MGFM) attention mechanism is calculated, such as... Figure 3 As shown.

[0045] First, calculate the query, key, and value vectors for the features of the CT and PET branches respectively, and generate the QKV:

[0046]

[0047] in, It is the three-dimensional feature tensor of the i-th medical imaging modality after downsampling by a k-level encoder, where the subscript i∈{ct, pet} is used to distinguish between computed tomography (CT) and positron emission tomography (PET) imaging modalities. It is obtained through a learnable linear transformation matrix. and Generate a sequence of query vectors, a sequence of key vectors, and a sequence of value vectors: Q i K i and V i .

[0048] Then, bidirectional attention is calculated:

[0049]

[0050] in This represents matrix multiplication, where C is the feature dimension of the query vector, and the Softmax function is used to normalize the attention weights. The transpose operation T in the formula adjusts the matrix dimension to match the requirements of matrix multiplication, and the scaling factor... Gradient calculation used to stabilize attention weights.

[0051] In order to selectively control the flow of information, this invention introduces a gating mechanism:

[0052]

[0053] in, A learnable weight matrix specifically designed for self-attention gating in CT modalities. For the learnable weight matrix specifically designed for PET modal self-attention gating, in cross-modal gating computation, Learnable parameters specifically designed for handling CT-to-PET cross-modal interactions. This represents the cross-modal learnable gating weight matrix from PET to CT. σ is the sigmoid activation function, used to restrict the gating weights between 0 and 1, ensuring that the gating mechanism can adaptively adjust the contribution of different modal features, thereby achieving effective fusion of multimodal information.

[0054] In addition, a global context-aware gate mechanism is designed to adjust the overall information exchange rate and achieve dynamic adjustment of multimodal information exchange:

[0055]

[0056] The specific process is as follows: This mechanism first samples the features of the CT modality. and PET modal sampling features Perform channel-level concatenation to form a joint feature representation. The global contextual information of the joint feature is extracted through a global average pooling (AvgPool) operation, and then passed through a learnable weight matrix W. g The linear transformation and sigmoid activation function σ processing enable the network to adaptively determine the proportion of global information retained for each modality based on different combinations of imaging features (such as the correspondence between the metabolic activity of the tumor region and the anatomical structure). This ultimately generates a pair of global gating weights. This design achieves optimal multimodal information integration at both global and local scales. It enables the network to dynamically balance the contribution ratio between anatomical information from CT and functional metabolic information from PET, tailored to the imaging characteristics of different cases, significantly improving the robustness of multimodal feature fusion.

[0057] The output of CRA is:

[0058]

[0059] Where ⊙ represents the dot product. After restoring the original spatial dimension through upsampling, the original features are fused through residual connections to obtain the output of this module:

[0060]

[0061] Region-Confined Attention (RCA) is as follows:

[0062] The output features from CRA are reorganized into a fixed size (W). h W w W d The 3D window uses the same MGFM bidirectional reciprocal gating attention mechanism as RCA for CT and PET features within each window. This allows for granular detail modeling within a smaller spatial region, optimizing local relationships, and yielding the final output.

[0063] The lesion-focused adaptive weighting network LFAWN (Lesion-Focused Adaptive Weighting Network) used in the decoding stage is as follows:

[0064] This network aims to enhance tumor-related features from multimodal features to achieve accurate segmentation. Specifically, the CDSGF module generates features... Upsampling features of (k+1) decoder blocks Cascade to create integrated CT and PET feature representations The prediction map obtained by upsampling using the (k+1)th decoder Separate foreground and background features:

[0065]

[0066] Where ⊙ denotes the dot product. Then, global average pooling is applied to the foreground and background features. To obtain the channel-by-channel intensity vector: These vectors are then passed through a multilayer perceptron (MLP) with an activation function to output importance weights for foreground and background features, thus obtaining a refined feature representation:

[0067]

[0068] Where σ(MLP(V) fg )) and σ(MLP(V g It can adaptively select foreground and background-independent features for optimization, with σ(MLP(V)) fg For example, for any given channel, when the corresponding foreground weight exceeds 0, the channel information is emphasized to optimize tumor-related features; otherwise, the channel's contribution is attenuated. In this way, the network can be ensured to focus its attention on the most diagnostically significant regions.

[0069] This invention first introduces some default formulas for semi-supervised learning: the training set consists of two subsets: a CT-PET paired-label dataset with N annotated samples. and a dataset with M unlabeled images Therefore, the entire training set is Input 3D image Suppose an image Then provide its true value Y i ∈{0,1} H×W×D However, if Therefore, its true value is unavailable. f φ (.) is used to represent a segmentation model with parameter set φ.

[0070] Multi-scale pyramid depth supervision:

[0071] In this invention, 3D V-Net is used as the backbone. To extract hierarchical hidden representations, an auxiliary layer is added after each block in the decoding stage to form a hierarchical supervised component. Multi-scale prediction is considered as a model optimization of deep supervised learning. For the prediction graph p of the k-th level decoder... k The predicted values ​​are upsampled to be the same as the ground truth labels (GT), and the cross-entropy loss L between the predicted and ground truth labels at different scales is calculated. bce and Dice loss L dice To obtain multi-scale depth supervision, the hierarchical supervision segmentation loss is formally defined as:

[0072] Supervised loss L sup It can be represented as:

[0073]

[0074] in, Y represents the prediction graph output by the k-th level student network decoder. i Represents the actual ground conditions, ω k This represents the weight coefficient assigned to the k-th level prediction. The weight coefficient reflects the importance of the prediction results at each scale. The experiment in this invention uses a predefined weight configuration: ω k ∈[1,2,3,4,5], to balance the contribution of deep and shallow features.

[0075] Multi-scale uncertainty-aware pyramid for semi-supervised segmentation:

[0076] Unlike existing works that use multi-scale prediction to accelerate the optimization process and improve performance in a fully supervised environment, this invention uses hierarchical feature representations for semi-supervised segmentation. To overcome the dual limitations of single-source supervision signals and the propagation of noisy pseudo-labels in traditional consistency training, a multi-scale uncertainty-aware pyramid model (MUP-SSL) for semi-supervised learning is proposed.

[0077] The synergistic combination of pyramidal supervision and uncertainty-guided learning creates a virtuous cycle: the student model receives increasingly reliable supervision across multiple scales, which in turn enables the teacher model to generate increasingly higher-quality targets through an exponential moving average update mechanism. This two-way improvement drives continuous enhancement of Segmentation Performance throughout the training process.

[0078] It addresses the fundamental challenge of pseudo-label reliability through uncertainty quantification combined with multi-scale learning. The model selectively learns from confident predictions while leveraging hierarchical feature representations, enabling simultaneous learning across low-level texture and high-level semantic features. This avoids knowledge bias at a single level of abstraction, allowing for more effective use of unlabeled data and reducing reliance on large, annotated datasets.

[0079] For unlabeled data, this invention introduces consistency regularization by encouraging multi-scale prediction similarity. This invention designs a multi-scale pyramid uncertainty-aware consistency loss to minimize the difference (i.e., variance) between predictions at different scales. Comprehensive pyramid supervision is implemented throughout the student model architecture, with the consistency loss calculated through the teacher network on the feature maps of each layer of the student network. Specifically:

[0080]

[0081] in Let f represent the k-th layer prediction function of the student model. t (·) represents the final layer prediction function of the teacher network, d k Let represent the mean squared error between the student model's prediction at layer k and the teacher model's prediction. This multi-level consistency calculation ensures that the network maintains consistency with the teacher model from shallow local texture features to deep global semantic features. (θ′, ξ′) and (θ, ξ) represent the weights of the teacher and student models, respectively, and different perturbations (e.g., adding noise to the input and network loss). (M⊙d) k ) v Apply consistency constraints only to trusted regions. This is particularly important because lesion boundaries are often blurred in medical image segmentation, leading to high uncertainty. Forcing consistency in these regions could cause error propagation. k ∈[1, 2, 3, 4, 5] are the weight coefficients assigned to the k-th layer, reflecting the relative importance of features at different levels; M is an uncertainty mask that achieves "credible region focusing"; |Ω M | represents the number of valid voxels in the mask; K=5 is the number of decoding layers, used for layer balancing to avoid over-representation of high-confidence regions in the loss; |Ω M |+K+∈ ensures that only the number of voxels in the reliable region is considered during normalization,∈ avoids the gradient explosion problem when the mask is too sparse in the early stage of training, andλ(t) is the consistency weight that is dynamically adjusted as the training process progresses. This reflects an important learning strategy: in the early stage of training, the model mainly relies on limited labeled data to build basic feature representations; as training progresses, the reliance on unlabeled data is gradually increased to expand the model's generalization ability.

[0082] A core challenge in semi-supervised learning is preventing the propagation of errors from noisy pseudo-labels. Inspired by existing work UA-MT, we seamlessly integrate uncertainty quantification with pyramid supervision, applying consistency constraints only to high-confidence regions. For each unlabeled sample, the teacher model not only generates a target prediction but also estimates the uncertainty of the prediction using Monte Carlo Dropout, as detailed below:

[0083] The constrained uncertainty graph U is calculated from information entropy:

[0084]

[0085] Where the average prediction probability The following results were obtained by performing T=8 Monte Carlo sampling operations on the teacher model:

[0086]

[0087] T is the number of Monte Carlo samples, and the average predicted probability. (θ, ξ) represent the weights of the student model and the different perturbations (e.g., adding noise to the input and network loss), f t (·) represents the final layer prediction function of the teacher network.

[0088] The uncertainty mask M estimates the uncertainty of the prediction using Bayesian Monte Carlo sampling: M = 1 U<τ(t) A dynamic threshold function is introduced to convert the uncertainty map into a binary mask, thereby selectively filtering high-uncertainty regions. This voxel-level uncertainty guidance mechanism ensures that the consistency loss only applies to reliable regions, significantly reducing the interference of noise signals on the training process. Its dynamic threshold τ(t) varies with the number of training iterations:

[0089] τ(t)=τ min +(τ max -τ min )·σ ramp (t)

[0090] Where τ min and τ max These are the minimum and maximum values ​​of the threshold, σ. ramp (t) control τ(t) from τ min Smooth transition to τ max The progress.

[0091] This invention empirically sets the EMA decay α to 0.99 (Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[J]. Advances in neural information processing systems, 2017, 30.). For uncertainty estimation, T=8 is set to balance the quality of uncertainty estimation and training efficiency. This invention also uses the same Gaussian ramp paradigm to adjust the dynamic threshold. As training continues, less and less data is filtered out, allowing students to gradually learn from relatively certain to uncertain cases.

[0092] Hierarchical uncertainty integration and dynamic feedback:

[0093] One innovation of this invention lies in integrating an uncertainty perception mechanism with a multi-layered supervision strategy to create a dynamic, self-correcting knowledge transfer framework. At each network layer i, this invention utilizes the high-confidence uncertainty graph U of the teacher network to guide consistent training at the corresponding layer. This hierarchical uncertainty perception mechanism ensures that the knowledge transfer process can guide the reliability constraints of features at different abstraction levels of the student network in real time. As training progresses, the teacher model is updated using an exponential moving average (EMA) mechanism: the student and teacher networks share the same network backbone, and the teacher network weights are updated using the exponential moving average (EMA) parameters of the student network weights. The teacher network weight update strategy in step t is represented as θ′. t =ηθ′ t-1 +(1-η)θ t η controls the EMA update rate, and the influence of the teacher network is weighed by the current student network and the teacher network weights at t-1.

[0094] This dynamic feedback loop drives the co-evolution of student and teacher models. As training progresses, the teacher model generates increasingly reliable supervision signals, while the student model acquires more accurate feature representations. Through the UPAS mechanism, the MUP-SSL model significantly improves the efficiency of semi-supervised learning, especially in tasks highly dependent on feature quality, such as medical image segmentation. Experimental results show that this uncertainty-guided multi-layer pyramid supervision strategy effectively prevents error propagation while fully utilizing the hierarchical representation capabilities of deep network architectures.

[0095] The unsupervised regularization loss and the deep supervised loss are combined to jointly optimize the network. The training objective is:

[0096]

[0097] in It is a supervisory item. It is an unsupervised consistency regularization term, which, according to (Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[J]. Advances in neural information processing systems, 2017, 30.; Yu L, Wang S, Li X, et al. Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation[C] / / Medical image computing and computer assisted intervention-MICCAI 2019: 22nd international conference, Shenzhen, China, October 13-17, 2019, procedures, part II 22. Springer International Publishing, 2019: 605-613.), uses a time-dependent Gaussian warm-up function. To control the balance between supervised loss and unsupervised consistency loss, where t represents the current training step size, t max This is the maximum training step size. This design ensures that the target loss is dominated by the supervised loss term at the beginning and avoids the network from getting stuck in degeneracy, i.e., unable to obtain meaningful target predictions for unlabeled data. This design achieves two key goals: (1) filtering reliable regions through uncertainty masks to avoid interference from misleading pseudo-labels; (2) achieving full-stack feature alignment from low-level texture to high-level semantics through multi-level consistency constraints. As training progresses, the teacher model generates increasingly reliable supervised signals through exponential moving average (EMA) updates, forming a self-improving dynamic learning system.

[0098] In the experimental stage, we used two publicly available datasets to evaluate the performance of the proposed model: the soft tissue sarcoma (STS) dataset (Oreiller V, Andrearczyk V, Jreige M, et al. Head and neck tumorsegmentation in PET / CT: the HECKTOR challenge[J]. Medical image analysis, 2022, 77: 102336.) and the head and neck tumor (Hecktor) 2022 dataset (Vallières M, Freeman CR, Skamene SR, et al. A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities[J]. Physics in Medicine & Biology, 2015, 60(14): 5471.).

[0099] The STS dataset contains PET / CT images of 51 patients diagnosed with soft tissue sarcoma of the extremities. Acquired from different scanning devices, it exhibits high heterogeneity. The CT images in this dataset have a planar resolution of 512×512 pixels. Each CT and PET image contains 91 to 311 slices, with 15 to 82 slices annotating the tumor region. The annotated tumor volume ranges from 17.35 cm³. 3 Up to 2332.72cm 3 The CT slices had a pixel spacing of 0.98mm × 0.98mm and a slice thickness of 3.27mm. The PET images had a planar resolution of 128 × 128 pixels, with the same number of slices as the CT images. The pixel spacing within each slice was 3.91mm × 3.91mm, and the slice thickness was 3.27mm. Forty cases from the STS dataset were randomly selected in an 8:2 ratio as the training set, and the remaining 11 cases were used as the test set.

[0100] The Hecktor 2022 dataset contains PET / CT images of 524 patients diagnosed with oropharyngeal and head and neck cancer, acquired from nine different medical centers, involving various scanning equipment and imaging protocols. The CT images in this dataset have a planar resolution of 512×512 pixels, and each CT and PET image contains 91 to 348 slices, with 16 to 73 slices annotating the tumor region. The tumor annotation volume ranges from 0.79 cm³. 3 Up to 186cm3 The pixel pitch within CT slices ranged from 0.98 mm to 1.36 mm, and the slice thickness was 3.27 mm. PET images had a planar resolution of 128 × 128 pixels, with the same number of slices as CT, a pixel pitch within slices ranging from 2.73 mm to 5.47 mm, and a slice thickness of 3.27 mm. Since this competition included the dual segmentation objectives of primary head and neck tumor lesions (GTVp) and affected lymph nodes (GTVn), this study planned to focus only on GTVp segmentation. An experimental dataset was constructed by selecting cases with clearly defined GTVp lesions (a total of 508 cases). The experiment randomly selected 406 patients at an 8:2 ratio as the training set and the remaining 102 patients as the test set.

[0101] Preprocessing the image data before the experiment can make the training data more suitable for the network input, which helps to accelerate network convergence and improve feature extraction.

[0102] For the STS dataset, this invention processes the data using a method similar to EFNet. First, nearest-neighbor interpolation is performed on the label images, followed by B-spline interpolation to resample to an isotropic resolution of 1mm × 1mm × 1mm. Second, the Huntsfield unit (HU) intensity of the CT data is calculated, and the values ​​are truncated to [-110, 190]. The standard uptake value (SUV) of the PET data is calculated, and adaptive contrast adjustment is performed according to thresholds (1%, 99.8%) to enhance image contrast and eliminate the influence of noise and other outliers. Third, min-max normalization and z-score normalization are performed on the CT and PET images respectively, and the labels are binarized. Finally, the images are uniformly cropped to obtain a final image with a size of 256*256*256.

[0103] For the Hecktor 2022 dataset, based on existing literature (Sun L, Sakaridis C, Liang J, et al. Event-based fusion for motion deblurring with cross-modal attention[C] / / European conference on computer vision.Cham: Springer Nature Switzerland, 2022: 412-428.), the images were resampled to an isotropic resolution of 1mm×1mm×1mm. The second step involved calculating the Huntsfield unit (HU) intensity of the CT data, truncating the values ​​to the range [-1024, 1024], and mapping them to [-1, 1]. PET pixel values ​​were then converted to standard uptake values ​​(SUV). The third step involved performing min-max normalization and z-score normalization on the CT and PET images, respectively, and binarizing the labels. Finally, the images were uniformly cropped to obtain a final image with a size of 200*200*200.

[0104] This framework was implemented in PyTorch on Ubuntu 20.04 NVIDIA A600, using the SGD optimizer to update network parameters (weight decay = 0.0001, momentum = 0.9). The initial learning rate was set to 0.01, divided by 10 every 2500 iterations. A total of 20500 iterations were trained as the network converged. The batch size was 2, consisting of one annotated image and one unannotated image. In this invention, a randomized 128×128×64 pixel size was used as the network input. The window size of the backbone network was set to (8, 8, 4). According to Tarvainen and Valpola (Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[J]. Advances in neural information processing systems, 2017, 30.), the decay parameter η of the exponential moving average (EMA) was 0.99, and a sliding window strategy was used to obtain the final segmentation result. We immediately employed standard data augmentation techniques to avoid overfitting (Yu L, Cheng JZ, Dou Q, et al. Automatic 3D cardiovascular MR segmentation with densely-connected volumetric convnets[C] / / Medical Image Computing and Computer-Assisted Intervention-MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part II 20. Springer International Publishing, 2017: 287-295.), including random flipping and rotations of 90, 180, and 270 degrees along the axial plane. A hierarchical mechanism was introduced for the model output, with the weights of the supervised loss and unsupervised consistency loss specified as {0.5, 0.4, 0.05, 0.05}.

[0105] This invention employs four widely used metrics for quantitative evaluation: Dice score, 95% Hausdorff distance (HD95), precision, and recall. Dice score, precision, and recall are expressed as percentages (%) in the experiment. The formulas for calculating the four evaluation metrics are as follows:

[0106]

[0107] Where TP, FP, TN, and FN represent true-positive, false-positive, true-negative, and false-negative segmentation voxels, and A, B, S(A), and S(B) represent the surface voxel sets of the label, prediction, label, and prediction results, respectively.

[0108] Figure 4 The diagram shows a comparison of DICE box plots for tumor segmentation methods, comparing the segmentation results of other methods in a single modality with the segmentation results of the method of the present invention in a multimodal fusion approach.

[0109] The advantages of the method of the present invention are as follows: Figure 4 The results clearly demonstrate this. On the Hecktor dataset, the method of this invention achieves a positional accuracy of approximately 0.8, outperforming other methods on CT with a median accuracy hovering around 0.5 and PET around 0.75. Furthermore, the method of this invention exhibits enhanced robustness, as evidenced by its smaller interquartile range (IQR). The superiority of the method of this invention is also evident on the STS dataset.

[0110] From these figures, we can conclude that by integrating complementary feature information from CT and PET, the multimodal network of this invention significantly improves the segmentation accuracy of tumor regions, effectively overcoming the inherent limitations of single-modal imaging in segmentation. Furthermore, since certain tissue structures often have similar density characteristics to the surrounding tumor region in CT images, CT-based segmentation alone has a high error rate. This invention effectively reduces these errors by utilizing complementary information and intrinsic correlations in fused CT and PET data, and these errors can be well avoided by using intrinsic correlations in multimodal data. The above analysis demonstrates the superior performance of the method of this invention, and the improvement of the method over other excellent methods becomes more pronounced as the number of tags used increases.

[0111] A quantitative comparison of the multimodal fusion semi-supervised segmentation scheme of this invention and various multimodal fusion fully supervised segmentation methods is as follows: Figure 5 As shown.

[0112] All experiments maintained the same experimental setup. It is clear that the network of this invention, using only 20% of the labels, outperformed most other excellent methods with a dice coefficient of 75.98%. Furthermore, due to using only 10% of the labels, the method of this invention is highly competitive, with a dice coefficient only 3.06% lower than the state-of-the-art (SOTA) methods. This demonstrates that semi-supervised methods can achieve promising results when only limited labeled data is available. Moreover, under different label ratio settings, the method proposed in this invention consistently outperforms MMC, indicating that it can fully utilize unlabeled data and deliver performance improvements. Furthermore, when the label ratio is increased to 20%, the method of this invention achieves results very close to those of nnUNet learning from 100% annotated images. These results demonstrate that the method of this invention achieves accurate segmentation results with only a small portion of the training images labeled, which is feasible for reducing annotation costs in clinical practice.

[0113] Figure 6 Visual segmentation results of tumors using different unimodal methods and the semi-supervised network of this invention were visualized on the Hecktor 2022 and STS datasets, with 20% of the data labeled. As observed, the segmentation results of this invention (red areas) are closer to the Ground Truth (white areas) representing expert annotations, and the segmentation results are better than other methods.

[0114] Specifically, the first column shows the original image; columns 2 through 6 show the results of comparative experiments; and the last column shows the experimental results of this invention. As can be observed from the figure, different algorithms exhibit significant differences in segmentation accuracy, while the segmentation result of this invention (rightmost column) shows the highest degree of agreement with the true boundary. In this case, the method of this invention uses information from multiple modalities to segment simultaneously registered input images, with multiple images sharing a single mask. Comparative experiments segmented images from multiple modalities (segmentation results for CT and PET are shown in the figure), and the segmentation results are only applicable to the corresponding modality.

[0115] Figure 7 Based on the Hecktor2022 and STS datasets, the feature maps generated by the method of this invention at different encoder stages are visualized.

[0116] Figure 8 This is a visualization of heatmaps on PET and CT scans based on the Hecktor2022 and STS datasets. The yellow portion represents the tumor label, and the rightmost column is a visualization of the heatmaps during the training process.

[0117] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion, characterized in that, The system includes: The data acquisition module is used to acquire dual-modal data of PET and CT images and divide them into labeled and unlabeled data. A cross-modal adaptive fusion network module for tumor segmentation is used to independently extract features from CT and PET images during the encoding stage and perform dynamic feature fusion through a cross-modal attention mechanism. Specifically, cross-modal attention calculation is performed based on the cross-resolution attention mechanism (CRA). The CRA output is upsampled and combined with the residual connection, and then input into the region-limited attention mechanism (RCA) for further processing, transforming the original input data into an output with enhanced multimodal features. During the decoding stage, a lesion-focusing adaptive weighted network is used to enhance tumor-related features from multimodal features to achieve accurate segmentation. Specifically, the features generated by the k-th encoder are concatenated with the upsampled features of the k+1-th decoder to obtain a comprehensive CT and PET feature representation. The prediction map obtained by upsampling from the k+1-th decoder is used to separate foreground and background features, and a refined feature representation is obtained based on the importance weights of foreground and background features. The multi-scale pyramid deep supervision module is used to add an auxiliary layer after each decoder in the decoding stage to form a hierarchical supervision component. It considers multi-scale prediction as a model optimization for deep supervised learning, upsamples the decoder's prediction map to be the same as the real label, and calculates the cross-entropy loss (Dice loss) between the prediction and the real label at different scales to obtain multi-scale deep supervision. The multi-scale uncertainty-aware pyramid module for semi-supervised segmentation is used for the synergy of pyramid-based supervision and uncertainty-guided learning. The student model obtains increasingly reliable supervision at multiple scales, while the teacher model can generate increasingly high-quality targets through an exponential moving average update mechanism. Both the student model and the teacher model adopt the network architecture of a cross-modal adaptive fusion network module for tumor segmentation. Labeled data is trained by the student model through pyramid-style hierarchical supervision, while unlabeled data is trained by the teacher model to generate pseudo-labels after uncertainty correction, forming a hierarchical consistency loss. Hierarchical feature representation is used for semi-supervised tumor segmentation.

2. The hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 1, characterized in that, The Cross-Resolution Attention (CRA) mechanism precisely modulates the information flow through learnable global and local gate control units in the bidirectional reciprocal gated attention (MGFM) mechanism, capturing long-distance dependencies between CT and PET images at downsampled resolution. Subsequently, the region-restricted attention (RCA) mechanism divides the features into local window regions at the original resolution, and captures fine spatial details within the window by reapplying MGFM.

3. The hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 2, characterized in that, The Cross-Resolution Attention (CRA) mechanism also includes a global context-aware gate mechanism, which is used to adjust the overall information exchange rate and achieve dynamic adjustment of multimodal information exchange. Specifically, it first performs channel-dimensional concatenation on the CT modality downsampling features and PET modality downsampling features to form joint features; then, it extracts the global context information of the joint features through global average pooling, and then processes it through linear transformation of the learnable weight matrix and activation function, so that the network can adaptively determine the retention ratio of global information of each modality according to different combinations of imaging features.

4. The hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 1, characterized in that, Region-restricted attention (RCA) specifically involves recombining the output features from CRA into a fixed-size 3D window. Within each window, CT and PET features utilize the same MGFM bidirectional reciprocal gating attention mechanism as RCA, performing granular detail relationships within a smaller spatial region to obtain the final output.

5. A hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 1, characterized in that, In the multi-scale uncertainty-aware pyramid module of semi-supervised segmentation, for unlabeled data, the multi-scale pyramid uncertainty-aware consistency loss is used to minimize the difference between predictions at different scales. Comprehensive pyramid supervision is implemented in the entire architecture of the student model. The consistency loss is calculated for each layer of feature map of the student network through the teacher network. This multi-level consistency calculation ensures that the network maintains consistency with the teacher model from shallow local texture features to deep global semantic features.

6. A hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 1, characterized in that, In the multi-scale uncertainty perception pyramid module of semi-supervised segmentation, uncertainty quantification is integrated with pyramid supervision, and consistency constraints are applied only to high-confidence regions. For each unlabeled sample, the teacher model not only generates target predictions, but also estimates the uncertainty of the predictions through Monte Carlo Dropout technology.

7. A hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 6, characterized in that, In the semi-supervised segmentation multi-scale uncertainty perception pyramid module, the uncertainty perception mechanism is integrated with a multi-layer supervision strategy to create a dynamic self-correcting knowledge transfer framework. At each network level, the uncertainty graph with high confidence of the teacher network is used to guide the consistency training of the corresponding level. This hierarchical uncertainty perception mechanism ensures that the knowledge transfer process can guide the reliability constraints of the features of different abstraction levels of the student network in real time.

8. A hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 7, characterized in that, In the multi-scale uncertainty perception pyramid module of semi-supervised segmentation, the teacher model is updated through the exponential moving average (EMA) mechanism as training progresses: the student network and the teacher network share the same network backbone, and the teacher network weights are updated through the exponential moving average (EMA) parameter of the student network weights.

9. A hierarchical semi-supervised tumor segmentation system based on PET / CT dynamic cross-modal fusion according to claim 1, characterized in that, The multi-scale uncertainty perception pyramid module for semi-supervised segmentation combines unsupervised regularization loss and deep supervised loss to jointly optimize the network; and uses a time-dependent Gaussian warm-up function to control the balance between supervised loss and unsupervised consistency loss.

Images