A multi-modal brain tumor image test-time adaptation segmentation method based on category decoupling and related device

Abstract

The application discloses a kind of based on category decoupling multi-modal brain tumor image test time adaptive segmentation method and related device, it is related to medical image processing technical field, the method includes pre-training stage and inputs multi-modal image into exclusive encoder, is decoupled into the binary subtask of each foreground category by inverse mixing expert module and adaptively selects mode, after fusion, input multi-class segmentation branch obtains segmentation result;Based on subtask and multi-class segmentation branch feature map calculation structure alignment loss and combine source domain label iterative optimization, obtain segmentation model and source domain distribution;Test time adaptive stage applies model to unlabeled target domain data, estimates target domain distribution and takes source domain as anchor point, based on optimal transport calculation difference parameter adaptive update is carried out, after obtaining stable parameter by reliability check, output segmentation result.The application effectively improves the segmentation precision and robustness of model in cross-domain scene.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to an adaptive segmentation method and related apparatus for testing multimodal brain tumor images based on category decoupling. Background Technology

[0002] Multimodal medical image segmentation (such as FLAIR, T1ce, T2, and T1 modalities in brain tumor MRI) is of significant value in clinical auxiliary diagnosis. Existing techniques typically improve segmentation accuracy by fusing complementary information from different modalities. However, existing methods suffer from the following technical problems: (1) Imbalance in modal contributions: Existing fusion methods (such as direct splicing or simple weighting) assume that each modality contributes equally, ignoring the significant differences in the ability of different modalities to distinguish different categories (such as peritumoral edema, necrotic core and non-enhanced tumor core, enhanced tumor core), resulting in suboptimal fusion, especially with severe performance degradation under domain offset.

[0003] (2) Poor cross-domain generalization ability: When learning multi-class segmentation tasks, the model tends to over-rely on the fixed spatial relationships between classes in the source domain (e.g., edema always surrounds the core of the tumor in the adult brain). When applied to the target domain, the anatomical structure and lesion morphology change, causing the model's discrimination ability to drop sharply.

[0004] (3) Instability of test-time adaptation: Existing test-time adaptation methods mostly rely on minimizing prediction entropy or pseudo-labels. When the domain offset is large, noise predictions will accumulate and lead to incorrect adaptation. In addition, the lack of a mechanism to handle class distribution offsets leads to the majority class dominating the adaptation process, while minority classes (such as small-sized tumor regions) are suppressed.

[0005] Therefore, this application proposes a multimodal medical image segmentation method based on optimal transmission category decoupling and distribution alignment, aiming to address the aforementioned challenges. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal brain tumor image testing adaptive segmentation method and related device based on category decoupling, which can effectively improve the segmentation accuracy and robustness of the model in cross-domain scenarios.

[0007] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a test-time adaptive segmentation method for multimodal brain tumor images based on category decoupling, including: During the pre-training phase, medical images and corresponding labels of multiple modalities in the source domain are acquired. The medical images of each modality are then input into the corresponding modality-specific encoder for feature extraction to obtain feature maps of each modality. The medical images include background and multiple foreground categories.

[0008] The feature maps of each modality are concatenated and then input into the inverse hybrid expert module to obtain the binary segmentation map of each foreground category, the feature map of the subtask branch, and the weight vector of each modality.

[0009] Based on the weight vectors of each modality, multimodal fusion features are calculated, and these features are then input into multi-class segmentation branches for feature extraction and decoding, resulting in feature maps and results for the multi-class segmentation branches.

[0010] Based on the feature maps of each subtask branch and the feature maps of the multi-class segmentation branches, the structural alignment loss is calculated.

[0011] Based on the medical images and corresponding labels of each modality in the source domain, the results of the multi-class segmentation branches, the binary segmentation maps of each foreground category, the structure alignment loss, and the entropy regularization of the multi-modal weights to construct the loss function, the parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches are iteratively optimized to obtain the segmentation model and the feature distribution of each foreground category in the source domain.

[0012] During the adaptation phase of testing, the segmentation model is applied to unlabeled target domain medical images to estimate the feature distribution of each foreground category in the target domain. Using the feature distribution of each foreground category in the source domain as anchor points, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. Based on the transmission cost, the segmentation model parameters are adaptively updated. The reliability of the updated segmentation model parameters is verified to obtain stable adaptive segmentation model parameters.

[0013] Based on the stable adaptive segmentation model parameters, the segmentation model is used to infer the target domain image to obtain multi-class segmentation results.

[0014] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the adaptive segmentation method for testing multimodal brain tumor images based on category decoupling described above.

[0015] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the adaptive segmentation method for testing multimodal brain tumor images based on category decoupling described above.

[0016] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the adaptive segmentation method for testing multimodal brain tumor images based on category decoupling described above.

[0017] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a multimodal brain tumor image testing-time adaptive segmentation method and related apparatus based on category decoupling. The method includes a pre-training phase where multiple modalities of medical images and their corresponding labels are acquired from the source domain. Each modality's medical image is then input into its corresponding modality-specific encoder for feature extraction, resulting in feature maps for each modality. The medical images contain background and multiple foreground categories, providing a multimodal complementary information basis for subsequent category decoupling and adaptive modality selection, avoiding feature interference caused by differences in imaging principles between different modalities. The feature maps of each modality are concatenated and input into an inverse hybrid expert module to obtain binary segmentation maps for each foreground category, feature maps for sub-task branches, and weight vectors for each modality. Subsequently, based on the weight vectors of each modality, multimodal fusion features are calculated, and these features are input into multi-class segmentation branches for feature extraction and decoding, resulting in feature maps and results for the multi-class segmentation branches. This significantly improves cross-domain generalization ability and achieves differentiated and reasonable fusion of multimodal features. Based on the feature maps of each subtask branch and the feature maps of the multi-class segmentation branches, the structural alignment loss is calculated. By maintaining the consistency of the feature structure of the same category among different tasks, the decoupled binary task features are successfully structurally aligned with the multi-class task features. A loss function is constructed based on the medical images of each modality in the source domain and their corresponding labels, the results of the multi-class segmentation branches, the binary segmentation maps of each foreground category, the structural alignment loss, and entropy regularization of the multi-modal weights. The parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches are iteratively optimized to obtain the segmentation model and the feature distributions of each foreground category in the source domain. During the adaptation phase in testing, the segmentation model is applied to the unlabeled target domain medical image to estimate the feature distributions of each foreground category in the target domain. Using the feature distributions of each foreground category in the source domain as anchors, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. The segmentation model parameters are adaptively updated based on the transmission cost, and the reliability of the updated segmentation model parameters is verified to obtain stable adaptive model parameters. Finally, based on the stable adaptive model parameters, the segmentation model is used to infer the target domain image to obtain multi-class segmentation results. This application employs multi-task joint optimization to force the model to learn more generalized category-specific representations, while preserving the distribution of each category in the source domain as a fixed anchor point for adaptation during testing. It directly aligns the category-level feature distributions of the source and target domains without relying on prediction entropy or pseudo-labels, and can still work stably under large domain shifts. At the same time, it effectively addresses label distribution shifts through reliability verification, avoiding the suppression of minority classes by the majority class, thereby achieving model stability and improved accuracy. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This application provides an illustration of the application environment for an adaptive segmentation method for multimodal brain tumor image testing based on category decoupling, as provided in one embodiment of this application. Figure 2 This application provides an overall technical framework diagram of a multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling, as an embodiment of the present application. Figure 3 A flowchart illustrating an adaptive segmentation method for multimodal brain tumor images based on category decoupling, provided as an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] The test-time adaptive segmentation method for multimodal brain tumor images based on category decoupling provided in this application can be applied to, for example... Figure 1The application environment shown is illustrated. Terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send multiple modalities of medical images from the source domain to be processed to server 104. After receiving the multiple modalities of medical images from the source domain, server 104 inputs each modality of the medical image into its corresponding modality-specific encoder for feature extraction, obtaining feature maps for each modality. The medical images contain background and multiple foreground categories. The feature maps of each modality are concatenated and input into the inverse hybridization expert module to obtain binary segmentation maps for each foreground category, feature maps of sub-task branches, and weight vectors for each modality. Based on the weight vectors of each modality, multimodal fusion features are calculated, and these features are input into multi-class segmentation branches for feature extraction and decoding, obtaining feature maps and results for the multi-class segmentation branches. Based on the feature maps of each sub-task branch and the feature maps of the multi-class segmentation branches, structural alignment loss is calculated. The system uses medical images of each modality in the source domain and their corresponding labels, the results of multi-class segmentation branches, binary segmentation maps of each foreground category, structural alignment loss, and entropy regularization of the multi-modal weights to construct a loss function. It iterative optimization is performed on the parameters of each modality-specific encoder, inverse hybrid expert module, and multi-class segmentation branches to obtain the segmentation model and the feature distributions of each foreground category in the source domain. During the adaptation phase in the testing phase, the segmentation model is applied to the unlabeled target domain medical image to estimate the feature distributions of each foreground category in the target domain. Using the feature distributions of each foreground category in the source domain as anchors, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. Based on the transmission cost, the segmentation model parameters are adaptively updated, and the reliability of the updated segmentation model parameters is verified to obtain stable adaptive segmentation model parameters. Based on the stable adaptive segmentation model parameters, the segmentation model is used to infer the target domain image to obtain multi-class segmentation results. Server 104 can feed back the obtained multi-class segmentation results to terminal 102. Furthermore, in some embodiments, the adaptive segmentation method for testing multimodal brain tumor images based on category decoupling can also be implemented separately by the server 104 or the terminal 102. For example, the terminal 102 can directly process the medical images of multiple modalities in the source domain to be processed, or the server 104 can obtain the medical images of multiple modalities in the source domain to be processed from the data storage system and process them.

[0023] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0024] In one exemplary embodiment, such as Figure 2 As shown, the multimodal brain tumor image test-time adaptive segmentation method based on category decoupling provided in this application includes two main modules: a pre-training phase and a test-time adaptation phase. The pre-training phase ( Figure 2 Left side): After feature extraction, the source domain multimodal medical image is input into the iMoE-based class decoupling module to obtain multimodal weights; simultaneously, the GW-OT structure is used to align and constrain the feature structure consistency between the binary subtask branch and the multi-class segmentation main branch; finally, the multi-class segmentation result is output through the multi-class segmentation main branch. Adaptation phase during testing ( Figure 2 (Right side): The pre-trained model weights are loaded into the target domain, and class-level distribution alignment is performed based on OT. Combined with class adaptive weights and parameter rollback mechanism, the target domain segmentation map is output after reliability verification.

[0025] In one exemplary embodiment, such as Figure 3 As shown, a multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps A1 to A7. Wherein: Step A1: In the pre-training stage, acquire medical images of multiple modalities in the source domain, and input the medical images of each modality into the corresponding modality-specific encoder for feature extraction to obtain feature maps of each modality; the medical images include background and multiple foreground categories.

[0026] Step A2: After concatenating the feature maps of each modality, input them into the inverse hybrid expert module to obtain the binary segmentation map of each foreground category, the feature map of the subtask branch, and the weight vector of each modality.

[0027] Step A3: Based on the weight vectors of each modality, calculate the multimodal fusion features, and input the multimodal fusion features into the multi-class segmentation branch for feature extraction and decoding to obtain the feature map and result of the multi-class segmentation branch.

[0028] Step A4: Calculate the structure alignment loss based on the feature maps of each subtask branch and the feature maps of the multi-class segmentation branches.

[0029] Step A5: Based on the medical images of each modality in the source domain, the results of the multi-class segmentation branches, the binary segmentation maps of each foreground category, the structure alignment loss, and the entropy regularization of the multi-modal weights to construct a loss function, iterative optimization is performed on the parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches to obtain the segmentation model and the feature distribution of each foreground category in the source domain.

[0030] Step A6: During the adaptation phase of testing, the segmentation model is applied to the unlabeled target domain medical image to estimate the feature distribution of each foreground category in the target domain. Using the feature distribution of each foreground category in the source domain as the anchor point, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. Based on the transmission cost, the segmentation model parameters are adaptively updated. The reliability of the updated segmentation model parameters is verified to obtain stable adaptive segmentation model parameters.

[0031] Step A7: Based on the stable adaptive segmentation model parameters, the segmentation model is used to infer the target domain image to obtain multi-class segmentation results.

[0032] In this embodiment, the publicly available BraTS2020 adult brain tumor dataset is used as the source domain training data, and the BraTS2023-PED pediatric brain tumor and BraTS2024-MET adult metastatic tumor datasets are used as the target domain test data. All data include four modalities: FLAIR, T1ce, T1, and T2. During image preprocessing, all modalities are resampled to a uniform spatial resolution. Each modality-specific encoder uses a 3D U-Net encoder; each expert in the inverse hybrid expert module consists of a router (MLP), a decoder, and a segmentation head. In the feature decoupling stage of pre-training, the iMoE module includes three sub-tasks, corresponding to three foreground categories: enhanced tumor core, necrotic and non-enhanced tumor core, and peritumoral edema. The router for each sub-task generates a 4-dimensional weight vector and performs Top-2 selection. In the test-time adaptation stage, high-confidence pixels with a predicted probability greater than 0.9 are selected to estimate the feature distribution of each category in the target domain, and only the first three layers of the four encoders are updated. In reliability verification, the first and second thresholds are set according to the specific task and are used to determine whether the update is excessive or dominated by a major category.

[0033] Steps A1 to A7 above are implemented to unify the class decoupling pre-training and test-time distribution alignment of the multimodal segmentation model through optimal transport theory. In the pre-training phase, the multi-class segmentation task is decomposed into multiple independent binary sub-tasks using an inverse hybrid expert (iMoE) structure to learn class-specific representations and adaptively evaluate the importance of each modality. Simultaneously, the Gromov-Wasserstein distance is innovatively introduced to align the feature structures of the same class in the main multi-class branch and the binary sub-task branches, solving the alignment problem in heterogeneous feature spaces. In the test-time adaptation phase, instead of relying on unreliable prediction entropy or pseudo-labels, the Wasserstein distance between the source and target domains on the class-level feature distribution is directly calculated for distribution alignment. This is supplemented by class-adaptive reweighting and parameter rollback mechanisms to achieve stable and balanced domain adaptation.

[0034] In another exemplary embodiment of this application, in order to achieve adaptive modality selection for each class in multimodal brain tumor image segmentation and avoid the model's over-reliance on fixed spatial relationships between classes in the source domain, the complex multi-class segmentation task can be decoupled into multiple parallel binary subtasks corresponding to each foreground class. An inverse hybrid expert module adaptively evaluates the discriminative ability of each modality for different foreground classes, generating class-specific modality fusion weights, thereby forcing the model to learn more generalized class-specific representations. In this case, step A2 is replaced by steps B1 to B7: Step B1: The modal feature maps are spliced ​​together to obtain multimodal spliced ​​features.

[0035] Step B2: Input the multimodal splicing features into the inverse hybrid expert module to decouple the multi-classification task into multiple parallel binary segmentation subtasks corresponding to each foreground category.

[0036] Step B3: For each binary segmentation subtask, generate the weight vector for each modality through the router.

[0037] Step B4: Select the K largest weights from the weight vectors of each modality using Top-K selection, and perform weighted fusion of the modal features based on the retained weights to obtain the subtask fusion features.

[0038] Step B5: The subtask fusion features are extracted and decoded by the decoder to obtain the subtask branch feature map.

[0039] Step B6: Input the feature map of the subtask branch into the segmentation head for segmentation to obtain a binary segmentation map of the corresponding foreground category.

[0040] Step B7: Collect the modal weight vectors of each foreground category output by each binary segmentation subtask branch router.

[0041] In another exemplary embodiment of this application, in order to aggregate the modal importance specific to each foreground category to guide the multi-class segmentation main branch to perform multi-modal feature fusion, thereby obtaining a unified multi-class segmentation result, the above step A3 is replaced by the following steps C1 to C4: Step C1: Average the weight vectors of each modality across the foreground category dimension to obtain the global modality weights.

[0042] Step C2: Based on the global modal weight vector, the feature maps of each modality are weighted and fused to obtain multimodal fusion features.

[0043] Step C3: Input the multimodal fusion features into the multi-class segmentation branches for feature extraction to obtain feature maps of the multi-class segmentation branches.

[0044] Step C4: Decode and segment the feature maps of the multi-class segmentation branches to obtain the multi-class segmentation branch results.

[0045] In this embodiment, the multimodal fusion feature is calculated by the following formula: ; in, This is a multimodal fusion feature; Represents the multimodal features after fusion and the first modal features before fusion. Features of each modality This indicates the output of the iMoE. The weights of each modality; Indicates the number of modes.

[0046] In another exemplary embodiment of this application, in order to solve the problem of feature space heterogeneity caused by the different task complexities of the binary subtask branch and the multi-class segmentation main branch, and to effectively transfer the decoupled class-specific representation to the multi-class segmentation main branch by maintaining the consistency of the feature structure of the same category between different tasks, the above step A4 is replaced by the following steps D1 to D5: Step D1: Extract the first feature point set of each foreground category from the last layer feature map of each binary subtask branch.

[0047] Step D2: Extract the second feature point set for each foreground category from the last layer feature map of the multi-class segmentation branches.

[0048] Step D3: Extract key points from each of the first feature point sets and each of the second feature point sets to obtain the first key point set and the second key point set for each foreground category.

[0049] Step D4: Calculate the internal distance matrix of each feature point pair in the first keypoint set and the internal distance matrix of each feature point pair in the second keypoint set.

[0050] Step D5: Based on the Gromov-Wasserstein optimal transmission, calculate the difference between the two internal distance matrices to obtain the structural alignment loss.

[0051] In this embodiment, the two segmentation tasks (multi-class segmentation and binary segmentation) are independent. The class decoupling information from binary segmentation is not applied to multi-class segmentation. Therefore, Gromov-Wasserstein optimal transport (GW-OT) is introduced to achieve class feature structure alignment between tasks. The structure alignment loss is calculated by the following formula: ; in, This is for structural alignment loss; The total number of foreground categories; For the first An index of foreground categories; To reduce computational load, from category The set of key points extracted from feature points; This refers to the feature point index within the keypoint set in the binary subtask branch; For feature point indices within the keypoint set in multi-class segmentation branches; In the binary subtask branch, the category Feature points With point The Euclidean distance between them; In multi-class splitting branches, the category Feature points With point The Euclidean distance between them; The elements in the optimal transfer matrix represent the feature points in the binary subtask branch. Feature points in multi-class segmentation branches The probability of a match; The elements in the optimal transfer matrix represent the feature points in the binary subtask branch. Feature points in multi-class segmentation branches The probability of a match.

[0052] Based on the aforementioned structural alignment loss, the decoupled intra-class structure is passed to multi-class segmentation to achieve structural alignment, rather than forced point-to-point alignment.

[0053] In another exemplary embodiment of this application, in order to force the model to learn a more generalized category-specific representation and obtain the segmentation model and the feature distribution of each category in the source domain through joint optimization of multi-class segmentation loss, binary segmentation loss, structural alignment loss and entropy regularization of multimodal weights, step A5 above is replaced by the following steps E1 to E5: Step E1: Calculate the multi-class segmentation loss based on the results of the multi-class segmentation branches and the labels, calculate the binary segmentation loss based on the results of each binary segmentation branch and the labels, and calculate the entropy regularization loss based on the weight vector of each modality.

[0054] Step E2: Construct a total loss function based on the multi-class segmentation loss, each of the binary segmentation losses, the structure alignment loss, and the entropy regularization loss.

[0055] Step E3: Based on the total loss function, iteratively optimize the parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches to obtain the segmentation model.

[0056] Step E4: Use the segmentation model to perform forward inference on the source domain data to obtain a feature map.

[0057] Step E5: Based on the annotation labels of the source domain data, the features of pixels belonging to each foreground category in the feature map are statistically analyzed to obtain the feature distribution of each foreground category in the source domain.

[0058] In another exemplary embodiment of this application, in order to achieve category-level feature distribution alignment on unlabeled target domain data, reduce the inter-domain differences between the source and target domains through optimal transmission, and adaptively update model parameters to adapt to the target domain data distribution, the following steps F1 to F4 are replaced: Step F1: Apply the segmentation model to the target domain medical image and perform forward inference to obtain the target domain prediction probability.

[0059] Step F2: Filter pixels whose predicted probability is greater than a preset threshold, and estimate the feature distribution of each foreground category in the target domain based on the filtered pixels.

[0060] Step F3: Using the feature distribution of each foreground category in the source domain as anchor points, calculate the transmission cost between the source domain distribution and the target domain distribution through optimal transmission.

[0061] Step F4: The transmission costs of each foreground category are weighted and summed as the adaptation loss. Based on minimizing the adaptation loss, the first three layers of parameters of each modality-specific encoder are updated to obtain candidate parameters.

[0062] In this embodiment, based on the pre-training steps A1 to A5 described above, a segmentation model with category decoupling and structure alignment capabilities, along with prior feature distributions for each category in the source domain, can be obtained. However, due to the data distribution differences between the source and target domains, directly applying the pre-trained model to unlabeled target domain data will significantly reduce segmentation performance. To further improve the model's performance in the target domain, this embodiment proposes a test-time distribution alignment method based on standard optimal transfer (OT). For each foreground category, the source domain Gaussian class distribution obtained during the pre-training phase is used as an anchor point, and its difference from the target domain Gaussian class distribution is compared. Optimal transfer is then used to bring the target domain class distribution closer to the source domain class distribution. Gradient backpropagation updates the shallow parameters of the modality-specific encoder, allowing the model to adapt to the target domain's data distribution. Furthermore, due to label distribution shifts between different domains, the adaptation process may be dominated by larger categories, thereby suppressing performance improvements for smaller categories (such as enhanced tumors). Therefore, this embodiment uses two thresholds to measure the reliability of model updates: the first threshold is designed based on the entropy change before and after the update. If the entropy change of a certain class is higher than the first threshold, it is considered an over-update, and the loss weight of that class is reduced exponentially. The second threshold is defined based on the change in class proportion before and after the update. If the proportion change of any class is higher than the second threshold, it is considered that the update is overly biased towards a certain class (usually a large class), which may be dominated by the large class and have a negative impact on the performance improvement of other classes. Therefore, it is judged as an unreliable update, and the model parameters are rolled back to the state before the update. The above-mentioned adaptation process makes the feature distribution of each foreground class in the target domain as close as possible to the feature distribution of the corresponding class in the source domain. Its objective function is the adaptation loss, which is expressed by the following formula: ; in, Indicate category loss weights, Indicates the 2-Wasserstein distance. and These represent the categories in the multi-class segmentation branches in the source and target domains, respectively. Characteristic distribution, and These represent the categories in the binary subtask branches of the source and target domains, respectively. Characteristic distribution, subscript Represents the source domain. Indicates the target domain. Indicates multiple types of splitting branches, This represents a binary partition branch.

[0063] The transmission cost is 2-Wasserstein distance, calculated using the following formula: ; in, and Representing the categories in the source and target domains respectively. The characteristic distribution; and Representing the categories in the source and target domains respectively. The eigenmean vector, and Representing the categories in the source and target domains respectively. The characteristic covariance matrix, Represents the trace of a matrix.

[0064] In another exemplary embodiment of this application, in order to constrain the reliability of model parameter updates during the adaptation phase at test time, prevent the accumulation of noise predictions due to large domain offsets, and avoid the majority class dominating adaptation and suppressing minority class performance due to class distribution offsets, a stable and balanced domain adaptation is achieved through adaptive reweighting and parameter rollback mechanisms. Therefore, the reliability verification of the updated segmentation model parameters in step A6 above to obtain stable adaptive segmentation model parameters is replaced by the following steps G1 to G8: Step G1: Monitor the entropy changes and category proportion changes of each foreground category prediction before and after re-inference.

[0065] Step G2: Determine whether the entropy change of any foreground category in each foreground category is greater than a first threshold, and obtain the first determination result.

[0066] Step G3: If the first judgment result is yes, then reduce the Wasserstein distance loss weight of the foreground category according to the exponential formula.

[0067] Step G4, otherwise, the Wasserstein distance loss weight for the foreground category is 1; Step G5: Determine whether the change in the proportion of any foreground category among the foreground categories is greater than the second threshold, and obtain the second determination result.

[0068] Step G6: If the second judgment result is yes, then roll back to the parameters before the update.

[0069] Step G7, otherwise, accept this update and use the candidate parameters as the current encoder parameters, and jump to step F1.

[0070] Step G8 continues until the preset stopping condition is met, and the last accepted parameter is used as the stable adaptive model parameter.

[0071] In this embodiment, the exponential formula is expressed by the following equation: ; in, Indicate category loss weights, Indicate category The change in entropy before and after the parameter update. Indicate category The first threshold, This represents a minimal constant to prevent the denominator from being zero.

[0072] The rollback can be represented as: ; in, Indicates the first The candidate parameters are updated in the next iteration. This represents the last accepted stable parameter. Indicate category The change in the category ratio before and after the parameter update. Indicate category The second threshold.

[0073] In another exemplary embodiment of this application, in order to verify the effectiveness of the method proposed in this application, this embodiment conducted experimental verification on a publicly available multimodal MRI dataset of brain tumors.

[0074] 1. Experimental setup This application uses the BraTS2020 adult brain tumor dataset as the source domain training data, and the BraTS2023-PED pediatric brain tumor dataset and the BraTS2024-MET adult metastatic tumor dataset as the target domain test data. All datasets include four modalities: FLAIR, T1ce, T1, and T2. Image preprocessing resamples all modalities to a uniform spatial resolution. A 3D U-Net encoder is used for each modality's dedicated encoder; the inverse hybridization expert module consists of a multilayer perceptron; the inverse hybridization expert module contains three sub-tasks corresponding to three foreground categories: tumor core, edema, and enhanced tumor; K=2 is used in Top-K selection. The Dice similarity coefficient is used as the evaluation metric.

[0075] 2. Cross-domain generalization performance verification To verify the effectiveness of the category decoupling pre-training mechanism proposed in this application, the method provided was tested in a large-domain offset scenario, ranging from adult brain tumor data (BraTS2020) to pediatric brain tumor data (BraTS2023-PED), and compared with the existing state-of-the-art method, M2FTrans. Experimental results show that the average Dice coefficient of this method is improved by 2.13% compared to M2FTrans, especially in the tumor core region, where the improvement reaches 5.05%. These results demonstrate that by decoupling the multi-class segmentation task into multiple independent binary sub-tasks, the model no longer learns fixed spatial relationships between classes in the source domain (e.g., edema always surrounds the tumor core), but instead focuses on learning the intrinsic discriminative features of each class. This forces the model to learn more generalized class-specific representations, significantly improving cross-domain segmentation accuracy.

[0076] 3. Verification of the rationality of multimodal fusion To verify the effectiveness of the modality selection mechanism of the inverse hybrid expert module, this application analyzes the modality weight distribution output by the routers of each binary subtask branch. Experimental results show that the model automatically assigns higher T1ce modality weights to the "enhanced tumor" category. This result is highly consistent with prior clinical medical knowledge—enhanced tumors typically exhibit the most significant high signal in the T1ce modality—and outperforms traditional unified weighting or simple concatenation fusion methods. These results demonstrate that this application can adaptively evaluate the discriminative contribution of different modalities for each foreground category and generate category-specific optimal fusion weights, significantly improving the discriminative power of the fusion features.

[0077] 4. Verification of structural alignment effectiveness To verify the effectiveness of the Gromov-Wasserstein structural alignment constraint, this application conducted an ablation study. Experimental results show that after adding the Gromov-Wasserstein structural alignment constraint, the Dice coefficient of the most difficult-to-segment enhanced tumor region was increased by 3.0% alone. These results demonstrate that this application aligns the feature structures of the same class in the "multi-class segmentation main branch" and the "binary subtask branch" through Gromov-Wasserstein optimal transfer. It does not require strict point-to-point matching of features between the two branches, but rather aligns their internal pairwise distance relationships, thus cleverly solving the feature space heterogeneity problem caused by tasks of different complexities.

[0078] 5. Adaptability and stability verification during testing. To verify the effectiveness of the test-time adaptation phase of this application, our method was compared with the TENT method based on entropy minimization on the PED dataset. Experimental results show that the TENT method experiences a 1.26% decrease in average Dice under large domain shifts, resulting in performance degradation. In contrast, our method, through class distribution alignment based on Wasserstein distance, achieves a stable gain of 1.7% in average Dice without relying on unreliable prediction confidence. Furthermore, the class adaptive reweighting and parameter rollback mechanisms effectively address the label distribution shift problem, preventing the majority class (e.g., edema) from dominating the optimization direction during adaptation and suppressing the performance improvement of the minority class (e.g., small-sized enhancing tumors). These results demonstrate that our application's class-level distribution alignment strategy based on optimal transmission fundamentally reduces inter-domain differences, maintains stable operation under large domain shifts, and exhibits superior robustness and balance compared to existing test-time adaptation methods.

[0079] This application also provides an application scenario in which the above-mentioned category-decoupled multimodal brain tumor image testing-time adaptive segmentation method is applied. Specifically, the category-decoupled multimodal brain tumor image testing-time adaptive segmentation method provided in this embodiment can be applied in medical image-assisted diagnosis scenarios. Medical image-assisted diagnosis scenarios include image acquisition, image processing, and clinical diagnosis; multimodal brain tumor images enter the image processing chain from the image acquisition stage, and through human-machine collaboration, obtain corresponding lesion segmentation results, which then enter the downstream clinical diagnosis and treatment stage. The category-decoupled multimodal brain tumor image testing-time adaptive segmentation method provided in this embodiment belongs to the intelligent segmentation stage in the image processing chain. Specifically, in the processing of multimodal MRI images of brain tumors, brain tumor lesions can be segmented based on a collaborative approach of AI automatic segmentation and doctor manual review, that is, adding corresponding lesion region segmentation annotations to the multimodal MRI images.

[0080] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores multi-class segmentation results. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program can implement an adaptive segmentation method for multimodal brain tumor image testing based on class decoupling.

[0081] Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0082] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0083] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0084] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0085] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0086] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0087] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A test-time adaptive segmentation method for multimodal brain tumor images based on category decoupling, characterized in that, include: During the pre-training phase, medical images and corresponding labels of multiple modalities in the source domain are acquired. The medical images of each modality are then input into the corresponding modality-specific encoder for feature extraction to obtain feature maps of each modality. The medical images include background and multiple foreground categories. The feature maps of each modality are concatenated and then input into the inverse hybrid expert module to obtain the binary segmentation map of each foreground category, the feature map of the subtask branch, and the weight vector of each modality. Based on the weight vectors of each modality, multimodal fusion features are calculated, and these features are then input into multi-class segmentation branches for feature extraction and decoding, resulting in feature maps and results for the multi-class segmentation branches. Based on the feature maps of each subtask branch and the feature maps of the multi-class segmentation branches, the structural alignment loss is calculated. Based on the medical images and corresponding labels of each modality in the source domain, the results of the multi-class segmentation branches, the binary segmentation maps of each foreground category, the structure alignment loss, and the entropy regularization of the multi-modal weights to construct the loss function, the parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches are iteratively optimized to obtain the segmentation model and the feature distribution of each foreground category in the source domain. During the adaptation phase of testing, the segmentation model is applied to unlabeled target domain medical images to estimate the feature distribution of each foreground category in the target domain. Using the feature distribution of each foreground category in the source domain as anchor points, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. Based on the transmission cost, the segmentation model parameters are adaptively updated. The reliability of the updated segmentation model parameters is verified to obtain stable adaptive segmentation model parameters. Based on the stable adaptive segmentation model parameters, the segmentation model is used to infer the target domain image to obtain multi-class segmentation results.

2. The multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling according to claim 1, characterized in that, The concatenated feature maps of each modality are input into the inverse hybrid expert module to obtain binary segmentation maps of each foreground category, feature maps of subtask branches, and weight vectors of each modality, specifically including: The modal feature maps are then stitched together to obtain multimodal stitched features; The multimodal splicing features are input into the inverse hybrid expert module to decouple the multi-classification task into multiple parallel binary segmentation subtasks corresponding to each foreground category. For each binary segmentation subtask, a weight vector for each modality is generated through a router; The weight vectors of each modality are selected using Top-K and the K largest weights are retained. Based on the retained weights, the features of each modality are weighted and fused to obtain the subtask fusion features. The subtask fusion features are extracted and decoded by a decoder to obtain the subtask branch feature map; The subtask branch feature map is input into the segmentation head for segmentation to obtain a binary segmentation map corresponding to the foreground category; Collect the modal weight vectors of each foreground category output by each of the binary segmentation subtask routers.

3. The multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling according to claim 1, characterized in that, Based on the weight vectors of each modality, multimodal fusion features are calculated, and these features are then input into a multi-class segmentation branch for feature extraction and decoding. This yields feature maps and results for the multi-class segmentation branch, specifically including: The weight vectors of each modality are averaged over the foreground category dimension to obtain the global modality weights; Based on the global modality weight vector, the feature maps of each modality are weighted and fused to obtain multimodal fusion features; The multimodal fusion features are input into the multi-class segmentation branches for feature extraction to obtain feature maps of the multi-class segmentation branches; The feature maps of the multi-class segmentation branches are decoded and segmented to obtain the multi-class segmentation branch results.

4. The multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling according to claim 1, characterized in that, Based on the feature maps of each subtask branch and the feature maps of the multi-class segmentation branches, the structure alignment loss is calculated, specifically including: Extract the first feature point set of each foreground category from the last layer feature map of each binary subtask branch; Extract the second feature point set for each foreground category from the last layer feature map of the multi-class segmentation branches; Key points are extracted from each of the first feature point sets and each of the second feature point sets to obtain the first key point set and the second key point set for each foreground category. Calculate the internal distance matrix of each feature point pair in the first keypoint set, and the internal distance matrix of each feature point pair in the second keypoint set; Based on the Gromov-Wasserstein optimal transmission, the difference between the two internal distance matrices is calculated to obtain the structural alignment loss.

5. The multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling according to claim 1, characterized in that, Based on the medical images and corresponding labels of each modality in the source domain, the results of the multi-class segmentation branches, the binary segmentation maps of each foreground category, the structure alignment loss, and the entropy regularization of the multi-modal weights to construct the loss function, the parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches are iteratively optimized to obtain the segmentation model and the feature distribution of each foreground category in the source domain, specifically including: The multi-class segmentation loss is calculated based on the results of the multi-class segmentation branches and the labels, the binary segmentation loss is calculated based on the results of each binary segmentation branch and the labels, and the entropy regularization loss is calculated based on the weight vector of each modality. A total loss function is constructed based on the multi-class segmentation loss, each of the binary segmentation losses, the structure alignment loss, and the entropy regularization loss. The parameters of each modality-specific encoder, the inverse hybrid expert module, and the multi-class segmentation branches are iteratively optimized based on the total loss function to obtain the segmentation model; The segmentation model is used to perform forward inference on the source domain data to obtain a feature map; Based on the annotation labels of the source domain data, the features of pixels belonging to each foreground category in the feature map are statistically analyzed to obtain the feature distribution of each foreground category in the source domain.

6. The multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling according to claim 1, characterized in that, The segmentation model is applied to unlabeled target domain medical images to estimate the feature distribution of each foreground category in the target domain. Using the feature distribution of each foreground category in the source domain as anchors, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. The segmentation model parameters are then adaptively updated based on the transmission cost. Specifically, this includes: The segmentation model is applied to the target domain medical image and forward inference is performed to obtain the target domain prediction probability. Pixels whose predicted probability is greater than a preset threshold are filtered out, and the feature distribution of each foreground category in the target domain is estimated based on the filtered pixels. Using the feature distribution of each foreground category in the source domain as anchor points, the transmission cost between the source domain distribution and the target domain distribution is calculated through optimal transmission. The transmission costs of each foreground category are weighted and summed as the adaptation loss. Based on minimizing the adaptation loss, the first three layers of parameters of each modality-specific encoder are updated to obtain candidate parameters.

7. The multimodal brain tumor image testing-time adaptive segmentation method based on category decoupling according to claim 6, characterized in that, The updated segmentation model parameters are then validated for reliability to obtain stable adaptive segmentation model parameters, specifically including: Monitor the changes in entropy and category proportions of each foreground category prediction before and after re-inference; Determine whether the entropy change of any foreground category among the foreground categories is greater than a first threshold, and obtain a first determination result; If the first judgment result is yes, then the Wasserstein distance loss weight of the foreground category is reduced according to the exponential formula; Otherwise, the Wasserstein distance loss weight for the foreground category is 1; Determine whether the change in the proportion of any foreground category among the foreground categories is greater than a second threshold, and obtain a second determination result; If the second judgment result is yes, then roll back to the parameters before the update; Otherwise, accept this update and use the candidate parameters as the current encoder parameters, and jump to the step "apply the segmentation model to the target domain medical image and perform forward inference to obtain the target domain prediction probability"; The process continues until a preset stopping condition is met, at which point the last accepted parameter is used as the stable adaptive model parameter.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the steps of the adaptive segmentation method for multimodal brain tumor image testing based on category decoupling as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the adaptive segmentation method for testing multimodal brain tumor images based on category decoupling, as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the adaptive segmentation method for testing multimodal brain tumor images based on category decoupling, as described in any one of claims 1-7.

Images