A multi-modal brain tumor segmentation method based on spectrum semantic collaborative guidance

Abstract

The application claims a multi-modal brain tumor segmentation method based on spectrum semantic collaborative guidance, belonging to the technical field of computer vision and medical image processing. The method comprises the following steps: step 1. The application proposes a priori guided spectrum difference modal calibration module, which divides four sequences into structural flow and fluid flow according to the MRI imaging principle, and eliminates cross-modal feature interference from the source through a differential operator and a spectrum gate. Step 2. The application designs a global context interaction module to realize linear complexity long-range dependence modeling between double flows through block cross-attention. Step 3. The application designs a semantic guided high frequency gating module, which combines physical high frequency residual enhancement and deep supervision semantic bias to sharpen the tumor boundary while suppressing background noise, significantly improving segmentation accuracy and generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and medical image processing technology. Specifically, it is a multimodal brain tumor segmentation method based on spectral semantic collaborative guidance. Background Technology

[0002] Multimodal brain tumor segmentation is a critical task in clinical diagnosis and treatment planning. However, existing multimodal fusion methods generally employ an early channel splicing strategy, which involves directly merging the four modalities at the input end and feeding them into a single coding network. This approach ignores the inherent biophysical imaging differences between different modalities, leading to mutual interference between modal features and obscuring subtle pathological features at tumor boundaries.

[0003] Currently, existing methods mainly include CNN-based methods that extract volumetric features through 3D convolution, but are limited by local receptive fields and struggle to model long-range dependencies; Transformer-based methods capture global context through self-attention mechanisms, but suffer from high computational complexity and still rely on early splicing fusion methods; and state-space model-based methods achieve efficient modeling with linear complexity, but lack the ability to explicitly enhance high-frequency boundary information across modalities. Unfortunately, these methods do not adequately consider the spectral differences between modalities when processing multimodal data, leading to a bias in the low-pass filtering of spatial domain convolutions that suppresses high-frequency boundary details. Furthermore, standard skip connections inevitably introduce background noise from the encoder into the decoder during the transmission of spatial information, exacerbating boundary blurring and false positive segmentation problems. Therefore, effectively decoupling multimodal features and enhancing high-frequency boundary information while maintaining segmentation accuracy, computational efficiency, and generalization ability is a crucial issue in current multimodal brain tumor segmentation.

[0004] A search revealed that application publication number CN120070480B proposes a multimodal segmentation method and system for brain tumor imaging. This method identifies key regions of each modality through an attention layer, extracts features using parallel dynamic convolutional layers, fuses them, and then performs dual-channel decoupling and multimodal fusion. Finally, it obtains the brain tumor location information through attention weighting. However, this method has the following shortcomings: First, the modal fusion method treats all MRI sequences equally, failing to consider the inherent biophysical imaging differences between T1, T1ce, T2, and FLAIR sequences, leading to mutual interference between modal features. Second, feature extraction and fusion are both performed in the spatial domain, which, due to the low-pass filtering bias of spatial domain convolution, cannot explicitly enhance high-frequency boundary details, resulting in blurred tumor edges. Furthermore, the dual-channel decoupling only separates the channel and spatial attention of the fused combined features, failing to achieve cross-modal long-range dependency modeling between different modalities and neglecting the suppression of background noise in skip connections.

[0005] To address the aforementioned shortcomings, this invention proposes a multimodal brain tumor segmentation method based on spectral difference and semantic guidance. The method categorizes four modalities into structural flow and fluid flow using a priori-guided modal decoupling strategy, eliminating cross-modal feature interference at its source. Category-specific interaction operators are designed for different groups to accurately separate and enhance tumor and edema regions. A spectral difference module is designed to transform intra-group modal differences to the frequency domain and adaptively enhance high-frequency boundary information using learnable spectral gating, overcoming spatial domain low-pass filtering bias. A global context interaction module is proposed, using a block-based cross-attention mechanism to model long-range cross-modal dependencies between the two flows with linear complexity. A semantically guided high-frequency gating module, combined with physical high-frequency residual enhancement and deep-supervised semantic bias, effectively suppresses background noise introduced by skip connections while sharpening tumor boundaries. Through these systematic improvements, this invention achieves significant enhancements in boundary accuracy and generalization ability for multimodal brain tumor segmentation. Summary of the Invention

[0006] This invention aims to solve the problems of the prior art. It proposes a multimodal brain tumor segmentation method based on spectral semantic collaborative guidance. The technical solution of this invention is as follows:

[0007] A multimodal brain tumor segmentation method based on spectral semantic collaborative guidance includes the following steps:

[0008] Step 1: Based on the MRI imaging mechanism, the four modes of T1, T1ce, T2 and FLAIR are divided into structural flow and fluid flow. The pathological differences within the group are extracted by differential interaction and gated multiplication interaction, respectively. The obtained pathological difference features within the group are mapped to the frequency domain. The high-frequency boundary information is enhanced by learnable spectrum gating to obtain the dual-stream coding features after spectrum differential calibration.

[0009] Step 2: Input the calibrated structural flow and fluid flow features obtained in Step 1 into the dual-flow encoder. Extract multi-scale coding features through three-layer 3D convolution and downsampling. Select deep coding features and input them into the global context interaction module. This module adds 3D position coding to the features of the two flows and divides them into non-overlapping 3D blocks. Calculate the bidirectional semantic response of structural flow to fluid flow and fluid flow to structural flow through block cross attention. At the same time, combine block self attention to enhance the internal context expression and obtain the deep fusion features of the two flows after semantic interaction.

[0010] Step 3: Input the dual-stream fusion features obtained in Step 2 into the decoder for upsampling at each level, and fuse them with the skip connection features of the corresponding level of the encoder. Use the difference between the skip connection features and their low-frequency pooling components to extract the high-frequency boundary residuals. Combine the deep supervised segmentation results to generate a background semantic bias mask, guide cross-attention to suppress background noise and enhance the tumor boundary response. Finally, the brain tumor segmentation results with accurate tumor region localization and complete boundary details are obtained.

[0011] Furthermore, step 1 involves calibrating the intra-group modal differences of structural flow and fluid flow groups using frequency domain transformation and gating enhancement, specifically including the following steps:

[0012] Based on the differences in imaging response of different MRI sequences to anatomical structures, enhanced tumors, and edema regions, the four input modalities were divided into two physiological groups. T1ce and T1 constituted the structural flow group, which was used to characterize the structural differences between enhanced tumor regions and normal anatomical backgrounds. T2 and FLAIR constituted the fluid flow group, which was used to characterize the differences in fluid signals related to edema and cerebrospinal fluid inhibition.

[0013] For two modal feature maps within the same physiological group obtained through prior decoupling ,in Represents the real number field. The number of feature channels, , , These represent the depth, height, and width of the feature map, respectively; firstly, through element-wise interaction operators... Extracting pathological differences between modalities;

[0014]

[0015] in This represents a feature map showing the differences between cases across modalities. For group normalization operation, Represents the difference operator within a modal group;

[0016] For structural flow (T1ce, T1), the intensity difference between the tumor region and the normal anatomical background (T1) is enhanced using T1ce, and a subtraction operator is employed:

[0017]

[0018] in, Represents the structured flow interaction operator. This represents the feature map obtained after convolutional projection of the T1ce mode. This represents the feature map obtained after convolutional projection of the T1 mode;

[0019] For fluid flow (T2, FLAIR), taking advantage of the fact that both show high signal intensity for edema signals while cerebrospinal fluid signals are suppressed in FLAIR, a gated multiplication operator is employed:

[0020]

[0021] in Represents the fluid flow interaction operator. This represents the feature map obtained after convolutional projection of the T2 mode. This represents the feature map obtained after convolutional projection of the FLAIR mode. This represents element-wise multiplication. Activation ensures that only positive activation signals participate in the product, avoiding spurious positive responses from negative values.

[0022] Subsequently, the differences in characteristics The real part is obtained by mapping to the frequency domain using a three-dimensional fast Fourier transform. and the virtual part :

[0023]

[0024] in Represents a three-dimensional fast Fourier transform; This indicates that the transformed frequency domain consists of real components. For the imaginary part of the frequency domain, It is the imaginary unit.

[0025] To fully preserve spectral information, the real and imaginary parts are concatenated along the channel dimension to form... Next, a combination of two The bottleneck structure composed of convolutional layers learns spectral gating weights:

[0026]

[0027] in Represents the spectrum-gated weighted graph. and These represent the weights of the first and second convolutional layers in the spectrum-gated bottleneck structure, respectively. for Activation function for Activation function This indicates that the data is concatenated along the channel dimension to generate a weighted graph with the same dimension as the spectrum. ;

[0028] The learned weights are multiplied element-wise with the original spectrum, and the high-frequency enhanced detail map in the spatial domain is reconstructed using inverse 3D fast Fourier transform. :

[0029]

[0030] in Represents high-frequency enhanced detail feature map, This represents the inverse three-dimensional fast Fourier transform. This represents element-wise multiplication;

[0031] Finally, the enhanced high-frequency details are added to the original differential features, followed by convolution and... Spatial attention map is generated after activation. :

[0032]

[0033] in Represents the spatial attention weight map. express convolution;

[0034] Using this attention map The final output of the spectral differential mode calibration module can be obtained by performing residual calibration on the original modal features.

[0035] Step 2 achieves long-range dependency modeling between the two streams through a block-based cross-attention mechanism, specifically including the following steps:

[0036] Given dual-stream coding features calibrated by the spectral differential module ,in and Features from structural flow and fluid flow branches are obtained through three convolutions and downsampling. First, to preserve three-dimensional spatial location information, learnable 3D location codes are added to the features of the two branches to obtain location-enhanced features.

[0037] To achieve cross-modal interaction with linear complexity, the volume feature space is divided into two parts of size . Non-overlapping 3D blocks; for the first branch structural flow querying the second branch fluid flow unidirectional cross-attention process, through Convolution projects the features from both branches into a unified attention space; in each local block Within the block, independently compute scaled dot product cross attention, allowing each voxel to pay attention to all voxels within the same block of the other's flow. The formula is:

[0038]

[0039] in Indicates the index number of the 3D feature block. From block respectively The extracted query, key, and value subtensors, Scaling factor Indicates the first The feature output obtained by performing local cross-attention calculation within each block;

[0040] Complete all After the attention calculation for each block, the outputs of all blocks are merged without parameters according to their original spatial positions to form a result. Residual signals of the same size are used to update the first branch features through residual connections, where... , representing the number of voxels in a 3D block;

[0041] Symmetrically, the process of the second branch fluid flow querying the first branch structural flow is as follows: generating After performing the same intra-block attention calculation, they are merged; at the same time, in order to enhance the global context modeling within each branch, the block self-attention of each branch is calculated in parallel, and the output is added to the cross-attention output to form the final global context interaction feature.

[0042] Step 3 employs a semantically guided high-frequency gating module, combining physical high-frequency residual enhancement with deep supervision semantic bias, and specifically includes the following steps:

[0043] Given the skip connection features of a certain level of the encoder First, extract its high-frequency residuals. It is obtained by subtracting the low-frequency components of the original features after three-dimensional average pooling, as shown in the following formula:

[0044]

[0045] in This represents the skip connection features of the corresponding level encoder. This represents a three-dimensional average pooling operation; subsequently, pass Convolution and Activate the generated boundary weight map and perform residual enhancement on the semantically projected skip connection features to obtain the boundary enhanced features. :

[0046]

[0047] in For semantic projection convolution, for Activation function A learnable scaling factor. Features that indicate boundary enhancement;

[0048] Simultaneously, the segmentation output of this level of deep supervision is utilized. ,in For the number of categories, by After normalization, the background channel probabilities are used to construct a semantic bias mask. :

[0049]

[0050]

[0051] in This represents the segmentation map of the shallow output of the decoder. Represents the background probability plot, superscript This indicates retrieving the channel corresponding to the background class. This represents the obtained semantic bias mask; the semantic bias mask is then injected into the decoder for querying. Enhanced skip connectivity features with boundary In the calculation of cross-attention:

[0052]

[0053] in This represents the feature output of cross-attention computation with semantic bias. This indicates that the decoder upsampled features are used as the query key. The projection matrix generated from skip connection features. The attention weight of the background region is forced to be zero, so that the fused features are derived only from the tumor-related region.

[0054] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal brain tumor segmentation method based on spectral semantic collaborative guidance as described in any one of the claims.

[0055] An electronic device includes: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the multimodal brain tumor segmentation method based on spectral semantic co-guided as described in any one of the claims.

[0056] The advantages and beneficial effects of this invention are as follows:

[0057] This invention proposes a multimodal brain tumor segmentation method based on spectral semantic co-guidance, aiming to solve the problems of intermodal feature interference and boundary ambiguity in existing multimodal fusion methods. This method significantly improves the boundary accuracy and generalization ability of brain tumor segmentation through clinically prior-driven modal decoupling, frequency domain difference enhancement, and semantically guided boundary refinement. The main advantages and beneficial effects of this invention are as follows:

[0058] Enhanced modal discrimination capability (corresponding to the prior-guided modal decoupling strategy and step 1 in the claims): This method uses a prior-guided modal decoupling strategy to classify four MRI sequences into structural flow and fluid flow according to their biophysical characteristics, thereby eliminating cross-modal feature interference from the source and improving the model's ability to distinguish pathological information under different imaging mechanisms.

[0059] Improve boundary segmentation accuracy (corresponding to the spectrum difference module and step 1 in the claims): The spectrum difference module transforms the intra-group modal differences to the frequency domain, and adaptively enhances the high-frequency boundary information through learnable spectrum gating, overcoming the low-pass filtering bias of traditional spatial domain convolution, and effectively capturing and sharpening the tumor boundary.

[0060] Cross-dataset generalization capability (corresponding to the global context interaction module and step 2 in the claims): The global context interaction module realizes long-range dependency modeling between two streams through block cross attention, and is jointly optimized with composite loss function, so that the model can better adapt to multi-center and multi-parameter data distribution, thereby improving generalization performance.

[0061] Suppressing background noise (corresponding to the semantically guided high-frequency gating module and step 3 in the claims): The semantically guided high-frequency gating module combines physical high-frequency residual enhancement with deep supervision semantic bias, which effectively suppresses background noise introduced by skip connections while sharpening tumor boundaries, significantly improving the cleanliness and boundary fineness of the segmentation results.

[0062] The core innovation of this invention lies in the organic synergy of the technical solutions corresponding to claims 1-4 and their synergistic combination. Specifically, this invention constitutes a complete multimodal brain tumor segmentation scheme by combining clinical prior knowledge-driven modal decoupling, frequency domain high-frequency enhancement, linear complexity cross-modal interaction, and semantically guided boundary refinement.

[0063] 1. Existing technologies generally perform channel stitching or simple attention fusion directly on multimodal MRI. However, this invention designs a decoupling grouping strategy based on the biophysical imaging principles of different modalities, and designs category-specific interaction operators such as subtraction and gated multiplication for structural flow and fluid flow respectively. This idea of ​​explicitly encoding clinical knowledge into network structure design is extremely rare in existing methods.

[0064] 2. Existing frequency domain learning methods are mostly used for image generation or domain adaptation. This invention applies them to intra-group differential enhancement in multimodal brain tumor segmentation. By adaptively filtering boundary-related frequency components through learnable spectral gating, it is not a simple frequency domain filtering method and is not easily associated with conventional techniques.

[0065] 3. The semantically guided high-frequency gating module is not a simple superposition of attention mechanisms, but innovatively injects the semantic information output by deep supervision as a bias mask into the cross-attention calculation of skip connections, so as to achieve the synergistic effect of physical high-frequency enhancement and semantic background suppression. This dual-guided combination method is not a conventional approach in this field.

[0066] The synergistic relationship between the above modules—prior decoupling and difference operator operations provide clean feature grouping for frequency domain enhancement, frequency domain enhancement provides clearer boundary information for cross attention, and semantic gating provides reliable tumor region guidance for decoder fusion—constitutes a mutually supportive overall scheme, the synergistic effect of which is difficult to predict by a single module or simple combination. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the overall network model structure of a preferred embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the spectrum differential module.

[0069] Figure 3 This is a schematic diagram of the global context interaction module.

[0070] Figure 4 This is a schematic diagram of a semantically guided high-frequency gating module. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0072] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0073] This invention is based on an encoder-decoder network design, essentially for brain tumor segmentation, addressing the intermodal feature interference and boundary blurring issues caused by early channel stitching in multimodal MRI. The network segments tumor regions by learning the hierarchical structure of features in the image. It extracts useful features from the input MRI and progressively calibrates them through a prior-guided modal decoupling strategy and a spectral difference module, ultimately generating a high-level representation of the tumor region. This representation is then fed into the segmentation head for decision-making. During segmentation, we consider intermodal interference and boundary blurring issues, implementing frequency domain difference enhancement and semantically guided boundary refinement to mitigate the loss of high-frequency information and the introduction of background noise. Simultaneously, a composite loss function is designed, making the segmenter training more focused on boundary quality, further enhancing performance.

[0074] Figure 1 The overall network model structure of this invention is shown in the diagram, which includes the following steps:

[0075] Step 1: Based on the MRI imaging mechanism, the four modes of T1, T1ce, T2 and FLAIR are divided into structural flow and fluid flow. The pathological differences within the group are extracted by differential interaction and gated multiplication interaction, respectively. The obtained pathological difference features within the group are mapped to the frequency domain. The boundary related frequency components are enhanced by learnable spectrum gating to obtain the dual-stream coding features after spectrum differential calibration.

[0076] Step 2: Input the calibrated structural flow and fluid flow features obtained in Step 1 into the dual-flow encoder. Extract multi-scale coding features through three-layer 3D convolution and downsampling. Select deep coding features and input them into the global context interaction module. Add 3D position coding and divide them into non-overlapping 3D blocks. Calculate the bidirectional semantic response of structural flow to fluid flow and fluid flow to structural flow through block cross attention. At the same time, combine block self attention to enhance the internal context expression to obtain the deep fusion features of dual flow after semantic interaction.

[0077] Step 3: Input the dual-stream fusion features obtained in Step 2 into the decoder for upsampling at each level, and fuse them with the skip connection features of the corresponding level of the encoder. During the fusion process, the high-frequency boundary residual is extracted by using the difference between the skip connection features and their low-frequency pooling components. The background semantic bias mask is generated by combining the deep supervised segmentation results, guiding cross-attention to suppress background noise and enhance the tumor boundary response. Finally, the brain tumor segmentation results with accurate tumor region localization and complete boundary details are obtained.

[0078] Furthermore, step 1 involves calibrating the intra-group modal differences of structural flow and fluid flow groups using frequency domain transformation and gating enhancement, specifically including the following steps:

[0079] Based on the differences in imaging response of different MRI sequences to anatomical structures, enhanced tumors, and edema regions, the four input modalities were divided into two physiological groups. T1ce and T1 constituted the structural flow group, which was used to characterize the structural differences between enhanced tumor regions and normal anatomical backgrounds. T2 and FLAIR constituted the fluid flow group, which was used to characterize the differences in fluid signals related to edema and cerebrospinal fluid inhibition.

[0080] For two modal feature maps within the same physiological group obtained through prior decoupling ,in Represents the real number field. The number of feature channels, , , These represent the depth, height, and width of the feature map, respectively. First, an element-wise interaction operator is used... Extracting pathological differences between modalities:

[0081]

[0082] in This represents a feature map showing the differences between cases across modalities. For group normalization operation, Represents the difference operator within a modal group;

[0083] For structural flow (T1ce, T1), the intensity difference between the tumor region and the normal anatomical background (T1) is enhanced using T1ce, and a subtraction operator is employed:

[0084]

[0085] in, Represents the structured flow interaction operator. This represents the feature map obtained after convolutional projection of the T1ce mode. This represents the feature map obtained after convolutional projection of the T1 mode;

[0086] For fluid flow (T2, FLAIR), taking advantage of the fact that both show high signal intensity for edema signals while cerebrospinal fluid signals are suppressed in FLAIR, a gated multiplication operator is employed:

[0087]

[0088] in Represents the fluid flow interaction operator. This represents the feature map obtained after convolutional projection of the T2 mode. This represents the feature map obtained after convolutional projection of the FLAIR mode. This represents element-wise multiplication. Activation ensures that only positive activation signals participate in the product, avoiding spurious positive responses from negative values.

[0089] Subsequently, the differences in characteristics The real part is obtained by mapping to the frequency domain using a three-dimensional fast Fourier transform. and the virtual part :

[0090]

[0091] in This represents a three-dimensional Fast Fourier Transform. This indicates that the transformed frequency domain consists of real components. For the imaginary part of the frequency domain, It is the imaginary unit.

[0092] To fully preserve spectral information, the real and imaginary parts are concatenated along the channel dimension to form... Next, a combination of two The bottleneck structure composed of convolutional layers learns spectral gating weights:

[0093]

[0094] in Represents the spectrum-gated weighted graph. and These represent the weights of the first and second convolutional layers in the spectrum-gated bottleneck structure, respectively. for Activation function for The activation function generates a weight map with the same dimension as the spectrum. ;

[0095] The learned weights are multiplied element-wise with the original spectrum, and the high-frequency enhanced detail feature map in the spatial domain is reconstructed by inverse 3D fast Fourier transform. :

[0096]

[0097] in Represents high-frequency enhanced detail feature map, This represents the inverse three-dimensional fast Fourier transform. This represents element-wise multiplication. This indicates splicing along the channel dimension;

[0098] Finally, the enhanced high-frequency details are added to the original differential features, followed by convolution and... Spatial attention map is generated after activation. :

[0099]

[0100] in Represents the spatial attention weight map. express convolution;

[0101] Using this attention map The final output of the spectral differential mode calibration module can be obtained by performing residual calibration on the original modal features.

[0102] Furthermore, step 2 achieves long-range dependency modeling between the two streams through a block-based cross-attention mechanism, specifically including the following steps:

[0103] Given deep encoded features extracted from structural flow and fluid flow after spectral differential calibration and input into a dual-flow encoder. ,in and Features from structural flow and fluid flow branches are obtained through three convolutions and downsampling. First, to preserve three-dimensional spatial location information, learnable 3D location codes are added to the features of the two branches to obtain location-enhanced features.

[0104] To achieve cross-modal interaction with linear complexity, the volume feature space is divided into two parts of size . Non-overlapping 3D blocks; for the first branch structural flow querying the second branch fluid flow unidirectional cross-attention process, through Convolution projects the features from both branches into a unified attention space; in each local block Within the block, independently compute scaled dot product cross attention, allowing each voxel to pay attention to all voxels within the same block of the other's flow. The formula is:

[0105]

[0106] in Indicates the index number of the 3D feature block. From block respectively The extracted query, key, and value subtensors, Scaling factor Indicates the first The feature output obtained by performing local cross-attention calculation within each block;

[0107] Complete all After the attention calculation for each block, the outputs of all blocks are merged without parameters according to their original spatial positions to form a result. Residual signals of the same size are used to update the first branch features through residual connections, where... , representing a voxel of a three-dimensional block;

[0108] Symmetrically, the process of the second branch fluid flow querying the first branch structural flow is as follows: generating After performing the same intra-block attention calculation, they are merged; at the same time, in order to enhance the global context modeling within each branch, the block self-attention of each branch is calculated in parallel, and the output is added to the cross-attention output to form the features after global context interaction;

[0109] Furthermore, step 3 employs a semantically guided high-frequency gating module, combining physical high-frequency residual enhancement with deep supervision semantic bias, specifically including the following steps:

[0110] Given the skip connection features of a certain level of the encoder First, extract its high-frequency residuals. It is obtained by subtracting the low-frequency components of the original features after three-dimensional average pooling, as shown in the following formula:

[0111]

[0112] in This represents the skip connection features of the corresponding level encoder. This represents a three-dimensional average pooling operation; subsequently, pass Convolution and Activate the generated boundary weight map and perform residual enhancement on the semantically projected skip connection features to obtain the boundary enhanced features. :

[0113]

[0114] in For semantic projection convolution, for Activation function A learnable scaling factor. Features that indicate boundary enhancement;

[0115] Simultaneously, the segmentation map output by this level of deep supervision is utilized. ,in For the number of categories, by After normalization, the background channel probabilities are used to construct a semantic bias mask. :

[0116]

[0117]

[0118] in This represents the segmentation map of the shallow output of the decoder. Represents the background probability plot, superscript This indicates retrieving the channel corresponding to the background class. This represents the obtained semantic bias mask; the semantic bias mask is then injected into the decoder for querying. Enhanced skip connectivity features with boundary In the calculation of cross-attention:

[0119]

[0120] in This represents cross-attention computation with semantic bias. This indicates that the decoder upsampled features are used as the query key. The projection matrix generated from skip connection features. The attention weight of the background region is forced to be zero, so that the fused features are derived only from the tumor-related region.

[0121] This invention introduces a priori-guided modal decoupling and spectral-spatial interaction strategy to design a multimodal brain tumor segmentation method based on spectral-semantic collaborative guidance. This approach promotes effective learning of the model in multimodal data and improves its ability to segment tumor boundaries. The network enhances the sensitivity of the feature extractor to high-frequency information through frequency domain difference enhancement and semantically guided boundary refinement, enabling the model to learn richer and more generalized feature representations, thereby improving the model's accuracy in fine-grained tumor boundary segmentation. This invention effectively solves the problems of intermodal interference and boundary ambiguity caused by early channel splicing in traditional multimodal brain tumor segmentation methods. Furthermore, the spectral difference module and semantically guided gating mechanism of this invention can effectively aggregate feature information from different stages, optimizing each stage for different frequency components and semantic levels, enhancing the model's adaptability to various data distributions. This strategy not only improves segmentation accuracy but also enhances the model's generalization ability to multi-center, multi-parameter clinical scenarios. Compared with other brain tumor segmentation methods, this invention demonstrates superior performance on multiple benchmark datasets. The method achieves higher Dice scores and lower Hausdorff distances across different data distributions, demonstrating its robustness and generalization ability in handling multimodal brain tumor segmentation problems. In practical applications, especially with multicenter clinical data, the method significantly improves the model's segmentation performance and boundary quality.

[0122] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0123] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0125] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A multimodal brain tumor segmentation method based on spectral semantic collaborative guidance, characterized in that, Includes the following steps: Step 1: Based on the MRI imaging mechanism, the four modes of T1, T1ce, T2 and FLAIR are divided into structural flow and fluid flow. The pathological differences within the group are extracted by differential interaction and gated multiplication interaction, respectively. The obtained pathological difference features within the group are mapped to the frequency domain. The high-frequency boundary information is enhanced by learnable spectrum gating to obtain the dual-stream coding features after spectrum differential calibration. Step 2: Input the calibrated structural flow and fluid flow features obtained in Step 1 into the dual-flow encoder. Extract multi-scale coding features through three-layer 3D convolution and downsampling. Select deep coding features and input them into the global context interaction module. This module adds 3D position coding to the features of the two flows and divides them into non-overlapping 3D blocks. Calculate the bidirectional semantic response of structural flow to fluid flow and fluid flow to structural flow through block cross attention. At the same time, combine block self attention to enhance the internal context expression and obtain the deep fusion features of the two flows after semantic interaction. Step 3: Input the dual-stream fusion features obtained in Step 2 into the decoder for upsampling at each level, and fuse them with the skip connection features of the corresponding level of the encoder. Use the difference between the skip connection features and their low-frequency pooling components to extract the high-frequency boundary residuals. Combine the deep supervised segmentation results to generate a background semantic bias mask, guide cross-attention to suppress background noise and enhance the tumor boundary response. Finally, the brain tumor segmentation results with accurate tumor region localization and complete boundary details are obtained.

2. The multimodal brain tumor segmentation method based on spectral semantic collaborative guidance according to claim 1, characterized in that, Step 1, which calibrates the intra-group modal differences of structural flow and fluid flow groups by performing frequency domain transformation and gating enhancement, specifically includes the following steps: Based on the differences in imaging response of different MRI sequences to anatomical structures, enhanced tumors, and edema regions, the four input modalities were divided into two physiological groups. T1ce and T1 constituted the structural flow group, which was used to characterize the structural differences between enhanced tumor regions and normal anatomical backgrounds. T2 and FLAIR constituted the fluid flow group, which was used to characterize the differences in fluid signals related to edema and cerebrospinal fluid inhibition. For two modal feature maps within the same physiological group obtained through prior decoupling ,in Represents the real number field. The number of feature channels, , , These represent the depth, height, and width of the feature map, respectively; firstly, through element-wise interaction operators... Extracting pathological differences between modalities; ； in This represents a feature map showing the differences between cases across modalities. For group normalization operation, Represents the difference operator within a modal group; For structural flow (T1ce, T1), the intensity difference between the tumor region and the normal anatomical background (T1) is enhanced using T1ce, and a subtraction operator is employed: ； in, Represents the structured flow interaction operator. This represents the feature map obtained after convolutional projection of the T1ce mode. This represents the feature map obtained after convolutional projection of the T1 mode; For fluid flow (T2, FLAIR), taking advantage of the fact that both show high signal intensity for edema signals while cerebrospinal fluid signals are suppressed in FLAIR, a gated multiplication operator is employed: ； in Represents the fluid flow interaction operator. This represents the feature map obtained after convolutional projection of the T2 mode. This represents the feature map obtained after convolutional projection of the FLAIR mode. This represents element-wise multiplication. Activation ensures that only positive activation signals participate in the product, avoiding false positive responses from negative values; Subsequently, the differences in characteristics The real part is obtained by mapping to the frequency domain using a three-dimensional fast Fourier transform. and the virtual part : ； in Represents a three-dimensional fast Fourier transform; This indicates that the transformed frequency domain consists of real components. For the imaginary part of the frequency domain, The imaginary unit; To fully preserve spectral information, the real and imaginary parts are concatenated along the channel dimension to form... Next, a combination of two The bottleneck structure composed of convolutional layers learns spectral gating weights: ； in Represents the spectrum-gated weighted graph. and These represent the weights of the first and second convolutional layers in the spectrum-gated bottleneck structure, respectively. for Activation function for Activation function This indicates that the data is concatenated along the channel dimension to generate a weighted graph with the same dimension as the spectrum. ; The learned weights are multiplied element-wise with the original spectrum, and the high-frequency enhanced detail map in the spatial domain is reconstructed using inverse 3D fast Fourier transform. : ； in This represents a high-frequency enhanced detail feature map. This represents the inverse three-dimensional fast Fourier transform. This represents element-wise multiplication; Finally, the enhanced high-frequency details are added to the original differential features, followed by convolution and... Spatial attention map is generated after activation. : ； in Represents the spatial attention weight map. express convolution; Using this attention map The final output of the spectral differential mode calibration module can be obtained by performing residual calibration on the original modal features.

3. The multimodal brain tumor segmentation method based on spectral semantic collaborative guidance according to claim 2, characterized in that, Step 2 achieves long-range dependency modeling between the two streams through a block-based cross-attention mechanism, specifically including the following steps: Given dual-stream coding features calibrated by the spectral differential module ,in and Features from structural flow and fluid flow branches are obtained through three convolutions and downsampling. First, to preserve three-dimensional spatial location information, learnable 3D location codes are added to the features of the two branches to obtain location-enhanced features. To achieve cross-modal interaction with linear complexity, the volume feature space is divided into two parts of size . Non-overlapping 3D blocks; for the first branch structural flow querying the second branch fluid flow unidirectional cross-attention process, through Convolution projects the features from both branches into a unified attention space; in each local block Within the block, independently compute scaled dot product cross attention, allowing each voxel to pay attention to all voxels within the same block of the other's flow. The formula is: ； in Indicates the index number of the three-dimensional feature block. From block respectively The extracted query, key, and value subtensors, Scaling factor Indicates the first The feature output obtained by performing local cross-attention calculation within each block; Complete all After the attention calculation for each block, the outputs of all blocks are merged without parameters according to their original spatial positions to form a result. Residual signals of the same size are used to update the first branch features through residual connections, where , representing the number of voxels in a 3D block; Symmetrically, the process of the second branch fluid flow querying the first branch structural flow is as follows: generating After performing the same intra-block attention calculation, they are merged; at the same time, in order to enhance the global context modeling within each branch, the block self-attention of each branch is calculated in parallel, and the output is added to the cross-attention output to form the final global context interaction feature.

4. The multimodal brain tumor segmentation method based on spectral semantic collaborative guidance according to claim 3, characterized in that, Step 3 employs a semantically guided high-frequency gating module, combining physical high-frequency residual enhancement with deep supervision semantic bias, and specifically includes the following steps: Given the skip connection features of a certain level of the encoder First, extract its high-frequency residuals. It is obtained by subtracting the low-frequency components of the original features after three-dimensional average pooling, as shown in the following formula: ； in This represents the skip connection features of the corresponding level encoder. This represents a three-dimensional average pooling operation; subsequently, pass Convolution and Activate the generated boundary weight map and perform residual enhancement on the semantically projected skip connection features to obtain the boundary enhanced features. : ； in For semantic projection convolution, for Activation function A learnable scaling factor. Features that indicate boundary enhancement; Simultaneously, the segmentation output of this level of deep supervision is utilized. ,in For the number of categories, by After normalization, the background channel probabilities are used to construct a semantic bias mask. : in This represents the segmentation map of the shallow output of the decoder. Represents the background probability plot, superscript This indicates retrieving the channel corresponding to the background class. This represents the obtained semantic bias mask; Inject semantic bias mask into decoder query Enhanced skip connectivity features with boundary In the calculation of cross-attention: ； in This represents the feature output of cross-attention computation with semantic bias. This indicates that the decoder upsampled features are used as the query key. The projection matrix generated from skip connection features. The attention weight of the background region is forced to be zero, so that the fused features are derived only from the tumor-related region.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal brain tumor segmentation method based on spectral semantic collaborative guidance as described in any one of claims 1-4.

6. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute a computer program stored in the memory to enable the electronic device to perform the multimodal brain tumor segmentation method based on spectral semantic collaborative guidance as described in any one of claims 1-4.

Images