Incomplete modality brain tumor segmentation system based on cue-driven attention

Abstract

The application provides an incomplete modality brain tumor segmentation system based on prompt-driven attention, relates to the field of artificial intelligence, and a double-branch fusion encoder extracts features of different abstraction levels through two branches, can reserve spatial information while extracting high-level semantics, a modality missing prompt synthesizer generates modality missing specific prompts for the current modality combination through learnable general prompts, integrates the modality missing specific prompts through intra-modality and inter-modality attention modules, further integrates catalyst information for key modalities through enhanced active attention, improves the processing capacity of the model for missing data, significantly improves the performance of processing multi-modality MRI data and the accuracy of processing downstream tasks, and a catalyst synthesizer module identifies and strengthens the role of key modalities, ensures that even in the case of missing key modalities, the performance of downstream tasks can be maintained through generated catalysts.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to a cue-driven attention-based incomplete modality brain tumor segmentation system. Background Technology

[0002] Brain tumors have garnered significant attention due to their potentially destructive impact on neurological function. Malignant tumors, in particular, can invade surrounding healthy tissues, leading to more severe neurological dysfunction and posing a serious threat to patient life. Therefore, accurate characterization of brain tumors is crucial for developing treatment plans, assessing patient prognosis, and monitoring disease progression. In clinical practice, multimodal magnetic resonance imaging (MRI) plays an increasingly important role, with common modalities including T1-weighted (T1), enhanced T1-weighted (T1c), fluid-attenuated inversion recovery (FLAIR), and T2-weighted (T2) images. By integrating these multimodal data, clinicians can gain a more comprehensive understanding of the biological characteristics and heterogeneity of tumors, thereby providing higher-quality personalized treatment and prognostic assessments.

[0003] Recently, deep learning-based multimodal methods have been developed and have shown remarkable performance in brain tumor segmentation. While these methods are impressive, they are primarily designed to process complete, full-modality data. However, in clinical settings, the effectiveness of multimodal MRI in the diagnosis and treatment of brain tumors largely depends on the availability and quality of all relevant modalities. Due to technical issues such as equipment movement, hardware malfunctions, or patient-related factors, some MRI modalities may be lost or degraded in quality; this condition, known as incomplete or missing modalities, poses a challenge to accurate segmentation of brain tumors.

[0004] To address the issue of missing modalities, the academic community has proposed several solutions, mainly falling into two categories: One category attempts to transfer knowledge from the full-modal model to the missing modality to compensate for insufficient information; this method is often referred to as "co-training." While effective, these methods typically require training a dedicated model for each possible combination of missing modalities, which is not only resource-intensive but also often fails to fully utilize the information provided by other modalities, thus affecting the overall model performance. The other category attempts to synthesize missing modal data to achieve full-modal segmentation. This method uses a unified model to handle all missing modalities. Although logically sound, this approach often has limited ability to reconstruct missing modalities and is computationally expensive in practice. Furthermore, current methods have not considered the specificity of the T1c modality. When the T1c modality is missing, the segmentation performance of almost all models in enhanced tumor regions significantly decreases, highlighting the importance of the T1c modality in capturing key information about enhanced tumor regions.

[0005] In particular, many methods utilize prompts in both training specialized and unified networks. While the concept of prompts was initially introduced in natural language processing, recent research has yielded promising results in computer vision and multimodal tasks. However, existing models employing prompts require manual intervention and cannot automatically synthesize necessary prompts based on missing modalities. Furthermore, current methods often synthesize prompts directly based on modal combinations, failing to delve into the shared information provided by different modal combinations. This results in synthesized prompts containing insufficient information, thus impacting performance on subsequent segmentation tasks.

[0006] Therefore, despite significant progress in processing incomplete modal MRI data, some challenges remain. A key issue is how to better integrate cueing techniques into the processing of incomplete modal objects and uncover hidden knowledge in key modalities. Success in this area would significantly improve the accuracy and efficiency of brain tumor segmentation when certain MRI modalities are lacking, thus providing more reliable support for clinical diagnosis and treatment decisions. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to propose an incomplete modality brain tumor segmentation system based on cue-driven attention, comprising an image data acquisition module, a training sample dataset construction module, and an incomplete modality brain tumor segmentation module.

[0008] The image data acquisition module is used to acquire MRI image data of all modalities of the brain, as well as the segmentation map corresponding to the MRI image data of all modalities of the brain. The segmentation map corresponding to the MRI image data of all modalities of the brain includes brain tumor labels. It is also used to transmit the MRI image data of all modalities of the brain and the segmentation map corresponding to the MRI image data of all modalities of the brain to the training sample dataset construction module.

[0009] The training sample dataset construction module receives MRI image data of all brain modalities and corresponding segmentation maps of the MRI image data of all brain modalities. It then performs modality missing processing on the MRI image data of all brain modalities to obtain modality-missing image data. This modality-missing image data is further preprocessed to obtain preprocessed modality data, and a modality missing mask is determined simultaneously. 0 The modality missing mask 0The system represents the presence or absence of each modality in all modalities of the preprocessed image data; the preprocessed modal data and modality missing mask are used as input samples, and the segmentation maps corresponding to the MRI image data of all brain modalities are used as output samples; the input samples and output samples constitute training samples, and multiple training samples are used to construct a training sample dataset; the training sample dataset is then transmitted to the incomplete modality brain tumor segmentation module.

[0010] The incomplete modality brain tumor segmentation module is used to receive training sample datasets and train the incomplete modality object processing model based on the cue-driven attention mechanism based on the training sample datasets. The incomplete modality object processing model based on the cue-driven attention mechanism includes a dual-branch fusion encoder, a modality missing cue synthesizer, a cue-driven attention fusion generator, and a catalyst synthesizer.

[0011] Specifically, the preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modes in the preprocessed modal data; the fusion features of all modes in the preprocessed modal data are then combined with a mode missing mask. 0 The input is fed into the modality missing cues synthesizer to obtain modality missing specific cues P. spec ; The fusion features and mode missing mask of all modes in the preprocessed modal data 0 Inputting the catalyst synthesizer, we obtain the loss value L of the catalyst synthesizer's optimization objective function. con and catalyst F Act The fusion features of all modes in the preprocessed modal data, and the catalyst F output from the catalyst synthesizer are then analyzed. Act Modal Missing Mask 0 and the modality missing specific cue P spec The input is fed into a cue-driven attention fusion processor to obtain the enhanced activation feature representation F. cross ; representing enhanced activation characteristics as F cross The input is fed into the decoder to obtain a segmentation map corresponding to the preprocessed modality data. This segmentation map includes the predicted tumor label. Simultaneously, based on the output sample and the segmentation map corresponding to the preprocessed modality data, the decoder's loss value L is obtained. decoder ;

[0012] Loss value L obtained from the catalyst synthesizer con The loss value L of the decoder decoder The total loss value is obtained, specifically expressed by the following formula:

[0013] L total =L con +Ldecoder ;

[0014] Among them, L total This represents the total loss value;

[0015] During model training, the parameters of the incomplete modal object processing model based on the cue-driven attention mechanism are updated based on the total loss value, and finally the trained incomplete modal object processing model based on the cue-driven attention mechanism is obtained.

[0016] Optionally, in the specific implementation process, the image data acquisition module is also used to acquire MRI image data of the brain with real modal missingness, and transmit the MRI image data of the brain with real modal missingness to the training sample dataset construction module;

[0017] The training sample dataset construction module is also used to preprocess the MRI image data of the brain with missing real modalities to obtain preprocessed MRI image data, and transmit the preprocessed MRI image data to the incomplete modal brain tumor segmentation module.

[0018] The incomplete modality brain tumor segmentation module is also used to receive preprocessed MRI image data and process the preprocessed MRI image data through a trained incomplete modality object processing model based on cue-driven attention mechanism to obtain a segmentation map corresponding to the MRI image data with real modality missing of the brain. The segmentation map corresponding to the MRI image data with real modality missing of the brain includes the label of the brain tumor.

[0019] Optionally, the incomplete modality brain tumor segmentation module processes the preprocessed modal data using a dual-branch fusion encoder to obtain the fusion features of all modalities in the preprocessed modal data, including:

[0020] For each modal data in the preprocessed modal data, the modal data is processed by the first branch of the dual-branch fusion encoder, specifically through the following formula:

[0021]

[0022] Among them, M m This refers to the modal data of the m-th mode in the preprocessed modal data. For M m The feature representation after the l-th layer of the first branch in the dual-branch fusion encoder. For M mThe feature representation after the (l-1)th layer of the first branch in the dual-branch fusion encoder is DS, which represents the downsampling operation, and ConvBlock, which represents a convolutional block consisting of group normalization, ReLU activation function and 3×3 kernel size convolution.

[0023] For each modal data in the preprocessed modal data, the modal data is processed by the second branch of the dual-branch fusion encoder, specifically through the following formula:

[0024]

[0025] in, For M m The feature representation after the l-th layer of the second branch in the dual-branch fusion encoder. For M m Feature representation after the (l-1)th layer of the second branch in the dual-branch fusion encoder;

[0026] For M m Feature representation after the 5th layer of the first branch in the dual-branch fusion encoder and M m Feature representation after the 5th layer of the second branch in the dual-branch fusion encoder Feature fusion is performed to obtain the fused feature F. m Specifically, this is achieved through the following formula:

[0027]

[0028] Among them, F m Let CA represent the fusion feature of the m-th modality;

[0029] in, (m=1,2,…,N M D represents the depth of the preprocessed modal data, W represents the width of the preprocessed modal data, H represents the height of the preprocessed modal data, and C represents the number of channels of the preprocessed modal data.

[0030] Optional, Modality Missing Mask 0 , For the Bernoulli index used to indicate the m-th modality, when the MRI image data of the m-th modality is absent in the MRI image data of all brain modalities, The value is 0 when the m-th modality of MRI image data exists in all modalities of brain MRI image data. The value of N is 1. M The total number of modes;

[0031] Based on this, the incomplete modality brain tumor segmentation module will use the fusion features of all modalities and the modality missing mask in the preprocessed modal data. 0 The input is fed into the modality missing cues synthesizer to obtain modality missing specific cues P. spec ,include:

[0032] General prompts for modal missing Perform random initialization, the general hint P for the missing modality. uni It contains common information implied by all modality missing cases, where D represents the depth of the preprocessed modal data, W represents the width of the preprocessed modal data, and H represents the height of the preprocessed modal data. N P This represents the total number of all possible modal missing cases.

[0033] By cascading operations, the modality missing mask is... 0 The fusion features of all modes in the preprocessed modal data are merged to obtain... Specifically, this is expressed by the following formula:

[0034]

[0035] in, As an intermediate variable, Modality Missing Mask 0 The Bernoulli index used to indicate the first mode. Modality Missing Mask 0 The Bernoulli index is used to indicate the second mode. Modality Missing Mask 0 Used to indicate the Nth M The Bernoulli indices for each modality, where F1 represents the fusion feature of the first modality and F2 represents the fusion feature of the second modality. For the Nth M Fusion features of each modality;

[0036] Through three fusion modules, intermediate variables are processed. General hints for modal missing P uni By splicing, we obtain N P The set of modality missing cues corresponding to each modality missing case Specifically, this is achieved through the following formula:

[0037]

[0038] Here, PF represents three consecutive fusion modules, each containing a convolutional layer, batch normalization, and a ReLU activation function; Split represents segmenting features along the channel dimension to obtain N. P A prompt with 1 channel; P1 represents the modality missing prompt corresponding to the first type of modality missing, and P2 represents the modality missing prompt corresponding to the second type of modality missing. Indicates the relationship with the Nth P Modality missing information corresponding to each modality missing condition;

[0039] Based on the modality missing mask 0 ,exist Select from the options to obtain a specific cue P indicating modality loss. spec .

[0040] Optional, Modality Missing Mask 0 , For the Bernoulli index used to indicate the m-th modality, when the MRI image data of the m-th modality is absent in the MRI image data of all brain modalities, The value is 0 when the m-th modality of MRI image data exists in all modalities of brain MRI image data. The value of N is 1. M The total number of modes;

[0041] Based on this, the incomplete modality brain tumor segmentation module will use the fusion features of all modalities and the modality missing mask in the preprocessed modal data. 0 Inputting the catalyst synthesizer, we obtain the loss value L of the catalyst synthesizer's optimization objective function. con and catalyst F Act ,include:

[0042] Determine the mask for missing modes 0 The Bernoulli index, used to indicate whether a specified mode is 1, is used in the mode missing mask. 0 When the Bernoulli index used to indicate a specified mode is 1, the catalyst F output from the catalyst synthesizer is determined according to the first method. Act In the modality missing mask 0 When the Bernoulli index used to indicate a specified mode is 0, the catalyst F output from the catalyst synthesizer is determined according to the second method. Act ;

[0043] The first method includes:

[0044] Modality missing specific cue P spec Determine the initial catalyst YAct The Bernoulli index used to indicate a specified mode is set to 0 to obtain a new mode absence mask. 1 Furthermore, based on the new modality missing mask 1 The first catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. And calculate the initial catalyst Y Act and the first catalyst The loss value between them is expressed by the following formula:

[0045] Y Act =F spec ;

[0046]

[0047]

[0048] Here, MF represents three consecutive modules, each containing a convolutional layer, batch normalization, and a ReLU activation function, and MSE represents the mean squared error. For the new modality missing mask 1 The Bernoulli index used to indicate the first mode. For the new modality missing mask 1 Used to indicate the Nth M Bernoulli index for each modality, F1 is the fusion feature of the first modality. For the Nth M The fusion features of each modality, L con For the initial catalyst Y Act and the first catalyst The loss value between;

[0049] At this point, the initial catalyst Y... Act Catalyst F, as the output of the catalyst synthesizer Act ;

[0050] The second method includes:

[0051] Modality Missing Mask 0 Rename to Modality Missing Mask 1 Based on modality missing mask 1 The first catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. Randomly select modality missing mask 1 Take a Bernoulli index with a value of 1 that indicates a mode, reset its value to 0, and obtain a new mode missing mask. 2 According to the new modality missing mask2 The second catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. And calculate the first catalyst Second catalyst The loss value between them is expressed by the following formula:

[0052]

[0053]

[0054]

[0055] in, For the new modality missing mask 2 The Bernoulli index used to indicate the first mode. For the new modality missing mask 2 The middle is used to indicate the Nth M Bernoulli index for each modality, L con As the first catalyst Second catalyst The loss value between;

[0056] At this time, the first catalyst Catalyst F, as the output of the catalyst synthesizer Act .

[0057] Optionally, the cue-driven attention fusion unit includes an intramodal attention module, an intramodal attention module, and an enhanced activation attention module;

[0058] Based on this, the incomplete modality brain tumor segmentation module incorporates the fusion features of all modalities in the preprocessed modal data, and the catalyst F output by the catalyst synthesizer. Act Modal Missing Mask 0 and the modality missing specific cue P spec The input is fed into a cue-driven attention fusion processor to obtain the enhanced activation feature representation F. cross ,include:

[0059] Within the intramodal attention module, the fused feature F m and modality missing specific cue P spec The process yields the intra-modal sequence; within the inter-modal attention module, the intra-modal sequence and the modal missing mask output by the intra-modal attention module are processed. 0 and modality missing specific cue P spec The process is performed to obtain the intermodal feature representation F. globalIn the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is... global Catalyst F output from the catalyst synthesizer Act And modality-specific cue P spec The process yields the enhanced activation feature representation F. cross .

[0060] Optionally, within the intramodal attention module, the fused feature F m and modality missing specific cue P spec The process yields the intramodal sequence, including:

[0061] F fusion feature F m Flattened into a shape of length L seq The sequence was analyzed, and features were obtained through linear projection and the addition of positional embeddings. P spec By performing convolution, flattening, and normalization operations with parameters as the first parameter, the key of the intramodal attention module is obtained. and the value of the intramodal attention module Features Key to intramodal attention modules and the value of the intramodal attention module After processing by multi-scale self-attention MSSAP and multilayer perceptron MLP, intramodal sequences are obtained. Specifically, this is achieved through the following formula:

[0062]

[0063]

[0064] Here, MLP represents a multilayer perceptron, which includes a linear input layer, a non-linear activation layer, and a linear output layer. m As intermediate variables, all Forming intramodal sequences.

[0065] Optionally, in the inter-modal attention module, the intra-modal sequence and the modal missing mask output by the intra-modal attention module are... 0 and modality missing specific cue P spec The process is performed to obtain the intermodal feature representation F. global ,include:

[0066] By cascading operations, the modality missing mask is... 0 The intramodal sequence is merged with the intramodal sequence, and positional embeddings are added to obtain the intermodal sequence S. global Specifically, this is achieved through the following formula:

[0067]

[0068] Where Pos represents position embedding. For intramodal sequences, Modality Missing Mask 0 ;

[0069] Modality missing specific cue P spec By performing convolution, flattening, and normalization operations with the second parameter, the key of the intermodal attention module is obtained. The value of the intermodal attention module

[0070] The intermodal sequence S global Key to intermodal attention modules The value of the intermodal attention module After processing by multi-scale self-attention MSSAP and multilayer perceptron MLP, the intermodal feature representation F is obtained. global Specifically, it is expressed by the following formula:

[0071]

[0072] F global =MLP(y global )+y global ;

[0073] Among them, y global It is an intermediate variable.

[0074] Optionally, in the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is... global Catalyst F output from the catalyst synthesizer Act And modality-specific cue P spec The process yields the enhanced activation feature representation F. cross ,include:

[0075] For the intermodal feature representation F global After flattening, deforming, and layer normalization, the enhanced activation attention sequence S is obtained. cross For catalyst F Act Flattening, deformation, and layer normalization were performed to obtain the catalyst sequence S. Act ;

[0076] Specific cue P for modality missing spec Perform convolution, flattening, and normalization operations with a third parameter to obtain the key for enhancing the activation attention module. and the value of the enhanced activation attention module

[0077] Will enhance the activation attention sequence S cross Catalyst sequence S Act , Enhance the key to activating the attention module and the value of the enhanced activation attention module After processing by multimodal cross-attention (MMCAP) and multilayer perceptron (MLP), the enhanced activation feature representation F is obtained. cross Specifically, this is achieved through the following formula:

[0078]

[0079] F cross =MLP(y cross )+y cross ;

[0080] Among them, y cross It is an intermediate variable.

[0081] Optionally, the training sample dataset construction module performs modality missing processing on MRI image data of all brain modalities to obtain modality missing image data, and then preprocesses the modality missing image data to obtain preprocessed modality data, specifically including:

[0082] Data corruption and random masking were performed on MRI image data of all brain modalities to obtain image data with missing modalities. Then, image denoising, image cropping pre-adjustment, intensity normalization, spatial normalization, field correction and intracranial extraction were performed on the image data with missing modalities to obtain preprocessed modal data.

[0083] The beneficial effects of adopting the above technical solution are as follows:

[0084] 1. More efficient feature extraction: The dual-branch fusion encoder extracts features at different levels of abstraction through two branches, which can preserve the spatial information of details while extracting high-level semantics. This effectively balances the need to capture details and abstract features, which is especially crucial for complex multimodal analysis.

[0085] 2. Improved Cue Generation: The modal missing cue synthesizer generates modal missing cue specifics for the current modal combination based on learnable general cues and the modal combinations in the input data. This method effectively utilizes shared information between different modal missing cases and improves the completion of missing information through this shared information, which is crucial for enhancing the accuracy and robustness of downstream tasks.

[0086] 3. More scientific use of cue information: This invention not only integrates specific cues for modal missing data by employing intra-modal and inter-modal attention modules, but also further integrates catalyst information for key modalities by enhancing activation attention, thereby improving the model's ability to process missing data and significantly enhancing the performance of processing multimodal MRI data and the accuracy of processing downstream tasks.

[0087] 4. More reasonable supplementation or enhancement of key modal information: The catalyst synthesizer module identifies and enhances the role of key modes, ensuring that the performance of downstream tasks can be maintained by the generated catalyst even when key modes are missing. This strategy significantly improves the model's adaptability and accuracy on incomplete modal data.

[0088] In summary, this invention provides a highly flexible and effective framework by comprehensively considering the collection, preprocessing, encoding, and missing modality handling of multimodal data, as well as through the design of innovative dual-branch encoders, attention mechanisms, and catalyst synthesizers. It can be used for incomplete modal MRI brain tumor segmentation and is also applicable to a wider range of incomplete modal data analysis tasks. Attached Figure Description

[0089] Figure 1 This is a schematic diagram of the structure of an incomplete modality brain tumor segmentation system based on cue-driven attention in an embodiment of the present invention;

[0090] Figure 2 This is a schematic diagram of the data processing flow of an incomplete modality brain tumor segmentation module in an embodiment of the present invention;

[0091] Figure 3 This is a schematic diagram of the structure of an incomplete modality brain tumor segmentation module in an embodiment of the present invention;

[0092] Figure 4 This is a schematic diagram of the structure of a modality missing prompt synthesizer according to an embodiment of the present invention;

[0093] Figure 5 This is a schematic diagram of a catalyst synthesizer according to an embodiment of the present invention. Detailed Implementation

[0094] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0095] To address the problems of existing technologies, this invention provides a cue-driven attention-based incomplete modality brain tumor segmentation system. This system aims to solve the performance degradation and poor generalization capabilities caused by existing methods for handling missing modalities, which fail to effectively utilize cue information and fully exploit the hidden knowledge of key modalities. Combined with... Figure 1 The incomplete modality brain tumor segmentation system based on cue-driven attention includes an image data acquisition module, a training sample dataset construction module, and an incomplete modality brain tumor segmentation module.

[0096] The image data acquisition module is used to acquire MRI image data of all modalities of the brain, as well as the segmentation map corresponding to the MRI image data of all modalities of the brain. The segmentation map corresponding to the MRI image data of all modalities of the brain includes brain tumor labels. It is also used to transmit the MRI image data of all modalities of the brain and the segmentation map corresponding to the MRI image data of all modalities of the brain to the training sample dataset construction module.

[0097] The study involves acquiring MRI image data for all brain modalities, including T1, T1c, FLAIR, and T2 modalities. One approach is to access public datasets, such as BraTS, TCIA, and ISLES, which are publicly available medical imaging datasets for academic research and teaching. Alternatively, collaboration with medical institutions can be used. If public datasets cannot meet specific research needs, collaboration with hospitals or other medical institutions can be considered. Another approach is to collect data autonomously using specialized MRI equipment and qualified medical personnel, ensuring rigorous ethical review and informed consent from participants.

[0098] The training sample dataset construction module receives MRI image data of all brain modalities and corresponding segmentation maps of the MRI image data of all brain modalities. It then performs modality missing processing on the MRI image data of all brain modalities to obtain modality-missing image data. This modality-missing image data is further preprocessed to obtain preprocessed modality data, and a modality missing mask is determined simultaneously. 0 The modality missing mask 0 The system represents the presence or absence of each modality in all modalities of the preprocessed image data; the preprocessed modal data and modality missing mask are used as input samples, and the segmentation maps corresponding to the MRI image data of all brain modalities are used as output samples; the input samples and output samples constitute training samples, and multiple training samples are used to construct a training sample dataset; the training sample dataset is then transmitted to the incomplete modality brain tumor segmentation module.

[0099] It should be noted that when constructing the training sample dataset, in order to simulate the possibility of missing MRI modalities in the real world, the following strategies are needed to generate missing modality data from the full modality data. Specifically, modality missing processing is performed on the MRI image data of all brain modalities. This includes: data corruption, which simulates modality missingness by introducing noise or reducing data quality in certain modalities, such as blurring images or reducing resolution; and random masking, which randomly selects certain modalities from a portion of the dataset for masking, such as dividing a dataset containing N modalities into 2... N -1 subsets, each containing only one combination of modes.

[0100] Preprocessing is performed on image data lacking modalities to obtain preprocessed modal data, specifically including:

[0101] (1) Image denoising: MRI images are often affected by various noises, such as Gaussian noise, salt and pepper noise, etc. Denoising methods include using median filtering, Gaussian filtering, nonlocal mean filtering and other techniques.

[0102] (2) Image cropping and adjustment: cropping the image can remove uninteresting areas, that is, background areas other than brain regions, thereby reducing the computational burden of subsequent processing. Adjusting the image size can ensure that the images input into the model have a consistent size.

[0103] (3) Intensity normalization: The intensity values ​​of MRI images can vary due to differences in scanning equipment. Intensity normalization reduces this variation by adjusting the intensity distribution of the image. Methods include z-score normalization (converting pixel values ​​to a distribution with a mean of 0 and a standard deviation of 1) or scaling the intensity to a fixed range (e.g., 0 to 1 or -1 to 1);

[0104] (4) Spatial standardization: Multiple images are aligned to a common spatial coordinate system for spatial standardization, which is accomplished by registering all images to a standard brain image template, such as the Montreal Neuroscience Institute MNI coordinate system.

[0105] (5) Off-field correction: MRI images are often affected by uneven magnetic field intensity distribution, resulting in uneven brightness in the image. Off-field correction can correct this intensity unevenness and restore the true tissue contrast of the image;

[0106] (6) Intracranial extraction: Removal of non-brain tissues, such as skull, skin and other extracranial tissues, from brain MRI images.

[0107] MRI image preprocessing is a crucial step to ensure data quality and consistency, laying the foundation for subsequent image analysis and machine learning model training. Data preprocessing is not only a fundamental task but also a key element in optimizing model performance, ensuring analysis quality, and simplifying model implementation. The correct preprocessing strategy can significantly improve the success rate of the entire project.

[0108] The incomplete modality brain tumor segmentation module is used to receive training sample datasets and train the incomplete modality object processing model based on the cue-driven attention mechanism based on the training sample datasets. The incomplete modality object processing model based on the cue-driven attention mechanism includes a dual-branch fusion encoder, a modality missing cue synthesizer, a cue-driven attention fusion generator, and a catalyst synthesizer.

[0109] Specifically, in combination Figure 2 and Figure 3 The preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modes in the preprocessed modal data; the fusion features of all modes in the preprocessed modal data and the mode missing mask are then combined. 0 The input is fed into the modality missing cues synthesizer to obtain modality missing specific cues P. spec ; The fusion features and mode missing mask of all modes in the preprocessed modal data 0 Inputting the catalyst synthesizer, we obtain the loss value L of the catalyst synthesizer's optimization objective function. con and catalyst F Act The fusion features of all modes in the preprocessed modal data, and the catalyst F output from the catalyst synthesizer are then analyzed. Act Modal Missing Mask 0 and the modality missing specific cue P spec The input is fed into a cue-driven attention fusion processor to obtain the enhanced activation feature representation F. cross ; representing enhanced activation characteristics as F cross The input is fed into the decoder to obtain a segmentation map corresponding to the preprocessed modality data. This segmentation map includes the predicted tumor label. Simultaneously, based on the output sample and the segmentation map corresponding to the preprocessed modality data, the decoder's loss value L is obtained. decoder ;

[0110] The dual-branch fusion encoder consists of two branches. The first branch adopts the encoding part of U-Net, reducing the spatial resolution of the feature map through a series of convolutional and downsampling layers while increasing the number of channels to extract more abstract features. It employs a five-level encoding structure, with each level containing two convolutional blocks. Each convolutional block consists of group normalization, a ReLU activation function, and a 3×3 kernel convolution. The stride of the downsampling operation is set to 2. The second branch uses a convolutional encoder that maintains constant resolution to avoid spatial information loss due to downsampling. It retains the basic design of the first branch but removes the downsampling step.

[0111] The dual-branch fusion encoder cross-fuses features at different levels of abstraction between two branches, which can effectively capture high-level semantic information while preserving key spatial information and local details. This is extremely important for subsequent segmentation tasks or other downstream tasks.

[0112] Based on this, a dual-branch fusion encoder is used to process the preprocessed modal data to obtain the fusion features of all modes in the preprocessed modal data, including:

[0113] For each modal data in the preprocessed modal data, the modal data is processed by the first branch of the dual-branch fusion encoder, specifically through the following formula:

[0114]

[0115] Among them, M m This refers to the modal data of the m-th mode in the preprocessed modal data. For M m The feature representation after the l-th layer of the first branch in the dual-branch fusion encoder. For M m The feature representation after the (l-1)th layer of the first branch in the dual-branch fusion encoder is DS, which represents the downsampling operation, and ConvBlock, which represents a convolutional block consisting of group normalization, ReLU activation function and 3×3 kernel size convolution.

[0116] For each modal data in the preprocessed modal data, the modal data is processed by the second branch of the dual-branch fusion encoder, specifically through the following formula:

[0117]

[0118] in, For M m The feature representation after the l-th layer of the second branch in the dual-branch fusion encoder. For M mFeature representation after the (l-1)th layer of the second branch in the dual-branch fusion encoder;

[0119] Furthermore, the feature maps of the two branches of the dual-branch fusion encoder are fused at the highest level through multi-scale cross-attention operations, enabling the model to simultaneously capture high-level semantic information and low-level spatial details. This mechanism unifies the extraction and fusion of multi-scale features into a single attention layer, with different attention heads focusing on different scales of the input features, effectively extracting and integrating coarse-grained and fine-grained features. Specifically, for M... m Feature representation after the 5th layer of the first branch in the dual-branch fusion encoder and M m Feature representation after the 5th layer of the second branch in the dual-branch fusion encoder Feature fusion is performed to obtain the fused feature F. m Specifically, this is achieved through the following formula:

[0120]

[0121] Among them, F m Let CA represent the fusion feature of the m-th modality;

[0122] in, (m=1,2,…,N M D represents the depth of the preprocessed modal data, W represents the width of the preprocessed modal data, H represents the height of the preprocessed modal data, and C represents the number of channels of the preprocessed modal data.

[0123] The modality missing information synthesizer addresses this issue by recognizing the undeniable correlations between different modal missing conditions in multimodal MRI. Accurately identifying and effectively utilizing these correlations can significantly improve the completion process of missing modal information, thereby enhancing the accuracy of downstream tasks. Considering the complexity and tediousness of manually establishing correlations between different modal missing conditions, this invention proposes a modality missing information synthesizer. This module uses a learnable cue called a general modality missing information cue to describe the correlations between different modal missing conditions and uses this cue to generate modality missing information-specific cuees for all modality missing conditions. This approach provides a new perspective and tool for understanding and processing missing conditions in multimodal data.

[0124] Among them, the modality missing mask 0 , For the Bernoulli index used to indicate the m-th modality, when the MRI image data of the m-th modality is absent in the MRI image data of all brain modalities, The value is 0 when the m-th modality of MRI image data exists in all modalities of brain MRI image data. The value of N is 1. M The total number of modes;

[0125] Based on this, the incomplete modality brain tumor segmentation module will use the fusion features of all modalities and the modality missing mask in the preprocessed modal data. 0 The input is fed into the modality missing cues synthesizer to obtain modality missing specific cues P. spec , combined Figure 4 ,include:

[0126] General prompts for modal missing Perform random initialization, the general hint P for the missing modality. uni It contains common information implied by all modal missing cases, wherein the present invention will include the general modal missing clue P. uni The first three dimensions are set to be the same as the first three dimensions of the feature representation output by the encoder, and the number of channels is set to the total number of missing modal cases, i.e. Where D represents the depth of the preprocessed modal data, W represents the width of the preprocessed modal data, and H represents the height of the preprocessed modal data. N P This represents the total number of all possible modal missing cases.

[0127] By cascading operations, the modality missing mask is... 0 The fusion features of all modes in the preprocessed modal data are merged to obtain... Specifically, this is expressed by the following formula:

[0128]

[0129] in, As an intermediate variable, Modality Missing Mask 0 The Bernoulli index used to indicate the first mode. Modality Missing Mask 0 The Bernoulli index is used to indicate the second mode. Modality Missing Mask 0 The middle is used to indicate the Nth M The Bernoulli indices for each modality, where F1 represents the fusion feature of the first modality and F2 represents the fusion feature of the second modality. For the Nth M Fusion features of each modality;

[0130] Through three fusion modules, intermediate variables are processed. General hints for modal missing P uni By splicing, we obtain N P The set of modality missing cues corresponding to each modality missing case Specifically, this is achieved through the following formula:

[0131]

[0132] Here, PF represents three consecutive fusion modules, each containing a convolutional layer, batch normalization, and a ReLU activation function. The convolutional layers are used to gradually adjust the feature dimensions, adjusting them to N. P Split represents segmenting features along the channel dimension to obtain N P A prompt with 1 channel; P1 represents the modality missing prompt corresponding to the first type of modality missing, and P2 represents the modality missing prompt corresponding to the second type of modality missing. Indicates the relationship with the Nth P Modality missing information corresponding to each modality missing condition.

[0133] Based on the modality missing mask 0 ,exist Select from the options to obtain a specific cue P indicating modality loss. spec Based on the modality missing mask 0 ,exist Select from the options to obtain a specific cue P indicating modality loss. spec In this invention, there are 4 modes, resulting in a total of 15 possible modes missing, i.e., N. P The value is 15, when the modality is missing mask. 0 When the value is [1,1,1,0], select The 14th missing state case, namely P 14 As a specific cue for modality absence P spec .

[0134] In this way, the model incorporates prior knowledge about the missing modalities, effectively improving the accuracy and robustness of downstream tasks.

[0135] Catalyst Synthesizer for a Specific Modality: This invention is based on the assumption that the T1c modality plays a more important role in enhanced tumor segmentation tasks within multimodal MRI data. A review of papers on multimodal MRI brain tumor segmentation methods reveals that in enhanced tumor regions, if the missing modality includes the T1c modality, the score is significantly lower than when the T1c modality is not missing. This invention refers to the T1c modality, which provides more crucial information, as the specific modality M. specConsidering that catalysts can provide the conditions for chemical reactions to occur and promote their progress, this invention, based on the concept of catalysts, proposes a catalyst synthesizer module. This module is designed for a specific mode M. spec Synthesis catalyst F Act The aim is to further explore and integrate M spec The information contained herein is used to address the issue of poor performance of downstream tasks when this modality is missing.

[0136] Considering that the input modality missing mask can be used to represent modality missing conditions, this invention will classify and discuss the levels of modality information missing. When M spec When it exists, its corresponding Mask is said to be the most informative; that is, as long as the T1c mode exists, the Mask is the most informative. spec When information is missing, the corresponding Mask is said to be of secondary richness. Furthermore, if in M... spec When a modality is missing, an additional existing modality is left blank; the corresponding mask is called the least informative. For example, consider a modality missing mask in the order T1, T1c, FLAIR, T2. If the modality missing mask is [1,1,1,0], meaning modality T1 exists, modality T1c exists, modality FLAIR exists, and modality T2 does not exist, then modality T1c exists, making this mask the most informative. If the modality missing mask is [1,0,1,0], meaning modality T1 exists, modality T1c does not exist, modality FLAIR exists, and modality T2 does not exist, then modality T1c does not exist, making this mask the second most informative. If the modality missing mask is [1,0,0,0], meaning modality T1 exists, modality T1c does not exist, modality FLAIR does not exist, modality T2 does not exist, then modality T1c does not exist, making this mask the least informative.

[0137] In determining the catalyst, this invention divides the training sample dataset into two groups: one for cases where the T1c mode exists in the input sample and the other for cases where the T1c mode does not exist. When the T1c mode exists in the input sample, the modal missing mask is the most informative. The value of the T1c mode in the modal missing mask is set to 0, resulting in a second-most informative modal missing mask. The catalyst is then determined based on the most informative and second-most informative modal missing masks. When the T1c mode does not exist in the input sample, the modal missing mask is the second-most informative. The value of one mode other than the T1c mode in the modal missing mask is set to 0, resulting in a least informative modal missing mask. The catalyst is then determined based on the second-most informative and least informative modal missing masks.

[0138] Based on this, the incomplete modality brain tumor segmentation module will use the fusion features of all modalities and the modality missing mask in the preprocessed modal data. 0 Inputting the catalyst synthesizer, we obtain the loss value L of the catalyst synthesizer's optimization objective function. con and catalyst F Act , combined Figure 5 ,include:

[0139] Determine the mask for missing modes 0 The Bernoulli index, used to indicate whether a specified mode is 1, is used in the mode missing mask. 0 When the Bernoulli index used to indicate a specified mode is 1, the catalyst F output from the catalyst synthesizer is determined according to the first method. Act In the modality missing mask 0 When the Bernoulli index used to indicate a specified mode is 0, the catalyst F output from the catalyst synthesizer is determined according to the second method. Act ;

[0140] The first method includes:

[0141] Modality missing specific cue P spec Determine the initial catalyst Y Act The Bernoulli index used to indicate a specified mode is set to 0 to obtain a new mode absence mask. 1 Furthermore, based on the new modality missing mask 1 The first catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. And calculate the initial catalyst Y Act and the first catalyst The loss value between them is expressed by the following formula:

[0142] Y Act =F spec ;

[0143]

[0144]

[0145] Here, MF represents three consecutive modules, each containing a convolutional layer, batch normalization, and a ReLU activation function, and MSE represents the mean squared error. For the new modality missing mask 1 The Bernoulli index used to indicate the first mode. For the new modality missing mask 1 The middle is used to indicate the Nth MBernoulli index for each modality, F1 is the fusion feature of the first modality. For the Nth M The fusion features of each modality, L con For the initial catalyst Y Act and the first catalyst The loss value between;

[0146] At this point, the initial catalyst Y... Act Catalyst F, as the output of the catalyst synthesizer Act ;

[0147] The second method includes:

[0148] Modality Missing Mask 0 Rename to Modality Missing Mask 1 Based on modality missing mask 1 The first catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. Randomly select modality missing mask 1 Take a Bernoulli index with a value of 1 that indicates a mode, reset its value to 0, and obtain a new mode missing mask. 2 According to the new modality missing mask 2 The second catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. And calculate the first catalyst Second catalyst The loss value between them is expressed by the following formula:

[0149]

[0150]

[0151]

[0152] in, For the new modality missing mask 2 The Bernoulli index used to indicate the first mode. For the new modality missing mask 2 Used to indicate the Nth M Bernoulli index for each modality, L con As the first catalyst Second catalyst The loss value between;

[0153] At this time, the first catalyst Catalyst F, as the output of the catalyst synthesizer Act .

[0154] In this way, the catalyst synthesizer can transfer knowledge between heterogeneous missing modalities, rather than being limited to fixed modal combinations as in traditional pairwise joint training. This strategy significantly improves the model's performance when specific modes are unavailable.

[0155] The cue-driven attention fusion processor includes an intra-modal attention module, an inter-modal attention module, and an enhanced activation attention module. Through these three attention modules, the cue-driven attention fusion processor progressively embeds cue information into the model, thereby improving the model's performance and robustness.

[0156] For cue-based learning, the configuration and placement of cues are crucial. Most methods typically inject cues into the input layer or shallow feature extraction layer to guide the model in performing downstream tasks. However, this approach can lead to the dilution or even loss of cue information at deeper levels of the model. To address this issue, this invention designs a cue-driven attention fusion processor.

[0157] Based on this, combined Figure 2 The fusion features of all modes in the preprocessed modal data, and the catalyst F output from the catalyst synthesizer are then used to... Act Modal Missing Mask 0 and the modality missing specific cue P spec The input is fed into a cue-driven attention fusion processor to obtain the enhanced activation feature representation F. cross ,include:

[0158] Within the intramodal attention module, the fused feature F m and modality missing specific cue P spec The process yields the intra-modal sequence; within the inter-modal attention module, the intra-modal sequence and the modal missing mask output by the intra-modal attention module are processed. 0 and modality missing specific cue P spec The process is performed to obtain the intermodal feature representation F. global In the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is... global Catalyst F output from the catalyst synthesizer Act And modality-specific cue P spec The process yields the enhanced activation feature representation F. cross .

[0159] The intramodal attention module is designed to fuse features F m Modal Missing Specific Cue P specThe information is fused through a cue-driven, multi-scale self-attention MSSAP. Based on this, within the intramodal attention module, the fused features F are... m and modality missing specific cue P spec The process yields the intramodal sequence, including:

[0160] F fusion feature F m Flattened into a shape of length L seq The sequence was analyzed, and features were obtained through linear projection and the addition of positional embeddings. P spec By performing convolution, flattening, and normalization operations with parameters as the first parameter, the key of the intramodal attention module is obtained. and the value of the intramodal attention module Features Key to intramodal attention modules and the value of the intramodal attention module After processing by multi-scale self-attention MSSAP and multilayer perceptron MLP, intramodal sequences are obtained. Specifically, this is achieved through the following formula:

[0161]

[0162]

[0163] Here, MLP represents a multilayer perceptron, which includes a linear input layer, a non-linear activation layer, and a linear output layer. m As intermediate variables, all Forming intramodal sequences.

[0164] Furthermore, the intermodal attention module aims to aggregate all pre-aggregated modal information and modality-specific cues. By fusing different modalities, the complementarity between them can be better utilized to explore local and global dependencies and improve model performance. Based on this, in the inter-modal attention module, the intra-modal sequence and modality missing mask output by the intra-modal attention module are processed. 0 and modality missing specific cue P spec The process is performed to obtain the intermodal feature representation F. global ,include:

[0165] By cascading operations, the modality missing mask is... 0 The intramodal sequence is merged with the intramodal sequence, and positional embeddings are added to obtain the intermodal sequence S. global Specifically, this is achieved through the following formula:

[0166]

[0167] Where Pos represents position embedding. For intramodal sequences, Modality Missing Mask 0 ;

[0168] Modality missing specific cue P spec By performing convolution, flattening, and normalization operations with the second parameter, the key of the intermodal attention module is obtained. The value of the intermodal attention module

[0169] The intermodal sequence S global Key to intermodal attention modules The value of the intermodal attention module After processing by multi-scale self-attention MSSAP and multilayer perceptron MLP, the intermodal feature representation F is obtained. global Specifically, it is expressed by the following formula:

[0170]

[0171] F global =MLP(y global )+y global ;

[0172] Among them, y global It is an intermediate variable.

[0173] Furthermore, the enhanced activation attention module is designed to integrate F, which incorporates information from all modalities. cross and catalyst F synthesized for specific modes Act Further integration. This invention refers to the mode that provides key information as a specific mode; the input of this module is F. global P spec And the catalyst F synthesized for this specific modal data Act The three are integrated through multimodal cross-attention (MMCAP) to output F. cross Based on this, in the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is... global Catalyst F output from the catalyst synthesizer Act And modality-specific cue P spec The process yields the enhanced activation feature representation F. cross ,include:

[0174] For the intermodal feature representation F global After flattening, deforming, and layer normalization, the enhanced activation attention sequence S is obtained. cross For catalyst F ActFlattening, deformation, and layer normalization were performed to obtain the catalyst sequence S. Act ;

[0175] Specific cue P for modality missing spec Perform convolution, flattening, and normalization operations with a third parameter to obtain the key for enhancing the activation attention module. and the value of the enhanced activation attention module

[0176] Will enhance the activation attention sequence S cross Catalyst sequence S Act , Enhance the activation of the attention module and the value of the enhanced activation attention module After processing by multimodal cross-attention (MMCAP) and multilayer perceptron (MLP), the enhanced activation feature representation F is obtained. cross Specifically, this is achieved through the following formula:

[0177]

[0178] F cross =MLP(y cross )+y cross ;

[0179] Among them, y cross It is an intermediate variable.

[0180] Loss value L obtained from the catalyst synthesizer con The loss value L of the decoder decoder The total loss value is obtained, specifically expressed by the following formula:

[0181] L total =L con +L decoder ;

[0182] Among them, L total This represents the total loss value;

[0183] During model training, the parameters of the incomplete modal object processing model based on the cue-driven attention mechanism are updated based on the total loss value, and finally the trained incomplete modal object processing model based on the cue-driven attention mechanism is obtained.

[0184] In specific implementation, the image data acquisition module is also used to acquire MRI image data of the brain with actual modal loss, i.e. Figure 2 The missing modal data of the actual task in the training sample dataset are obtained, and the MRI image data of the brain with real missing modalities are transmitted to the training sample dataset construction module.

[0185] The training sample dataset construction module is also used to preprocess the MRI image data of the brain with missing real modalities to obtain preprocessed MRI image data, and transmit the preprocessed MRI image data to the incomplete modal brain tumor segmentation module.

[0186] The incomplete modality brain tumor segmentation module is also used to receive preprocessed MRI image data and process the preprocessed MRI image data using a trained incomplete modality object processing model based on a cue-driven attention mechanism, to obtain a segmentation map corresponding to the MRI image data of the brain with actual modal missing features. Figure 2 The prediction results show that the segmentation maps corresponding to the MRI image data with missing actual brain modalities include labels for brain tumors.

[0187] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A cue-driven attention-based incomplete modality brain tumor segmentation system, characterized in that, It includes an image data acquisition module, a training sample dataset construction module, and an incomplete modality brain tumor segmentation module; The image data acquisition module is used to acquire MRI image data of all modalities of the brain, as well as the segmentation map corresponding to the MRI image data of all modalities of the brain. The segmentation map corresponding to the MRI image data of all modalities of the brain includes brain tumor labels. It is also used to transmit the MRI image data of all modalities of the brain and the segmentation map corresponding to the MRI image data of all modalities of the brain to the training sample dataset construction module. The training sample dataset construction module receives MRI image data of all brain modalities and corresponding segmentation maps of the MRI image data of all brain modalities. It then performs modality missing processing on the MRI image data of all brain modalities to obtain modality-missing image data. This modality-missing image data is further preprocessed to obtain preprocessed modality data, and a modality missing mask is determined simultaneously. The modality missing mask The system represents the presence or absence of each modality in all modalities of the preprocessed image data; the preprocessed modal data and modality missing mask are used as input samples, and the segmentation maps corresponding to the MRI image data of all brain modalities are used as output samples; the input samples and output samples constitute training samples, and multiple training samples are used to construct a training sample dataset; the training sample dataset is then transmitted to the incomplete modality brain tumor segmentation module. The incomplete modality brain tumor segmentation module is used to receive training sample datasets and train the incomplete modality object processing model based on cue-driven attention mechanism based on the training sample datasets. The incomplete modality object processing model based on cue-driven attention mechanism includes a dual-branch fusion encoder, a modality missing cue synthesizer, a cue-driven attention fusion generator, and a catalyst synthesizer. Specifically, the preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modes in the preprocessed modal data; the fusion features of all modes in the preprocessed modal data and the mode missing mask are then combined. Inputting the modality missing cues into the modality missing cues synthesizer produces modality missing cues. ; The fusion features and mode missing mask of all modes in the preprocessed modal data Input the catalyst synthesizer and obtain the loss value of the optimization objective function of the catalyst synthesizer. and catalyst The fusion features of all modes in the preprocessed modal data, and the catalyst output from the catalyst synthesizer are then analyzed. Modal Missing Mask and the specific prompt for the missing modality The input is fed into a cue-driven attention fusion processor to obtain an enhanced activation feature representation. ; representing enhanced activation characteristics The input is fed into the decoder to obtain a segmentation map corresponding to the preprocessed modality data. This segmentation map contains the predicted tumor label. Simultaneously, based on the output sample and the segmentation map corresponding to the preprocessed modality data, the decoder's loss value is obtained. ; Loss value obtained from catalyst synthesizer and the loss value of the decoder The total loss value is obtained, specifically expressed by the following formula: ； in, This represents the total loss value; During model training, the parameters of the incomplete modal object processing model based on the cue-driven attention mechanism are updated based on the total loss value, and finally the trained incomplete modal object processing model based on the cue-driven attention mechanism is obtained.

2. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that, In the specific implementation process, the image data acquisition module is also used to acquire MRI image data of the brain with real modal missingness, and transmit the MRI image data of the brain with real modal missingness to the training sample dataset construction module; The training sample dataset construction module is also used to preprocess the MRI image data of the brain with missing real modalities to obtain preprocessed MRI image data, and transmit the preprocessed MRI image data to the incomplete modal brain tumor segmentation module. The incomplete modality brain tumor segmentation module is also used to receive preprocessed MRI image data and process the preprocessed MRI image data through a trained incomplete modality object processing model based on cue-driven attention mechanism to obtain a segmentation map corresponding to the MRI image data with real modality missing of the brain. The segmentation map corresponding to the MRI image data with real modality missing of the brain includes the label of the brain tumor.

3. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that, The incomplete modality brain tumor segmentation module uses a dual-branch fusion encoder to process the preprocessed modal data, obtaining the fusion features of all modalities in the preprocessed modal data, including: For each modal data in the preprocessed modal data, the modal data is processed by the first branch of the dual-branch fusion encoder, specifically through the following formula: ； in, This refers to the modal data of the m-th mode in the preprocessed modal data. for The feature representation after the l-th layer of the first branch in the dual-branch fusion encoder. for Feature representation after the (l-1)th layer of the first branch in the dual-branch fusion encoder This indicates a downsampling operation. This represents a convolutional block consisting of grouped normalization, ReLU activation function, and a 3×3 kernel-sized convolution; For each modal data in the preprocessed modal data, the modal data is processed by the second branch of the dual-branch fusion encoder, specifically through the following formula: ； in, for The feature representation after the l-th layer of the second branch in the dual-branch fusion encoder. for Feature representation after the (l-1)th layer of the second branch in the dual-branch fusion encoder; right Feature representation after the 5th layer of the first branch in the dual-branch fusion encoder and Feature representation after the 5th layer of the second branch in the dual-branch fusion encoder Perform feature fusion to obtain fused features Specifically, this is achieved through the following formula: ； in, For the fusion feature of the m-th mode, This indicates a cross-attention operation; in, , This indicates the depth of the preprocessed modal data. This indicates the width of the preprocessed modal data. This indicates the height of the preprocessed modal data. This indicates the number of channels in the preprocessed modal data.

4. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that, Modal Missing Mask , For use in indicating the first Bernoulli index of the _th modality, in all modalities of brain MRI image data When MRI image data for a particular modality is unavailable, The value is 0, which is the [value] in all modalities of MRI image data of the brain. When MRI image data of each modality exists, The value is 1. The total number of modes; Based on this, the incomplete modality brain tumor segmentation module will incorporate the fusion features of all modalities and the modality missing mask from the preprocessed modal data. Inputting the modality missing cues into the modality missing cues synthesizer produces modality missing cues. ,include: General prompts for missing modes Perform random initialization; the general prompt for modality missing. It contains the common information implied by all modal missing cases, among which, This indicates the depth of the preprocessed modal data. This indicates the width of the preprocessed modal data. This indicates the height of the preprocessed modal data. , This represents the total number of all possible modal missing cases. Modal missing mask through cascading operations The fusion features of all modes in the preprocessed modal data are merged to obtain... Specifically, it is expressed by the following formula: ； in, As an intermediate variable, Modality Missing Mask The middle is used to indicate the first Bernoulli index for each modality, Modality Missing Mask The middle is used to indicate the first Bernoulli index for each modality, Modality Missing Mask The middle is used to indicate the first Bernoulli index for each modality, For the fusion features of the first modality, For the fusion features of the second modality, For the first Fusion features of each modality; Through three fusion modules, intermediate variables are processed. General prompts for modal missing By splicing, we can obtain the same as The set of modality missing cues corresponding to each modality missing case Specifically, this is achieved through the following formula: ； in, This represents three consecutive fusion modules, each containing a convolutional layer, batch normalization, and a ReLU activation function. This represents the segmentation features along the channel dimension to obtain... A message indicating that the number of channels is 1; This indicates a mode missing information corresponding to the first type of mode missing. This indicates a mode missing information corresponding to the second type of mode missing. Indicates the relationship with the first Modality missing information corresponding to each modality missing condition; Based on modality missing mask ,exist Select from the options to get specific prompts for modality loss. .

5. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that, Modal Missing Mask , For use in indicating the first Bernoulli index of the _th modality, in all modalities of brain MRI image data When MRI image data for a particular modality is unavailable, The value is 0, which is the [value] in all modalities of MRI image data of the brain. When MRI image data of each modality exists, The value is 1. The total number of modes; Based on this, the incomplete modality brain tumor segmentation module will incorporate the fusion features of all modalities and the modality missing mask from the preprocessed modal data. Input the catalyst synthesizer and obtain the loss value of the optimization objective function of the catalyst synthesizer. and catalyst ,include: Determine the mask for missing modes The Bernoulli index, used to indicate whether a specified mode is 1, is used in the mode missing mask. When the Bernoulli index used to indicate a specified mode is 1, the catalyst output from the catalyst synthesizer is determined according to the first method. In the modality missing mask When the Bernoulli index used to indicate a specified mode is 0, the catalyst output from the catalyst synthesizer is determined according to the second method. ; The first method includes: Modality missing specific prompt Determine the initial catalyst The Bernoulli index used to indicate the specified mode is set to 0 to obtain a new mode missing mask. Furthermore, based on the new modality missing mask The first catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. And calculate the initial catalyst and the first catalyst The loss value between them is expressed by the following formula: ； ; ; in, This represents three consecutive modules, each containing a convolutional layer, batch normalization, and a ReLU activation function. MSE represents the mean squared error. For the new modality missing mask The middle is used to indicate the first Bernoulli index for each modality, For the new modality missing mask The middle is used to indicate the first Bernoulli index for each modality, For the fusion features of the first modality, For the first The fusion features of each modality as the initial catalyst and the first catalyst The loss value between; At this point, the initial catalyst... The catalyst output from the catalyst synthesizer ; The second method includes: Modality missing mask Rename to Modality Missing Mask Based on modal missing mask The first catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. Randomly select a mode missing mask A Bernoulli index with a value of 1 used to indicate a mode is reset to 0 to obtain a new mode missing mask. According to the new modality missing mask The second catalyst was synthesized by combining the fusion features of all modes in the preprocessed modal data. And calculate the first catalyst Second catalyst The loss value between them is expressed by the following formula: ； ； ； in, For the new modality missing mask The middle is used to indicate the first Bernoulli index for each modality, For the new modality missing mask The middle is used to indicate the first Bernoulli index for each modality, As the first catalyst Second catalyst The loss value between; At this time, the first catalyst The catalyst output from the catalyst synthesizer .

6. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that, The cue-driven attention fusion module includes an intramodal attention module, an intermodal attention module, and an enhanced activation attention module; Based on this, the incomplete modality brain tumor segmentation module incorporates the fusion features of all modalities in the preprocessed modal data and the catalyst output from the catalyst synthesizer. Modal Missing Mask and the specific prompt for the missing modality The input is fed into a cue-driven attention fusion processor to obtain an enhanced activation feature representation. ,include: Within the intramodal attention module, the fused features Modality-specific prompts The process yields the intra-modal sequence; within the inter-modal attention module, the intra-modal sequence and the modal missing mask output by the intra-modal attention module are processed. Modality-specific prompts The process is performed to obtain the inter-modal feature representation. ; In the enhanced activation attention module, the inter-modal feature representation of the output of the inter-modal attention module is... Catalyst output from the catalyst synthesizer and specific prompts for modal absence The process is performed to obtain an enhanced activation feature representation. .

7. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 6, characterized in that, Within the intramodal attention module, the fused features Modality-specific prompts The process yields the intramodal sequence, including: Fusion features Flattened into a length of The sequence was analyzed, and features were obtained through linear projection and the addition of positional embeddings. ;Will By performing convolution, flattening, and normalization operations with parameters as the first parameter, the key of the intramodal attention module is obtained. and the value of the intramodal attention module ; Features Key to intramodal attention modules and the value of the intramodal attention module After multi-scale self-attention and multilayer perceptron The processing yields the intramodal sequence. Specifically, this is achieved through the following formula: ； ; in, This represents a multilayer perceptron, consisting of a linear input layer, a non-linear activation layer, and a linear output layer. As intermediate variables, all Forming intramodal sequences.

8. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 6, characterized in that, In the intermodal attention module, the intramodal sequence and modal missing mask output by the intramodal attention module are... Modality-specific prompts The process is performed to obtain the inter-modal feature representation. ,include: Modal missing mask through cascading operations Merge the intramodal sequences with the intermodal sequences and add positional embeddings to obtain the intermodal sequences. Specifically, this is achieved through the following formula: ； in, Position embedding, { , } represents an intramodal sequence, { } is a modality missing mask ; Modality missing specific prompt By performing convolution, flattening, and normalization operations with the second parameter, the key of the intermodal attention module is obtained. The value of the intermodal attention module ; intermodal sequences Key to intermodal attention modules The value of the intermodal attention module After multi-scale self-attention and multilayer perceptron The processing yields inter-modal feature representations. Specifically, it is expressed by the following formula: ； ； in, It is an intermediate variable.

9. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 6, characterized in that, In the enhanced activation attention module, the inter-modal feature representation output by the inter-modal attention module is... Catalyst output from the catalyst synthesizer and specific prompts for modal absence The process is performed to obtain an enhanced activation feature representation. ,include: Intermodal feature representation After flattening, deforming, and layer normalization, an enhanced activation attention sequence is obtained. , for catalyst Flattening, deformation, and layer normalization were performed to obtain the catalyst sequence. ; Specific prompts for modality missing Perform convolution, flattening, and normalization operations with a third parameter to obtain the key for enhancing the activation attention module. and the value of the enhanced activation attention module ; Will enhance activation attention sequence Catalyst sequence , Enhance the activation of the attention module and the value of the enhanced activation attention module After multimodal cross attention and multilayer perceptron The processing yields an enhanced activation feature representation. Specifically, this is achieved through the following formula: ； ； in, It is an intermediate variable.

10. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that, The training sample dataset construction module performs modality loss processing on MRI image data of all brain modalities to obtain modality-loss-free image data. This modality-loss-free image data is then preprocessed to obtain preprocessed modality data, specifically including: Data corruption and random masking were performed on MRI image data of all brain modalities to obtain image data with missing modalities. Then, image denoising, image cropping pre-adjustment, intensity normalization, spatial normalization, field correction and intracranial extraction were performed on the image data with missing modalities to obtain preprocessed modal data.

Images