Incomplete modal brain tumor segmentation system for driving attention based on prompt

Abstract

The invention provides an incomplete modal brain tumor segmentation system based on prompt-driven attention, and relates to the field of artificial intelligence, a double-branch fusion encoder extracts features of different abstraction levels through two branches, and can reserve detailed spatial information while extracting high-level semantics, so that the segmentation accuracy is improved. The mode loss prompt synthesizer generates a mode loss specific prompt aiming at the current mode combination condition through a learnable universal prompt, integrates the mode loss specific prompt by adopting an intra-mode attention module and an inter-mode attention module, further integrates catalyst information aiming at a key mode by enhancing activation attention, and further integrates the catalyst information aiming at the key mode. According to the method, the processing capacity of the model for missing data is improved, the performance of processing multi-mode MRI data and the accuracy of processing downstream tasks are remarkably improved, and the catalyst synthesizer module ensures that the performance of the downstream tasks can be maintained through the generated catalyst even under the condition that the key mode is missing by identifying and enhancing the effect of the key mode.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an incomplete modality brain tumor segmentation system based on cue-driven attention. Background Art

[0002] Brain tumors have attracted widespread attention due to their potential destructive effects on neurological function. Malignant tumors, in particular, may invade surrounding healthy tissues, leading to more severe neurological dysfunction and seriously threatening the patient's life safety. Therefore, accurate characterization of brain tumors is crucial for formulating treatment plans, assessing patient prognosis, and monitoring disease progression. In clinical practice, multimodal magnetic resonance imaging (MRI) plays an increasingly important role, and common modalities include T1-weighted (T1), enhanced T1-weighted (T1c), fluid-attenuated inversion recovery (FLAIR), and T2-weighted (T2) images. By integrating these multimodal data, clinicians can more comprehensively understand the biological characteristics and heterogeneity of tumors, thereby providing better personalized treatment and prognostic assessment.

[0003] Recently, multimodal methods based on deep learning have been developed and have shown excellent results in the field of brain tumor segmentation. Despite the impressive performance of these methods, they are mainly designed to process complete full-modality data. However, in clinical settings, the effectiveness of multimodal MRI in the diagnosis and treatment of brain tumors depends largely on the availability and quality of all relevant modalities. Due to technical issues such as device motion, hardware failure, or patient factors, some MRI modalities may be lost or degraded in quality. This situation, known as incomplete or missing modalities, poses a challenge to the accurate segmentation of brain tumors.

[0004] In response to this missing modality problem, the academic community has proposed a variety of solutions, which can be divided into two categories: one is to try to transfer the knowledge of the full modality model to the missing modality to make up for the lack of information. This method is usually called "co-training". Although these methods are effective, they usually require training dedicated models for each possible missing modality combination, which is not only resource-intensive, but also often fails to fully utilize the information provided by other modalities, thus affecting the overall performance of the model. Another method is to try to synthesize the missing modality data to achieve full modality segmentation. This method handles all missing modality situations through a unified model. Although this method seems logically perfect, it is often limited in its ability to reconstruct the missing modality and has a high computational cost in actual operation. In addition, none of the current methods pay attention to the particularity of the T1c modality. When the T1c modality is missing, the segmentation performance of the enhanced tumor area of ​​almost all models will drop significantly, which highlights the importance of the T1c modality in capturing key information of the enhanced tumor area.

[0005] In particular, whether in training a dedicated network or in training a unified network, many methods use prompts. The concept of prompts was originally introduced in the field of natural language processing, but recently many researchers have used prompts in computer vision and multimodal tasks and achieved ideal results. However, existing models that use prompts require manual intervention and cannot automatically synthesize the required prompts based on the lack of modalities. In addition, existing methods often directly synthesize corresponding prompts based on the modal combination, and fail to deeply explore the common information provided by different modal combinations, resulting in too little information in the synthesized prompts, which affects the performance of subsequent segmentation tasks.

[0006] Therefore, although the existing technology has made a lot of progress in the processing of incomplete modality MRI, there are still some problems to a greater or lesser extent. How to better integrate the prompting technology into the processing of incomplete modality objects and mine the hidden knowledge of key modalities has become a problem that needs to be solved urgently. If successful, it will significantly improve the accuracy and efficiency of brain tumor segmentation when certain MRI modality data is missing, thereby providing more reliable support in clinical diagnosis and treatment decisions. Summary of the invention

[0007] In view of the deficiencies of the prior art, the object of the present invention is to propose an incomplete modality brain tumor segmentation system based on cue-driven attention, comprising an image data acquisition module, a training sample data set construction module and an incomplete modality brain tumor segmentation module;

[0008] The image data acquisition module is used to obtain MRI image data of all brain modalities and segmentation maps corresponding to the MRI image data of all brain modalities, wherein the segmentation maps corresponding to the MRI image data of all brain modalities include brain tumor labels, and is also used to transmit the MRI image data of all brain modalities and the segmentation maps corresponding to the MRI image data of all brain modalities to the training sample data set construction module;

[0009] The training sample data set construction module is used to receive MRI image data of all brain modalities and segmentation maps corresponding to the MRI image data of all brain modalities, and perform modality missing processing on the MRI image data of all brain modalities to obtain modality missing image data, and then preprocess the modality missing image data to obtain the preprocessed modality data, and at the same time determine the modality missing mask Mask 0 , the modality missing mask Mask 0Characterize whether each modality in all modalities of the preprocessed image data exists; use the preprocessed modality data and the modality missing mask as input samples, and use the segmentation map corresponding to the MRI image data of all modalities of the brain as output samples; the input samples and output samples constitute training samples, and multiple training samples are constructed to obtain a training sample data set; transmit the training sample data set to the incomplete modality brain tumor segmentation module;

[0010] The incomplete modality brain tumor segmentation module is used to receive a training sample data set, and perform model training on the incomplete modality object processing model based on the prompt-driven attention mechanism based on the training sample data set, wherein the incomplete modality object processing model based on the prompt-driven attention mechanism includes a dual-branch fusion encoder, a modality missing prompt synthesizer, a prompt-driven attention fusion device, and a catalyst synthesizer;

[0011] Specifically, the preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modalities in the preprocessed modal data; the fusion features of all modalities in the preprocessed modal data and the modality missing mask Mask are combined. 0 Input to the modality missing prompt synthesizer to obtain the modality missing specific prompt P spec ; Combine the fusion features of all modalities and the modality missing mask Mask in the preprocessed modal data 0 , input the catalyst synthesizer, and obtain the loss value L of the optimization objective function of the catalyst synthesizer con and catalyst F Act ; The fusion features of all modes in the preprocessed modal data and the catalyst F output by the catalyst synthesizer Act , modality missing mask Mask 0 and the modality lacks specific prompt P spec , input to the cue-driven attention fusion device to obtain the enhanced activation feature representation F cross ; Enhance the activation feature representation F cross The segmentation map corresponding to the preprocessed modality data is input into the decoder to obtain the segmentation map corresponding to the preprocessed modality data, which contains the predicted tumor label. At the same time, based on the output sample and the segmentation map corresponding to the preprocessed modality data, the loss value L of the decoder is obtained. decoder ;

[0012] The loss value L obtained based on the catalyst synthesizer con and the decoder loss L decoder , and the total loss value is obtained, which is specifically expressed by the following formula:

[0013] L total =L con +Ldecoder ;

[0014] Among them, L total is the total loss value;

[0015] During the model training process, the parameters of the incomplete modal object processing model based on the prompt-driven attention mechanism are updated based on the total loss value, and finally a trained incomplete modal object processing model based on the prompt-driven attention mechanism is obtained.

[0016] Optionally, in a specific implementation process, the image data acquisition module is also used to acquire real modality-missing MRI image data of the brain, and transmit the real modality-missing MRI image data of the brain to the training sample data set construction module;

[0017] The training sample data set construction module is also used to preprocess the real modality-missing MRI image data of the brain to obtain preprocessed MRI image data, and transmit the preprocessed MRI image data to the incomplete modality brain tumor segmentation module;

[0018] The incomplete modality brain tumor segmentation module is also used to receive the preprocessed MRI image data, and process the preprocessed MRI image data through the trained incomplete modality object processing model based on the prompt-driven attention mechanism to obtain a segmentation map corresponding to the real modality-missing MRI image data of the brain. The segmentation map corresponding to the real modality-missing MRI image data of the brain includes a label of the brain tumor.

[0019] Optionally, in the incomplete modality brain tumor segmentation module, the preprocessed modality data is processed by a dual-branch fusion encoder to obtain fusion features of all modalities in the preprocessed modality data, including:

[0020] The modal data of each modality in the modal data after preprocessing is processed by the first branch in the dual-branch fusion encoder, which is specifically implemented by the following formula:

[0021]

[0022] Among them, M m is the modal data of the mth mode in the modal data after preprocessing, M m The feature representation after the lth layer of the first branch in the dual-branch fusion encoder is M mThe feature representation after the l-1th layer of the first branch in the dual-branch fusion encoder, DS represents the downsampling operation, and ConvBlock represents the convolution block consisting of group normalization, ReLU activation function and 3×3 kernel size convolution;

[0023] The modal data of each modality in the modal data after preprocessing is processed by the second branch in the dual-branch fusion encoder, which is specifically implemented by the following formula:

[0024]

[0025] in, M m The feature representation after the lth layer of the second branch in the dual-branch fusion encoder is M m Feature representation after the l-1th layer of the second branch in the dual-branch fusion encoder;

[0026] To M m Feature representation after the 5th layer of the first branch in the dual-branch fusion encoder and M m Feature representation after the 5th layer of the second branch in the dual-branch fusion encoder Perform feature fusion to obtain fusion feature F m , which is specifically achieved through the following formula:

[0027]

[0028] Among them, F m is the fusion feature of the m-th modality, CA represents the cross attention operation;

[0029] in, (m=1,2,…,N M ), D represents the depth of the modal data after preprocessing, W represents the width of the modal data after preprocessing, H represents the height of the modal data after preprocessing, and C represents the number of channels of the modal data after preprocessing.

[0030] Optional, modality missing mask Mask 0 , is a Bernoulli index for indicating the mth mode. When the MRI image data of the mth mode does not exist in the MRI image data of all the brain modes, The value of is 0. When the MRI image data of the mth modality exists in the MRI image data of all modalities of the brain, The value of N is 1, M is the total number of modes;

[0031] Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the modality missing mask Mask 0 Input to the modality missing prompt synthesizer to obtain the modality missing specific prompt P spec ,include:

[0032] General hint for modal missing Perform random initialization, the modality lacks the general prompt P uni Contains the common information implied by all modal missing situations, where D represents the depth of the modal data after preprocessing, W represents the width of the modal data after preprocessing, and H represents the height of the modal data after preprocessing. N P indicates the total number of all cases where the modality is missing;

[0033] Through the cascade operation, the modality missing mask Mask 0 Combine the fusion features of all modes in the preprocessed modal data to obtain It is specifically expressed by the following formula:

[0034]

[0035] in, is the intermediate variable, Mask for modal missing 0 The Bernoulli index used to indicate the first mode in , Mask for modal missing 0 The Bernoulli index used to indicate the second mode in , Mask for modal missing 0 Used to indicate the Nth M The Bernoulli index of each modality, F1 is the fusion feature of the first modality, and F2 is the fusion feature of the second modality. For N M The fusion features of the modalities;

[0036] Through three fusion modules, the intermediate variables and modality missing general prompt P uni Splice and get P A collection of modality missing prompts corresponding to the modality missing situations This is achieved specifically through the following formula:

[0037]

[0038] Among them, PF represents three consecutive fusion modules, each module contains a convolution layer, batch normalization and ReLU activation function, and Split represents the segmentation of features along the channel dimension to obtain N P The number of channels is 1; P1 represents the modality missing prompt corresponding to the first modality missing situation, and P2 represents the modality missing prompt corresponding to the second modality missing situation. Indicates that the Nth P The modality missing prompts corresponding to the modality missing situations;

[0039] Mask is missing according to the modality 0 ,exist Select from the list to get the modality missing specific prompt P spec .

[0040] Optional, modality missing mask Mask 0 , is a Bernoulli index for indicating the mth mode. When the MRI image data of the mth mode does not exist in the MRI image data of all the brain modes, The value of is 0. When the MRI image data of the mth modality exists in the MRI image data of all modalities of the brain, The value of N is 1, M is the total number of modes;

[0041] Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the modality missing mask Mask 0 , input the catalyst synthesizer, and obtain the loss value L of the optimization objective function of the catalyst synthesizer con and catalyst F Act ,include:

[0042] Determine the missing mask of the modality 0 Indicates whether the value of the Bernoulli index of the specified mode is 1, and the mask Mask is missing in the mode 0 When the value of the Bernoulli index indicating the specified mode is 1, the catalyst F output by the catalyst synthesizer is determined according to the first method. Act , in the modality missing mask Mask 0 When the value of the Bernoulli index indicating the specified mode is 0, the catalyst F output by the catalyst synthesizer is determined according to the second method. Act ;

[0043] The first method includes:

[0044] The modal missing specific prompt P spec Determine the initial catalyst YAct , set the value of the Bernoulli indicator used to indicate the specified mode to 0, and obtain the new mode missing mask Mask 1 , and then based on the new modality missing mask Mask 1 The first catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data And calculate the initial catalyst Y Act and the first catalyst The loss value between is specifically expressed by the following formula:

[0045] Y Act =F spec ;

[0046]

[0047]

[0048] Among them, MF represents three consecutive modules, each module contains a convolution layer, batch normalization and ReLU activation function, MSE represents mean square error, Missing mask for new modality 1 The Bernoulli index used to indicate the first mode in , Missing mask for new modality 1 Used to indicate the Nth M The Bernoulli index of the modalities, F1 is the fusion feature of the first modality, For N M The fusion features of the modalities, L con is the initial catalyst Y Act and the first catalyst The loss value between

[0049] At this time, the initial catalyst Y Act Catalyst F as output from the Catalyst Synthesizer Act ;

[0050] The second method includes:

[0051] The modal missing mask Mask 0 Renamed to modality missing mask Mask 1 , based on the modality missing mask Mask 1 The first catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data Randomly select the modality missing mask Mask 1 A Bernoulli indicator with a value of 1 in the modality is reset to 0 to obtain a new modality missing mask Mask 2 , according to the new modality missing mask Mask2 The second catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data And calculate the first catalyst and the second catalyst The loss value between is specifically expressed by the following formula:

[0052]

[0053]

[0054]

[0055] in, Missing mask for new modality 2 The Bernoulli index used to indicate the first mode in , Missing mask for new modality 2 Used to indicate the Nth M The Bernoulli index of the mode, L con The first catalyst and the second catalyst The loss value between

[0056] At this time, the first catalyst Catalyst F as output from the Catalyst Synthesizer Act .

[0057] Optionally, the cue-driven attention fuser includes an intra-modality attention module, an intra-modality attention module, and an enhanced activation attention module;

[0058] Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the catalyst F output by the catalyst synthesizer. Act , modality missing mask Mask 0 and the modality lacks specific prompt P spec , input to the cue-driven attention fusion device to obtain the enhanced activation feature representation F cross ,include:

[0059] In the intra-modal attention module, the fusion feature F m and modality-specific missing prompt P spec Processing is performed to obtain an intra-modal sequence; in the inter-modal attention module, the intra-modal sequence output by the intra-modal attention module and the modality missing mask Mask 0 and modality-specific missing prompt P spec Processing is performed to obtain the inter-modal feature representation F global; In the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module global , the catalyst F output by the catalyst synthesizer Act , and the modality-specific missing hint P spec After processing, the enhanced activation feature representation F is obtained cross .

[0060] Optionally, in the intra-modal attention module, the fusion feature F m and modality-specific missing prompt P spec Processing is performed to obtain the intra-modal sequence, including:

[0061] The fusion feature F m Flatten into a length of L seq The sequence is then embedded by linear projection and positional embedding. P spec The key of the intra-modal attention module is obtained through the convolution operation with the first parameter, flattening operation, and normalization operation. and the value of the intra-modal attention module The characteristics Keys for intra-modal attention modules and the value of the intra-modal attention module After being processed by multi-scale self-attention MSSAP and multi-layer perceptron MLP, the intra-modal sequence is obtained This is achieved specifically through the following formula:

[0062]

[0063]

[0064] Among them, MLP represents a multi-layer perceptron, including a linear input layer, a nonlinear activation layer and a linear output layer, y m is an intermediate variable, all Composition of intra-modal sequences.

[0065] Optionally, in the inter-modality attention module, the intra-modality sequence and the modality missing mask Mask output by the intra-modality attention module are 0 and modality-specific missing prompt P spec Processing is performed to obtain the inter-modal feature representation F global ,include:

[0066] Through the cascade operation, the modality missing mask Mask 0 Merge with the intra-modal sequence and add position embedding to obtain the inter-modal sequence S global , which is specifically achieved through the following formula:

[0067]

[0068] Among them, Pos represents position embedding, is the intra-modal sequence, Mask for modal missing 0 ;

[0069] The modal missing specific prompt P spec The key of the inter-modal attention module is obtained through the convolution operation with the second parameter, flattening operation, and normalization operation. and the value of the inter-modal attention module

[0070] The inter-modal sequence S global , the key of the inter-modal attention module and the value of the inter-modal attention module After being processed by multi-scale self-attention MSSAP and multi-layer perceptron MLP, the inter-modal feature representation F is obtained. global , which is specifically expressed by the following formula:

[0071]

[0072] F global =MLP(y global )+y global ;

[0073] Among them, y global is an intermediate variable.

[0074] Optionally, in the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is global , the catalyst F output by the catalyst synthesizer Act , and the modality-specific missing hint P spec After processing, the enhanced activation feature representation F is obtained cross ,include:

[0075] For the inter-modal feature representation F global Flatten, deform, and normalize the layers to obtain the enhanced activation attention sequence S cross , for catalyst F Act Flatten, deform, and normalize the layers to obtain the catalyst sequence S Act ;

[0076] Missing specific hints for modality P spec Perform convolution operation, flattening operation, and normalization operation with the third parameter as the parameter to obtain the key of the enhanced activation attention module and enhance the value of the activated attention module

[0077] The activation attention sequence S will be enhanced cross , catalyst sequence S Act , enhance the key to activate the attention module and enhance the value of the activated attention module After processing by multimodal cross attention MMCAP and multi-layer perceptron MLP, the enhanced activation feature representation F is obtained. cross , which is specifically achieved through the following formula:

[0078]

[0079] F cross =MLP(y cross )+y cross ;

[0080] Among them, y cross is an intermediate variable.

[0081] Optionally, in the training sample data set construction module, modality missing processing is performed on the MRI image data of all modalities of the brain to obtain modality missing image data, and then preprocessing is performed on the modality missing image data to obtain the modality data after preprocessing, specifically including:

[0082] The MRI image data of all brain modalities are subjected to data corruption and random shielding to obtain the image data with missing modalities. The image data with missing modalities are then subjected to image denoising, image cropping pre-adjustment, intensity normalization, spatial normalization, bias field correction and intracranial extraction in sequence to obtain the modal data after preprocessing.

[0083] The beneficial effects of adopting the above technical solution are:

[0084] 1. More effective feature extraction: The dual-branch fusion encoder extracts features at different levels of abstraction through two branches, which can retain detailed spatial information while extracting high-level semantics, effectively balancing the needs of capturing details and abstract features, which is particularly critical for complex multimodal analysis.

[0085] 2. Better prompt generation method: The modality missing prompt synthesizer generates modality missing specific prompts for the current modality combination based on the modality combination of the input data through learnable general prompts. This method can effectively utilize the shared information between different modality missing situations and improve the completion of missing information through this shared information, which is crucial to improving the accuracy and robustness of downstream tasks.

[0086] 3. More scientific use of prompt information: The present invention not only integrates the missing modality-specific prompts by adopting intra-modality and inter-modality attention modules, but also further integrates the catalyst information for key modalities by enhancing activation attention, thereby improving the model's ability to handle missing data, and significantly improving the performance of processing multimodal MRI data and the accuracy of processing downstream tasks.

[0087] 4. More reasonably complete or enhance key modal information: The catalyst synthesizer module identifies and strengthens the role of key modalities, ensuring that the performance of downstream tasks can be maintained through the generated catalysts even when key modalities are missing. This strategy significantly improves the model's adaptability and accuracy on incomplete modal data.

[0088] In summary, the present invention provides a highly flexible and effective framework by comprehensively considering the collection, preprocessing, encoding and missing modality processing of multimodal data, as well as through the design of innovative dual-branch encoder, attention mechanism, and catalyst synthesizer. It can be used for incomplete modality MRI brain tumor segmentation as well as for a wider range of incomplete modality data analysis tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 Schematic diagram of the structure of an incomplete modality brain tumor segmentation system based on cue-driven attention in an embodiment of the present invention;

[0090] Figure 2 Schematic diagram of a data processing flow of an incomplete modality brain tumor segmentation module in an embodiment of the present invention;

[0091] Figure 3 is a schematic structural diagram of an incomplete modality brain tumor segmentation module in an embodiment of the present invention;

[0092] Figure 4 is a structural schematic diagram of a modality missing prompt synthesizer in an embodiment of the present invention;

[0093] Figure 5 It is a schematic structural diagram of a catalyst synthesizer in an embodiment of the present invention. DETAILED DESCRIPTION

[0094] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0095] In view of the problems existing in the prior art, the present invention provides an incomplete modality brain tumor segmentation system based on cue-driven attention, so as to solve the problem that the existing methods for processing missing modalities fail to reasonably utilize cue information and fully explore the hidden knowledge of key modalities, resulting in performance degradation and poor generalization ability. Figure 1 ,The incomplete modality brain tumor segmentation system based on cue-driven attention includes an image data acquisition module, a training sample dataset building module and an incomplete modality brain tumor segmentation module;

[0096] The image data acquisition module is used to obtain MRI image data of all brain modalities and segmentation maps corresponding to the MRI image data of all brain modalities, wherein the segmentation maps corresponding to the MRI image data of all brain modalities include brain tumor labels, and is also used to transmit the MRI image data of all brain modalities and the segmentation maps corresponding to the MRI image data of all brain modalities to the training sample data set construction module;

[0097] Among them, all brain modalities include T1, T1c, FLAIR, and T2 modalities. MRI image data of all brain modalities are obtained. In one implementation method, public data sets can be accessed and obtained from public data sets. Many research institutions and universities provide public medical imaging data sets for academic research and teaching, such as BraTS, TCIA, and ISLES; they can be obtained in cooperation with medical institutions. If public data sets cannot meet specific research needs, you can consider cooperating with hospitals or other medical institutions. In another implementation method, you can collect data independently, using professional MRI equipment and qualified medical personnel to perform scanning procedures, and collect data independently under the guarantee of strict ethical review and informed consent of the participants.

[0098] The training sample data set construction module is used to receive MRI image data of all brain modalities and segmentation maps corresponding to the MRI image data of all brain modalities, and perform modality missing processing on the MRI image data of all brain modalities to obtain modality missing image data, and then preprocess the modality missing image data to obtain the preprocessed modality data, and at the same time determine the modality missing mask Mask 0 , the modality missing mask Mask 0 Characterize whether each modality in all modalities of the preprocessed image data exists; use the preprocessed modality data and the modality missing mask as input samples, and use the segmentation map corresponding to the MRI image data of all modalities of the brain as output samples; the input samples and output samples constitute training samples, and multiple training samples are constructed to obtain a training sample data set; transmit the training sample data set to the incomplete modality brain tumor segmentation module;

[0099] It should be noted that when constructing the training sample dataset, in order to simulate the situation in the real world where some MRI modalities may be missing, the following strategies need to be adopted to generate missing modal data from the full modality data, that is, to perform modality missing processing on the MRI image data of all modalities of the brain, including: data corruption, simulating modality missing by introducing noise in some modalities or reducing data quality, such as blurring the image or reducing the resolution; random masking, randomly selecting some modalities in a part of the dataset for masking, such as dividing a dataset containing N modalities into 2 N -1 subsets, each containing only one mode combination.

[0100] Preprocess the image data with missing modalities to obtain preprocessed modal data, including:

[0101] (1) Image denoising: MRI images are often affected by various noises, such as Gaussian noise, salt and pepper noise, etc. Denoising methods include using median filtering, Gaussian filtering, non-local mean filtering, etc.

[0102] (2) Image cropping and adjustment: cropping images can remove areas of no interest, that is, background areas other than the brain area, thereby reducing the computational burden of subsequent processing. Adjusting the image size can ensure that the images input to the model have a consistent size.

[0103] (3) Intensity normalization: The intensity values ​​of MRI images can vary due to differences in scanning equipment. Intensity normalization reduces this variation by adjusting the intensity distribution of the image. Methods include z-score normalization (converting pixel values ​​to a distribution with a mean of 0 and a standard deviation of 1) or scaling the intensity to a fixed range (e.g., 0 to 1 or -1 to 1);

[0104] (4) spatial normalization, aligning multiple images to a common spatial coordinate system for spatial normalization, which is accomplished by registering all images to a standard brain image template, such as the Montreal Neurological Institute (MNI) coordinate system;

[0105] (5) Bias field correction: MRI images are often affected by uneven magnetic field intensity distribution, resulting in uneven brightness in the image. Bias field correction can correct this intensity unevenness and restore the true tissue contrast of the image;

[0106] (6) Intracranial extraction, which removes non-brain tissues such as skull, skin, and other extracranial tissues from brain MRI images.

[0107] Preprocessing of MRI images is a key step to ensure data quality and consistency, laying a solid foundation for subsequent image analysis and machine learning model training. Data preprocessing is not only basic work, but also a key link in optimizing model performance, ensuring analysis quality, and simplifying model implementation. The correct preprocessing strategy can significantly improve the success rate of the entire project.

[0108] The incomplete modality brain tumor segmentation module is used to receive a training sample data set, and perform model training on the incomplete modality object processing model based on the prompt-driven attention mechanism based on the training sample data set, wherein the incomplete modality object processing model based on the prompt-driven attention mechanism includes a dual-branch fusion encoder, a modality missing prompt synthesizer, a prompt-driven attention fusion device, and a catalyst synthesizer;

[0109] Specific, combined Figure 2 and Figure 3 , the preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modalities in the preprocessed modal data; the fusion features of all modalities in the preprocessed modal data and the modality missing mask Mask 0 Input to the modality missing prompt synthesizer to obtain the modality missing specific prompt P spec ; Combine the fusion features of all modalities and the modality missing mask Mask in the preprocessed modal data 0 , input the catalyst synthesizer, and obtain the loss value L of the optimization objective function of the catalyst synthesizer con and catalyst F Act ; The fusion features of all modes in the preprocessed modal data and the catalyst F output by the catalyst synthesizer Act , modality missing mask Mask 0 and the modality lacks specific prompt P spec , input to the cue-driven attention fusion device to obtain the enhanced activation feature representation F cross ; Enhance the activation feature representation F cross The segmentation map corresponding to the preprocessed modality data is input into the decoder to obtain the segmentation map corresponding to the preprocessed modality data, which contains the predicted tumor label. At the same time, based on the output sample and the segmentation map corresponding to the preprocessed modality data, the loss value L of the decoder is obtained. decoder ;

[0110] The dual-branch fusion encoder consists of two branches. The first branch adopts the encoding part of U-Net, which reduces the spatial resolution of the feature map through a series of convolution and downsampling layers, while increasing the number of channels to extract more abstract features. It adopts a five-level encoding structure, each level contains two convolution blocks, and each convolution block consists of group normalization, ReLU activation function and convolution with a kernel size of 3×3. The step size of the downsampling operation is set to 2. The second branch adopts a convolution encoder that maintains a constant resolution to avoid the loss of spatial information caused by downsampling, and it retains the basic design of the first branch, but removes the downsampling step.

[0111] The two branches in the dual-branch fusion encoder cross-fuse features at different levels of abstraction, which can effectively capture high-level semantic information while retaining key spatial information and local detail information, which is extremely important for subsequent segmentation tasks or other downstream tasks.

[0112] Based on this, the preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modalities in the preprocessed modal data, including:

[0113] The modal data of each modality in the modal data after preprocessing is processed by the first branch in the dual-branch fusion encoder, which is specifically implemented by the following formula:

[0114]

[0115] Among them, M m is the modal data of the mth mode in the modal data after preprocessing, M m The feature representation after the lth layer of the first branch in the dual-branch fusion encoder is M m The feature representation after the l-1th layer of the first branch in the dual-branch fusion encoder, DS represents the downsampling operation, and ConvBlock represents the convolution block consisting of group normalization, ReLU activation function and 3×3 kernel size convolution;

[0116] The modal data of each modality in the modal data after preprocessing is processed by the second branch in the dual-branch fusion encoder, which is specifically implemented by the following formula:

[0117]

[0118] in, M m The feature representation after the lth layer of the second branch in the dual-branch fusion encoder is M mFeature representation after the l-1th layer of the second branch in the dual-branch fusion encoder;

[0119] Furthermore, the feature maps of the two branches of the dual-branch fusion encoder are fused at the highest level through a multi-scale cross-attention operation, enabling the model to capture both high-level semantic information and low-level spatial details. This mechanism unifies the extraction and fusion of multi-scale features into one attention layer, with different attention heads focusing on different scales of the input features, effectively extracting and integrating coarse-grained and fine-grained features. Specifically, for M m Feature representation after the 5th layer of the first branch in the dual-branch fusion encoder and M m Feature representation after the 5th layer of the second branch in the dual-branch fusion encoder Perform feature fusion to obtain fusion feature F m , which is specifically achieved through the following formula:

[0120]

[0121] Among them, F m is the fusion feature of the m-th modality, CA represents the cross attention operation;

[0122] in, (m=1,2,…,N M ), D represents the depth of the modal data after preprocessing, W represents the width of the modal data after preprocessing, H represents the height of the modal data after preprocessing, and C represents the number of channels of the modal data after preprocessing.

[0123] Among them, modality missing prompt synthesizer: There is undoubtedly a correlation between different modality missing situations in multimodal MRI. By accurately identifying and effectively utilizing these correlations, the process of completing the missing modality information can be significantly improved, thereby improving the accuracy of downstream tasks. Considering the complexity and tediousness of manually establishing the correlation between different modality missing situations, the present invention proposes a modality missing prompt synthesizer. This module uses a learnable prompt called a modality missing general prompt to describe the correlation between different modality missing situations, and uses this prompt to generate modality missing specific prompts for all modality missing situations. This method provides new perspectives and tools for understanding and processing missing situations in multimodal data.

[0124] Among them, the modality missing mask Mask 0 , is a Bernoulli index for indicating the mth mode. When the MRI image data of the mth mode does not exist in the MRI image data of all the brain modes, The value of is 0. When the MRI image data of the mth modality exists in the MRI image data of all modalities of the brain, The value of N is 1, M is the total number of modes;

[0125] Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the modality missing mask Mask 0 Input to the modality missing prompt synthesizer to obtain the modality missing specific prompt P spec , combined with Figure 4 ,include:

[0126] General hint for modal missing Perform random initialization, the modality lacks the general prompt P uni Contains the common information contained in all modality loss situations, wherein the present invention uses the modality loss general prompt P uni The first three dimensions of are set to be the same as the first three dimensions of the feature representation output by the encoder, and the number of channels is set to the total number of missing modal cases, i.e. Wherein, D represents the depth of the modal data after preprocessing, W represents the width of the modal data after preprocessing, and H represents the height of the modal data after preprocessing. N P indicates the total number of all cases where the modality is missing;

[0127] Through the cascade operation, the modality missing mask Mask 0 Combine the fusion features of all modes in the preprocessed modal data to obtain It is specifically expressed by the following formula:

[0128]

[0129] in, is the intermediate variable, Mask for modal missing 0 The Bernoulli index used to indicate the first mode in , Mask for modal missing 0 The Bernoulli index used to indicate the second mode in , Mask for modal missing 0 Used to indicate the Nth M The Bernoulli index of each modality, F1 is the fusion feature of the first modality, and F2 is the fusion feature of the second modality. For N M The fusion features of the modalities;

[0130] Through three fusion modules, the intermediate variables and modality missing general prompt P uni Splice and get P A collection of modality missing prompts corresponding to the modality missing situations This is achieved specifically through the following formula:

[0131]

[0132] Among them, PF represents three consecutive fusion modules, each module contains a convolution layer, batch normalization and ReLU activation function. The convolution layer is used to gradually adjust the feature dimension to N. P , Split means splitting the features along the channel dimension to obtain N P The number of channels is 1; P1 represents the modality missing prompt corresponding to the first modality missing situation, and P2 represents the modality missing prompt corresponding to the second modality missing situation. Indicates that the Nth P The modality missing prompts corresponding to the modality missing situations.

[0133] Mask is missing according to the modality 0 ,exist Select from the list to get the modality missing specific prompt P spec , according to the modality missing mask Mask 0 ,exist Select from the list to get the modality missing specific prompt P spec , where the present invention has 4 modes, and there are 15 types of mode missing conditions, namely N P The value is 15, when the modality lacks the mask 0 When [1,1,1,0], select The 14th state is missing, that is, P 14 , as a modality-specific missing hint P spec .

[0134] In this way, the model incorporates prior knowledge about the missing modality, effectively improving the accuracy and robustness of downstream tasks.

[0135] Catalyst synthesizer for specific modality: This invention is based on the assumption that T1c modality plays a more important role in enhancing tumor segmentation tasks in multimodal MRI data. By summarizing the papers on multimodal MRI brain tumor segmentation methods, it can be found that in the enhanced tumor area, if the missing modality includes T1c modality, the score will be significantly lower than the case where there is no missing T1c modality. This invention calls T1c, which provides more critical information, a specific modality M. specConsidering that in chemical reactions, catalysts can provide conditions for chemical reactions to occur and promote the reaction, the present invention proposes a catalyst synthesizer module based on the concept of catalyst. The module targets a specific mode M spec Synthesis of Catalyst F Act , aims to further explore and integrate M spec This can solve the problem of poor performance of downstream tasks when this modality is missing.

[0136] Considering that the input modality missing mask Mask can be used to indicate the modality missing situation, the present invention will discuss the level of modality information missing by classification. spec When M exists, the corresponding Mask is said to be the most informative. That is, as long as the T1c mode exists, the Mask is the most informative. spec When it is missing, the corresponding Mask is said to be the second richest information. Furthermore, if spec When missing, another existing modality is set to zero, and its corresponding Mask is said to be the most information-poor. For example, taking the modality missing mask Mask with the modality order of T1, T1c, FLAIR, and T2 as an example, if the modality missing mask Mask is [1,1,1,0], that is, the T1 modality exists, the T1c modality exists, the FLAIR modality exists, and the T2 modality does not exist, and the T1c modality exists at this time, then the Mask is the most information-rich. If the modality missing mask Mask is [1,0,1,0], that is, the T1 modality exists, the T1c modality does not exist, the FLAIR modality exists, and the T2 modality does not exist, and the T1c modality does not exist at this time, then the Mask is the second most information-rich. If the modality missing mask Mask is [1,0,0,0], that is, the T1 modality exists, the T1c modality does not exist, the FLAIR modality does not exist, and the T2 modality does not exist, and the T1c modality does not exist at this time, then the Mask is the least information-rich.

[0137] When determining the catalyst, the present invention divides the training sample data set into two groups, when the T1c mode exists in the input sample and when the T1c mode does not exist in the input sample, when the T1c mode exists in the input sample, the modality missing mask is the most information-rich at this time, and the value of the T1c mode in the modality missing mask is set to 0 to obtain the modality missing mask with the second richest information, and then determine the catalyst based on the modality missing mask with the richest information and the modality missing mask with the second richest information; when the T1c mode does not exist in the input sample, the modality missing mask is the second richest information at this time, and the value of one mode except the T1c mode in the modality missing mask is set to 0 to obtain the modality missing mask with the poorest information, and then determine the catalyst based on the modality missing mask with the second richest information and the modality missing mask with the poorest information.

[0138] Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the modality missing mask Mask 0 , input the catalyst synthesizer, and obtain the loss value L of the optimization objective function of the catalyst synthesizer con and catalyst F Act , combined with Figure 5 ,include:

[0139] Determine the missing mask of the modality 0 Indicates whether the value of the Bernoulli index of the specified mode is 1, and the mask Mask is missing in the mode 0 When the value of the Bernoulli index indicating the specified mode is 1, the catalyst F output by the catalyst synthesizer is determined according to the first method. Act , in the modality missing mask Mask 0 When the value of the Bernoulli index indicating the specified mode is 0, the catalyst F output by the catalyst synthesizer is determined according to the second method. Act ;

[0140] The first method includes:

[0141] The modal missing specific prompt P spec Determine the initial catalyst Y Act , set the value of the Bernoulli indicator used to indicate the specified mode to 0, and obtain the new mode missing mask Mask 1 , and then based on the new modality missing mask Mask 1 The first catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data And calculate the initial catalyst Y Act and the first catalyst The loss value between is specifically expressed by the following formula:

[0142] Y Act =F spec ;

[0143]

[0144]

[0145] Among them, MF represents three consecutive modules, each module contains a convolution layer, batch normalization and ReLU activation function, MSE represents mean square error, Missing mask for new modality 1 The Bernoulli index used to indicate the first mode in , Missing mask for new modality 1 Used to indicate the Nth MThe Bernoulli index of the modalities, F1 is the fusion feature of the first modality, For N M The fusion features of the modalities, L con is the initial catalyst Y Act and the first catalyst The loss value between

[0146] At this time, the initial catalyst Y Act Catalyst F as output from the Catalyst Synthesizer Act ;

[0147] The second method includes:

[0148] The modal missing mask Mask 0 Renamed to modality missing mask Mask 1 , based on the modality missing mask Mask 1 The first catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data Randomly select the modality missing mask Mask 1 A Bernoulli indicator with a value of 1 in the modality is reset to 0 to obtain a new modality missing mask Mask 2 , according to the new modality missing mask Mask 2 The second catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data And calculate the first catalyst and the second catalyst The loss value between is specifically expressed by the following formula:

[0149]

[0150]

[0151]

[0152] in, Missing mask for new modality 2 The Bernoulli index used to indicate the first mode in , Missing mask for new modality 2 Used to indicate the Nth M The Bernoulli index of the mode, L con The first catalyst and the second catalyst The loss value between

[0153] At this time, the first catalyst Catalyst F as output from the Catalyst Synthesizer Act .

[0154] In this way, Catalyst Synthesizer is able to transfer knowledge between heterogeneous missing modal cases, rather than being limited to fixed modality combinations as in traditional pairwise joint training. This strategy significantly improves the performance of the model when a specific modality is not available.

[0155] The cue-driven attention fuser includes an intra-modality attention module, an inter-modality attention module, and an enhanced activation attention module. Through the three attention modules of intra-modality attention, inter-modality attention, and enhanced activation attention, the cue-driven attention fuser gradually embeds the cue information into the model, thereby improving the performance and robustness of the model.

[0156] For cue-based learning, the configuration and location of cues are crucial. Most methods usually inject cues into the input layer or shallow feature extraction layer to guide the model to perform downstream tasks. However, this approach may cause the cue information to be diluted or even lost at deeper levels of the model. To solve this problem, the present invention designs a cue-driven attention fuser.

[0157] Based on this, combined Figure 2 , the fusion features of all modes in the preprocessed modal data and the catalyst F output by the catalyst synthesizer Act , modality missing mask Mask 0 and the modality lacks specific prompt P spec , input to the cue-driven attention fusion device to obtain the enhanced activation feature representation F cross ,include:

[0158] In the intra-modal attention module, the fusion feature F m and modality-specific missing prompt P spec Processing is performed to obtain an intra-modal sequence; in the inter-modal attention module, the intra-modal sequence output by the intra-modal attention module and the modality missing mask Mask 0 and modality-specific missing prompt P spec Processing is performed to obtain the inter-modal feature representation F global ; In the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module global , the catalyst F output by the catalyst synthesizer Act , and the modality-specific missing hint P spec After processing, the enhanced activation feature representation F is obtained cross .

[0159] The intra-modal attention module aims to integrate the fusion feature F m With modality-specific missing prompt P specThe information of is fused through a cue-driven multi-scale self-attention MSSAP. Based on this, in the intra-modality attention module, the fused feature F m and modality-specific missing prompt P spec Processing is performed to obtain the intra-modal sequence, including:

[0160] The fusion feature F m Flatten into a length of L seq The sequence is then embedded by linear projection and positional embedding. P spec The key of the intra-modal attention module is obtained through the convolution operation with the first parameter, flattening operation, and normalization operation. and the value of the intra-modal attention module The characteristics Keys for intra-modal attention modules and the value of the intra-modal attention module After being processed by multi-scale self-attention MSSAP and multi-layer perceptron MLP, the intra-modal sequence is obtained This is achieved specifically through the following formula:

[0161]

[0162]

[0163] Among them, MLP represents a multi-layer perceptron, including a linear input layer, a nonlinear activation layer and a linear output layer, y m is an intermediate variable, all Composition of intra-modal sequences.

[0164] Furthermore, the inter-modal attention module aims to aggregate all the modal information and modality-specific cues. The inter-modality attention module is used to integrate the intra-modality sequence and the modality missing mask output by the intra-modality attention module. 0 and modality-specific missing prompt P spec Processing is performed to obtain the inter-modal feature representation F global ,include:

[0165] Through the cascade operation, the modality missing mask Mask 0 Merge with the intra-modal sequence and add position embedding to obtain the inter-modal sequence S global , which is specifically achieved through the following formula:

[0166]

[0167] Among them, Pos represents position embedding, is the intra-modal sequence, Mask for modal missing 0 ;

[0168] The modal missing specific prompt P spec The key of the inter-modal attention module is obtained through the convolution operation with the second parameter, flattening operation, and normalization operation. and the value of the inter-modal attention module

[0169] The inter-modal sequence S global , the key of the inter-modal attention module and the value of the inter-modal attention module After being processed by multi-scale self-attention MSSAP and multi-layer perceptron MLP, the inter-modal feature representation F is obtained. global , which is specifically expressed by the following formula:

[0170]

[0171] F global =MLP(y global )+y global ;

[0172] Among them, y global is an intermediate variable.

[0173] Furthermore, the enhanced activation attention module is to integrate the F cross and catalyst F for specific mode synthesis Act Further integration. The present invention refers to the modality that provides key information as a specific modality. The input of this module is F global , P spec , and the catalyst F synthesized for this specific modal data Act The three are integrated through multimodal cross attention MMCAP and the output F cross Based on this, in the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is global , the catalyst F output by the catalyst synthesizer Act , and the modality-specific missing hint P spec After processing, the enhanced activation feature representation F is obtained cross ,include:

[0174] For the inter-modal feature representation F global Flatten, deform, and normalize the layers to obtain the enhanced activation attention sequence S cross , for catalyst F ActFlatten, deform, and normalize the layers to obtain the catalyst sequence S Act ;

[0175] Missing specific hints for modality P spec Perform convolution operation, flattening operation, and normalization operation with the third parameter as the parameter to obtain the key of the enhanced activation attention module and enhance the value of the activated attention module

[0176] The activation attention sequence S will be enhanced cross , catalyst sequence S Act , enhance the key to activate the attention module and enhance the value of the activated attention module After processing by multimodal cross attention MMCAP and multi-layer perceptron MLP, the enhanced activation feature representation F is obtained. cross , which is specifically achieved through the following formula:

[0177]

[0178] F cross =MLP(y cross )+y cross ;

[0179] Among them, y cross is an intermediate variable.

[0180] The loss value L obtained based on the catalyst synthesizer con and the decoder loss L decoder , and the total loss value is obtained, which is specifically expressed by the following formula:

[0181] L total =L con +L decoder ;

[0182] Among them, L total is the total loss value;

[0183] During the model training process, the parameters of the incomplete modal object processing model based on the prompt-driven attention mechanism are updated based on the total loss value, and finally a trained incomplete modal object processing model based on the prompt-driven attention mechanism is obtained.

[0184] In the specific implementation process, the image data acquisition module is also used to acquire the real modality-missing MRI image data of the brain, that is, Figure 2 The missing modality data of the actual task in the training sample dataset is transmitted to the training sample dataset construction module;

[0185] The training sample data set construction module is also used to preprocess the real modality-missing MRI image data of the brain to obtain preprocessed MRI image data, and transmit the preprocessed MRI image data to the incomplete modality brain tumor segmentation module;

[0186] The incomplete modality brain tumor segmentation module is also used to receive the preprocessed MRI image data, and process the preprocessed MRI image data through the trained incomplete modality object processing model based on the prompt-driven attention mechanism to obtain a segmentation map corresponding to the real modality-missing MRI image data of the brain, that is, Figure 2 In the prediction results, the segmentation map corresponding to the MRI image data with missing real modality of the brain includes the label of the brain tumor.

[0187] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. An incomplete modality brain tumor segmentation system based on cue-driven attention, characterized in that It includes an image data acquisition module, a training sample data set construction module and an incomplete modality brain tumor segmentation module; The image data acquisition module is used to obtain MRI image data of all brain modalities and segmentation maps corresponding to the MRI image data of all brain modalities, wherein the segmentation maps corresponding to the MRI image data of all brain modalities include brain tumor labels, and is also used to transmit the MRI image data of all brain modalities and the segmentation maps corresponding to the MRI image data of all brain modalities to the training sample data set construction module; The training sample data set construction module is used to receive MRI image data of all brain modalities and segmentation maps corresponding to the MRI image data of all brain modalities, and perform modality missing processing on the MRI image data of all brain modalities to obtain modality missing image data, and then preprocess the modality missing image data to obtain the preprocessed modality data, and at the same time determine the modality missing mask Mask 0 , the modality missing mask Mask 0 Characterize whether each modality in all modalities of the preprocessed image data exists; use the preprocessed modality data and the modality missing mask as input samples, and use the segmentation map corresponding to the MRI image data of all modalities of the brain as output samples; the input samples and output samples constitute training samples, and multiple training samples are constructed to obtain a training sample data set; transmit the training sample data set to the incomplete modality brain tumor segmentation module; The incomplete modality brain tumor segmentation module is used to receive a training sample data set, and perform model training on the incomplete modality object processing model based on the prompt-driven attention mechanism based on the training sample data set, wherein the incomplete modality object processing model based on the prompt-driven attention mechanism includes a dual-branch fusion encoder, a modality missing prompt synthesizer, a prompt-driven attention fusion device, and a catalyst synthesizer; Specifically, the preprocessed modal data is processed by a dual-branch fusion encoder to obtain the fusion features of all modalities in the preprocessed modal data; the fusion features of all modalities in the preprocessed modal data and the modality missing mask Mask are combined. 0 Input to the modality missing prompt synthesizer to obtain the modality missing specific prompt P spec ; Combine the fusion features of all modalities and the modality missing mask Mask in the preprocessed modal data 0 , input the catalyst synthesizer, and obtain the loss value L of the optimization objective function of the catalyst synthesizer con and catalyst F Act ; The fusion features of all modes in the preprocessed modal data and the catalyst F output by the catalyst synthesizer Act , modality missing mask Mask 0 and the modality lacks specific prompt P spec , input to the cue-driven attention fusion device to obtain the enhanced activation feature representation F cross ; Enhance the activation feature representation F cross The segmentation map corresponding to the preprocessed modality data is input into the decoder to obtain the segmentation map corresponding to the preprocessed modality data, which contains the predicted tumor label. At the same time, based on the output sample and the segmentation map corresponding to the preprocessed modality data, the loss value L of the decoder is obtained. decoder ; The loss value L obtained based on the catalyst synthesizer con and the decoder loss L decoder , and the total loss value is obtained, which is specifically expressed by the following formula: L total =L con +L decoder ; Among them, L total is the total loss value; During the model training process, the parameters of the incomplete modal object processing model based on the prompt-driven attention mechanism are updated based on the total loss value, and finally a trained incomplete modal object processing model based on the prompt-driven attention mechanism is obtained.

2. The cue-driven attention-based incomplete modality brain tumor segmentation system according to claim 1, characterized in that: In a specific implementation process, the image data acquisition module is also used to acquire real modality-missing MRI image data of the brain, and transmit the real modality-missing MRI image data of the brain to the training sample data set construction module; The training sample data set construction module is also used to preprocess the real modality-missing MRI image data of the brain to obtain preprocessed MRI image data, and transmit the preprocessed MRI image data to the incomplete modality brain tumor segmentation module; The incomplete modality brain tumor segmentation module is also used to receive the preprocessed MRI image data, and process the preprocessed MRI image data through the trained incomplete modality object processing model based on the prompt-driven attention mechanism to obtain a segmentation map corresponding to the real modality-missing MRI image data of the brain. The segmentation map corresponding to the real modality-missing MRI image data of the brain includes a label of the brain tumor.

3. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that: In the incomplete modality brain tumor segmentation module, the preprocessed modality data is processed by a dual-branch fusion encoder to obtain fusion features of all modalities in the preprocessed modality data, including: The modal data of each modality in the modal data after preprocessing is processed by the first branch in the dual-branch fusion encoder, which is specifically implemented by the following formula: Among them, M m is the modal data of the mth mode in the modal data after preprocessing, M m The feature representation after the lth layer of the first branch in the dual-branch fusion encoder is M m The feature representation after the l-1th layer of the first branch in the dual-branch fusion encoder, DS represents the downsampling operation, and ConvBlock represents the convolution block consisting of group normalization, ReLU activation function and 3×3 kernel size convolution; The modal data of each modality in the modal data after preprocessing is processed by the second branch in the dual-branch fusion encoder, which is specifically implemented by the following formula: in, M m The feature representation after the lth layer of the second branch in the dual-branch fusion encoder is M m Feature representation after the l-1th layer of the second branch in the dual-branch fusion encoder; To M m Feature representation after the 5th layer of the first branch in the dual-branch fusion encoder and M m Feature representation after the 5th layer of the second branch in the dual-branch fusion encoder Perform feature fusion to obtain fusion feature F m , which is specifically achieved through the following formula: Among them, F m is the fusion feature of the m-th modality, CA represents the cross attention operation; in, D represents the depth of the modal data after preprocessing, W represents the width of the modal data after preprocessing, H represents the height of the modal data after preprocessing, and C represents the number of channels of the modal data after preprocessing.

4. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that: Modality missing mask 0 , is a Bernoulli index for indicating the mth mode. When the MRI image data of the mth mode does not exist in the MRI image data of all the brain modes, The value of is 0. When the MRI image data of the mth modality exists in the MRI image data of all modalities of the brain, The value of N is 1, M is the total number of modes; Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the modality missing mask Mask 0 Input to the modality missing prompt synthesizer to obtain the modality missing specific prompt P spec ,include: General hint for modal missing Perform random initialization, the modality lacks the general prompt P uni Contains the common information implied by all modal missing situations, where D represents the depth of the modal data after preprocessing, W represents the width of the modal data after preprocessing, and H represents the height of the modal data after preprocessing. N P indicates the total number of all cases where the modality is missing; Through the cascade operation, the modal missing mask Mask 0 Combine the fusion features of all modes in the preprocessed modal data to obtain It is specifically expressed by the following formula: in, is the intermediate variable, Mask for modal missing 0 The Bernoulli index used to indicate the first mode in , Mask for modal missing 0 The Bernoulli index used to indicate the second mode in , Mask for modal missing 0 Used to indicate the Nth M The Bernoulli index of each modality, F1 is the fusion feature of the first modality, and F2 is the fusion feature of the second modality. For N M The fusion features of the modalities; Through three fusion modules, the intermediate variables and modality missing general prompt P uni Splice and get P A collection of modality missing prompts corresponding to the modality missing situations This is achieved specifically through the following formula: Among them, PF represents three consecutive fusion modules, each module contains a convolution layer, batch normalization and ReLU activation function, and Split represents the segmentation of features along the channel dimension to obtain N P The number of channels is 1; P1 represents the modality missing prompt corresponding to the first modality missing situation, and P2 represents the modality missing prompt corresponding to the second modality missing situation. Indicates that the Nth P The modality missing prompts corresponding to the modality missing situations; Mask is missing according to the modality 0 ,exist Select from the list to get the modality missing specific prompt P spec .

5. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that: Modality missing mask 0 , is a Bernoulli index for indicating the mth mode. When the MRI image data of the mth mode does not exist in the MRI image data of all the brain modes, The value of is 0. When the MRI image data of the mth modality exists in the MRI image data of all modalities of the brain, The value of N is 1, M is the total number of modes; Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the modality missing mask Mask 0 , input the catalyst synthesizer, and obtain the loss value L of the optimization objective function of the catalyst synthesizer con and catalyst F Act ,include: Determine the missing mask of the modality 0 Indicates whether the value of the Bernoulli index of the specified mode is 1, and the mask Mask is missing in the mode 0 When the value of the Bernoulli index indicating the specified mode is 1, the catalyst F output by the catalyst synthesizer is determined according to the first method. Act , in the modality missing mask Mask 0 When the value of the Bernoulli index indicating the specified mode is 0, the catalyst F output by the catalyst synthesizer is determined according to the second method. Act ; The first method includes: The modal missing specific prompt P spec Determine the initial catalyst Y Act , set the value of the Bernoulli indicator used to indicate the specified mode to 0, and obtain the new mode missing mask Mask 1 , and then based on the new modality missing mask Mask 1 The first catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data And calculate the initial catalyst Y Act and the first catalyst The loss value between is specifically expressed by the following formula: Y Act =F spec ; Among them, MF represents three consecutive modules, each module contains a convolution layer, batch normalization and ReLU activation function, MSE represents mean square error, Missing mask for new modality 1 The Bernoulli index used to indicate the first mode in , Missing mask for new modality 1 Used to indicate the Nth M The Bernoulli index of the modalities, F1 is the fusion feature of the first modality, For N M The fusion features of the modalities, L con is the initial catalyst Y Act and the first catalyst The loss value between At this time, the initial catalyst Y Act Catalyst F as output from the Catalyst Synthesizer Act ; The second method includes: The modal missing mask Mask 0 Renamed to modality missing mask Mask 1 , based on the modality missing mask Mask 1 The first catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data Randomly select the modality missing mask Mask 1 A Bernoulli indicator with a value of 1 in the modality is reset to 0 to obtain a new modality missing mask Mask 2 , according to the new modality missing mask Mask 2 The second catalyst is synthesized by combining the fusion features of all modes in the preprocessed modal data And calculate the first catalyst and the second catalyst The loss value between is specifically expressed by the following formula: in, Missing mask for new modality 2 The Bernoulli index used to indicate the first mode in , Missing mask for new modality 2 Used to indicate the Nth M The Bernoulli index of the mode, L con The first catalyst and the second catalyst The loss value between At this time, the first catalyst Catalyst F as output from the Catalyst Synthesizer Act .

6. The cue-driven attention-based incomplete modality brain tumor segmentation system according to claim 1, characterized in that: The cue-driven attention fuser includes an intra-modality attention module, an intra-modality attention module, and an enhanced activation attention module; Based on this, the incomplete modality brain tumor segmentation module combines the fusion features of all modalities in the preprocessed modality data and the catalyst F output by the catalyst synthesizer. Act , modality missing mask Mask 0 and the modality lacks specific prompt P spec , input to the cue-driven attention fusion device to obtain the enhanced activation feature representation F cross ,include: In the intra-modal attention module, the fusion feature F m and modality-specific missing prompt P spec Processing is performed to obtain an intra-modal sequence; in the inter-modal attention module, the intra-modal sequence output by the intra-modal attention module and the modality missing mask Mask 0 and modality-specific missing prompt P spec Processing is performed to obtain the inter-modal feature representation F global ; In the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module global , the catalyst F output by the catalyst synthesizer Act , and the modality-specific missing hint P spec After processing, the enhanced activation feature representation F is obtained cross .

7. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 6, characterized in that: In the intra-modal attention module, the fusion feature F m and modality-specific missing prompt P spec Processing is performed to obtain the intra-modal sequence, including: The fusion feature F m Flatten into a length of L seq The sequence is then embedded by linear projection and positional embedding. P spec The key of the intra-modal attention module is obtained through the convolution operation with the first parameter, flattening operation, and normalization operation. and the value of the intra-modal attention module The characteristics Keys for intra-modal attention modules and the value of the intra-modal attention module After being processed by multi-scale self-attention MSSAP and multi-layer perceptron MLP, the intra-modal sequence is obtained This is achieved specifically through the following formula: Among them, MLP represents a multi-layer perceptron, including a linear input layer, a nonlinear activation layer and a linear output layer, y m is an intermediate variable, all Composition of intra-modal sequences.

8. The cue-driven attention-based incomplete modality brain tumor segmentation system according to claim 6, characterized in that: In the inter-modality attention module, the intra-modality sequence and modality missing mask Mask output by the intra-modality attention module are 0 and modality-specific missing prompt P spec Processing is performed to obtain the inter-modal feature representation F global ,include: Through the cascade operation, the modal missing mask Mask 0 Merge with the intra-modal sequence and add position embedding to obtain the inter-modal sequence S global , which is specifically achieved through the following formula: Among them, Pos represents position embedding, is the intra-modal sequence, Mask for modal missing 0 ; The modal missing specific prompt P spec The key of the inter-modal attention module is obtained through the convolution operation with the second parameter, flattening operation, and normalization operation. and the value of the inter-modal attention module The inter-modal sequence S global , the key of the inter-modal attention module and the value of the inter-modal attention module After being processed by multi-scale self-attention MSSAP and multi-layer perceptron MLP, the inter-modal feature representation F is obtained. global , which is specifically expressed by the following formula: F global =MLP(and global )+y global ; Among them, y global is an intermediate variable.

9. The cue-driven attention-based incomplete modality brain tumor segmentation system according to claim 6, characterized in that: In the enhanced activation attention module, the inter-modal feature representation F output by the inter-modal attention module is global , the catalyst F output by the catalyst synthesizer Act , and the modality-specific missing hint P spec After processing, the enhanced activation feature representation F is obtained cross ,include: For the inter-modal feature representation F global Flatten, deform, and normalize the layers to obtain the enhanced activation attention sequence S cross , for catalyst F Act Flatten, deform, and normalize the layers to obtain the catalyst sequence S Act ; Missing specific hints for modality P spec Perform convolution operation, flattening operation, and normalization operation with the third parameter as the parameter to obtain the key of the enhanced activation attention module and enhance the value of the activated attention module The activation attention sequence S will be enhanced cross , catalyst sequence S Act , enhance the key to activate the attention module and enhance the value of the activated attention module After processing by multimodal cross attention MMCAP and multi-layer perceptron MLP, the enhanced activation feature representation F is obtained. cross , which is specifically achieved through the following formula: F cross =MLP(and cross )+y cross ; Among them, y cross is an intermediate variable.

10. The incomplete modality brain tumor segmentation system based on cue-driven attention according to claim 1, characterized in that: In the training sample data set construction module, modality missing processing is performed on the MRI image data of all modalities of the brain to obtain modality missing image data, and then preprocessing is performed on the modality missing image data to obtain the modality data after preprocessing, specifically including: The MRI image data of all brain modalities are subjected to data corruption and random shielding to obtain the image data with missing modalities. The image data with missing modalities are then subjected to image denoising, image cropping pre-adjustment, intensity normalization, spatial normalization, bias field correction and intracranial extraction in sequence to obtain the modal data after preprocessing.

Images