Multi-modal brain tumor image segmentation system based on sparse attention mechanism

Abstract

The invention discloses a multi-modal brain tumor image segmentation system based on a sparse attention mechanism, and relates to the technical field of medical image processing, and the system comprises a data obtaining unit which is used for obtaining multi-modal brain tumor data and carrying out the preprocessing of the multi-modal brain tumor data to obtain multi-modal data; the feature extraction unit is used for extracting a first specific feature Fm of each mode in the multi-mode data based on a first encoder; the feature fusion unit is used for obtaining a first feature f1 based on the first specific feature Fm and a first fusion token Ff; based on a modal sparse mask fusion Transform model, interacting the first feature f1 to obtain a second feature Z, based on the second feature Z, extracting a weight IMm of each modal, based on the weight IMm, obtaining a third feature # imgabs0 #, based on spatial dimensions, and based on the third feature # imgabs0 #, obtaining a second feature Z; the prediction unit is used for inputting the fourth feature # imgabs3 # into a brain tumor segmentation model to obtain a first loss L1, obtaining a prediction segmentation result based on the first loss L1, and obtaining a prediction segmentation result based on the prediction segmentation result; the problems that an existing segmentation mode based on shared feature representation is insensitive to brain tumor information, information redundancy occurs in brain tumor feature fusion when modals are lost, and important brain tumor information is ignored can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multimodal brain tumor image segmentation system based on a sparse attention mechanism. Background Art

[0002] Brain tumor is one of the top ten malignant tumors. Due to the complex internal structure of the brain, accurate segmentation of brain tumor is of great significance. Magnetic resonance imaging (MRI) modalities include T1-weighted (T1), contrast-enhanced T1-weighted (T1c), T2-weighted (T2), and fluid-attenuated inversion recovery (FLAIR) modalities. The multimodal information of brain tumors complements each other. The FLAIR modality highlights the key information of the entire tumor, and the T1c modality highlights the core area of ​​the brain tumor. In recent years, great progress has been made in brain tumor segmentation using complete multimodal magnetic resonance images (MRI). However, in actual clinical environments, poor image quality, patient movement during scanning, and acquisition time constraints can all lead to the loss of brain tumor magnetic resonance image modalities.

[0003] For the missing brain tumor segmentation, researchers have proposed three types of segmentation methods. The first is a segmentation method based on modality image synthesis, which synthesizes the image of the missing modality from the existing modality, such as the generative adversarial network (GAN), but this method requires cumbersome calculations and cannot synthesize the key features of the missing modality; the second method is a segmentation method based on knowledge distillation, which first trains the teacher model in the complete brain tumor MRI modality. When the teacher model is fully converged, the student model of the missing modality is trained together with the teacher model. Although this method can improve the feature learning performance in the missing modality student network, it is still difficult to learn the important features of the missing modality from all modalities; the third method is a segmentation method based on shared feature representation, which fuses the features of the available modalities into the potential common space and projects the features extracted in the available modalities into the segmentation space. This segmentation method does not require cumbersome calculations and a large amount of computing resources.

[0004] However, this segmentation method has the following problems: (1) Tumor lesions usually show irregular shapes, and some areas have a wide range of lesions, so more features need to be extracted for detailed inspection. The ordinary feature U-NET model cannot fully capture the contextual information of brain tumors, is insensitive to brain tumor information, and has low perception ability of the model; (2) The lack of a certain modality of brain tumors will cause the brain tumor to over-rely on other modalities when performing feature fusion. Although learnable fusion tokens are introduced for brain tumor feature fusion, it will also cause the model to prefer information from a certain modality and ignore other important information when fusion; and during the fusion process, if the query vector and the key vector calculate all the attention weights, the attention weight is low, which proves that the correlation between the query key pairs is low. When the attention weight is used to perform weighted summation on the value vector, less brain tumor information will be captured, resulting in redundant information exchange, interfering with the process of brain tumor information exchange, and ignoring important brain tumor information. Summary of the invention

[0005] In order to solve the problems that the existing segmentation method based on shared feature representation is insensitive to brain tumor information and that brain tumor feature fusion in the absence of modality leads to information redundancy and neglect of important brain tumor information, the present invention provides a multimodal brain tumor image segmentation system based on a sparse attention mechanism, the system comprising:

[0006] A data acquisition unit: used for acquiring multimodal brain tumor data, and preprocessing the multimodal brain tumor data to obtain multimodal data;

[0007] Feature extraction unit: used to extract the first specific feature F of each modality in the multimodal data based on the first encoder m ;

[0008] Feature fusion unit: used to combine the first specific feature F m and the first fusion token F f , get the first feature f 1 ;Based on the modal sparse mask fusion Transformer model, the first feature f 1 Interact to obtain the second feature Z, and based on the second feature Z, extract the weight IM of each modality m , based on the weights IM m Get the third feature Based on the spatial dimension, the third feature Plastic surgery to obtain the fourth feature

[0009] Prediction unit: used to transform the fourth feature A brain tumor segmentation model is input to obtain a first loss L1, and a predicted segmentation result is obtained based on the first loss L1.

[0010] The principle of the invention is as follows: the multimodal features of brain tumors in multimodal MRI images are selected, focusing on the local detail features of available modalities and the modal features of brain tumors of all available modalities; a mask is introduced to make public multimodal brain tumor data into a simulation data set, simulating 15 missing modal combinations of missing brain tumors, which can improve the brain tumor segmentation effect.

[0011] The first encoder is used to extract features for each modality. The first encoder adds a network structure combined with MobileNetV3 reverse residual on the basis of the convolutional network, and adopts a hierarchical feature extraction method to gradually extract and enhance features. The MobileNetV3 reverse residual module includes the SE convolution attention mechanism (Squeeze-and-Excitation). The SE convolution attention mechanism adaptively adjusts the weights of the channel so that it can learn the global information within the modality and emphasize useful brain tumor information features. It dynamically emphasizes useful brain tumor feature information, focuses on the local detail features of brain tumors, improves the sensitivity to brain tumor information, more accurately captures the location of brain tumors, suppresses healthy brain tissue information, and improves the ability to extract brain tumor features of each modality, thereby improving the accuracy of brain tumor segmentation.

[0012] An intra-modality self-attention mechanism is introduced for all available modalities to perform intra-modal feature fusion of brain tumors. A learnable fusion token is introduced for all available modalities to perform cross-modal feature fusion between available modal brain tumors. A sparse mask matrix is ​​introduced to reduce feature information redundancy and increase attention to important brain tumor features, thereby improving the accuracy and generalization ability of model segmentation.

[0013] For the feature fusion of missing modalities, a sparse attention mechanism is introduced to perform feature fusion of missing modal brain tumors. Transformer can efficiently capture global features. The sparse self-attention mechanism sets the weight of invalid positions to 0, and sets the non-attention positions of each attention head to 0 according to the number of categories, reducing the interference of healthy brain tissue information on the brain tumor fused with the missing modality, suppressing healthy brain tissue information, highlighting valuable brain tumor information, and increasing attention to important brain tumor features. Feature fusion aggregates brain tumor-related marker information, filters irrelevant healthy brain tissue information, and selectively extracts useful feature information for modality fusion, thereby improving the accuracy of model segmentation.

[0014] In the decoder, the multi-scale void attention mechanism is introduced into the missing modality brain tumor segmentation task to reduce the problem of brain tumor information loss during the skip connection process of the brain tumor, obtain more spatial information of the brain tumor, effectively integrate the feature information from the encoder, increase the receptive field and contextual information, increase the attention to the brain tumor features, help the model to extract brain tumor information more finely, better capture the boundary information of the brain tumor, and more accurately determine the boundary of the brain tumor, thereby improving the segmentation performance of the brain tumor.

[0015] Multimodal imaging of brain tumors refers to the process of using multiple MRI sequences to obtain information about tumor tissue characteristics. It includes T1-weighted (T1), contrast-enhanced T1-weighted (T1c), T2-weighted (T2), and fluid-attenuated inversion recovery (FLAIR) modalities.

[0016] Sparse Attention is a technique used in the Transformer model to reduce computational complexity. The core idea is to introduce sparsity in the self-attention calculation, that is, instead of having every position in the sequence perform attention calculations with all other positions, only some positions are selected for calculation.

[0017] Learnable Fusion Tokens: used for feature fusion between different modalities. They do not directly represent data of any specific modality, but as part of the model, they learn how to integrate information from different modalities through optimization during training. These fusion tokens can dynamically adjust their parameters to adapt to different data representations and interactions between modalities, thereby obtaining cross-modal brain tumor information.

[0018] Latent space is a low-dimensional space used to represent data in machine learning. It refers to a compressed representation of all useful information contained in the data.

[0019] MobileNetV3 reverse residual module: including expansion convolution, depthwise convolution, projection convolution and SE (Squeeze and Excitation blocks) modules.

[0020] Modality Sparse Mask Fusion Transformer: A Transformer architecture for more efficient cross-modality feature fusion and addressing the modality missing problem. It uses a sparse masking mechanism to address the missing brain tumor modality and reduce feature redundancy, and combines the sparse masking mechanism with the traditional Transformer self-attention mechanism to achieve more effective feature fusion.

[0021] Multi-head modal sparse masked attention mechanism: It is a modal sparse masked attention mechanism formed by combining sparse mask and self-attention mechanism. Multi-head modal sparse masked attention mechanism allows the model to learn information in multiple subspaces in parallel, which can enhance the learning ability and expressiveness of the model.

[0022] Modal sparse masked attention mechanism: It is a modal sparse masked attention mechanism formed by combining sparse mask and self-attention mechanism.

[0023] Multi-Scale Dilated Spatial Attention (MSDA) is a multi-scale dilated spatial attention mechanism that can effectively fuse feature information of different scales and use dilated convolution to capture longer-distance dependencies, thereby improving the feature extraction ability of the model.

[0024] Channel fusion Transformer layer: A layer that uses a mask mechanism combined with the Transformer to reduce channel dimension redundancy.

[0025] Sparse coding: refers to a technique used to indicate which neurons or parameters are activated or retained during sparse training.

[0026] Sparse mask matrix: It is a sparse mask formed by combining the mask mechanism with the sparse attention mechanism.

[0027] Reshape: Changing the "shape" of an array (usually multidimensional) without changing its data.

[0028] Furthermore, the first specific feature F m It includes the first mode T1, the second mode T2, the third mode T1c and the fourth mode Flair. The first mode T1 is used to extract non-enhanced tumor area information, the second mode T2 is used to extract tumor area information of edema area and necrosis area, the third mode T1c is used to extract tumor area information of enhanced tumor area and non-enhanced tumor area, and the fourth mode Flair is used to extract complete tumor area information. An encoder is provided for each mode, and specific features are extracted in each mode. Each mode has different focuses, which improves the feature extraction ability of the encoder.

[0029] Furthermore, the feature extraction unit is further used to: extract the first specific feature F m The input is sent to the second encoder for segmentation and projection to the latent space to obtain the second loss L2 of the latent space projection. The decoder is used to decode the modality-specific features and calculate the regularization loss to learn more generalized features and improve the generalization of the model.

[0030] Furthermore, the first encoder includes five encoder layers, the first encoder layer includes three convolutional layers and one residual connection, and each layer from the second encoder layer to the fifth encoder layer includes three MobileNetV3 reverse residual modules and one residual connection. A layered feature extraction method is adopted to gradually extract and enhance features, and the MobileNetV3 reverse residual module includes an SE convolution attention mechanism. The SE convolution attention mechanism adaptively adjusts the weights of the channels so that it can learn the global information within the modality and emphasize useful brain tumor information features, dynamically emphasize useful brain tumor feature information, focus on local detail features of brain tumors, improve sensitivity to brain tumor information, more accurately capture the location of brain tumors, suppress healthy brain tissue information, and enhance the ability to extract brain tumor features of each modality, thereby improving the accuracy of brain tumor segmentation.

[0031] Furthermore, the second feature Z includes a second fusion token F containing cross-modal information. f1 and a second specific feature F containing global information within the modality m1 .

[0032] Furthermore, the modality sparse mask fusion Transformer model includes several layers of modality sparse mask fusion Transformer, and each layer of the modality sparse mask fusion Transformer includes a multi-head modality sparse mask attention mechanism, a feedforward neural network layer and a normalization layer.

[0033] Furthermore, the feature fusion unit specifically includes:

[0034] The first fusion unit is used to transform the first specific feature F m The first specific feature F m With the first fusion token F f and the first fusion token F f The position encoding is cascaded to obtain the first feature f 1 ;

[0035] The second fusion unit is used to transform the first feature f based on the modal sparse mask attention mechanism. 1 Project to the preset space to obtain the fifth feature Z m; Based on the modality sparse mask fusion Transformer model, the fifth feature Z m and the first feature f 1 The second feature Z is obtained by fusion, wherein the preset space includes a query space, a key space, and a value space;

[0036] The third fusion unit is used to extract the second fusion token F f1 The attention information of each modality in the attention tensor W j , for the attention information W j Slice to obtain the spatial weight W of each mode m ; The spatial weight W m Reshaping is performed to obtain the weight IM m Based on the weight IM m , for the second specific feature F m1 Weighted, the third feature is obtained

[0037] The fourth fusion unit is used to combine the third feature Reshape along the spatial dimension to obtain the sixth feature The sixth feature Cascade to obtain the seventh feature The second fusion token F f1 Plastic surgery to obtain the eighth feature Based on the modality mask Transformer layer, the seventh feature And the eighth feature Fusion is performed along the channel dimension to obtain the fourth feature

[0038] Furthermore, the brain tumor segmentation model includes a decoder, which includes a multi-scale void attention mechanism and a channel fusion Transformer layer. The multi-scale void attention mechanism is introduced into the missing modality brain tumor segmentation task to reduce the problem of brain tumor information loss during the skip connection process of the brain tumor, obtain more spatial information of the brain tumor, effectively integrate the feature information from the encoder, increase the receptive field and context information, increase the attention to the brain tumor features, help the model to extract brain tumor information more finely, better capture the boundary information of the brain tumor, and more accurately determine the boundary of the brain tumor, thereby improving the segmentation performance of the brain tumor.

[0039] Furthermore, the prediction unit specifically includes:

[0040] The first prediction unit is used to transform the fourth feature Input the decoder to obtain the third specific feature F m2, based on the channel fusion Transformer layer, the third specific feature F m2 And the fourth feature Perform fusion to obtain the ninth feature D1;

[0041] The second prediction unit is used to cascade the ninth feature D1 to obtain a final feature D, obtain the first loss L1 based on the final feature D, and obtain a predicted segmentation result based on the first loss L1.

[0042] Furthermore, the calculation method for obtaining the second loss L2 is:

[0043] L rgl =∑ m∈M L Dice (D rgl (E m (x m )),Y g )+L WCE (D rgl (E m (x m )),Y g );

[0044] Among them, L rgl represents the second loss L2, L Dice represents the Dice loss function, D rgl Represents the second encoder, E m Indicates the first encoder, L WCE represents the weighted cross entropy loss function, Y g represents the true annotation of brain tumor, x m represents any 3D input modality path, m represents the modality, and M represents the first specific feature F m , including a first mode T1, a second mode T2, a third mode T1c and a fourth mode Flair.

[0045] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0046] 1. The first encoder is used to extract features for each modality. The first encoder adds a network structure combined with MobileNetV3 reverse residual on the basis of the convolutional network, and adopts a hierarchical feature extraction method to gradually extract and enhance features. The MobileNetV3 reverse residual module includes the SE convolution attention mechanism (Squeeze-and-Excitation). The SE convolution attention mechanism adaptively adjusts the weights of the channel so that it can learn the global information within the modality and emphasize useful brain tumor information features. It dynamically emphasizes useful brain tumor feature information, focuses on the local detail features of brain tumors, improves the sensitivity to brain tumor information, more accurately captures the location of brain tumors, suppresses healthy brain tissue information, and improves the ability to extract brain tumor features of each modality, thereby improving the accuracy of brain tumor segmentation.

[0047] 2. Introduce the intra-modal self-attention mechanism for all available modalities, perform feature fusion within the brain tumor modality, introduce learnable fusion tokens for all available modalities, perform cross-modal feature fusion between available modal brain tumors, and introduce sparse mask matrices to reduce feature information redundancy and increase attention to important brain tumor features, thereby improving the accuracy and generalization ability of model segmentation.

[0048] 3. For the feature fusion of missing modalities, a sparse attention mechanism is introduced to perform feature fusion of missing modal brain tumors. Transformer can efficiently capture global features. The sparse self-attention mechanism sets the weight of invalid positions to 0, and sets the non-attention positions of each attention head to 0 according to the number of categories, reducing the interference of healthy brain tissue information on the brain tumor fused with missing modalities, suppressing healthy brain tissue information, highlighting valuable brain tumor information, and increasing attention to important brain tumor features. Feature fusion aggregates brain tumor-related marker information, filters irrelevant healthy brain tissue information, and selectively extracts useful feature information for modality fusion, thereby improving the accuracy of model segmentation.

[0049] 4. In the decoder, the multi-scale void attention mechanism is introduced into the missing modality brain tumor segmentation task to reduce the problem of brain tumor information loss during the skip connection process of the brain tumor, obtain more spatial information of the brain tumor, effectively integrate the feature information from the encoder, increase the receptive field and contextual information, increase the attention to the brain tumor features, help the model to extract brain tumor information more finely, better capture the boundary information of the brain tumor, and more accurately determine the boundary of the brain tumor, thereby improving the segmentation performance of the brain tumor. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;

[0051] Figure 1 is a schematic diagram of the structure of a multimodal brain tumor image segmentation system based on a sparse attention mechanism in the present invention;

[0052] Figure 2 It is a flowchart of the multimodal brain tumor image segmentation system based on the sparse attention mechanism in the present invention. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0055] Embodiment 1

[0056] refer to Figure 1-Figure 2 , this embodiment provides a multimodal brain tumor image segmentation system based on a sparse attention mechanism, the system comprising:

[0057] Data acquisition unit: used to acquire multimodal brain tumor data based on a magnetic resonance imaging (MRI) scanner, and to pre-process the multimodal brain tumor data based on a processor to obtain multimodal data; in this embodiment, the pre-processing operation may include cropping, normalization, random rotation, and random mirror flipping, etc. The processor may be a controller, a chip, a computer, and dedicated image processing software including 3D Slicer and FSL, etc. These devices contain computer programs, calculation model programs, or related calculation functions, etc.; the acquisition device is connected to the processor data, the processor may be built in the acquisition device, or may not be installed on the acquisition device, the acquisition device transmits the acquired data to the processor for processing, and the processor has a built-in encoder, a pre-trained modality sparse mask fusion Transformer model, and a pre-trained brain tumor segmentation model. In this embodiment, it may also include a memory, a voice acquisition device, and a voice processing chip, the memory is connected to both the magnetic resonance imaging scanner and the processor, and is used to store multimodal brain tumor data and multimodal data; the voice acquisition device is connected to the voice processing chip, the voice processing chip is connected to the processor, the voice acquisition device is used to acquire voice data, and the voice processing chip is used to process voice data and convert it into text or computer instructions, etc.

[0058] Feature extraction unit: used to extract the first specific feature F of each modality in the multimodal data based on the first encoder m ; Such as inputting the modal information of each modality into the corresponding first encoder to extract the specific features of each modality;

[0059] Among them, the first specific feature F m It includes a first modality T1, a second modality T2, a third modality T1c and a fourth modality Flair. The first modality T1 is used to extract non-enhanced tumor region information, the second modality T2 is used to extract tumor region information of edema region and necrotic region, the third modality T1c is used to extract tumor region information of enhanced tumor region and non-enhanced tumor region, and the fourth modality Flair is used to extract complete tumor region information.

[0060] Among them, the first encoder includes five encoder layers, the first encoder layer includes three convolution layers and one residual connection, and each layer from the second encoder layer to the fifth encoder layer includes three MobileNetV3 reverse residual modules and one residual connection. The first encoder consists of ordinary convolution and MobileNetV3 reverse residual. The encoder is divided into five layers, each with three sublayers. The first layer consists of three ordinary convolution layers, which are used to process the input image and extract basic features. The second to fifth layers are composed of MobileNetV3 reverse residuals, and each layer is composed of three MobileNetV3 reverse residual modules. The results of the previous layer are extracted in turn, and the information of the input features is retained through the residual connection. This encoder extracts the features of the input image layer by layer through stacked convolution layers and MobileNetV3 reverse residual modules, thereby obtaining the local detail features of each modality, that is, the first specific feature F m ;

[0061] Feature fusion unit: used to combine the first specific feature F m and the first fusion token F f , get the first feature f 1 ;Based on the modal sparse mask fusion Transformer model, the first feature f 1 Interact to obtain the second feature Z, and based on the second feature Z, extract the weight IM of each modality m , based on the weights IM m Get the third feature Based on the spatial dimension, the third feature Plastic surgery to obtain the fourth feature In this embodiment, the fusion token may be a learnable fusion token.

[0062] The second feature Z includes a second fusion token F containing cross-modal informationf1 and a second specific feature F containing global information within the modality m1 .

[0063] Among them, the modality sparse mask fusion Transformer model includes several layers of modality sparse mask fusion Transformer, and each layer of the modality sparse mask fusion Transformer includes a multi-head modality sparse mask attention mechanism, a feedforward neural network layer and a normalization layer.

[0064] Among them, the feature fusion unit specifically includes:

[0065] The first fusion unit is used to transform the first specific feature F m The first specific feature F m With the first fusion token F f and the first fusion token F f The position encoding is cascaded to obtain the first feature f 1 , which can be calculated as:

[0066] f 1 =Concat(f flair ,f t1 ,f t1c ,f t2 ,F f )+PE;

[0067] Among them, f 1 Represents the first feature f 1 , Concat means cascade, f t1 , f t2 , f t1c and f flair They represent the first mode T1, the second mode T2, the third mode T1c and the fourth mode Flair of the shaping, respectively. f Indicates the first fusion token F f , PE represents the first fusion token F f location;

[0068] Second fusion unit: for modality-based sparse masked attention mechanism and binary attention mask B∈{0,1} 5N×5N , the first feature f 1 Project to the preset space to obtain the fifth feature Z m , which can be calculated as:

[0069]

[0070] Z m=(Softmax(θ(Q,K,i,j))*s_mask())V;

[0071] Among them, E represents the calculation of the scaled dot product of the query vector and the key vector and then converted to a positive number, H represents the number of heads of the multi-head modality sparse mask attention mechanism, and c represents the number of feature channels. Represents the feature dimension of each head, which is used to scale the dot product to make the training more stable; Q, K and V represent the first specific feature F respectively. m The query vector, key vector and value vector of , i represents the index of the query vector, j represents the index of the key vector, j′ represents the index of the accessible modality, T represents the transpose, represents the attention weight after applying the mask, Z m Represents the fifth feature Z m , B i,j Indicates whether the attention score can be calculated between i and j. If not, the modality is missing. B i,j is 0, if so, the mode exists, B i,j is 1; E′ represents B i,j The attention score, B i,j′＝1 represents the sum of the attention scores of the key vector when index = 1, s_mask represents the sparse mask matrix, which implements the sparseness of attention, and * represents element-by-element multiplication;

[0072] Based on the modality sparse mask fusion Transformer model, the fifth feature Z m and the first feature f 1 The second feature Z is obtained by fusion, and the calculation method thereof can be:

[0073] C2←MSDA(Z m )+f 1 ;

[0074] Z←(FFN(LN(C2))+C2);

[0075] Among them, f 1 Represents the first feature f 1 Z m Represents the fifth feature Z m , MSDA represents the multi-head masked attention mechanism, FFN represents the feedforward neural network, LN represents normalization, Z represents the second feature Z, and C2 represents the relevant parameters in the fusion process.

[0076] The preset space includes a query space, a key space and a value space;

[0077] The third fusion unit is used to extract the second fusion token F f1 The attention information of each modality in the attention tensor Wj , for the attention information W j Slice to obtain the spatial weight W of each mode m ; The calculation method can be:

[0078]

[0079] W m =Split(W)∈R 1×N ;

[0080] Among them, W j Represents the attention information W j , H represents the number of heads of the multi-head modality sparse mask attention mechanism, N = h × w × d, h represents the height of the input feature map, w represents the width of the feature map, and d represents the depth of the feature map. represents the attention weight matrix obtained in the calculation of the first-layer modal sparse mask Transformer, W m Represents the spatial weight W m , Split represents the slicing operation, and W represents extracting different types of attention information from the attention tensor.

[0081] The spatial weight W m Reshaping is performed to obtain the weight IM m Based on the weight IM m , for the second specific feature F m1 Weighted, the third feature is obtained The calculation method can be:

[0082] IM m = reshape(W m ),IM m ∈R 1×h×w×d ;

[0083]

[0084] Among them, W m Represents the spatial weight W m , IM m Represents weight IM m , reshape represents the reshaping operation, h represents the height of the input feature map, w represents the width of the feature map, and d represents the depth of the feature map. Indicates the third feature F m1 Represents the second specific feature F m1 .

[0085] The fourth fusion unit is used to combine the third feature Reshape along the spatial dimension to obtain the sixth feature The sixth feature Cascade to obtain the seventh feature The calculation method can be:

[0086]

[0087] in, Indicates the sixth characteristic reshape represents a reshaping operation. Indicates the third feature Indicates the seventh characteristic Concat means cascade, N = h×w×d, h represents the height of the input feature map, w represents the width of the feature map, d represents the depth of the feature map, and c represents the number of feature channels.

[0088] The second fusion token F f1 Plastic surgery to obtain the eighth feature The calculation method can be:

[0089]

[0090] in, Indicates the eighth characteristic reshape indicates a reshape operation, F f1 Indicates the second fusion token F f1 , N = h×w×d, h represents the height of the input feature map, w represents the width of the feature map, d represents the depth of the feature map, and c represents the number of feature channels.

[0091] Based on the modality mask Transformer layer, the seventh feature And the eighth feature Fusion is performed along the channel dimension to obtain the fourth feature The calculation method is:

[0092]

[0093] Among them, λ α and ω α Both represent point-by-point convolution with a kernel size of 1. represents the depthwise convolution with a kernel size of 3, Ω α∈{q,k,v} (·) represents a mapping function. Indicates the eighth characteristic Indicates the seventh characteristic Concat means concatenation, q means learnable fusion token, The query vector obtained by mapping; k represents the fourth feature of each modality The key vector is obtained by projecting, and the key vector is obtained by concatenating the key vectors of the four modes in the channel dimension; v represents the fourth feature of each mode The value vector is obtained by projection, and the value vector is obtained by concatenating the value vectors of the four modes in the channel dimension. m represents the mode and M represents the first specific feature F m , including the first modality T1, the second modality T2, the third modality T1c and the fourth modality Flair. Att(q,k) represents the attention weight obtained by the self-attention mechanism, T represents the transposition, = h×w×d, h represents the height of the input feature map, w represents the width of the feature map, d represents the depth of the feature map, CFT represents the modality mask attention mechanism along the channel direction, φ i,j corresponds to the relationship between the query vector q and the key vector k, respectively. If k corresponds to the missing mode, then φ i,j is equal to 0, otherwise φ i,j is equal to 1, j′ represents the index of the accessible modality, i represents the index of the query vector, and j represents the index of the key vector. Indicates the eighth characteristic It represents a feedforward neural network with a group convolution of size 3 added between two fully connected layers in the channel dimension. L2 represents the modal mask Transformer of the L2 layer, and L2=2.

[0094] Prediction unit: used to transform the fourth feature A brain tumor segmentation model is input to obtain a first loss L1, a predicted segmentation result is obtained based on the first loss L1, and the predicted segmentation result is displayed on a display device, which is connected to the magnetic resonance imaging scanner, the processor and the memory. In this embodiment, the display device can be a display screen.

[0095] Among them, the brain tumor segmentation model includes a decoder, which includes a multi-scale void attention mechanism and a channel fusion Transformer layer.

[0096] The prediction unit specifically includes:

[0097] The first prediction unit is used to transform the fourth feature Input the decoder to obtain the third specific feature F m2 , based on the channel fusion Transformer layer, the third specific feature F m2 And the fourth feature Perform fusion to obtain the ninth feature D1;

[0098] The second prediction unit is used to cascade the ninth feature D1 to obtain a final feature D, obtain the first loss L1 based on the final feature D, and obtain a predicted segmentation result based on the first loss L1.

[0099] As the fourth characteristic The first three layers of the input decoder can reduce information loss, more easily capture brain tumor boundary information, and improve the quality of brain tumor segmentation. The input is weighted in the fourth layer and cascaded upsampled to the final output layer for brain tumor segmentation. The first loss L1 calculates the error between the predicted segmentation result and the true label, which can be calculated as:

[0100] L1=∑ m∈M L Dice (D f (Concat(E m (x m ))),Y g )+L WCE (D f (E m (x m )),Y g );

[0101] Among them, L1 l represents the first loss L1, L Dice represents the Dice loss function, D f Denotes the decoder, E m represents the first encoder, Concat represents cascade, L WCE represents the weighted cross entropy loss function, Y g represents the true annotation of brain tumor, x m represents any 3D input modality path, m represents the modality, and M represents the first specific feature F m , including a first mode T1, a second mode T2, a third mode T1c and a fourth mode Flair.

[0102] Embodiment 2

[0103] The feature extraction unit is further used to: extract the first specific feature F m The first specific feature F of each modality is input to the second encoder for segmentation and projection to the latent space to obtain the second loss L2 of the latent space projection. In this embodiment, the second encoder is a weight sharing encoder. m The input is sent to the shared second encoder for separate segmentation and projected into the shared latent space to obtain the second loss L2 of the latent space projection, and the loss can be calculated as follows:

[0104] L rgl =∑m∈M L Dice (D rgl (E m (x m )),Y g )+L WCE (D rgl (E m (x m )),Y g );

[0105] Among them, L rgl Represents the second loss L2, i.e., the regularization loss, L Dice represents the Dice loss function, D rgl Represents the second encoder, E m Indicates the first encoder, L WCE represents the weighted cross entropy loss function, Y g represents the true annotation of brain tumor, x m represents any 3D input modality path, m represents the modality, and M represents the first specific feature F m , including a first mode T1, a second mode T2, a third mode T1c and a fourth mode Flair.

[0106] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0107] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal brain tumor image segmentation system based on sparse attention mechanism, characterized by: The system comprises: A data acquisition unit: used for acquiring multimodal brain tumor data, and preprocessing the multimodal brain tumor data to obtain multimodal data; Feature extraction unit: used to extract the first specific feature F of each modality in the multimodal data based on the first encoder m ; Feature fusion unit: used to combine the first specific feature F m and the first fusion token F f , get the first feature f 1 ;Based on the modal sparse mask fusion Transformer model, the first feature f 1 Interact to obtain the second feature Z, and based on the second feature Z, extract the weight IM of each modality m , based on the weights IM m Get the third feature Based on the spatial dimension, the third feature Plastic surgery to obtain the fourth feature Prediction unit: used to transform the fourth feature A brain tumor segmentation model is input to obtain a first loss L1, and a predicted segmentation result is obtained based on the first loss L1.

2. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 1, characterized in that: The first specific feature F m It includes a first modality T1, a second modality T2, a third modality T1c and a fourth modality Flair. The first modality T1 is used to extract non-enhanced tumor region information, the second modality T2 is used to extract tumor region information of edema region and necrotic region, the third modality T1c is used to extract tumor region information of enhanced tumor region and non-enhanced tumor region, and the fourth modality Flair is used to extract complete tumor region information.

3. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 2, characterized in that: The feature extraction unit is also used to: The first specific feature F m The input is sent to the second encoder for segmentation and projection to the latent space to obtain the second loss L2 of the latent space projection.

4. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 1, characterized in that: The first encoder includes five encoder layers, the first encoder layer includes three convolutional layers and one residual connection, and each layer from the second encoder layer to the fifth encoder layer includes three MobileNetV3 reverse residual modules and one residual connection.

5. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 1, characterized in that: The second feature Z includes a second fusion token F containing cross-modal information f1 and a second specific feature F containing global information within the modality m1 .

6. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 5, characterized in that: The modality sparse mask fusion Transformer model includes several layers of modality sparse mask fusion Transformer, and each layer of the modality sparse mask fusion Transformer includes a multi-head modality sparse mask attention mechanism, a feedforward neural network layer and a normalization layer.

7. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 6, characterized in that: The feature fusion unit specifically includes: The first fusion unit is used to transform the first specific feature F m The first specific feature F m With the first fusion token F f and the first fusion token F f The position encoding is cascaded to obtain the first feature f 1 ; The second fusion unit is used to transform the first feature f based on the modal sparse mask attention mechanism. 1 Project to the preset space to obtain the fifth feature Z m ; Based on the modality sparse mask fusion Transformer model, the fifth feature Z m and the first feature f 1 The second feature Z is obtained by fusion, wherein the preset space includes a query space, a key space, and a value space; The third fusion unit is used to extract the second fusion token F f1 The attention information of each modality in the attention tensor W j , for the attention information W j Slice to obtain the spatial weight W of each mode m ; The spatial weight W m Reshaping is performed to obtain the weight IM m Based on the weight IM m , for the second specific feature F m1 Weighted, the third feature is obtained The fourth fusion unit is used to combine the third feature Reshape along the spatial dimension to obtain the sixth feature The sixth feature Cascade to obtain the seventh feature The second fusion token F f1 Plastic surgery to obtain the eighth feature Based on the modality mask Transformer layer, the seventh feature And the eighth feature Fusion is performed along the channel dimension to obtain the fourth feature 8. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 1, characterized in that: The brain tumor segmentation model includes a decoder that includes a multi-scale dilated attention mechanism and a channel fusion Transformer layer.

9. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 8, characterized in that: The prediction unit specifically includes: The first prediction unit is used to transform the fourth feature Input the decoder to obtain the third specific feature F m2 , based on the channel fusion Transformer layer, the third specific feature F m2 And the fourth feature Perform fusion to obtain the ninth feature D1; The second prediction unit is used to cascade the ninth feature D1 to obtain a final feature D, obtain the first loss L1 based on the final feature D, and obtain a predicted segmentation result based on the first loss L1.

10. The multimodal brain tumor image segmentation system based on sparse attention mechanism according to claim 3, characterized in that: The calculation method for obtaining the second loss L2 is: L rgl =∑ m∈M L Dice (D rgl (E m (x m )),Y g )+L WCE (D rgl (E m (x m )),Y g ); Among them, L rgl represents the second loss L2, L Dice represents the Dice loss function, D rgl Represents the second encoder, E m Indicates the first encoder, L WCE represents the weighted cross entropy loss function, Y g represents the true annotation of brain tumor, x m represents any 3D input modality path, m represents the modality, and M represents the first specific feature F m , including a first mode T1, a second mode T2, a third mode T1c and a fourth mode Flair.

Images