A method for magnetic resonance brain tumor segmentation based on reversible feature generation

Abstract

The application belongs to the field of digital image processing and pattern recognition, and particularly relates to a magnetic resonance brain tumor segmentation method based on reversible feature generation, which comprises the following steps: constructing and training a human brain tumor image segmentation model, inputting an image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model comprises a multi-modal parallel processing module, a global-local specific modal feature interaction module, a reversible prompt generation module and a grouping fusion guiding module; and the application can improve the segmentation accuracy of the magnetic resonance brain tumor image on the entire tumor region, the tumor core region and the enhanced tumor region.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing and pattern recognition, and in particular relates to a magnetic resonance brain tumor segmentation method based on reversible feature generation. Background Art

[0002] Brain tumor segmentation technology uses computer vision to identify and segment brain tumor regions. This technology is based on MRI data from the patient's brain. Specifically, it uses four image modalities: T1-weighted, contrast-enhanced T1-weighted (T1ce), T2-weighted, and T2 fluid-attenuated inversion recovery (FLAIR). Brain tumor segmentation technology is primarily used to provide clinicians with accurate auxiliary diagnostic information to support treatment decision-making. It has applications in various fields, including smart healthcare and biomedical research. However, in practice, one or more modalities are often missing due to image quality issues, contrast agent allergies, acquisition costs, or protocol limitations. Most existing methods are designed for the ideal case where all modalities are present, and their performance may be affected when modalities are missing. Furthermore, current methods often ignore or fail to fully utilize the global semantic information of brain tumors. Instead, they focus too much on extracting local features, failing to combine the advantages of global and local information. Summary of the Invention

[0003] To solve the above problems, the present invention provides a magnetic resonance brain tumor segmentation method based on reversible feature generation, constructs and trains a human brain tumor image segmentation model, and inputs the image to be processed into the trained human brain tumor image segmentation model to obtain the segmentation result; the human brain tumor image segmentation model includes a multimodal parallel processing module, a global-local specific modality feature interaction module, a reversible prompt generation module, and a group fusion guidance module.

[0004] The training process of the human brain tumor image segmentation model includes the following steps:

[0005] S1. Acquire raw brain tumor image data and preprocess it to obtain a preprocessed dataset, wherein the preprocessed data includes multiple image groups, each image group including images of different modalities;

[0006] S2. Input the image group into the multimodal parallel processing module to obtain detail feature maps and local feature maps for each type of image in the image group;

[0007] S3. All local feature maps are input into a global-local modality-specific feature interaction module to obtain multiple feature interaction maps, wherein the priorities of all image modalities are set in descending order: T1ce modality, T1 modality, Flair modality, and T2 modality;

[0008] S4. Input all feature interaction graphs into the reversible prompt generation module to obtain corresponding forward features and reverse features;

[0009] S5. Input all reverse features into the group fusion guidance module to obtain the total fusion feature;

[0010] S6. Calculate the first loss based on the forward features and the reverse features; calculate the second loss based on the result of the detail feature map passing through the independent modality decoder; calculate the third loss based on the result of the total fusion feature passing through the fusion modality decoder;

[0011] S7. Train the model parameters according to the loss until the model parameters converge.

[0012] Beneficial effects of the present invention:

[0013] The present invention can improve the segmentation accuracy of magnetic resonance brain tumor images for the entire tumor area, the tumor core area and the enhanced tumor area.

[0014] In actual clinical applications, obtaining complete quadrimodal brain MRI images may be limited by equipment, time, or patient conditions. Existing methods rely on complete quadrimodal images for segmentation. Missing images from any modality can lead to information loss, compromising segmentation accuracy. However, the present invention employs a reversible cue generation module to learn complementary information between modalities, thereby estimating information about the missing modality. This reduces reliance on complete data, mitigates the impact of missing data during the segmentation process, and maintains high segmentation accuracy. This method is adaptable to the presence of missing modalities, ensuring robustness across varying image quality and acquisition conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a training flow chart of the method of the present invention;

[0016] Figure 2 Schematic diagram of the model structure of the present invention;

[0017] Figure 3 Schematic diagram of the global-local specific modal feature interaction module structure of the present invention;

[0018] Figure 4 This is a schematic diagram of the structure of the reversible prompt generation module of the present invention;

[0019] Figure 5 This is a structural diagram of the group fusion guidance module of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] The present invention provides a magnetic resonance brain tumor segmentation method based on reversible feature generation, comprising constructing and training a human brain tumor image segmentation model, inputting the image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model comprises a multimodal parallel processing module, a global-local specific modality feature interaction module, a reversible prompt generation module, and a group fusion guidance module.

[0022] like Figure 1 、 Figure 2 As shown in Figure 2, the training process of the human brain tumor image segmentation model includes the following steps:

[0023] S1. Acquire original brain tumor image data and preprocess it to obtain a preprocessed data set, where the preprocessed data includes multiple image groups, each image group including images of different modalities.

[0024] Specifically, the original brain tumor image data includes original brain tumor images of different modalities from different patients. Each original brain tumor image is cropped to a minimum area containing non-zero pixels, and the cropped image size is ensured to be at least 128×128×128.

[0025] In the preprocessed data, each image set includes images from the same patient across different modalities. In this embodiment, an image set includes images from four modalities: T1ce, T1, Flair, and T2; each modality has one image. During training, the model is fed with image sets containing all modalities, and some modalities are subsequently randomly discarded to simulate missing images.

[0026] S2. Input the image group into the multimodal parallel processing module to obtain the detail feature map and local feature map of each modality image in the image group.

[0027] Specifically, the multimodal parallel processing module includes a ResUnet network, an FCA module, and a PE module. Step S2 obtains a local feature map of any image in the image group through the multimodal parallel processing module, including:

[0028] The image is fed into the ResUnet network, and the output of the fifth residual block in the ResUnet network is used as a detail feature map. The detail feature map is passed through the FCA module and the PE module to obtain a local feature map. The size of each local feature map is 1×128×8×8×8, where 1×128×8×8×8 represents the batch size, number of channels, length, width, and height from left to right.

[0029] The FCA module is mainly used for flattening operations, and the PE module is used for position encoding processing.

[0030] S3. All local feature maps are input into the global-local modality-specific feature interaction module to obtain multiple feature interaction maps; among them, priorities are set for all image modalities according to the degree of influence on the segmentation result, and the priorities are T1ce mode, T1 mode, Flair mode, and T2 mode from large to small.

[0031] Specifically, the global-local specific modality feature interaction module performs information interaction between modalities by selecting the highest-level modality among the available modalities (the currently owned modalities) as the interaction modality. The global-local specific modality feature interaction module includes a Cross-Attention Transformer module and a Swin Transformer module.

[0032] Preferably, the specific process of step S3 includes:

[0033] S31. All local feature maps Arrange in descending order of priority of the corresponding modalities, and use the first local feature map as the interactive modal feature map The modality corresponding to the first local feature map is selected as the interactive modality. If local feature maps for the four available modalities, T1ce, T1, Flair, and T2, are available, the local feature map corresponding to the T1ce modality is selected as the interactive modality feature map. If only the T1ce modality is missing, the local feature map corresponding to the T1 modality is selected as the interactive modality feature map. Here, i = 0, 1, …, 3 represents the modality category. When i = 0, it corresponds to the T1ce modality; when i = 1, it corresponds to the T1 modality; when i = 2, it corresponds to the Flair modality; and when i = 3, it corresponds to the T2 modality.

[0034] S32. For each local feature map, input it into the Cross-Attention Transformer module and the Swin Transformer module respectively to obtain the first feature and the second feature; the first feature and the second feature corresponding to the interactive modal feature map are recorded as the first interactive feature and the second interactive feature.

[0035] Specifically, if Figure 3 As shown, in step S32, any local feature map The process of inputting the Cross-AttentionTransformer module to obtain the first feature includes:

[0036] S321. Local feature map Input the LN layer to obtain vector matrices Q, K, and V; multiply the vector matrix Q and the vector matrix K, and then process them through the Softmax function layer. The processed result is multiplied by the vector matrix V to obtain the weight feature;

[0037] S322. The weighted features are passed through the Gabor filter and then combined with the local feature map Add up to get the attention feature map;

[0038] S323. Pass the attention feature map through the LN layer and the feedforward neural network to obtain an important feature map, and add the important feature map and the attention feature map to obtain the first feature.

[0039] The above process can be expressed by the following formula

[0040]

[0041] in

[0042]

[0043] Wq, Wk, and Wv all represent weight matrices, LayerNorm(·) represents the LN layer, G(·) represents the Gabor filter, and FFN(·) represents the feedforward neural network.

[0044] S33. For each local feature map, multiply its first feature by the second interactive feature to obtain a first fused feature, and multiply its second feature by the first interactive feature to obtain a second fused feature; and add the first fused feature and the second fused feature to obtain a feature interaction map.

[0045] Specifically, the local feature map Input the Cross-Attention Transformer module and SwinTransformer module respectively to convert the image data into long sequence data and obtain the first feature and the second feature The size of both features is 1×512×512; they can be expressed as

[0046]

[0047] Among them, CrossAttentionTransformer(·) represents the Cross-Attention Transformer module, and SwinTransformer(·) represents the Swin Transformer module.

[0048] Combine it with the interactive modal feature map The first interactive feature and the second interaction feature Fusion is expressed as

[0049]

[0050] Among them, R 1,i Represents the second interaction feature With local feature map The first characteristic The first fusion feature obtained by dot product, R 2,i Represents the first interaction feature With local feature map The second characteristic The second fusion feature obtained by dot product; W1 and W2 are weight parameters, W1+W2=1; Represents the feature interaction graph, whose size is 1×512×512; ⊙ represents the point-to-point multiplication of the feature matrix.

[0051] S4. Input all feature interaction graphs into the reversible prompt generation module to obtain the corresponding forward features and reverse features.

[0052] Specifically, the reversible prompt generation module generates cross-modal prompts that retain content based on the feature interaction graph. The reversible prompt generation module can generate high-quality prompt features for the missing modality from the available modalities. On the one hand, the prompt features obtain stronger discriminative information than the available modality, and on the other hand, they suppress the noise in the missing modality. With the reverse conduction and gradient update of the network model training process, the ability of the reversible prompt generation module to generate missing modality information is further improved.

[0053] The processing of the reversible prompt generation module can be divided into a forward process and a reverse process. In step S4, any feature interaction graph Input the reversible hint generation module to obtain the corresponding reverse features, such as Figure 4 Shown, including:

[0054] Forward process:

[0055] S41. Feature interaction graph Divide into first interaction graphs and the second interaction diagram

[0056] S42. The first interactive diagram After the linear network and the second interaction graph Add to get the first addition result The first addition result After the linear network and the first interaction graph Add to get the second addition result The first addition result Add the result with the second Splicing to get the forward feature f f,i , which can be expressed as

[0057]

[0058] Among them, L1(·) and L2(·) both represent linear neural networks, and Concat(·) represents the concatenation operation.

[0059] Reverse process:

[0060] S43. Dividing the forward feature into a first forward feature and a second forward feature;

[0061] S44. Subtract the first forward feature from the second forward feature after passing it through the linear network to obtain a first subtraction result; subtract the first subtraction result from the first forward feature after passing it through the linear network to obtain a second subtraction result; concatenate the first subtraction result and the second subtraction result to obtain a reverse feature f r,i .

[0062] Specifically, during the test, only the reverse process of the reversible hint generation module is executed to obtain the feature information of the missing modality. On the one hand, stronger discriminant information is obtained than that of the available modality, and on the other hand, the noise in the missing modality is suppressed.

[0063] S5. Input all reverse features into the group fusion guidance module to obtain the total fusion feature.

[0064] Specifically, the group fusion guidance module first groups the reverse features of T1, Flair, T2, and T1ce and fuses them according to different weights. Then, they are input into the attention module respectively. Secondly, the feature information of the four modalities is fused, and finally the three are fused together. The fused feature information not only retains the global semantic information of human brain tumors, but also enhances the expression ability of the corresponding local features of the four modalities.

[0065] like Figure 5 As shown, step S5 specifically includes:

[0066] S51. The reverse feature f of T1 mode r,T1 The inverse characteristic f of the Flair moder,Flair Perform weighted splicing to obtain the first weighted splicing feature f1, which can be expressed as

[0067] f1=Concat(W T1 ·f r,T1 ,W Flair ·f r,Flair )

[0068] W T1 、W Flair Represents weight.

[0069] S52. The reverse feature f of T2 mode r,T2 The inverse feature f of the T1ce mode r,T1ce Perform weighted splicing to obtain the second weighted splicing feature f2, which can be expressed as

[0070] f2=Concat(W T2 ·f r,T2 ,W T1ce ·f r,T1ce )

[0071] W T2 、W T1ce Represents weight.

[0072] S53. Weighted concatenation of all reverse features to obtain the third weighted concatenation feature f all , which can be expressed as

[0073] f all =Concat(W T1 ·f r,T1 ,W T2 ·f r,T2 ,W T1ce ·f r,T1ce ,W FLAIR ·f r,Flair )

[0074] S54. The first weighted splicing feature is processed by Cross Attention and a feedforward neural network to obtain a first processing result; the second weighted splicing feature is processed by Cross Attention and a feedforward neural network to obtain a second processing result; the third weighted splicing feature is processed by Cross Attention and a feedforward neural network to obtain a third processing result.

[0075] S55. The first processing result and the second processing result are concatenated and added to the third processing result to obtain the total fusion feature f final , which can be expressed as

[0076] f final=Concat(CrossAttention(f1),CrossAttention(f2))+CrossAttention(f all )

[0077] S6. Calculate the first loss based on the forward features and the reverse features; calculate the second loss based on the result of the detail feature map passing through the independent modality decoder; calculate the third loss based on the result of the total fusion feature passing through the fusion modality decoder.

[0078] Specifically, feature metric learning is performed based on forward features and reverse features, and mean square error loss is used for calculation. The formula is as follows:

[0079]

[0080] L rpf represents the forward loss, L rpr represents the reverse loss, N represents the number of samples, A feature interaction graph representing the selected interaction modality, It represents the forward feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. It represents the reverse feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. The nth feature interaction graph of modality type i.

[0081] Specifically, the second loss and then the third loss are calculated using the weighted cross entropy loss function, expressed as

[0082]

[0083] Where M represents the number of pixels, y mi represents the probability that the modality type of the mth pixel is i=0, 1, 2, 3, t m represents the true label of the m-th pixel, w i represents the weight of the m-th pixel.

[0084] S7. Train the model parameters according to the loss until the model parameters converge. By using the method model proposed by the present invention, inputting several modality images of a patient into the model, a segmentation result can be obtained.

[0085] In one embodiment, a magnetic resonance brain tumor segmentation method based on reversible feature generation is constructed as follows: Figure 2The network model structure shown was tested on the BraTS2018 dataset. In the field of human brain tumor segmentation research, the Dice similarity coefficient (DSC) is generally used to evaluate the performance of methods on human brain tumor segmentation datasets. Table 1 also compares Method 1 using the present invention, Method 2 using a fusion of a convolutional neural network and a Transformer, and Method 3 using region-aware fusion.

[0086] Table 1 Comparison of experimental results (%)

[0087]

[0088]

[0089] The above experimental results show that the method of the present invention has good segmentation performance for images of different modalities, and its overall average performance is better than other methods.

[0090] In the present invention, unless otherwise clearly stipulated and limited, the terms "installation", "setting", "connection", "fixation", "rotation" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0091] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A magnetic resonance imaging brain tumor segmentation method based on reversible feature generation, characterized in that: Constructing and training a human brain tumor image segmentation model, and inputting the image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model includes a multimodal parallel processing module, a global-local specific modality feature interaction module, a reversible prompt generation module, and a group fusion guidance module; The training process of the human brain tumor image segmentation model includes the following steps: S1. Acquire raw brain tumor image data and preprocess it to obtain a preprocessed dataset, wherein the preprocessed data includes multiple image groups, each image group including images of different modalities; S2. Input the image group into the multimodal parallel processing module to obtain detail feature maps and local feature maps of each modality image in the image group; S3. All local feature maps are input into a global-local modality-specific feature interaction module to obtain multiple feature interaction maps, wherein the priorities of all image modalities are set in descending order: T1ce modality, T1 modality, Flair modality, and T2 modality; The global-local specific modality feature interaction module includes a Cross-Attention Transformer module and a Swin Transformer module; step S3 specifically includes: S31. Arrange all local feature maps in descending order of priority according to the corresponding modalities, and use the first local feature map as the interactive modal feature map S32. For each local feature map, input it into the Cross-Attention Transformer module and the Swin Transformer module respectively to obtain the first feature and the second feature; the first feature and the second feature corresponding to the interactive modal feature map are recorded as the first interactive feature and the second interactive feature; S33. For each local feature map, multiply the first feature by the second interactive feature to obtain a first fused feature, and multiply the second feature by the first interactive feature to obtain a second fused feature; and add the first fused feature and the second fused feature to obtain a feature interaction map; S4. Input all feature interaction graphs into the reversible prompt generation module to obtain corresponding forward features and reverse features; In step S4, any feature interaction graph is input into the reversible prompt generation module to obtain the corresponding reverse feature, including: S41. The feature interaction diagram is divided into a first interaction diagram and a second interaction diagram; S42. Adding the first interaction graph to the second interaction graph after passing it through the linear network to obtain a first addition result; adding the first addition result to the first interaction graph after passing it through the linear network to obtain a second addition result; and concatenating the first addition result and the second addition result to obtain a forward feature; S43. Dividing the forward feature into a first forward feature and a second forward feature; S44. Subtracting the first forward feature from the second forward feature after passing it through the linear network to obtain a first subtraction result; subtracting the first subtraction result from the first forward feature after passing it through the linear network to obtain a second subtraction result; and concatenating the first subtraction result and the second subtraction result to obtain a reverse feature; S5. Input all reverse features into the group fusion guidance module to obtain the total fusion feature; Step S5 includes: S51. Perform weighted concatenation on the inverse features of the T1 modality and the Flair modality to obtain a first weighted concatenation feature; S52. Perform weighted concatenation of the reverse feature of the T2 modality and the reverse feature of the T1ce modality to obtain a second weighted concatenation feature; S53. Perform weighted concatenation of all reverse features to obtain a third weighted concatenated feature; S54. The first weighted splicing feature is processed by Cross Attention and a feedforward neural network to obtain a first processing result; the second weighted splicing feature is processed by Cross Attention and a feedforward neural network to obtain a second processing result; the third weighted splicing feature is processed by Cross Attention and a feedforward neural network to obtain a third processing result; S55. Concatenate the first processing result and the second processing result and add the result to the third processing result to obtain a total fusion feature; S6. Calculate the first loss based on the forward features and the reverse features; calculate the second loss based on the result of the detail feature map passing through the independent modality decoder; calculate the third loss based on the result of the total fusion feature passing through the fusion modality decoder; S7. Train the model parameters according to the loss until the model parameters converge.

2. The magnetic resonance imaging brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: The multimodal parallel processing module includes ResUnet network, FCA module and PE module; Step S2 obtains a local feature map of any image in the image group through a multimodal parallel processing module, including: The image is input into the ResUnet network, and the output of the fifth residual block in the ResUnet network is used as the detail feature map. The detail feature map is passed through the FCA module and the PE module to obtain the local feature map.

3. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: In step S32, the process of inputting any local feature map into the Cross-Attention Transformer module to obtain the first feature includes: S321. Input the local feature map into the LN layer to obtain vector matrices Q, K, and V; multiply the vector matrix Q and the vector matrix K, and then process them through the Softmax function layer. The processed result is multiplied by the vector matrix V to obtain the weight feature; S322. The weighted feature is passed through a Gabor filter and then added to the local feature map to obtain an attention feature map; S323. Pass the attention feature map through the LN layer and the feedforward neural network to obtain an important feature map, and add the important feature map and the attention feature map to obtain the first feature.

4. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: The first loss includes forward loss and reverse loss, which is expressed as L rpf represents the forward loss, L rpr represents the reverse loss, N represents the number of samples, A feature interaction graph representing the selected interaction modality, It represents the forward feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. It represents the reverse feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. The nth feature interaction graph of modality type i.

Images