Magnetic resonance brain tumor segmentation method based on reversible feature generation

Abstract

The invention belongs to the field of digital image processing and pattern recognition, and particularly relates to a magnetic resonance brain tumor segmentation method based on reversible feature generation, which comprises the following steps: constructing and training a human brain tumor image segmentation model, and inputting an image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model comprises a multi-modal parallel processing module, a global-local specific modal feature interaction module, a reversible prompt generation module and a grouping fusion guide module; according to the invention, the segmentation precision of the magnetic resonance brain tumor image on the whole tumor area, the tumor core area and the enhanced tumor area can be improved.

Description

Technical Field

[0001] The invention belongs to the field of digital image processing and pattern recognition, and in particular relates to a magnetic resonance brain tumor segmentation method based on reversible feature generation. Background Art

[0002] Brain tumor segmentation technology refers to the process of identifying and segmenting the tumor area in the brain using computer vision technology. This technology is based on the MRI data of the patient's brain. Specifically, brain tumor segmentation is achieved through images of four modalities: T1-weighted, contrast-enhanced T1-weighted (T1ce), T2-weighted, and T2 fluid-attenuated inversion recovery (FLAIR). Brain tumor segmentation technology is mainly used to provide clinicians with accurate auxiliary diagnostic information and support treatment decision-making. It can be applied in many fields such as smart medical care and biomedical research. However, in practice, due to image quality problems, contrast agent allergies, acquisition costs, or protocol restrictions, one or more modalities are often missing. Most existing methods are designed for all modalities to be complete in an ideal situation, and the performance may be affected in the case of modality missing. In addition, current methods often ignore or fail to fully utilize the global semantic information of brain tumors, but instead focus too much on the extraction of local features and fail to combine the advantages of global and local information. Summary of the invention

[0003] To solve the above problems, the present invention provides a magnetic resonance brain tumor segmentation method based on reversible feature generation, constructs and trains a human brain tumor image segmentation model, and inputs the image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model includes a multimodal parallel processing module, a global-local specific modality feature interaction module, a reversible prompt generation module, and a group fusion guidance module.

[0004] The training process of the human brain tumor image segmentation model includes the following steps:

[0005] S1. Acquire original brain tumor image data and perform preprocessing to obtain a preprocessed data set, wherein the preprocessed data includes multiple image groups, each image group includes images of different modalities;

[0006] S2. Input the image group into the multimodal parallel processing module to obtain a detail feature map and a local feature map of each type of image in the image group;

[0007] S3. All local feature maps are input into a global-local specific modality feature interaction module to obtain a plurality of feature interaction maps; wherein the priorities of all modalities of the image are set from large to small as T1ce modality, T1 modality, Flair modality and T2 modality;

[0008] S4. Input all feature interaction graphs into the reversible prompt generation module to obtain corresponding forward features and reverse features;

[0009] S5. Input all reverse features into the group fusion guidance module to obtain the total fusion feature;

[0010] S6. Calculate the first loss according to the forward feature and the reverse feature; calculate the second loss according to the result of the detail feature map passing through the independent modality decoder; calculate the third loss according to the result of the total fusion feature passing through the fusion modality decoder;

[0011] S7. Train the model parameters according to the loss until the model parameters converge.

[0012] Beneficial effects of the present invention:

[0013] The present invention can improve the segmentation accuracy of magnetic resonance brain tumor images for the entire tumor area, the tumor core area and the enhanced tumor area.

[0014] In actual clinical applications, obtaining complete four-modal brain MRI images may be limited by equipment, time or patient conditions. Existing methods basically rely on complete four-modal images for segmentation. When a certain modality image is missing, it may cause information loss and affect the segmentation accuracy. The present invention designs a reversible prompt generation module to learn the complementary information between modalities, and then estimate the relevant information of the missing modality, reducing the dependence on complete data, reducing the impact caused by missing data during the segmentation process, and maintaining a high segmentation accuracy. This method can adapt to the situation of missing modalities, so that the segmentation algorithm still has good robustness under different image quality and acquisition conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a training flow chart of the method of the present invention;

[0016] Figure 2 It is a schematic diagram of the structure of the model of the present invention;

[0017] Figure 3 It is a schematic diagram of the structure of the global-local specific modal feature interaction module of the present invention;

[0018] Figure 4 This is a schematic diagram of the structure of the reversible prompt generation module of the present invention;

[0019] Figure 5 It is a schematic diagram of the structure of the group fusion guidance module of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention provides a magnetic resonance brain tumor segmentation method based on reversible feature generation, comprising constructing and training a human brain tumor image segmentation model, inputting an image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model comprises a multi-modal parallel processing module, a global-local specific modality feature interaction module, a reversible prompt generation module, and a group fusion guidance module.

[0022] like Figure 1 , Figure 2 As shown in Figure 1, the training process of the human brain tumor image segmentation model includes the following steps:

[0023] S1. Acquire original brain tumor image data and perform preprocessing to obtain a preprocessed data set, wherein the preprocessed data includes multiple image groups, each image group includes images of different modalities.

[0024] Specifically, the original brain tumor image data includes original brain tumor images of different modalities of different patients. Each original brain tumor image is cropped to a minimum area containing non-zero pixels, and the cropped image size is ensured to be at least 128×128×128.

[0025] In the preprocessed data, each image group includes images of different modalities of the same patient. In the embodiment of the present invention, an image group includes images of four modalities, namely T1ce modality, T1 modality, Flair modality and T2 modality; each modality corresponds to one image. During the training process, the image group including all modalities is input into the model for training, and then some modalities are randomly discarded to simulate the missing situation.

[0026] S2. Input the image group into the multimodal parallel processing module to obtain the detail feature map and local feature map of each modality image in the image group.

[0027] Specifically, the multimodal parallel processing module includes a ResUnet network, an FCA module and a PE module; step S2 obtains a local feature map of any image in the image group through the multimodal parallel processing module, including:

[0028] The image is input into the ResUnet network, and the output of the fifth residual block in the ResUnet network is used as the detail feature map. The detail feature map is passed through the FCA module and the PE module to obtain the local feature map. The size of each local feature map is 1×128×8×8×8, where 1×128×8×8×8 refers to the batch size, number of channels, length, width, and height from left to right.

[0029] The FCA module is mainly used for flattening operations, and the PE module is used for position encoding processing.

[0030] S3. All local feature maps are input into the global-local specific modality feature interaction module to obtain multiple feature interaction maps; among them, priorities are set for all image modalities according to the degree of influence on the segmentation results, and the priorities are T1ce modality, T1 modality, Flair modality and T2 modality from large to small.

[0031] Specifically, the global-local specific modality feature interaction module performs information interaction between modalities by selecting the highest-level modality among the available modalities (the currently owned modality) as the interaction modality. The global-local specific modality feature interaction module includes a Cross-Attention Transformer module and a Swin Transformer module.

[0032] Preferably, the specific process of step S3 includes:

[0033] S31. All local feature maps Arrange the corresponding modes in descending order of priority, and use the first local feature map as the interactive mode feature map. That is, the mode corresponding to the first local feature map is selected as the interactive mode. If there are local feature maps of the four available modes, namely, T1ce mode, T1 mode, Flair mode and T2 mode, the local feature map corresponding to the T1ce mode is selected as the interactive mode feature map; if only the T1ce mode is missing, the local feature map corresponding to the T1 mode is selected as the interactive mode feature map. Among them, i=0,1,…,3 is the mode category, when i=0, it means corresponding to the T1ce mode, when i=1, it means corresponding to the T1 mode, when i=2, it means corresponding to the Flair mode, and when i=3, it means corresponding to the T2 mode.

[0034] S32. For each local feature map, input it into the Cross-Attention Transformer module and the Swin Transformer module respectively to obtain the first feature and the second feature; the first feature and the second feature corresponding to the interactive modal feature map are recorded as the first interactive feature and the second interactive feature.

[0035] Specifically, Figure 3 As shown, in step S32, any local feature map The process of inputting the Cross-AttentionTransformer module to obtain the first feature includes:

[0036] S321. Local feature map Input the LN layer to obtain vector matrices Q, K, and V; multiply the vector matrix Q and the vector matrix K and process them through the Softmax function layer, and multiply the processing result with the vector matrix V to obtain the weight feature;

[0037] S322. The weighted features are filtered by Gabor filter and then compared with the local feature map Add together to get the attention feature map;

[0038] S323. Pass the attention feature map through the LN layer and the feedforward neural network to obtain an important feature map, and add the important feature map and the attention feature map to obtain the first feature.

[0039] The above process can be expressed by the following formula:

[0040]

[0041] in

[0042]

[0043] Wq, Wk, and Wv all represent weight matrices, LayerNorm(·) represents LN layer, G(·) represents Gabor filter, and FFN(·) represents feedforward neural network.

[0044] S33. For each local feature map, multiply its first feature by the second interactive feature to obtain a first fused feature, and multiply its second feature by the first interactive feature to obtain a second fused feature; add the first fused feature and the second fused feature to obtain a feature interaction map.

[0045] Specifically, the local feature map Input the Cross-Attention Transformer module and the SwinTransformer module respectively to convert the image data into long sequence data and obtain the first feature And the second feature The size of both features is 1×512×512; they can be expressed as

[0046]

[0047] Among them, CrossAttentionTransformer(·) represents the Cross-Attention Transformer module, and SwinTransformer(·) represents the Swin Transformer module.

[0048] Combine it with the interactive modal feature map The first interactive feature and the second interaction feature The fusion is expressed as

[0049]

[0050] Among them, R 1,i Represents the second interaction feature With local feature map The first characteristic The first fusion feature obtained by dot product, R 2,i Represents the first interaction feature With local feature map The second characteristic The second fusion feature obtained by dot product; W1 and W2 are weight parameters, W1+W2=1; represents the feature interaction graph, whose size is 1×512×512; ⊙ represents the point-to-point multiplication of the feature matrix.

[0051] S4. Input all feature interaction graphs into the reversible prompt generation module to obtain the corresponding forward features and reverse features.

[0052] Specifically, the reversible prompt generation module generates cross-modal prompts that retain content based on the feature interaction graph. The reversible prompt generation module can generate high-quality prompt features for the missing modality from the available modalities. On the one hand, the prompt features obtain stronger discriminative information than the available modality, and on the other hand, they suppress the noise in the missing modality. With the reverse conduction and gradient update of the network model training process, the ability of the reversible prompt generation module to generate missing modality information is further improved.

[0053] The processing of the reversible hint generation module can be divided into a forward process and a reverse process. Any feature interaction graph in step S4 Input the reversible prompt generation module to obtain the corresponding reverse features, such as Figure 4 As shown, including:

[0054] Forward process:

[0055] S41. Feature interaction graph Divide into first interaction graph and the second interaction diagram

[0056] S42. The first interactive diagram After the linear network and the second interaction graph Add to get the first addition result Add the first result After the linear network and the first interaction graph Add to get the second addition result Add the first result Add the result with the second Concatenate to get the forward feature f f,i , which can be expressed as

[0057]

[0058] Among them, L1(·) and L2(·) both represent linear neural networks, and Concat(·) represents a concatenation operation.

[0059] Reverse process:

[0060] S43. Divide the forward feature into a first forward feature and a second forward feature;

[0061] S44. Subtract the first forward feature from the second forward feature after passing through the linear network to obtain a first subtraction result; subtract the first subtraction result from the first forward feature after passing through the linear network to obtain a second subtraction result; concatenate the first subtraction result and the second subtraction result to obtain a reverse feature f r,i .

[0062] Specifically, during the test, only the reverse process of the reversible hint generation module is executed to obtain the feature information of the missing modality. On the one hand, stronger discriminant information is obtained than that of the available modality, and on the other hand, the noise in the missing modality is suppressed.

[0063] S5. Input all reverse features into the group fusion guidance module to obtain the total fusion feature.

[0064] Specifically, the group fusion guidance module first groups the reverse features of T1, Flair, T2, and T1ce according to different weights and then inputs them into the attention module respectively. Secondly, the feature information of the four modalities is fused, and finally the three are fused together. The fused feature information not only retains the global semantic information of human brain tumors, but also enhances the expression ability of the corresponding local features of the four modalities.

[0065] like Figure 5 As shown, step S5 specifically includes:

[0066] S51. The inverse feature f of T1 mode r,T1 The inverse characteristic f of the Flair moder,Flair Perform weighted concatenation to obtain the first weighted concatenation feature f1, which can be expressed as

[0067] f1=Concat(W T1 ·f r,T1 ,W Flair ·f r,Flair )

[0068] W T1 , W Flair Represents weight.

[0069] S52. The inverse feature f of the T2 mode r,T2 The inverse feature f of the T1ce mode r,T1ce Perform weighted concatenation to obtain the second weighted concatenation feature f2, which can be expressed as

[0070] f2=Concat(W T2 ·f r,T2 ,W T1ce ·f r,T1ce )

[0071] W T2 , W T1ce Represents weight.

[0072] S53. Weighted concatenation of all reverse features to obtain a third weighted concatenation feature f all , which can be expressed as

[0073] f all =Concat(W T1 ·f r,T1 ,W T2 ·f r,T2 ,W T1ce ·f r,T1ce ,W FLAIR ·f r,Flair )

[0074] S54. The first weighted splicing feature is subjected to Cross Attention and a feedforward neural network to obtain a first processing result; the second weighted splicing feature is subjected to Cross Attention and a feedforward neural network to obtain a second processing result; the third weighted splicing feature is subjected to Cross Attention and a feedforward neural network to obtain a third processing result.

[0075] S55. The first processing result is concatenated with the second processing result and then added to the third processing result to obtain the total fusion feature f final , which can be expressed as

[0076] f final=Concat(CrossAttention(f1),CrossAttention(f2))+CrossAttention(f all )

[0077] S6. Calculate the first loss based on the forward features and the reverse features; calculate the second loss based on the result of the detail feature map passing through the independent modality decoder; calculate the third loss based on the result of the total fusion feature passing through the fusion modality decoder.

[0078] Specifically, feature metric learning is performed based on forward features and reverse features, and mean square error loss is used for calculation. The formula is as follows:

[0079]

[0080] L rpf represents the forward loss, L rpr represents the reverse loss, N represents the number of samples, a feature interaction graph representing the selected interaction modality, It represents the forward feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. It represents the reverse feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. The nth feature interaction graph of modality type i.

[0081] Specifically, the second loss and then the third loss are calculated using the weighted cross entropy loss function, expressed as

[0082]

[0083] Where M represents the number of pixels, y mi represents the probability that the modality type of the mth pixel is i=0, 1, 2, 3, t m represents the true label of the mth pixel, w i Represents the weight of the m-th pixel.

[0084] S7. Train the model parameters according to the loss until the model parameters converge. By using the method model proposed by the present invention, inputting several modality images of a patient into the model, a segmentation result can be obtained.

[0085] In one embodiment, a magnetic resonance brain tumor segmentation method based on reversible feature generation is constructed as follows: Figure 2The network model structure shown is tested on the BraTS2018 dataset. In the field of human brain tumor segmentation research, the Dice similarity coefficient (DSC) is generally used to evaluate the performance of the method on the human brain tumor segmentation dataset. At the same time, Table 1 compares the method 1 using the present invention, the method 2 using the fusion of convolutional neural network and Transformer, and the method 3 using region-aware fusion.

[0086] Table 1 Comparison of experimental results (%)

[0087]

[0088]

[0089] It can be seen from the above experimental results that the method of the present invention has good segmentation performance for images of different modalities, and its overall average performance is better than other methods.

[0090] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0091] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A magnetic resonance brain tumor segmentation method based on reversible feature generation, characterized in that: Constructing and training a human brain tumor image segmentation model, and inputting the image to be processed into the trained human brain tumor image segmentation model to obtain a segmentation result; the human brain tumor image segmentation model includes a multi-modal parallel processing module, a global-local specific modality feature interaction module, a reversible prompt generation module, and a group fusion guidance module; The training process of the human brain tumor image segmentation model includes the following steps: S1. Acquire original brain tumor image data and perform preprocessing to obtain a preprocessed data set, wherein the preprocessed data includes multiple image groups, each image group includes images of different modalities; S2. Input the image group into the multi-modal parallel processing module to obtain a detail feature map and a local feature map of each modality image in the image group; S3. All local feature maps are input into a global-local specific modality feature interaction module to obtain a plurality of feature interaction maps; wherein the priorities of all modalities of the image are set from large to small as T1ce modality, T1 modality, Flair modality and T2 modality; S4. Input all feature interaction graphs into the reversible prompt generation module to obtain corresponding forward features and reverse features; S5. Input all reverse features into the group fusion guidance module to obtain the total fusion feature; S6. Calculate the first loss according to the forward feature and the reverse feature; calculate the second loss according to the result of the detail feature map passing through the independent modality decoder; calculate the third loss according to the result of the total fusion feature passing through the fusion modality decoder; S7. Train the model parameters according to the loss until the model parameters converge.

2. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: The multimodal parallel processing module includes ResUnet network, FCA module and PE module; Step S2 obtains a local feature map of any image in the image group through a multi-modal parallel processing module, including: The image is input into the ResUnet network, the output of the 5th residual block in the ResUnet network is used as the detail feature map, and the detail feature map is passed through the FCA module and the PE module to obtain the local feature map.

3. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: The global-local specific modality feature interaction module includes a Cross-Attention Transformer module and a Swin Transformer module; step S3 specifically includes: S31. Arrange all local feature maps in descending order according to the priority of the corresponding modality, and take the first local feature map as the interactive modality feature map S32. for each local feature map, input it into the Cross-Attention Transformer module and the Swin Transformer module respectively to obtain the first feature and the second feature; the first feature and the second feature corresponding to the interactive modal feature map are recorded as the first interactive feature and the second interactive feature; S33. For each local feature map, multiply its first feature by the second interactive feature to obtain a first fused feature, and multiply its second feature by the first interactive feature to obtain a second fused feature; add the first fused feature and the second fused feature to obtain a feature interaction map.

4. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 3, characterized in that: In step S32, the process of inputting any local feature map into the Cross-Attention Transformer module to obtain the first feature includes: S321. Input the local feature map into the LN layer to obtain vector matrices Q, K, and V; multiply the vector matrix Q and the vector matrix K and process them through the Softmax function layer, and multiply the processing result with the vector matrix V to obtain the weight feature; S322. Add the weighted feature to the local feature map after passing it through the Gabor filter to obtain an attention feature map; S323. Pass the attention feature map through the LN layer and the feedforward neural network to obtain an important feature map, and add the important feature map and the attention feature map to obtain the first feature.

5. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: Any feature interaction graph in step S4 is input into the reversible prompt generation module to obtain the corresponding reverse feature, including: S41. Dividing the feature interaction graph into a first interaction graph and a second interaction graph; S42. Add the first interaction graph to the second interaction graph after passing through the linear network to obtain a first addition result; add the first addition result to the first interaction graph after passing through the linear network to obtain a second addition result; and concatenate the first addition result and the second addition result to obtain a forward feature; S43. Divide the forward feature into a first forward feature and a second forward feature; S44. Subtract the first forward feature from the second forward feature after passing through a linear network to obtain a first subtraction result; subtract the first subtraction result from the first forward feature after passing through a linear network to obtain a second subtraction result; and concatenate the first subtraction result and the second subtraction result to obtain a reverse feature.

6. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: Step S5 includes: S51. weighted concatenation of the reverse feature of the T1 mode and the reverse feature of the Flair mode to obtain a first weighted concatenation feature; S52. weighted concatenation of the reverse feature of the T2 mode and the reverse feature of the T1ce mode to obtain a second weighted concatenation feature; S53. Weighted concatenation of all the reverse features is performed to obtain a third weighted concatenation feature; S54. The first weighted splicing feature is subjected to Cross Attention and a feedforward neural network to obtain a first processing result; the second weighted splicing feature is subjected to Cross Attention and a feedforward neural network to obtain a second processing result; the third weighted splicing feature is subjected to Cross Attention and a feedforward neural network to obtain a third processing result; S55. Concatenate the first processing result and the second processing result and add them to the third processing result to obtain a total fusion feature.

7. The magnetic resonance brain tumor segmentation method based on reversible feature generation according to claim 1, characterized in that: The first loss includes forward loss and reverse loss, expressed as L rpf represents the forward loss, L rpr represents the reverse loss, N represents the number of samples, a feature interaction graph representing the selected interaction modality, It represents the forward feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. It represents the reverse feature obtained by the reversible prompt generation module for the nth feature interaction graph of modality type i. The nth feature interaction graph of modality type i.

Images