Multimodal brain tumor image segmentation method, device, electronic device and storage medium

Abstract

The present invention provides a multimodal brain tumor image segmentation method, device, electronic device and storage medium, belonging to the field of medical image segmentation technology, the method comprises: obtaining a first brain tumor image and a second brain tumor image; extracting features of each MRI image in the first brain tumor image and the second brain tumor image based on a dual-branch network and a spatial position activation module, respectively, to obtain first target image features and second target image features; obtaining a segmentation result according to the first target image features and the second target image features. The present invention applies multimodal multi-scale feature fusion to MRI image segmentation, the dual-branch network and the spatial position activation module mutually guide the feature extraction of MRI images of different modalities in the feature extraction stage, and can more accurately extract multimodal features, and also utilizes a multi-scale feature aggregation module to fuse the extracted multi-scale features in the feature fusion stage, so as to further improve the accuracy of the image segmentation task.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a multimodal brain tumor image segmentation method, device, electronic equipment and storage medium. Background Art

[0002] In the actual brain tumor treatment process, doctors usually identify and analyze the patient's brain glioma lesion area based on different modal magnetic resonance imaging (MRI) images. The convolutional network (U-Net) for biomedical image segmentation has shown strong performance in the field of medical image segmentation. Its encoder-decoder framework with jump connections has become the mainstream framework for brain tumor image segmentation. Many subsequent network models use U-Net as the backbone network. With the rise of the self-attention model transformer, the transformer is applied to the U-Net model to enhance the feature extraction effect of brain tumor image segmentation.

[0003] The existing brain tumor image segmentation methods only enhance the effect of model feature extraction, but do not achieve the fusion of different modality data, and the accuracy of image segmentation tasks is low. Summary of the invention

[0004] The present invention provides a multimodal brain tumor image segmentation method, device, electronic device and storage medium to solve the defects in the prior art that only the effect of model feature extraction is enhanced but the fusion of different modal data is not realized, resulting in low accuracy of image segmentation tasks.

[0005] The present invention provides a multimodal brain tumor image segmentation method, comprising:

[0006] Acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image;

[0007] Based on a preset dual-branch network and a spatial position activation module, feature extraction is performed on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0008] A segmentation result is obtained according to the first target image feature and the second target image feature.

[0009] According to a multimodal brain tumor image segmentation method provided by the present invention, the dual-branch network includes a first branch unit and a second branch unit, the first branch unit includes a first convolution block and at least one second convolution block, the second branch unit includes a third convolution block and at least one fourth convolution block, each of the second convolution blocks and each of the fourth convolution blocks are respectively connected to one of the spatial position activation modules;

[0010] Correspondingly, the preset dual-branch network and spatial position activation module respectively extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image, including:

[0011] Based on the first convolution block and the second convolution block, sequentially extract features of the first brain tumor image to obtain first image features;

[0012] Based on the third convolution block and the fourth convolution block, sequentially extracting features from the second brain tumor image to obtain second image features;

[0013] Extracting features of the first image based on the spatial position activation module corresponding to the second convolution block to obtain first image spatial features;

[0014] Extracting features of the second image based on the spatial position activation module corresponding to the fourth convolution block to obtain second image spatial features;

[0015] Obtaining the first target image feature according to the first image feature and the second image space feature;

[0016] The second target image feature is obtained according to the second image feature and the first image space feature.

[0017] According to a multimodal brain tumor image segmentation method provided by the present invention, obtaining the first target image feature according to the first image feature and the second image space feature includes:

[0018] The first target image feature is obtained by adding the product of the first image feature and the second image space feature to the first image feature;

[0019] Correspondingly, obtaining the second target image feature according to the second image feature and the first image space feature includes:

[0020] The product of the second image feature and the first image space feature is added to the second image feature to obtain the second target image feature.

[0021] According to a multimodal brain tumor image segmentation method provided by the present invention, the output end of the first branch unit and the output end of the second branch unit are respectively connected to a decoder to determine the loss of the first branch unit and the loss of the second branch unit. The loss of the first branch unit and the loss of the second branch unit are also used to input into a preset total loss function to update the weight parameters of the dual-branch network.

[0022] According to a multimodal brain tumor image segmentation method provided by the present invention, obtaining a segmentation result according to the first target image feature and the second target image feature includes:

[0023] Based on a preset multi-scale feature aggregation module, feature fusion is performed on the first target image feature and the second target image feature to obtain an image fusion feature;

[0024] The image fusion feature is decoded based on a preset decoder to obtain the segmentation result.

[0025] According to a multimodal brain tumor image segmentation method provided by the present invention, the multi-scale feature aggregation module includes a pyramid convolution unit, an upsampling unit, an attention unit and an aggregation unit, the pyramid convolution unit is used to extract the global information and the local information of the first target image feature and the second target image feature respectively; the upsampling unit is used to upsample and splice the global information and the local information to obtain the target feature; the attention unit is used to reconstruct the target feature; the aggregation unit is used to aggregate the global information, the local information and the reconstructed features to obtain the image fusion feature.

[0026] According to a multimodal brain tumor image segmentation method provided by the present invention, the feature reconstruction of the target feature includes:

[0027] Determine a one-dimensional attention map and a two-dimensional attention map according to the target feature;

[0028] The target features are respectively element-wise multiplied according to the one-dimensional attention map and the two-dimensional attention map, and a splicing operation is performed on the products to complete feature reconstruction.

[0029] The present invention also provides a multimodal brain tumor image segmentation device, comprising:

[0030] An acquisition module, configured to acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image;

[0031] a first determination module, configured to extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image based on a preset dual-branch network and a spatial position activation module, respectively, to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0032] The second determination module is used to obtain a segmentation result according to the first target image feature and the second target image feature.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal brain tumor image segmentation method as described above is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal brain tumor image segmentation method as described in any one of the above is implemented.

[0035] The multimodal brain tumor image segmentation method, device, electronic device and storage medium provided by the present invention obtain a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image each include at least one modality MRI image; based on a preset dual-branch network and a spatial position activation module, feature extraction is performed on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image, respectively, to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image; a segmentation result is obtained based on the first target image feature and the second target image feature. The present invention applies multimodal feature fusion to MRI image segmentation. The dual-branch network and the spatial position activation module can guide each other in the feature extraction of MRI images of different modalities in the feature extraction stage, prevent information loss caused by single-scale features, and thereby more accurately extract multimodal features, thereby improving the accuracy of image segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the flow of the multimodal brain tumor image segmentation method provided by the present invention;

[0038] Figure 2 It is a structural schematic diagram of a multi-modal mutual conduction fusion network provided by the present invention;

[0039] Figure 3 The present invention provides Figure 1 Schematic diagram of the process of step S120;

[0040] Figure 4 It is a structural schematic diagram of the spatial position activation module provided by the present invention;

[0041] Figure 5 The present invention provides Figure 1 Schematic diagram of the process of step S130;

[0042] Figure 6 It is a structural schematic diagram of the multi-scale feature aggregation module provided by the present invention;

[0043] Figure 7 is a comparison diagram of image segmentation results of the present invention and the existing method;

[0044] Figure 8 It is a structural schematic diagram of a multimodal brain tumor image segmentation device provided by the present invention;

[0045] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0048] In the prior art, the fusion models of transformer and U-Net models include SwinUNet, UNETR, SegTransVAE and NestedFormer. Specifically, SwinUNet builds a symmetric encoder-decoder architecture with skip connections based on the Swin Transformer block. Self-attention from local to global is implemented in the encoder; in the decoder, the global features are upsampled to the input resolution and the corresponding pixel-level segmentation prediction is performed. UNETR uses a pure transformer as an encoder to learn contextual information from embedded input patches. The representation extracted from the transformer encoder is merged with the decoder through multi-resolution skip connections to predict the segmentation output. SegTransVAE is built on the encoder-decoder architecture and uses a transformer with a variational autoencoder (VAE) branch connected to the network to reconstruct the input image together with the segmentation. NestedFormer is also based on a multi-encoder and single decoder structure, performs nested multimodal fusion on high-level representations of different modalities, and applies modality-sensitive gating (MSG) at a lower scale to achieve more efficient skip connections.

[0049] However, the above fusion models only enhance the effect of model feature extraction, and do not specifically design multimodal data fusion methods based on the complementarity of multimodal MRI data. At the same time, these existing methods ignore the characteristics of data at different scales, and thus cannot well fuse global and local semantic information.

[0050] To this end, the present invention designs a multi-scale based multimodal mutual conduction fusion network, which can be used for multimodal medical image segmentation, represented as MMFA-Net, including 1) a dual-branch network, used as an encoder to extract features of different modalities; 2) a spatial position activation (SPA) module, which helps the encoder to more accurately extract features of different modalities; 3) a multi-scale feature aggregation (MFA) module, which is used to fuse multi-scale features between different modalities. The network provided by the present invention can more accurately extract more comprehensive features of multiple modalities, and at the same time, more thoroughly fuse features of different scales and modalities. The present invention also designs a module for fusing features of different scales, so that semantic information of different dimensions can be more fully utilized to improve the accuracy in downstream segmentation tasks.

[0051] The present invention is described in detail below with reference to the embodiments.

[0052] Figure 1 FIG. 1 is a flow chart of the multimodal brain tumor image segmentation method provided by the present invention. Figure 1 As shown, the present invention provides a multimodal brain tumor image segmentation method, which is implemented based on a multimodal mutual induction fusion network and includes:

[0053] S110, acquiring a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image; there are many differences between multimodal data, and images of different modalities can be combined into a first brain tumor image and a second brain tumor image based on the structural similarity between the multimodal images. Specifically, MRI images generally include four modalities: T1, T2, T1ce, and Flair. The T1ce modality image is formed after adding a developer to the T1 modality image, and the image structure is relatively similar. The present invention combines the T1 modality image and the T1ce modality image as the first brain tumor image, and combines the T2 modality image and the Flair modality image as the second brain tumor image.

[0054] S120, performing feature extraction on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image based on a preset dual-branch network and a spatial position activation module, respectively, to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0055] Specifically, the dual-branch network is used to extract detailed information of the first brain tumor image and the second brain tumor image respectively. In the feature extraction stage, the features of each modality image guide the other modality image to learn each other's position prior knowledge through the spatial position activation module. Since different modality images focus on different areas, the spatial position activation module uses the position prior of the current modality image to guide the other modality image to focus on the spatially significant features of the current modality image.

[0056] S130: Obtain a segmentation result according to the first target image feature and the second target image feature.

[0057] Specifically, the first target image feature and the second target image feature are fused to generate a final image segmentation result.

[0058] It can be understood that the present invention applies multimodal feature fusion to MRI image segmentation. The dual-branch network and the spatial position activation module can guide each other in the feature extraction of MRI images of different modalities in the feature extraction stage, preventing information loss caused by single-scale features, thereby being able to extract multimodal features more accurately, thereby improving the accuracy of image segmentation tasks.

[0059] Figure 2 is a schematic diagram of the structure of the multi-modal mutual conduction fusion network provided by the present invention, such as Figure 2 As shown, based on the above embodiment, as an optional embodiment, the dual-branch network includes a first branch unit and a second branch unit, the first branch unit includes a first convolution block and at least a second convolution block, the second branch unit includes a third convolution block and at least a fourth convolution block, and each of the second convolution blocks and each of the fourth convolution blocks are respectively connected to a spatial position activation module. Specifically, the first convolution block and the third convolution block are only used to extract detailed features of the image and are not connected to the spatial position activation module. The second convolution block and the fourth convolution block are both connected to the spatial position activation module, and their number can be set according to demand. The local window mechanism of the convolution block can effectively extract detailed information of the MRI image. In addition, the characteristic of shared parameters of the convolution block can reduce the parameters of the network.

[0060] Optionally, the dual-branch network uses 3D U-Net as the backbone network, and the network uses independent branches to extract modal features. The convolution block of the dual-branch network is expressed as and in, is the first convolutional block, is the second convolutional block, is the third convolutional block, It is the fourth convolutional block.

[0061] The output end of the first branch unit and the output end of the second branch unit are respectively connected to a decoder to determine the loss of the first branch unit and the loss of the second branch unit. The loss of the first branch unit and the loss of the second branch unit are also used to input a preset total loss function to update the weight parameters of the dual-branch network. Specifically, during the dual-branch network training process, non-modal features can be extracted separately, the features extracted by each branch are directly input into the decoder, and the losses of the two branches are calculated and added to the overall loss function during the iteration, so as to iterate and predict more accurately. After the training is completed, the dual-branch network can be used to extract features of images of different modalities.

[0062] Figure 3 The present invention provides Figure 1 The flowchart of step S120 is as follows: Figure 3 As shown, correspondingly, the preset dual-branch network and spatial position activation module respectively extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image, including:

[0063] S310, extracting features of the first brain tumor image in sequence based on the first convolution block and the second convolution block to obtain first image features;

[0064] S320, extracting features of the second brain tumor image based on the third convolution block and the fourth convolution block in sequence to obtain second image features; the order of S310 and S320 can be adjusted;

[0065] S330, extracting features of the first image based on the spatial position activation module corresponding to the second convolution block to obtain first image spatial features;

[0066] S340, extracting features of the second image features based on the spatial position activation module corresponding to the fourth convolution block to obtain second image spatial features; the order of S330 and S340 can be adjusted;

[0067] S350, obtaining the first target image feature according to the first image feature and the second image space feature;

[0068] S360, obtaining the second target image feature according to the second image feature and the first image space feature. The order of S350 and S360 is adjustable.

[0069] Optionally, obtaining the first target image feature according to the first image feature and the second image space feature includes:

[0070] The first target image feature is obtained by adding the product of the first image feature and the second image space feature to the first image feature;

[0071] Correspondingly, obtaining the second target image feature according to the second image feature and the first image space feature includes:

[0072] The product of the second image feature and the first image space feature is added to the second image feature to obtain the second target image feature.

[0073] Figure 4 is a schematic diagram of the structure of the spatial position activation module provided by the present invention, such as Figure 4 As shown, the output features of each second convolution block and each fourth convolution block are and After the spatial position activation module, we get and Then the output feature representation calculation method of each module-specific encoding in each segment is as follows:

[0074]

[0075] in, represents element-by-element addition, Stands for element-wise multiplication.

[0076] It can be understood that the present invention provides a technical solution of the structural composition of a dual-branch network. The encoder part after combining the dual-branch network and the spatial position activation module (SPA) guides different modalities to perform feature extraction in the feature extraction stage. The dual-branch network provides information to each other while preventing information loss caused by single-scale features, thereby being able to extract multi-modal and multi-scale features more accurately.

[0077] Figure 5 The present invention provides Figure 1 The flowchart of step S130 is as follows: Figure 5 As shown, based on the above embodiment, as an optional embodiment, obtaining a segmentation result according to the first target image feature and the second target image feature includes:

[0078] S510, performing feature fusion on the first target image feature and the second target image feature based on a preset multi-scale feature aggregation module to obtain an image fusion feature;

[0079] S520, decoding the image fusion features based on a preset decoder to obtain the segmentation result. The decoder uses the aggregated features of the four stages of the encoder as input. In order to process features of different scales, all inputs are processed to the same scale through bilinear interpolation operations, and finally fused through splicing and convolution to generate the final segmentation result.

[0080] Figure 6 is a schematic diagram of the structure of the multi-scale feature aggregation module provided by the present invention, such as Figure 6 As shown, optionally, the multi-scale feature aggregation module includes a pyramid convolution unit, an upsampling unit, an attention unit and an aggregation unit, wherein the pyramid convolution unit is used to extract the global information and the local information of the first target image feature and the second target image feature, respectively; the upsampling unit is used to upsample and concatenate the global information and the local information to obtain the target feature; the attention unit is used to reconstruct the target feature; the aggregation unit is used to aggregate the global information, the local information and the reconstructed features to obtain the image fusion feature.

[0081] Optionally, the reconstructing the target feature includes:

[0082] According to the target features, a one-dimensional attention map and a two-dimensional attention map are determined; according to the one-dimensional attention map and the two-dimensional attention map, the target features are respectively element-wise multiplied, and the products are concatenated to complete feature reconstruction.

[0083] Optionally, the present invention proposes a multi-scale feature aggregation module to ensure the effective fusion of cross-modal features. The input of the multi-scale feature aggregation (MFA) module is the aggregated features of two different network branches of the dual-branch network:

[0084]

[0085] The MFA module is based on the input fusion feature F i More specifically, the MFA module constructs a pyramid convolution structure. First, a global average pooling layer and a 7x7x7 3D convolution layer are used to extract the global representation. Then, a 3x3x3 3D ​​convolution layer and a 5x5x5 D convolution layer are used to capture the local detail information of the feature. The outputs of these multi-scale convolution layers are finally upsampled to H×W×D and then concatenated. The concatenated features are passed through a 1x1x1 convolution layer to obtain the target features. Target characteristics After an attention layer, feature reconstruction is performed. Specifically, given a feature map The attention unit uses the input features to infer a 1D attention map on the channel. and the attention map in 2D space The calculation formula of the reconstructed features is as follows:

[0086]

[0087] Among them, [·,·] is the concatenation operation, is element-wise multiplication.

[0088] The calculation formula of image fusion features is as follows:

[0089]

[0090] It can be understood that the present invention uses a multi-scale feature aggregation module to more comprehensively fuse multi-modal and multi-scale features, thereby using the fused features in downstream segmentation tasks, showing excellent image segmentation performance.

[0091] In summary, the present invention is a brand-new multi-scale feature aggregation network for medical image segmentation based on multi-modal MRI data. Through a dual-branch network, semantic features can be independently extracted from MRI images of different modalities. The spatial position activation module can be used to fuse the position information of image features of different modalities between the two branches. The multi-scale feature aggregation module can connect the features of different scales extracted by the dual-branch network to obtain rich global and local information, and combine the rich information to further improve the segmentation accuracy. The present invention extracts and fuses the multi-scale features of multi-modal MRI images of brain tumors to improve the segmentation performance of the lesion area.

[0092] The performance data of the present invention and the existing methods on the BraTS2020 dataset are shown in Table 1.

[0093]

[0094] Table 1

[0095] The performance data of the present invention and the existing methods on the BraTS2021 dataset are shown in Table 2.

[0096]

[0097] Table 2

[0098] The performance data of the present invention and the existing methods on the ISBI dataset are shown in Table 3.

[0099]

[0100]

[0101] Table 3

[0102] The research data of ablation experiments using the present invention are shown in Tables 4 and 5. Table 4 is the impact data of different input combinations on multimodal data, and Table 5 is the ablation research data of different modules of the present invention on the BraTS2021 dataset.

[0103]

[0104] Table 4

[0105]

[0106] Table 5

[0107] In Tables 1 to 5, the evaluation index Dice coefficient represents the similarity between the segmentation result and the actual value, and the larger the value, the more similar they are; the HD95 coefficient represents the distance between the segmentation result and the actual value, and the smaller the value, the more similar they are. Compared with the prior art, the evaluation Dice score of the present invention is increased by about 3%.

[0108] Figure 7 is a comparison diagram of the image segmentation results of the present invention and the existing method, such as Figure 7 As shown in the figure, from left to right are the input image, Groundtruth segmentation result, 3D UNet segmentation result, UNETR segmentation result, SegTransVAE segmentation result, NestedFormer segmentation result, and segmentation result of the present invention. The DBN and SPA modules of the present invention can better extract features of different scales and different modalities, while the MFA module can better fuse the global and local information in these features, so that the accuracy of brain tumor segmentation of the present invention is better.

[0109] The multimodal brain tumor image segmentation device provided by the present invention is described below. The multimodal brain tumor image segmentation device described below and the multimodal brain tumor image segmentation method described above can be referred to each other.

[0110] Figure 8 is a schematic diagram of the structure of the multimodal brain tumor image segmentation device provided by the present invention, such as Figure 8 As shown, the present invention also provides a multimodal brain tumor image segmentation device, comprising:

[0111] An acquisition module 810 is configured to acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image;

[0112] A first determination module 820 is used to extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image based on a preset dual-branch network and a spatial position activation module, to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0113] The second determination module 830 is configured to obtain a segmentation result according to the first target image feature and the second target image feature.

[0114] As an embodiment, the dual-branch network includes a first branch unit and a second branch unit, the first branch unit includes a first convolution block and at least one second convolution block, the second branch unit includes a third convolution block and at least one fourth convolution block, each of the second convolution blocks and each of the fourth convolution blocks are respectively connected to one of the spatial position activation modules;

[0115] Correspondingly, the first determining module 820 is further configured to:

[0116] Based on the first convolution block and the second convolution block, sequentially extract features of the first brain tumor image to obtain first image features;

[0117] Based on the third convolution block and the fourth convolution block, sequentially extracting features from the second brain tumor image to obtain second image features;

[0118] Extracting features of the first image based on the spatial position activation module corresponding to the second convolution block to obtain first image spatial features;

[0119] Extracting features of the second image based on the spatial position activation module corresponding to the fourth convolution block to obtain second image spatial features;

[0120] Obtaining the first target image feature according to the first image feature and the second image space feature;

[0121] The second target image feature is obtained according to the second image feature and the first image space feature.

[0122] As an embodiment, the first determining module 820 is further configured to:

[0123] The first target image feature is obtained by adding the product of the first image feature and the second image space feature to the first image feature;

[0124] Correspondingly, obtaining the second target image feature according to the second image feature and the first image space feature includes:

[0125] The product of the second image feature and the first image space feature is added to the second image feature to obtain the second target image feature.

[0126] As an embodiment, the output end of the first branch unit and the output end of the second branch unit are respectively connected to a decoder to determine the loss of the first branch unit and the loss of the second branch unit. The loss of the first branch unit and the loss of the second branch unit are also used to input into a preset total loss function to update the weight parameters of the dual-branch network.

[0127] As an embodiment, the second determining module 830 is further configured to:

[0128] Based on a preset multi-scale feature aggregation module, feature fusion is performed on the first target image feature and the second target image feature to obtain an image fusion feature;

[0129] The image fusion feature is decoded based on a preset decoder to obtain the segmentation result.

[0130] As an embodiment, the multi-scale feature aggregation module includes a pyramid convolution unit, an upsampling unit, an attention unit and an aggregation unit. The pyramid convolution unit is used to extract the global information and local information of the first target image feature and the second target image feature respectively; the upsampling unit is used to upsample and splice the global information and the local information to obtain the target feature; the attention unit is used to reconstruct the target feature; the aggregation unit is used to aggregate the global information, local information and reconstructed features to obtain the image fusion feature.

[0131] As an embodiment, the second determining module 830 is further configured to:

[0132] Determine a one-dimensional attention map and a two-dimensional attention map according to the target feature;

[0133] The target features are respectively element-wise multiplied according to the one-dimensional attention map and the two-dimensional attention map, and a splicing operation is performed on the products to complete feature reconstruction.

[0134] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the multimodal brain tumor image segmentation method, which includes:

[0135] Acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image;

[0136] Based on a preset dual-branch network and a spatial position activation module, feature extraction is performed on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0137] A segmentation result is obtained according to the first target image feature and the second target image feature.

[0138] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0139] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can perform a multimodal brain tumor image segmentation method, the method comprising:

[0140] Acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image;

[0141] Based on a preset dual-branch network and a spatial position activation module, feature extraction is performed on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0142] A segmentation result is obtained according to the first target image feature and the second target image feature.

[0143] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program is implemented to perform a multimodal brain tumor image segmentation method, the method comprising:

[0144] Acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image;

[0145] Based on a preset dual-branch network and a spatial position activation module, feature extraction is performed on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image;

[0146] A segmentation result is obtained according to the first target image feature and the second target image feature.

[0147] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal brain tumor image segmentation method, characterized in that: include: Acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image; Based on a preset dual-branch network and a spatial position activation module, feature extraction is performed on each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image; The spatial position activation module uses the position prior of the current modality image to guide the other modality image to focus on the spatial salient features of the current modality image; Obtaining a segmentation result according to the first target image feature and the second target image feature; The dual-branch network includes a first branch unit and a second branch unit, the first branch unit includes a first convolution block and at least one second convolution block, the second branch unit includes a third convolution block and at least one fourth convolution block, and each of the second convolution blocks and each of the fourth convolution blocks are respectively connected to one of the spatial position activation modules; Correspondingly, the preset dual-branch network and spatial position activation module respectively extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image, including: Based on the first convolution block and the second convolution block, sequentially extract features of the first brain tumor image to obtain first image features; Based on the third convolution block and the fourth convolution block, sequentially extracting features from the second brain tumor image to obtain second image features; Extracting features of the first image based on the spatial position activation module corresponding to the second convolution block to obtain first image spatial features; Extracting features of the second image based on the spatial position activation module corresponding to the fourth convolution block to obtain second image spatial features; Obtaining the first target image feature according to the first image feature and the second image space feature; Obtaining the second target image feature according to the second image feature and the first image space feature; The obtaining the first target image feature according to the first image feature and the second image space feature includes: The first target image feature is obtained by adding the product of the first image feature and the second image space feature to the first image feature; Correspondingly, obtaining the second target image feature according to the second image feature and the first image space feature includes: The product of the second image feature and the first image space feature is added to the second image feature to obtain the second target image feature.

2. The multimodal brain tumor image segmentation method according to claim 1, characterized in that: The output end of the first branch unit and the output end of the second branch unit are respectively connected to a decoder to determine the loss of the first branch unit and the loss of the second branch unit. The loss of the first branch unit and the loss of the second branch unit are also used to input into a preset total loss function to update the weight parameters of the dual-branch network.

3. The multimodal brain tumor image segmentation method according to claim 1, characterized in that: The obtaining a segmentation result according to the first target image feature and the second target image feature includes: Based on a preset multi-scale feature aggregation module, feature fusion is performed on the first target image feature and the second target image feature to obtain an image fusion feature; The image fusion feature is decoded based on a preset decoder to obtain the segmentation result.

4. The multimodal brain tumor image segmentation method according to claim 3, characterized in that: The multi-scale feature aggregation module includes a pyramid convolution unit, an upsampling unit, an attention unit and an aggregation unit. The pyramid convolution unit is used to extract the global information and the local information of the first target image feature and the second target image feature respectively; the upsampling unit is used to upsample and splice the global information and the local information to obtain the target feature; the attention unit is used to reconstruct the target feature; the aggregation unit is used to aggregate the global information, the local information and the reconstructed features to obtain the image fusion feature.

5. The multimodal brain tumor image segmentation method according to claim 4, characterized in that: The reconstructing the target feature includes: Determine a one-dimensional attention map and a two-dimensional attention map according to the target feature; The target features are respectively element-wise multiplied according to the one-dimensional attention map and the two-dimensional attention map, and a splicing operation is performed on the products to complete feature reconstruction.

6. A multimodal brain tumor image segmentation device, characterized in that: include: An acquisition module, configured to acquire a first brain tumor image and a second brain tumor image, wherein the first brain tumor image and the second brain tumor image both include at least one modality MRI image; a first determination module, configured to extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image based on a preset dual-branch network and a spatial position activation module, respectively, to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image; The spatial position activation module uses the position prior of the current modality image to guide the other modality image to focus on the spatial salient features of the current modality image; A second determination module, configured to obtain a segmentation result according to the first target image feature and the second target image feature; The dual-branch network includes a first branch unit and a second branch unit, the first branch unit includes a first convolution block and at least one second convolution block, the second branch unit includes a third convolution block and at least one fourth convolution block, and each of the second convolution blocks and each of the fourth convolution blocks are respectively connected to one of the spatial position activation modules; Correspondingly, the preset dual-branch network and spatial position activation module respectively extract features from each of the MRI images in the first brain tumor image and each of the MRI images in the second brain tumor image to obtain a first target image feature corresponding to the first brain tumor image and a second target image feature corresponding to the second brain tumor image, including: Based on the first convolution block and the second convolution block, sequentially extract features of the first brain tumor image to obtain first image features; Based on the third convolution block and the fourth convolution block, sequentially extracting features from the second brain tumor image to obtain second image features; Extracting features of the first image based on the spatial position activation module corresponding to the second convolution block to obtain first image spatial features; Extracting features of the second image based on the spatial position activation module corresponding to the fourth convolution block to obtain second image spatial features; Obtaining the first target image feature according to the first image feature and the second image space feature; Obtaining the second target image feature according to the second image feature and the first image space feature; The obtaining the first target image feature according to the first image feature and the second image space feature includes: The first target image feature is obtained by adding the product of the first image feature and the second image space feature to the first image feature; Correspondingly, obtaining the second target image feature according to the second image feature and the first image space feature includes: The product of the second image feature and the first image space feature is added to the second image feature to obtain the second target image feature.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multimodal brain tumor image segmentation method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal brain tumor image segmentation method according to any one of claims 1 to 5 is implemented.

Images