A three-dimensional multi-modal medical image lesion segmentation method
By combining modal grouping and attention mechanisms, an asymmetric structure of multi-branch encoding and single-branch decoding is constructed, which solves the shortcomings of three-dimensional multimodal medical image segmentation models in terms of data understanding and computational overhead, and achieves more efficient lesion segmentation performance and universality.
Patent Information
- Application Number
- CN202210818953.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-12
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2042-07-12
AI Technical Summary
Existing 3D multimodal medical image segmentation models are inadequate in terms of data understanding and prior knowledge modeling, which limits the improvement of model performance and results in high computational costs, especially when the data scale is small.
We employ a method that integrates image masking and attention mechanisms. We construct independent encoders through modal grouping, utilize class-independent attention masks and feature filtering modules, and combine residual context ideas to construct an asymmetric structure of multi-branch encoding and single-branch decoding, thereby reducing computational overhead and improving segmentation performance.
It improves segmentation performance with low data scale, reduces computational overhead, and enhances the model's universality and segmentation effect, especially outperforming traditional methods in capturing lesion details.
Smart Images
Figure CN115187616B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical image processing and understanding, and in particular to a three-dimensional multimodal medical image lesion segmentation method based on a neural network. Background Art
[0002] Medical image segmentation is a research hotspot in medical image analysis and understanding. The rapid development and widespread application of convolutional neural networks (CNNs) in natural image processing has spurred the field to focus on deep learning (DL) and computer vision (CV). Numerous medical image segmentation models have been proposed, achieving remarkable results in computer-assisted disease diagnosis. However, we note that lesion segmentation models and systems for 3D multimodal medicine are relatively limited, lacking a clear understanding and modeling of the specific semantics reflected by each imaging modality. While multimodal images clearly correspond to the actual final lesion type, existing systems and models fail to integrate prior knowledge of imaging modalities with model architecture design. This creates a bottleneck in the performance of 3D multimodal medical image segmentation, and despite the large size of the models, performance has not significantly improved. Therefore, the design of 3D multimodal medical image segmentation models requires further optimization and improvement through data understanding.
[0003] Properly modeling data understanding and prior knowledge into neural networks is the primary guiding principle for designing model structures. Currently, the most commonly used architecture for lesion segmentation in 3D multimodal medical images is the encoder-decoder architecture, which is not significantly different from the segmentation models used for unimodal medical images and natural images. Consequently, conventional methods for extracting multimodal image features inevitably suffer from modal information aliasing, significant noise interference, and weakly targeted multimodal information extraction. To address these shortcomings, employing the attention mechanism as a method for filtering feature maps in the model's intermediate layers is both practical and appropriate.
[0004] Attention mechanisms are a relatively mature field in computer vision, and related methods are widely used and have achieved good results. On the one hand, attention mechanisms exhibit strong dynamic characteristics, which empirically impose certain data requirements and are less effective with low data sets. On the other hand, attention mechanisms based on correlation metrics have high computational overhead, creating a significant computational bottleneck in 3D medical imaging tasks. Therefore, the design and application of attention mechanisms in medical imaging require the addition of certain constraints, balancing performance and computational overhead while ensuring that the constraints are reasonable and universal. Summary of the Invention
[0005] In response to the above technical problems, the present invention provides a three-dimensional multimodal medical image lesion segmentation method that integrates image masking and attention mechanism, which ensures segmentation performance under low data scale conditions, reduces computational overhead, and improves the universality of the model.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] The present invention provides a three-dimensional multimodal medical image lesion segmentation method, comprising the following steps:
[0008] S1. Collect 3D multimodal imaging data from the medical imaging database and align the modality types of all data.
[0009] S2, perform data preprocessing;
[0010] S3. Group modalities based on their semantic differences. Semantically similar modalities are concatenated in the channel dimension. For the same group of modalities, an independent encoder is constructed.
[0011] S4. Multiple independent encoders output the encoded features of their respective modality groups. Based on the differences in the performance of each modality group on the lesion area, a decoder branch is constructed to predict a category-independent attention mask covering the entire lesion area. This attention mask is used to provide position information and feature filtering for the subsequent decoder.
[0012] S5. The decoder is constructed using a weight-sharing approach. The category-independent attention mask is used again between the encoder and decoder. The attention mechanism and residual context are combined to perform feature noise filtering. Finally, the feature map with the same spatial resolution as the original input image is responsible for predicting the final segmentation result.
[0013] Furthermore, the modality grouping and encoder construction process in step S3 is as follows:
[0014] S301. For common MRI modalities, T1 and its derivatives belong to the same group, and T2 and its derivatives belong to the same group. If PET-CT is used, it is mixed into the T2 group and coded together. If other imaging modalities are used, all modalities under these other imaging modalities belong to the same group.
[0015] S302, for each group mode Build independent encoders, each of which consists of 5 stages, where the first stage takes the grouped modalities as input without changing their spatial resolution and produces feature maps In the next four stages, the kth stage outputs features from the previous stage. As input, the feature map space size is reduced to half of its original size at the beginning of each stage, and the output feature map
[0016] S303、In the final stage of the encoder, concatenate all the feature maps outputted by the encoder in the channel dimension as the preliminary encoding result F E，k .
[0017] Further, in step S302, each stage contains two groups of standard convolution blocks, and each convolution block contains a 3x3x3 regular convolution, a normalization layer, and a nonlinear function.
[0018] Further, in step S4, the single-channel attention mask step irrelevant to the prediction category is as follows:
[0019] S401, obtain the feature map outputted by the last stage of the T2 group encoder Construct a lightweight decoding branch that takes as input, restores the spatial size to of the original image size through 2 times of upsampling, and outputs a single-channel attention mask MA through a final 1x1 convolution;
[0020] S402, in the original data label, if the final prediction target contains C categories, generate a single-channel all-zero matrix with the same spatial size, and element by element judge whether the single element of the C-channel label is 1. If the element meets the condition, mark 1 at the corresponding position of the all-zero matrix, and finally generate a category-independent mask as the supervision information outputted in the previous step.
[0021] S403, in each forward propagation of the training process, first perform forward propagation of the T2 group encoder, and perform category-independent attention mask prediction through the lightweight decoding branch of S401. The result will be used in the subsequent feature filtering step.
[0022] Further, the single-channel attention mask outputted in step S401 is aligned with the spatial size of the feature map outputted by each stage of the encoder through a cubic spline interpolation algorithm.
[0023] Further, the specific steps of the decoder and feature filtering in step S5 are as follows:
[0024] S501, obtain the output feature maps of each stage of the encoder and the single-channel attention mask M A in S401;
[0025] S502, construct a feature filtering module, which contains two branches: in the upper branch, multiply the single-channel attention mask MA and the feature map element by element followed by 3 basic convolution blocks, a single convolution block contains 3x3x3 convolution and normalization and nonlinear function max(0.01*x, x); in the lower branch, the single-channel attention mask MA is multiplied with the feature map followed by 3 asymmetric convolution blocks in series on the channel, a single convolution block contains 1x3x3 convolution, 3x1x1 convolution, normalization and nonlinear function max(0.01*x, x); the two branches are summed element-wise after outputting to obtain filtered features and replace the original feature map as the input of the next stage of the encoder;
[0026] S503, a feature-shared decoder is constructed, the feature-shared decoder has only one branch, and the outputs of the encoders as the input of the decoder; the decoder is symmetrical with the single-branch structure of the encoder, has 4 stages, and each stage takes the output F D，k-1 of the previous stage and the filtered features of the same stage as the input as the input, F D，k-1 is first expanded in space size to twice the original size after transconvolution, and then added element-wise with F , and then a basic convolution block is passed to obtain the decoding output F D，k of the stage.
[0027] S504, the final output F D，0 of the decoder is obtained.
[0028] Compared with the prior art, the present application has the following beneficial effects:
[0029] The present application provides a three-dimensional multi-modal medical image lesion segmentation method, which combines image masks and attention mechanisms, and introduces multi-modal general priors as a strong and standardized attention mechanism constraint means. At the same time, in order to avoid the attention mask predicted by the model from filtering the features too strongly and losing the details of the lesion, a feature filtering module with a feature complementary strategy is constructed by combining the residual idea and the context. The overall model is an asymmetric configuration of multi-branch encoding and single-branch decoding. The encoding stage extracts features combined with modal priors, and a lightweight decoding branch is used to predict the overall region of the lesion from T2 group as a class-independent attention mask. Based on the mask and the feature filtering module, a multiplexed feature transfer between the decoder and the encoder is constructed. The method of the present application has achieved a good balance between segmentation performance and computational overhead, and has improved the rationality and universality of the additional constraints. BRIEF DESCRIPTION OF DRAWINGS
[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained based on these drawings.
[0031] Figure 1 The multi-branch encoding and single-branch decoding overall model structure provided by the embodiment of the present application is based on the U-Net network.
[0032] Figure 2 The basic feature filtering module structure provided by the embodiment of the present application.
[0033] Figure 3 The optimized result map provided by the embodiment of the present application is taken as an example of brain tumor lesion segmentation. DETAILED DESCRIPTION
[0034] In order to better understand the technical solutions, the following will make a detailed description of the method of the present application in combination with the drawings.
[0035] The present application provides a three-dimensional multi-modal medical image lesion segmentation method, comprising the following steps:
[0036] Step S1, collect the data in the medical image database, and in the embodiment, the nuclear magnetic resonance image of the brain tumor patient is taken as an example, mainly including T1, T1ce, T2 and Flair four modes.
[0037] Step S2, data preprocessing, first, motion correction and spatial normalization are performed on the original data. Then only the whole brain region is retained, and all invalid black regions are deleted. Finally, data enhancement is performed during training, including edge clipping, individual level gray scale normalization, elastic deformation, histogram specification and image clipping of limited foreground region.
[0038] Step S3, according to the semantic difference of the mode, the mode is grouped, and the modes with similar semantics are connected in series in the channel dimension. For the same group of modes, an independent encoder is constructed.
[0039] Firstly, grouping is performed according to the modality type rather than the specific pathological type. For all given modalities, grouping is performed according to general clinical diagnostic experience. For the general modality of magnetic resonance imaging, T1 and its derived types (such as T1ce) belong to the same group, and T2 and its derived types (such as Flair) belong to the same group. If PET-CT is used, it is mixed into the T2 group for common coding. If other imaging methods are additionally used, all modalities under the other imaging methods belong to the same group. Taking brain tumors as an example, T2 and Flair belong to one group, and other modalities belong to another group. Another key standard for grouping is that a certain group of modalities needs to fully reflect the overall area of the lesion, and this group is denoted as the mth group. It needs to be noted that some modalities may need to be used repeatedly, so the modalities included in each group are not strictly mutually exclusive. K modality groups are formed in this step.
[0040] Then, for each modality group An independent encoder is constructed, where R represents the set of all real numbers, C i represents the number of image channels, D represents the image depth, H represents the image height, and W represents the image width. The same structure and different parameters of the encoder are used for all groups. Each encoder contains 5 stages, and each stage contains two groups of standard convolution blocks, and each convolution block contains a 3x3x3 regular convolution, a normalization layer, and a nonlinear function. The first stage takes the grouped modalities as input and does not change the spatial size of the image features, producing a feature map In the remaining four stages, the step size of the first convolution in each stage is 2, reducing the spatial resolution by half. The output feature of the kth stage of the ith encoder is taken as the input of the k+1th stage. The spatial size of the feature map is reduced to half of the original at the beginning of each stage, and the output feature map
[0041] Finally, the final stage of each encoder is used to concatenate the feature maps output by all encoders in the channel dimension to produce the preliminary encoding result F E， k.
[0042] Step S4, the multiple independent encoders output the encoding features of each modality group. According to the priori of the multi-modal image, a lightweight decoding branch is constructed for the mth group to predict the class-independent attention mask M A The lightweight of the decoding branch is that only 4 basic convolution blocks are used, and the feature map is only restored to 1 / 4 of the original input spatial size. The output class-independent attention mask is aligned with the spatial size of the feature map output by the encoder at each stage through a three-order spline interpolation algorithm.
[0043] Specifically, a lightweight decoding branch is first constructed for the feature map output by the last stage of the T2 group encoder, which takes As input, the spatial size is restored to the original image size through 2 upsampling. And output the single-channel attention mask M through the final 1×1 convolution A ;
[0044] Then, in the original data label, if the final prediction target contains C categories, a single-channel all-zero matrix of the same spatial size is generated, and the element-by-element judgment of whether a single element of the C channel label is 1 is made. The elements that meet the conditions are marked as 1 in the corresponding position of the all-zero matrix, and finally a category-independent mask is generated as the supervision information output in the previous step;
[0045] Finally, during each forward propagation of the training process, the forward propagation of the T2 group encoder is performed first, and the category-independent attention mask prediction is performed through the lightweight decoding branch. The obtained result will be used in the subsequent feature filtering step.
[0046] Step S5: Output feature maps of each stage of the encoder Preliminary coding result F E，k and category-independent attention mask M A Based on this, we build the decoder and feature filtering modules.
[0047] First, construct the feature filtering module. This module consists of two branches: In the upper branch, the attention mask M A And the output feature maps of each stage of the encoder Perform element-wise product Followed by 3 basic convolution blocks, a single convolution block contains 3×3×3 convolution and normalization and nonlinear function max(0.01*x, x). In the lower branch, the output feature maps of each stage of the encoder are With the attention mask M A The convolution blocks are connected in series on the channel, followed by three asymmetric convolution blocks. A single convolution block contains 1×3×3 convolution, 3×1×1 convolution, normalization, and nonlinear function max(0.01*x, x). When the two branches are propagated forward, the output is summed element by element to obtain the filtered features and replace the original ones. As the input of the next stage of the encoder, it is also provided to the subsequent decoder for use; for example, the encoders of two independent branches each generate their own feature maps (i=0, 1) in the first stage. After filtering in this step, the two are combined into one feature map.
[0048] Then the decoder is constructed. The decoder is symmetrical with the single branch structure of the encoder and consists of 4 stages. The kth stage contains a deconvolution with a convolution kernel size of 2×2×2 and a stride of 2, which transforms the feature F of the previous stage into D，k-1 The spatial size is doubled (denoted as F′ D，k ), and 1 basic convolution block. In the forward propagation, first FE，k with F' D，k Element-wise addition, the result is sent to the basic convolution block, output the result F of the kth decoding stage D，k .
[0049] The decoder finally outputs the feature F D，0 After a single 1x1x1 convolution, the final segmentation prediction result is output, which does not contain the bias term.
[0050] The overall model is trained in an end-to-end manner. The overall loss function contains three parts: segmentation prediction loss, deep supervision loss and auxiliary decoding loss. The three loss functions are constructed using Focal Loss and Dice Loss . The mathematical expression is as follows:
[0051]
[0052]
[0053] The overall optimization loss function during training is expressed as:
[0054]
[0055] All the above hyperparameters take γ = 1.5, α i = 1.0, β D = 1.2, β F = 0.8.
[0056] The basic model of the application is shown in Figure 1 and Figure 2 . Figure 1 The encoding, decoding and auxiliary branch structure of the overall model are shown. Figure 1 Y ins represents the "class-independent mask" mentioned in this paper. It includes "class-independent single-channel mask" and "class-independent attention mask", both of which are represented by this symbol. Keyword: class-independent, mask. Sn represents the preliminary encoding result F E，k , where n and k are equivalent.
[0057] Figure 2 The structure of the feature filtering module is shown. Figure 2 The feature filtering module shown in Figure 1 is directly embedded sp，n or MF all instead. The only difference between MF sp，n and MF all is that MF sp，n truncates M AThe gradient of the auxiliary branch is thus allowed to be trained offline.
[0058] Take brain tumor lesion segmentation as an example, the final segmentation result is as shown in the following figure. Figure 3 Figure 3 The figure shows three cases, each case from left to right is the T1, T1ce, T2, Flair sequence (modality) of the magnetic resonance image, the brain tumor lesion label, the segmentation result of the basic model (3D UNet) with wide contrast, and the segmentation result of the model designed by the application. Figure 3 As can be seen from the figure, compared with the lesion label, the model of the application can better capture the detailed features of the lesion, and is superior to the segmentation result of the basic model. This shows that the model and method of the application are effective.
[0059] The application designs a three-dimensional multi-modal medical image lesion segmentation method. In view of the multi-modal problem, firstly, an asymmetric structure of multi-encoding and single decoding is designed in combination with the encoding and decoding neural network structure and according to the prior knowledge of the three-dimensional multi-modal medical image, and an attention mask which is independent of the category and directly reflects the lesion area (foreground) is constructed by using the performance difference of different modalities of medical images, so as to provide position information for subsequent feature decoding. Then, for the decoding structure of the segmentation network, the application converts the direct prediction mode based on the three-dimensional space image into a sequence mode, reduces the side effects caused by reducing the spatial resolution on the feature slice, and simultaneously uses an asymmetric convolution mode to enhance the prediction of the lesion shape change of the slice within a certain range. Since the gray difference of different modalities of medical images is large, the traditional encoding method is easy to introduce large noise in the early encoding, and the application again uses the foreground features independent of the lesion type, designs a feature filtering module based on the spatial mask and the spatial attention idea of re-weighting, and combines the attention mask to be used for feature denoising of the middle layer of the model. When training the network, supervision is introduced from multiple branches, feature interaction between encoding and decoding is established, and there is no additional annotation overhead. Unlike previous work, the attention used by the application combines the ideas of context and residual learning, which effectively avoids the loss of lesion details caused by re-weighting and rough foreground prediction. The method comprehensively utilizes different modalities of medical images, can fully utilize the characteristics of the modalities to introduce priori and constraints into the model, and thus obtains better segmentation performance. Finally, the overall structure of the application can further integrate the idea of spatial position division to perform instance-level lesion positioning and segmentation. The overall system has universality for three-dimensional multi-modal medical image segmentation tasks.
[0060] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A three-dimensional multi-modality medical image lesion segmentation method, characterized by, Comprising the following steps: S1, collecting three-dimensional multi-modal image data from a medical image database, aligning the modal types of all data; the modal types include T1, T1ce, T2 and Flair four modalities; S2, data preprocessing; S3, according to the semantic difference of modalities, the modalities are grouped, and the modalities with similar semantics are connected in series in the channel dimension, and for the same group of modalities, an independent encoder is constructed; S4, the multiple independent encoders output the encoding features of each modality group, and according to the performance difference of each modality group on the lesion area, a decoder branch is constructed to predict a class-independent and whole-lesion-area covering attention mask, which is used to provide position information and feature filtering for the subsequent decoder; the steps of predicting a class-independent and whole-lesion-area covering attention mask are as follows: S401、to the feature map output by the last stage of the T2 group encoder A light decoding branch is constructed, which takes as input, restores the spatial size to of the original image size through 2 times upsampling, and outputs the attention mask M through the final 1x1 convolution A The output attention mask is aligned with the spatial size of the feature map output by each stage of the encoder through a cubic spline interpolation algorithm. S402, in the original data label, if the final prediction target contains C categories, a single-channel all-zero matrix with the same spatial size is generated, and it is judged element by element whether the single element of the C-channel label is 1, and the elements that meet the conditions are marked as 1 in the all-zero matrix corresponding position, and finally a class-independent mask is generated as the supervision information output in the last step; S403, during each forward propagation in the training process, first perform forward propagation of the T2 group encoder, and then perform class-independent attention mask prediction through the lightweight decoding branch of S401, and the obtained result will be used for the subsequent feature filtering step; S5, a decoder is constructed by using weight sharing, and the class-independent attention mask is used again between encoding and decoding, combined with attention mechanism and residual context for feature noise filtering, and finally a feature map with a spatial resolution comparable to the original input image is output to predict the final segmentation result.
2. The three-dimensional multi-modality medical image lesion segmentation method of claim 1, wherein, The modality grouping and encoder construction process in step S3 is as follows: S301, for the general modalities of nuclear magnetic resonance images, T1 and its derivative types belong to the same group, T2 and its derivative types belong to the same group, and if PET-CT is used, it is mixed into the T2 group for common coding; if other imaging methods are additionally used, all modalities under other imaging methods belong to the same group; S302, for each group mode Build independent encoders, each of which consists of 5 stages, where the first stage takes the grouped modalities as input without changing their spatial resolution and produces feature maps In the next four stages, the kth stage outputs features from the previous stage. As input, the feature map space size is reduced to half of its original size at the beginning of each stage, and the output feature map S303、In the final stage of the encoder, concatenate all the feature maps output by the encoder in the channel dimension as the preliminary encoding result F E，k .
3. The three-dimensional multi-modality medical image lesion segmentation method of claim 2, wherein, In step S302, each stage contains two groups of standard convolution blocks, and each convolution block contains a 3*3*3 regular convolution, a normalization layer and a nonlinear function.
4. The three-dimensional multi-modality medical image lesion segmentation method of claim 1, wherein, The specific steps of the decoder and feature filtering in step S5 are as follows: S501, obtaining output feature maps of each stage of the encoder and the attention mask M in S401 A ; S502, construct a feature filtering module, which includes two branches: in the upper branch, the attention mask M is A With feature map Perform element-wise product Followed by 3 basic convolution blocks, a single convolution block contains 3×3×3 convolution and normalization and nonlinear function max(0.01*x, x); in the lower branch, the attention mask M A With feature map The convolution blocks are connected in series on the channel, followed by three asymmetric convolution blocks. A single convolution block contains 1×3×3 convolution, 3×1×1 convolution, normalization, and nonlinear function max(0.01*x,x); during forward propagation, the two branches output element-by-element summation to obtain the filtered features and replace the original feature map. As input to the next stage of the encoder; S503, construct a feature-sharing decoder. The feature-sharing decoder has only one branch, which filters the output of each encoder through the feature filtering module. As the input of the decoder; The decoder is symmetric to the encoder with a single branch structure, having 4 stages, each stage taking the output F D,k-1 of the previous stage as input As input, F D，k-1 is first upsampled by a factor of 2 using transposed convolution Element-wise addition is then performed with the filtered features of the same level D，k , followed by a basic convolutional block to obtain the decoded output F S504, final output F of the decoder D,0 The final partition prediction result is output after a single 1x1x1 convolution.