A three-dimensional image automatic segmentation method, system, device and medium

Abstract

The application discloses a kind of three-dimensional image automatic segmentation method, system, equipment and medium, including coding stage: input three-dimensional image data to be segmented into automatic segmentation network model, using continuous 3D convolution to extract image original feature, the local feature and global feature of image are perceived, obtain multi-modal feature, determine the feature similarity of multi-modal feature on the same channel dimension;Fusion stage: according to the feature similarity of multi-modal feature on the same channel dimension, the feature of different modal characteristics is spliced on the same channel, obtains fusion feature, and the fusion feature is reduced dimension processing;Decoding stage: by jump connection, multi-modal feature is connected from coding stage to decoding stage, the fusion feature after dimension reduction processing is restored to the same proportion as original feature, and output segmentation data. Utilize global feature and local feature, effectively maintain multi-modal image feature, effectively utilize the complementarity stereo feature between different modes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method, system, device, and medium for automatic segmentation of three-dimensional images. Background Technology

[0002] Currently, most research on automatic segmentation of medical images focuses on specific single modalities. However, with the diversification of medical imaging equipment and the increasing complexity of pathological features, many clinical applications require the analysis of medical images from multiple modalities. These multimodal methods help doctors more accurately measure the morphology and function of organs or tissues and develop more effective treatment plans.

[0003] In recent years, most researchers have focused on applying convolutional neural networks (CNNs) to multimodal medical image segmentation. However, CNN-based methods overemphasize local image features, making it difficult to capture long-range spatial dependencies. Furthermore, they focus almost exclusively on local image features, neglecting global representations. Therefore, these CNN-based methods struggle to achieve optimal segmentation results.

[0004] Existing methods for fusing features from two modalities have several problems: The first method merges the two modal images into a single image before feature extraction. This simple method can be directly segmented using existing single-modal segmentation networks, but it results in significant loss of original information and fails to fully utilize the complementary information between the modalities. The second method extracts features from the two modal images separately and then fuses them after obtaining their respective segmentation results. Similarly, this method can be easily accomplished using single-modal networks, but it ignores the correlation between the two modalities.

[0005] For multimodal segmentation, we consider two of the most challenging problems: how to effectively utilize complementary information from different modalities, and how to maintain accurate global morphological and locational dependencies between organs or tissues. These are key to making multimodal image segmentation more efficient than single-modal image segmentation. Summary of the Invention

[0006] The technical problem this invention aims to solve is the inability to effectively utilize the complementary stereo features between different modes during multimodal segmentation. The goal is to provide an automatic three-dimensional image segmentation method, system, device, and medium. By perceiving the local and global features of an image, the feature similarity of multimodal features in the same channel dimension is determined. Features of different modalities in the same channel are stitched together, and multimodal features are connected from the encoding stage to the decoding stage via skip connections. Multiple modalities are segmented as a whole, effectively preserving multimodal image features and utilizing the complementary stereo features between different modes by employing both global and local features.

[0007] This invention is achieved through the following technical solution:

[0008] The first aspect of this invention provides a method for automatic segmentation of three-dimensional images, comprising the following specific steps:

[0009] Encoding stage: Input the 3D image data to be segmented into the automatic segmentation network model, use continuous 3D convolution to extract the original features of the image, perceive the local and global features of the image, obtain multimodal features, and determine the feature similarity of multimodal features in the same channel dimension;

[0010] Fusion stage: Based on the feature similarity of multimodal features in the same channel dimension, the features of different modal features in the same channel are concatenated to obtain fused features, and the fused features are then subjected to dimensionality reduction processing;

[0011] Decoding stage: Multimodal features are connected from the encoding stage to the decoding stage through skip connections, restoring the fused features after dimensionality reduction to the same proportion as the original features, and outputting segmented data.

[0012] This invention perceives the local and global features of an image, determines the feature similarity of multimodal features in the same channel dimension, concatenates the features of different modal features in the same channel, connects the multimodal features from the encoding stage to the decoding stage through skip connections, segments multiple modalities as a whole, and effectively preserves multimodal image features by utilizing global and local features, and effectively utilizes the complementary stereo features between different modes.

[0013] Furthermore, determining the feature similarity of multimodal features in the same channel dimension also includes:

[0014] Obtain horizontal and vertical 3D local window data of the 3D image data to be segmented, obtain position codes in the horizontal and vertical directions, and determine the self-attention of the horizontal and vertical 3D local windows respectively.

[0015] The self-attention of the horizontal 3D local window and the self-attention of the vertical 3D local window are spliced ​​together to obtain the complete 3D self-attention.

[0016] Based on the complete 3D self-attention, a 3D constrained multi-head self-attention module is constructed. Based on the constructed 3D constrained multi-head self-attention module, parameter constraints are applied to the query matrix, key matrix, and value matrix to determine the global and local similarities between 3D image patches.

[0017] Furthermore, the self-attention stitching includes: normalizing the input 3D image data to be segmented based on MLP.

[0018] Furthermore, the perception of local and global features of the image includes:

[0019] CNN is used to perceive local features of the image, and Transformer is used to perceive global features of the image.

[0020] By sharing local and global feature parameters, features from different modalities jointly guide the automatic segmentation network model to learn features.

[0021] Furthermore, before splicing the features of different modalities on the same channel, the process also includes:

[0022] Feature mapping is performed by sharing weights, so that features of different modalities are kept in the same feature space;

[0023] The fusion weights of multiple modal features are learned to automatically select different modalities.

[0024] Furthermore, when the features of different modalities on the same channel are spliced ​​together, the different modalities are spliced ​​together in an interleaved manner.

[0025] Furthermore, it also includes training an automatic segmentation network model to output feature maps at different scales for deep supervision, the training steps including:

[0026] Calculate the cross-entropy loss and soft dice loss for all outputs, sum the cross-entropy loss and soft dice loss to obtain the sum of all losses across multiple scales.

[0027] A second aspect of the present invention provides a three-dimensional image automatic segmentation system, comprising:

[0028] The encoder includes a 3D image embedding module, a 3D Transformer module, and a 3D collaborative learning downsampling module;

[0029] A fusion unit, comprising a Transformer-based channel-interleaved adaptive feature fusion module;

[0030] The decoder includes a 3D Transformer module, an upsampling module, and a 3D expansion module;

[0031] Encoding stage: The 3D image data to be segmented is input into the automatic segmentation network model. The original features of the image are extracted by the 3D image embedding module using continuous 3D convolution. The local and global features of the image are perceived by the 3D collaborative learning downsampling module to obtain multimodal features. The feature similarity of multimodal features in the same channel dimension is determined by the 3D Transformer module.

[0032] Fusion stage: Based on the feature similarity of multimodal features in the same channel dimension, the features of different modal features in the same channel are spliced ​​together by the Transformer-based adaptive channel interleaving feature fusion module to obtain fused features, and the dimensionality reduction of the fused features is then performed.

[0033] Decoding stage: Multimodal features are connected from the encoding stage to the decoding stage through skip connections, feature mapping is performed again through the 3D Transformer module, the dimensionality-reduced fused features are restored to the same scale as the original features through the upsampling module, and the segmented data is output through the 3D extension module.

[0034] A third aspect of the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for automatic segmentation of three-dimensional images.

[0035] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for automatic segmentation of three-dimensional images.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] By perceiving the local and global features of the image, the feature similarity of multimodal features in the same channel dimension is determined. Features of different modalities in the same channel are stitched together. Multimodal features are connected from the encoding stage to the decoding stage through skip connections. Multiple modalities are segmented as a whole. By utilizing global and local features, multimodal image features are effectively preserved, and complementary stereo features between different modes are effectively utilized. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0039] Figure 1 This is a flowchart of the automatic three-dimensional image segmentation process in an embodiment of the present invention;

[0040] Figure 2 The image data to be segmented is collected in the embodiments of the present invention;

[0041] Figure 3 This is a schematic diagram of the encoder in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the fusion device in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the decoder in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the three-dimensional fusion segmentation network design in an embodiment of the present invention;

[0045] Figure 7 This is a schematic diagram of the three-dimensional Transformer module in an embodiment of the present invention;

[0046] Figure 8 This is a schematic diagram of a three-dimensional constrained multi-head self-attention module in an embodiment of the present invention;

[0047] Figure 9 The violin images are from the BraTS2021 datasets DSC, Jaccard, HD95, and RVD in this embodiment of the invention.

[0048] Figure 10 This is a graph showing the relationship between DSC score, training loss, and validation loss and epochs in the comparative experiment of this invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0050] Currently, most research on automated medical image segmentation focuses on specific single modalities, such as computed tomography (CT), positron emission tomography (PET), or magnetic resonance imaging (MRI). Popular segmentation methods, such as UNET, have achieved remarkable results in single-modal image segmentation, establishing symmetrical network structures for efficient encoding and decoding. Most research today relies on this mature segmentation architecture. However, single-modal images cannot fully reflect appropriate pathological conditions. In most cases, organs or lesion areas are visible in two different modes, such as... Figure 2 As shown in the first column of (a) and (b). However, the other columns show that people can only locate organs or lesion areas in one modality using human vision.

[0051] With the diversification of medical imaging equipment and the increasing complexity of pathological features, many clinical applications require the analysis of medical images from multiple modalities. These multimodal methods help physicians more accurately measure the morphology and function of organs or tissues and develop more effective treatment plans. However, most existing unimodal networks are challenging to apply directly to multimodal segmentation tasks. Therefore, a fast, efficient, and robust multimodal segmentation method is of great research significance.

[0052] In recent years, most researchers have focused on applying convolutional neural networks (CNNs) to multimodal medical image segmentation. Zhao et al. proposed a multi-branch segmentation model based on a fully convolutional network for segmenting non-small cell carcinoma in PET-CT images. Xu et al. proposed a multimodal segmentation method based on a cascaded network model (WNet) for segmenting multiple myeloma systemic bone lesions in PET-CT images. Alqazzaz et al. proposed a SegNet-based method for segmenting brain tumors in MRI images. 2 Multimodal segmentation methods have been proposed. While these methods have contributed to the research on multimodal image segmentation, they treat different modalities as two independent segmentation tasks, failing to effectively utilize the complementary information between the modalities. Furthermore, CNN-based methods overemphasize local image features, making it difficult to capture long-range spatial dependencies. They also focus almost exclusively on local image features, neglecting global representations. Therefore, these CNN-based methods struggle to achieve optimal segmentation results.

[0053] For multimodal segmentation, we consider two of the most challenging problems: how to effectively utilize complementary information from different modalities, and how to maintain accurate global morphological and locational dependencies between organs or tissues. These are key to making multimodal image segmentation more efficient than single-modal image segmentation.

[0054] Fusing features from two modalities can be categorized into three methods. The first method merges the two modalities into a single image before feature extraction. This simple approach can directly utilize existing single-modal segmentation networks. However, this results in significant loss of original information and fails to fully leverage the complementary information between the modalities. For example, Myronenko et al. proposed a multimodal 3D MRI tumor segmentation method based on an encoder-decoder architecture. The second method extracts features from the two modalities separately and then fuses the segmentation results. Similarly, this method can be easily accomplished using a single-modal network, but it ignores the correlation between the two modalities. For example, Nie et al. proposed a multimodal segmentation method for segmenting white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) in an infant's brain. For each modality image, it trains only one network and then fuses their last feature for subsequent segmentation. The third method fuses feature maps during feature extraction. Compared to the first two methods, it considers the correspondence between other modalities while extracting features from each modality, enabling the model to learn more accurate feature information and effectively improving segmentation accuracy. Fu et al. proposed a multimodal spatial attention module (MSAM) to automatically enhance tumor-related regions and suppress dissimilar tumor regions. Xue et al. proposed a multimodal segmentation strategy combining shared downsampling of multi-scale features. These studies show that different fusion methods yield different segmentation results. Therefore, considering complementary information during feature extraction is key to solving the multimodal segmentation problem.

[0055] The Transformer is a mainstream model in Natural Language Processing (NLP) used to model long-range sequence-to-sequence tasks. Based on the multi-head self-attention (MSA) mechanism, this architecture can model global information and more effectively represent global feature relationships. Recently, some works have introduced the Transformer into medical image segmentation tasks and achieved satisfactory results, especially for unimodal image segmentation. For example, Chen combined the Transformer and CNN to design the TransUNet model for medical image segmentation, which uses the Transformer to extract global contextual information by encoding the feature maps of the CNN into a sequence of contextual information. Hatamizadeh's UNETR model uses the Transformer as an encoder to learn the sequence representation of the input, effectively capturing global multi-scale features and achieving 3D medical image segmentation. Cao proposed the Swin-Unet model, which uses the Transformer entirely for medical image segmentation. Valanarasu studied a gated axial attention model that extends the application of the Transformer in medical images.

[0056] While these methods have effectively demonstrated the feasibility of the Transformer model in medical image segmentation, many challenges remain in applying it to multimodal medical image segmentation. Existing research has only seen a few studies attempt to apply the Transformer to multimodal medical image segmentation. For example, Sun proposed a multimodal segmentation method based on Transformer and CNN, which uses two parallel and independent paths to encode the images for CNN and Transformer respectively. Wang proposed a novel encoder-decoder network structure called TransBTS, which utilizes 3D CNN and Transformer for multimodal MRI brain tumor segmentation. However, these methods still do not fully consider the correspondence between modes, treating the two modalities as independent segmentation tasks. Their focus remains on the multimodal relationship between the CNN and Transformer models, rather than the image itself. Because visual Transformer models require high computational costs, existing Transformer-based segmentation methods are mainly based on 2D images, lacking direct segmentation of 3D medical images. Zhou proposed nnFormer, a powerful 3D medical image segmentation model that employs an alternating architecture of Transformer and CNN.

[0057] This invention addresses the aforementioned problems and proposes a powerful 3D fusion segmentation network (AMTNet), which extends the application of Transformer in multimodal medical image segmentation. While the overall structure of our proposed method follows a typical U-shaped architecture, we have made many effective and significant changes to the feature encoding, fusion, and decoding parts. This segmentation network can effectively utilize complementary stereo features between different modes.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a method for automatic three-dimensional image segmentation, comprising the following specific steps:

[0060] Encoding stage: Input the 3D image data to be segmented into the automatic segmentation network model, use continuous 3D convolution to extract the original features of the image, perceive the local and global features of the image, obtain multimodal features, and determine the feature similarity of multimodal features in the same channel dimension;

[0061] Fusion stage: Based on the feature similarity of multimodal features in the same channel dimension, the features of different modal features in the same channel are concatenated to obtain fused features, and the fused features are then subjected to dimensionality reduction processing;

[0062] Decoding stage: Multimodal features are connected from the encoding stage to the decoding stage through skip connections, restoring the fused features after dimensionality reduction to the same proportion as the original features, and outputting segmented data.

[0063] The proposed method follows a typical U-shaped network structure. By perceiving the local and global features of the image, it determines the feature similarity of multimodal features in the same channel dimension, concatenates the features of different modalities in the same channel, and connects the multimodal features from the encoding stage to the decoding stage through skip connections. It segments multiple modalities as a whole, effectively preserves multimodal image features by utilizing global and local features, and effectively utilizes the complementary stereo features between different modes.

[0064] In some possible embodiments, determining the feature similarity of multimodal features along the same channel dimension further includes:

[0065] Obtain horizontal and vertical three-dimensional local window data of the three-dimensional image data to be segmented, obtain position codes in the horizontal and vertical directions, and determine the self-attention of the horizontal and vertical three-dimensional local windows respectively.

[0066] The self-attention of the horizontal 3D local window and the self-attention of the vertical 3D local window are spliced ​​together to obtain the complete 3D self-attention.

[0067] Based on the complete 3D self-attention, a 3D constrained multi-head self-attention module is constructed. Based on the constructed 3D constrained multi-head self-attention module, parameter constraints are applied to the query matrix, key matrix, and value matrix to determine the global and local similarities between 3D image patches.

[0068] In some possible embodiments, stitching with self-attention includes: normalizing the input 3D image data to be segmented based on MLP.

[0069] In some possible embodiments, perceiving local and global features of an image includes:

[0070] CNN is used to perceive local features of the image, and Transformer is used to perceive global features of the image.

[0071] By sharing local and global feature parameters, features from different modalities jointly guide the automatic segmentation network model to learn features.

[0072] In some possible embodiments, before splicing features of different modalities on the same channel, the method further includes:

[0073] Feature mapping is performed by sharing weights, so that features of different modalities are kept in the same feature space;

[0074] The fusion weights of multiple modal features are learned to automatically select different modalities.

[0075] In some possible embodiments, when features of different modalities on the same channel are spliced ​​together, the different modalities are spliced ​​in an interleaved manner.

[0076] In some possible embodiments, the training also includes training an automatic segmentation network model to output feature maps at different scales for deep supervision, the training steps including:

[0077] Calculate the cross-entropy loss and soft dice loss for all outputs, sum the cross-entropy loss and soft dice loss to obtain the sum of all losses across multiple scales.

[0078] The second aspect of this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a three-dimensional image automatic segmentation method.

[0079] The third aspect of this invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for automatic segmentation of three-dimensional images.

[0080] Example 2

[0081] like Figures 3-5 As shown, this embodiment provides a three-dimensional image automatic segmentation system, including:

[0082] The encoder includes a 3D-Embedding module, a 3D-Transformer module, and a 3D Co-Learn Down-sampling (CDs) module.

[0083] The fusion unit includes a Transformer-based channel interleaved adaptive feature fusion module (AITF);

[0084] The decoder includes a 3D Transformer module, an up-sampling module, and a 3D Expanding module.

[0085] Encoding stage: The 3D image data to be segmented is input into the automatic segmentation network model. The original features of the image are extracted by the 3D image embedding module using continuous 3D convolution. The local and global features of the image are perceived by the 3D collaborative learning downsampling module to obtain multimodal features. The feature similarity of multimodal features in the same channel dimension is determined by the 3D Transformer module.

[0086] Fusion stage: Based on the feature similarity of multimodal features in the same channel dimension, the features of different modal features in the same channel are spliced ​​together by the Transformer-based adaptive channel interleaving feature fusion module to obtain fused features, and the dimensionality reduction of the fused features is then performed.

[0087] Decoding stage: Multimodal features are connected from the encoding stage to the decoding stage through skip connections, feature mapping is performed again through the 3D Transformer module, the dimensionality-reduced fused features are restored to the same scale as the original features through the upsampling module, and the segmented data is output through the 3D extension module.

[0088] In some possible embodiments, the encoding phase specifically includes:

[0089] 3D-Embedding module:

[0090] like Figures 6-8 As shown, in the first part of the network encoding stage, the embedding layer divides the input image into small blocks, similar to a step in the Visual Transformer (VIT). As previously stated, this invention focuses on multimodal 3D medical image segmentation, so we must embed a 3D image instead of a 2D image. Therefore, in this module, we will embed the 3D input... Convert to a high-dimensional tensor Where H′×W′×S′ represents the shape of the embedding block and C represents the sequence length, this invention uses continuous 3D convolutions to extract features in the 3D-Embedding module, which allows for more detailed voxel-level encoding and facilitates accurate segmentation tasks. After convolution, GeLU and LayerNorm are applied. In actual experiments, the convolution size in 3D-Embedding can be precisely adjusted according to the input.

[0091] 3D-Transformer module:

[0092] Multi-head self-attention (MSA) is the core of the Transformer, used to calculate the similarity between image patches. Generally, the MSA mechanism in Visual Interpreter Systems (VIT) is designed for 2D images and is not suitable for 3D medical images. Furthermore, it calculates global similarity, leading to significant computational overhead. Many studies have shown that calculating local similarity can reduce computational workload and achieve better results through appropriate design. To effectively utilize the spatial characteristics of 3D medical images, this invention designs a 3D constrained multi-head self-attention module (R-MSA), which, when calculating the similarity between 3D image patches, constrains the parameter R... qk and R v To constrain the convergence of the model, as shown in equations (2) and (3).

[0093] In addition to R-MSA, this invention also utilizes the concept of local windows to reduce the computational cost of the 3D Transformer. We divide the 3D self-attention computation process into two distinct parts using a multi-head mechanism (the size of the input image determines whether the third dimension is separated). We use half of the head to compute the self-attention of the horizontal 3D local window, while the remaining half is used to compute the vertical 3D local window. We then concatenate the results of these two branches to obtain the complete 3D self-attention computation result. The detailed calculation formula is shown below.

[0094] Assume the lth th The input to the layer 3D-Transformer module is The calculation of 3D-qkv is shown in equation (1):

[0095]

[0096] Where q, k, and v represent the query, key, and value matrices in the Transformer, respectively. The complete R-MSA calculation is as follows: in and The self-attention in the vertical and horizontal directions are represented by equations (2) and (3), respectively:

[0097]

[0098]

[0099] Where p h and p v R represents the position code in two directions respectively. qk and R v Represents the constraint parameters. In the 3D Transformer module, the last part of the self-attention calculation is the Multilayer Perceptron (MLP), as shown in Equation (4):

[0100]

[0101] LayerNorm is a technique for normalizing input data, which can effectively reduce the training time of the network, while MLP is used to enhance the non-linear modeling ability of the model.

[0102] Co-Learn Down-sampling Module for 3D Collaborative Learning:

[0103] The advantage of Transformers is their ability to perceive global feature relationships within an image. However, maintaining sensitivity to local features remains crucial for fine-grained medical image segmentation tasks. As mentioned earlier, CNNs can perceive local pixel relationships within an image. Therefore, this invention designs a collaborative learning downsampling module that combines the advantages of Transformers and CNNs. We introduce CNNs to compensate for the network's lack of perception of local features. This combination can further reduce the size of feature maps and computational workload. By sharing parameters, different modal features can jointly guide the model in learning features, strengthening the information interaction between the two modalities. Simultaneously, it can eliminate some misleading features, thereby achieving efficient multimodal joint segmentation.

[0104] In some possible embodiments, the fusion phase specifically includes:

[0105] After the encoding stage, the feature maps generated by different modalities have certain differences. Therefore, fusing them using ordinary addition operations cannot effectively represent the features and importance of each modality. Since the feature maps of two modalities have certain similarities on the same channel, we interleave the feature maps during concatenation. Therefore, this invention designs a Transformer-based Adaptive Channel Interleaved Feature Fusion Module (AITF), such as... Figure 2 As shown in (b).

[0106] This module mainly consists of two parts: a 3D Transformer and a channel-interleaved fusion operation. This invention inputs features generated by two modalities into the 3D Transformer module, performing feature mapping through shared weights. This ensures that feature maps from different modalities remain in the same feature space, thereby eliminating features detrimental to segmentation. In multimodal imaging, different modal images scanned for the same body part exhibit high feature similarity and complementarity. This similarity is reflected in the region's location, shape, size, and other features requiring segmentation. Based on this, this invention assumes that after mapping to the same feature space, the high-dimensional features of the two modalities obtained after the encoding stage have the same feature similarity in the same channel dimension. Therefore, in the AITF module, we designed a novel feature fusion method: channel-interleaved feature fusion. Feature fusion methods typically fall into two categories: the first is simply adding the values ​​of two feature maps in the same dimension; the second is stitching two feature maps together in one channel, then reducing the dimensionality of the stitched feature map and restoring it to its original scale. Our designed channel-interleaved fusion stitches together features from different feature maps in the same channel, and then performs dimensionality reduction on the stitched feature map.

[0107] Furthermore, before channel concatenation, we teach the model the fusion weights of the two modalities, automatically allowing the model to select the importance of different modalities. This channel-interleaved fusion method preserves similar features across different modalities as much as possible, reducing the interference of misleading features in segmentation. Finally, we connect the original features input to the AITF module with the fused features using residual connections, ensuring that the fused features also retain the specificity of their respective modalities—the complementary features we mentioned.

[0108] In some possible embodiments, the decoding stage specifically includes:

[0109] To keep the network structure as simple as possible, the decoding part is set up similarly to the encoding part. In the decoding branch, the fused features output by the AITF module are gradually restored to the same scale as the input by the upsampling module. At this stage, our model combines the encoded and decoded features through skip connections. We also designed a 3D-Transformer module to remap the features for more satisfactory results. Finally, the model outputs the segmentation result through a 3D expansion module, with the same scale as the input image.

[0110] Our model outputs feature maps at different scales during the training phase for deep supervision. Specifically, in addition to the final output, two feature maps at different scales (or more than two, which can be adjusted according to the actual experimental process) can be obtained during the decoding phase to make the model converge more stably. The model is shown in Equation (5), and the final training loss function is the sum of all losses at the three scales, as shown in Equation (6).

[0111] L all (s,h,w)=w ce ×L ce -log(-L dice )×w dice (5)

[0112]

[0113] Example 3

[0114] This embodiment provides an experiment using the BraTS2021 dataset.

[0115] The BraTS2021 dataset contains four MRI sequences: native T1-weighted images, contrast-enhanced T1-weighted images (T1GD), T2-weighted images, and T2 fluid attenuation inversion recovery images (T2-Flair). These can be broadly categorized into two types: T1 (native T1 and T1GD) and T2 (T2-weighted and T2-Flair). T1 sequences effectively display anatomical structures, while T2 sequences effectively show the signal of tissue lesions. Specifically, T1GD shows brighter areas in vascular regions and darker areas in non-vascularized tumor regions, which is detrimental to tumor segmentation. Conversely, compared to T2 sequences, Flair sequences effectively display the circumference of the tumor site and clearly show swollen areas. For tumor segmentation, this invention selected native T1 and T2-Flair sequences to test our model. Unlike the prostate dataset, the BraTS dataset has multiple regions requiring segmentation. Instead of predicting three mutually exclusive sub-regions corresponding to the segmentation label, this invention predicts three overlapping regions: enhanced tumor (ET, original region), tumor core or TC (ET+NCR), and the entire tumor or WT (ED+TC).

[0116] Table 1.Experiments on BraTS2021(the best results are bolded).

[0117]

[0118] Table 1 shows the quantitative results for the ET, TC, and WT regions. The fourth row (AVG) in each test method represents the corresponding mean. The best results are bolded in the table. Compared to these values, the proposed method achieves the best DSC (0.851), HD95 (2.736), Jaccard (0.759), and RVD (0.136) in these regions. For comparison, the DSCs of the MSAM, WNet, MAML, TCSM, and MFNet methods are 1.3% to 6.6% lower than the proposed method. Furthermore, the proposed method shows slightly lower performance compared to results on the Prostate dataset, which may be due to the complexity of the segmented regions on BraTS2021.

[0119] To demonstrate the generality of the proposed segmentation method on the BraTS2021 dataset, Figure 9 Violin plots for the four metrics are shown. These figures demonstrate that our method significantly outperforms the compared methods on all four metrics. These performances are similar to those on the Prostate dataset. In conclusion, the combined results demonstrate its strong advantages in multimodal image segmentation.

[0120] We are still Figure 10 The diagram shows the loss curves of our method and the comparison methods on the BraTS2021 dataset. It's clear that MFNet converges quickly, but its segmentation ability is relatively low. In contrast, TCSM, MAML, WNet, and MSAM all exhibit greater volatility than on the Prostate dataset, and their final segmentation results are lower than our proposed method. While our method also shows some volatility, it is much more stable than the comparison methods. Compared to the Prostate dataset, its variation is minimal, indicating that our method has better adaptability.

[0121] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic segmentation of three-dimensional images, characterized in that, Specifically, the following steps are included: Encoding stage: The 3D image data to be segmented is input into the automatic segmentation network model. Continuous 3D convolution is used to extract the original image features. Local and global features of the image are perceived to obtain multimodal features. The feature similarity of multimodal features in the same channel dimension is determined. Determining the feature similarity of multimodal features in the same channel dimension includes: Obtain horizontal and vertical 3D local window data of the 3D image data to be segmented, obtain position codes in the horizontal and vertical directions, and determine the self-attention of the horizontal and vertical 3D local windows respectively. The self-attention of the horizontal 3D local window and the self-attention of the vertical 3D local window are spliced ​​together to obtain the complete 3D self-attention. Based on the complete 3D self-attention, a 3D constrained multi-head self-attention module is constructed. Based on the constructed 3D constrained multi-head self-attention module, the query matrix, key matrix, and value matrix are subject to parameter constraints to determine the global similarity and local similarity between 3D image patches. The input to the l-th layer 3D-Transformer module is Then the calculation of 3D-qkv is as follows: ； Where q, k, and v represent the query, key, and value matrices in the Transformer, respectively. Represents the weights of the query matrix. Represents the weights of the key matrix. The weights of the value matrix are represented by the complete 3D constrained multi-head self-attention module R-MSA, which is calculated as follows: , This represents the input feature at the t-th time step / position of the l-th layer. and The self-attention values ​​represent the vertical and horizontal directions, respectively, and the calculation process is as follows: ； ； in and These represent position codes in two directions, respectively. and Representing the constraint parameters, d represents the scaling factor. In the 3D Transformer module, the final part of the self-attention calculation is the Multilayer Perceptron (MLP), and the calculation process is as follows: ； Fusion stage: Based on the feature similarity of multimodal features in the same channel dimension, the features of different modal features in the same channel are spliced ​​together by adaptive channel interleaving feature fusion based on Transformer to obtain fused features, and the dimensionality of the fused features is reduced. Before splicing the features of the different modalities on the same channel, the process includes: Feature mapping is performed by sharing weights, so that features of different modalities are kept in the same feature space; The fusion weights of multiple modal features are learned to automatically select different modalities; When different modal features are spliced ​​together on the same channel, the different modal features are spliced ​​together in an interleaved manner; Decoding stage: Multimodal features are connected from the encoding stage to the decoding stage through skip connections, restoring the fused features after dimensionality reduction to the same proportion as the original features, and outputting segmented data; Training phase: The automatic segmentation network model is trained and feature maps at different scales are output for deep supervision. The training steps include: calculating the cross-entropy loss and soft dice loss of all outputs, summing the cross-entropy loss and soft dice loss to obtain the sum of all losses at multiple scales.

2. The automatic three-dimensional image segmentation method according to claim 1, characterized in that, The self-attention stitching process includes: normalizing the input 3D image data to be segmented based on MLP.

3. The automatic three-dimensional image segmentation method according to claim 1, characterized in that, The perception of local and global features of the image includes: CNN is used to perceive local features of the image, and Transformer is used to perceive global features of the image. By sharing local and global feature parameters, features from different modalities jointly guide the automatic segmentation network model to learn features.

4. A three-dimensional image automatic segmentation system, applied to the three-dimensional image automatic segmentation method according to any one of claims 1-3, characterized in that, include: The encoder includes a 3D image embedding module, a 3D Transformer module, and a 3D collaborative learning downsampling module; A fusion unit, comprising a Transformer-based channel-interleaved adaptive feature fusion module; The decoder includes a 3D Transformer module, an upsampling module, and a 3D expansion module; Encoding stage: The 3D image data to be segmented is input into the automatic segmentation network model. The original features of the image are extracted by the 3D image embedding module using continuous 3D convolution. The local and global features of the image are perceived by the 3D collaborative learning downsampling module to obtain multimodal features. The feature similarity of multimodal features in the same channel dimension is determined by the 3D Transformer module. Fusion stage: Based on the feature similarity of multimodal features in the same channel dimension, the features of different modal features in the same channel are spliced ​​together by the Transformer-based adaptive channel interleaving feature fusion module to obtain fused features, and the dimensionality reduction of the fused features is then performed. Decoding stage: Multimodal features are connected from the encoding stage to the decoding stage through skip connections, feature mapping is performed again through the 3D Transformer module, the dimensionality-reduced fused features are restored to the same scale as the original features through the upsampling module, and the segmented data is output through the 3D extension module.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a three-dimensional image automatic segmentation method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a three-dimensional image automatic segmentation method as described in any one of claims 1 to 3.

Images