A coarse-fine attention skip connection enhanced three-dimensional medical image segmentation method

Abstract

The application discloses a three-dimensional medical image segmentation method based on coarse and fine attention jump connection enhancement, which is used for voxel-level automatic segmentation of multi-modal three-dimensional medical images. The method comprises the following steps: performing window division, feature embedding and window aggregation on multi-scale three-dimensional features extracted by an encoder to construct a prototype memory library containing hierarchical indexes and spatial indexes; generating query features according to current layer features of a decoder, and performing similarity matching with cross-layer prototype features in the prototype memory library to generate coarse-grained attention results; selecting candidate prototypes according to the coarse-grained matching results, backtracking corresponding local window features of the encoder, and generating fine-grained attention results; and fusing the coarse-grained attention results and the fine-grained attention results, enhancing the jump connection features, and inputting the jump connection features into the decoder to obtain three-dimensional medical image segmentation results. Through the application, the effectiveness of feature fusion between the encoder and the decoder can be improved, and the application is suitable for three-dimensional medical image segmentation tasks such as brain tumors.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer-aided diagnosis, and in particular to a three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections. Background Technology

[0002] 3D medical image segmentation is a crucial task in medical image processing and computer-aided diagnosis, aiming to perform voxel-level identification of organs, lesions, or tissue regions within 3D volumetric data. Taking brain tumor segmentation as an example, the Brain Tumor Segmentation Challenge (BraTS) has driven the development of automated brain tumor segmentation methods through multimodal magnetic resonance imaging and manually annotated results. However, 3D medical images are characterized by complex spatial structures, blurred tumor edges, and significant regional scale differences. How to simultaneously utilize high-level semantic information and local spatial details is a critical problem that automated segmentation needs to address.

[0003] In recent years, deep learning methods based on encoder-decoder structures have been widely used in medical image segmentation tasks. U-Net extracts contextual information by shrinking paths and recovers spatial localization capabilities by expanding paths, achieving good results in biomedical image segmentation. The 3D U-Net further extends two-dimensional convolutional operations to three-dimensional convolutional operations for dense three-dimensional segmentation of volume data. In this type of structure, skip connections are used to pass encoder features to the decoder to compensate for the spatial detail information lost during downsampling.

[0004] However, traditional skip connections often employ direct concatenation or addition, lacking explicit modeling of cross-layer semantic relevance. This can easily introduce redundant or semantically inconsistent information, affecting the feature fusion quality during the decoding stage. To address this issue, existing research has attempted to improve skip connections or enhance cross-layer feature interactions. For example, nested U-shaped networks (UNet++) reduce the semantic differences between encoder and decoder features through nested, dense skip paths, while attention-gated U-shaped networks (Attention U-Net) introduce attention gating mechanisms in skip connections to highlight target-related regions and suppress irrelevant background responses.

[0005] Furthermore, some methods enhance information transfer between the encoder and decoder from the perspectives of channel attention, spatial attention, or Transformer feature interaction. For example, the SE attention module, i.e., the compression-excitation module, enhances effective feature responses through spatial and channel recalibration; the Dual Cross-Attention mechanism enhances skip connections in U-Net-like networks through channel cross-attention and spatial cross-attention; the Swin Unet method, based on a shift-window Transformer, uses a U-shaped structure based on a shift-window visual transformer (Swin Transformer) for local-global semantic feature modeling; the U-Net Channel Transformer Network (UCTransNet) rethinks skip connections in U-Net from the channel dimension, replacing the original skip connections with channel Transformers and channel cross-attention to alleviate semantic differences between encoder and decoder features; and the Channel Cross Fusion Attention Transformer UNet (CFATransUnet) combines channel cross-fusion attention with a Transformer structure to enhance feature representation capabilities in medical image segmentation. The above methods can improve the effect of cross-layer feature fusion to some extent, but they mainly focus on single-stage feature recalibration, direct cross-layer attention interaction or Transformer feature modeling, and still cannot simultaneously take into account global semantic candidate selection and local structural detail backtracking.

[0006] Therefore, memory matching mechanisms in other visual tasks can provide a reference. These mechanisms establish a correlation between query features and candidate memory features to retrieve and match relevant regions. For example, the Hierarchical Memory Matching Network (HMMN) uses a hierarchical memory matching approach to guide high-resolution fine-grained matching with coarse-scale matching results, thus achieving a balance between matching efficiency and detail recovery. Restricted Memory Banks (RMem) further focus on efficient memory retrieval under limited memory bank size conditions to reduce the impact of redundant memory information on matching results. However, memory matching mechanisms in video object segmentation mainly target propagation between two-dimensional continuous video frames. Their memory units are usually derived from historical frame features, with the goal of maintaining the consistency of video objects in the temporal dimension. The skip connectivity enhancement problem in three-dimensional medical image segmentation is different. It deals with three-dimensional volume data with spatial continuity. The key is to select candidate regions related to the current decoding semantics from the encoder's multi-scale three-dimensional features and backtrack the corresponding three-dimensional local window details to enhance skip connectivity features. Therefore, existing memory matching mechanisms in video object segmentation cannot directly solve the problems of cross-layer semantic mismatch and insufficient 3D local structure recovery in 3D medical image segmentation.

[0007] In summary, while existing medical image segmentation methods can improve the feature interaction between the encoder and decoder to some extent, they still suffer from problems such as insufficient modeling of cross-layer semantic relevance and difficulty in coordinating global semantic constraints and local structural refinement. Furthermore, existing memory-based matching mechanisms primarily address target propagation between consecutive two-dimensional video frames and cannot be directly applied to the problem of skip connection feature enhancement in three-dimensional medical image segmentation. Therefore, existing technologies still require a skip connection enhancement method for three-dimensional medical image segmentation to establish cross-layer semantic associations between the encoder and decoder and achieve adaptive enhancement of skip connection features through global semantic retrieval and three-dimensional local window backtracking. Summary of the Invention

[0008] The purpose of this invention is to address the problems of insufficient modeling of cross-layer feature semantic correlation between encoder and decoder and difficulty in coordinating global semantic constraints and local structural detail recovery in existing three-dimensional medical image segmentation methods, and to provide a three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections.

[0009] The objective of this invention is achieved through the following technical solution: a three-dimensional medical image segmentation method enhanced by coarse and fine attention skip connections, comprising the following steps: (1) Input the multimodal three-dimensional medical image to be segmented into the encoder, perform window division and feature embedding on the multi-scale three-dimensional features output by the encoder, generate cross-layer prototype features, and record the hierarchical index and spatial index corresponding to the cross-layer prototype features to construct a prototype memory bank. (2) Generate query features based on the current layer features of the decoder, perform similarity matching between the query features and the cross-layer prototype features in the prototype memory obtained in step (1) to obtain coarse-grained attention weights, and generate coarse-grained attention results based on the coarse-grained attention weights. (3) Based on the coarse-grained attention weights obtained in step (2), select candidate prototypes from the prototype memory, backtrack the corresponding encoder local window features based on the hierarchical index and spatial index of the candidate prototypes, and generate fine-grained attention results based on the encoder local window features. (4) The coarse-grained attention result obtained in step (2) and the fine-grained attention result obtained in step (3) are fused to obtain a two-stage attention result. The two-stage attention result is used to weight and enhance the skip connection feature, and then input into the decoder to obtain the three-dimensional medical image segmentation result.

[0010] Further, step (1) includes the following sub-steps: (1.1) Input the multimodal 3D medical image to be segmented into the encoder, and extract 3D encoder features with different spatial resolutions and different semantic levels through multi-layer 3D convolution, normalization and nonlinear activation operations; (1.2) For the first Layer encoder features According to the preset window size The three-dimensional window is divided to obtain multiple three-dimensional local window features. , Indicates the number of the 3D local window; (1.3) Regarding the three-dimensional local window features Perform feature embedding mapping and in-window aggregation to obtain prototype features. ; (1.4) Map the prototype features obtained from different encoder levels to a unified feature dimension and perform cross-layer concatenation to form a cross-layer prototype feature set. Record the encoder level corresponding to each cross-layer prototype feature. and the position of the three-dimensional space window Build an index mapping table , Represents a cross-layer prototype feature set The prototype number in the document; (1.5) The cross-layer prototype feature set and index mapping table Combine these elements to build a prototype memory.

[0011] Furthermore, step (2) includes the following sub-steps: (2.1) Obtain the decoder's first... Layer features and for the decoder the first Layer features Perform feature mapping to obtain the query features; (2.2) Perform linear mapping on the cross-layer prototype feature set in the prototype memory to obtain the key feature matrix and the value feature matrix; (2.3) Calculate the similarity between the query features and the key feature matrix to obtain the relevance score matrix; (2.4) Normalize the correlation score matrix to obtain coarse-grained attention weights, and use the coarse-grained attention weights to perform weighted aggregation on the value feature matrix to generate coarse-grained attention results.

[0012] Furthermore, in sub-step (2.3), the similarity between the query feature and the key feature matrix is ​​calculated using dot product similarity.

[0013] Furthermore, step (3) includes the following sub-steps: (3.1) The coarse-grained attention weights are averaged and aggregated along the query position dimension to obtain the relevance score corresponding to each prototype; based on the relevance score, the top prototypes are selected from the prototype memory. A set of candidate prototype indices is obtained by identifying highly relevant candidate prototypes. (3.2) Based on the candidate prototype index set, query the encoder level and three-dimensional space window position corresponding to each candidate prototype in the index mapping table; (3.3) Based on the encoder level and the three-dimensional space window position, backtrack from the corresponding encoder feature layer and extract the encoder local window features; (3.4) Perform feature mapping and fine-grained modeling on the local window features of the encoder, and backfill the obtained fine-grained local window features to the spatial position of the corresponding encoder level. After count normalization, scale alignment and cross-layer aggregation, generate fine-grained attention results.

[0014] Furthermore, step (4) includes the following sub-steps: (4.1) The coarse-grained attention results are spliced ​​and fused with the fine-grained attention results, and the two-stage attention results are obtained through three-dimensional convolution and normalization operations; (4.2) Let the decoder be the first... The encoder skip connection feature corresponding to the layer is The two-stage attention results are compared with... Element-by-element addition yields the enhanced skip connection features; (4.3) The enhanced skip connection features are concatenated and convolved with the decoder upsampled features to obtain the current layer decoder output features; (4.4) Repeat steps (2) to sub-step (4.3) to perform coarse and fine two-stage attention enhancement on the skip connection features of each decoding stage and restore the spatial resolution step by step to obtain the final decoding features; perform three-dimensional convolution mapping and classification processing on the final decoding features to obtain the voxel-level segmentation results of the three-dimensional medical image.

[0015] Furthermore, during the model training phase, the voxel-level segmentation results obtained in sub-step (4.4) are compared with the corresponding manually labeled results, and the relevant mapping parameters of the encoder, decoder, prototype memory, and coarse and fine attention generation parameters are optimized based on Dice loss, cross-entropy loss, or a combination of Dice loss and cross-entropy loss.

[0016] The present invention also provides a three-dimensional medical image segmentation apparatus with enhanced coarse and fine attention skip connections, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections.

[0017] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described coarse-fine attention skip connection enhanced three-dimensional medical image segmentation method.

[0018] The beneficial effects of this invention are that by constructing a cross-layer prototype memory and introducing a coarse-grained two-stage attention mechanism, a more effective cross-layer semantic association can be established between the encoder and decoder. Coarse-grained attention is used to filter global candidate prototypes related to the current decoding semantics, while fine-grained attention is used to backtrack the corresponding 3D local window features and supplement local structural details. Compared to traditional skip connection methods that directly concatenate or add features, this invention can reduce the impact of redundant or semantically inconsistent features on the decoding process, improve the quality of skip connection feature fusion, and improve the segmentation effect of complex lesion regions, regions with blurred boundaries, and small-volume target regions. Attached Figure Description

[0019] Figure 1 This is a framework diagram of a three-dimensional medical image segmentation method enhanced by coarse and fine attention skip connections according to the present invention; Figure 2 This is a schematic diagram of the three-dimensional medical image segmentation network model in this invention; Figure 3This is a schematic diagram of the prototype memory module in this invention; Figure 4 This is a schematic diagram of the coarse and fine two-stage attention module in this invention; Figure 5 This is a visualization comparison of segmentation results on an adult glioma dataset in an embodiment of the present invention; Figure 6 This is a visual comparison of segmentation results on a pediatric glioma dataset in an embodiment of the present invention; Figure 7 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0021] The core technology of this invention addresses the problem of insufficient modeling of semantic relevance of skip connection features between encoders and decoders in 3D medical image segmentation networks. It constructs a cross-layer prototype memory and generates query representations based on decoder features. It then enhances skip connection features through coarse-grained semantic matching and fine-grained local window backtracking, thereby obtaining voxel-level segmentation results for 3D medical images.

[0022] This invention proposes a three-dimensional medical image segmentation method enhanced with coarse and fine attention skip connections, used for voxel-level segmentation of multimodal three-dimensional medical images, comprising the following steps: (1) Prototype memory construction: Windowing and feature embedding are performed on the multi-scale 3D features extracted by the encoder to generate searchable cross-layer prototype features. The hierarchical index (encoder level) and spatial index (spatial window position) corresponding to the cross-layer prototype features are recorded to construct the prototype memory. Specifically, this includes the following sub-steps: (1.1) Multi-scale 3D encoder feature extraction: The multi-modal 3D medical image to be segmented is input into the encoder, and 3D encoder features with different spatial resolutions and semantic levels are extracted through multi-layer 3D convolution, normalization, and nonlinear activation operations. Let the first... The characteristics of a layer encoder are: .

[0023] in, Represents the set of real numbers. Indicates the first Number of feature channels per layer , , These represent the dimensions of the feature in the vertical, horizontal, and front-back spatial directions, respectively.

[0024] (1.2) Three-dimensional window partitioning: the first window obtained in sub-step (1.1) Layer encoder features According to the preset window size By performing non-overlapping window partitioning, we obtain There are 3D local windows, each with a spatial dimension of [missing information]. In this embodiment, the spatial dimensions of each encoder level are set to the window size. Therefore, the integer multiple of , the th The number of 3D local windows obtained by feature partitioning from the layer encoder is:

[0025] in, Indicates the first The number of 3D local windows corresponding to the layer encoder features; Indicates the default window size.

[0026] Record No. The set of 3D local window features corresponding to the layer encoder features is as follows:

[0027] in, Indicates the first A set of 3D local window features obtained by layer encoder feature segmentation; Indicates the first The first layer encoder feature Three-dimensional local window features; Indicates the number of the 3D local window, and ;in, .

[0028] (1.3) Prototype feature generation: The three-dimensional local window obtained in sub-step (1.2) Intra-window feature aggregation is performed to obtain the corresponding prototype features. Specifically, feature embedding mapping operations are first used to map window features from different encoder levels to a unified feature dimension. The mapped local window features are obtained as follows:

[0029] in, ; Represents the mapped first Layer Local window features; Indicates the action on the first Feature embedding mapping operation of layer encoder window features, This represents the unified prototype feature dimension. The feature embedding mapping operation can be performed using... A 3D convolution implementation is used to map window features from different encoder levels to the same feature dimension.

[0030] Furthermore, the mapped local window features Perform in-window feature aggregation to obtain prototype features:

[0031] in, ; Indicates the first The first layer encoder feature The prototype features corresponding to each three-dimensional local window; This indicates that the voxel features within a 3D local window are subjected to average pooling.

[0032] Therefore, the first The prototype feature set corresponding to the layer encoder features is represented as follows:

[0033] in, ; Indicates by the first The set of prototype features generated by the layer encoder features.

[0034] (1.4) Construction of cross-layer prototype feature set and index mapping table: The prototype feature sets obtained from different encoder levels are mapped to a unified feature dimension and then concatenated across layers to form a cross-layer prototype feature set:

[0035] in, ; Represents a cross-layer prototype feature set; This indicates an operation that splices along the dimension of the number of prototypes; This indicates the number of encoder levels involved in building the prototype memory. Let represent the total number of cross-layer prototype features, and satisfy: .

[0036] Simultaneously, the encoder level and 3D spatial window position corresponding to each prototype feature are recorded to construct a prototype index mapping table:

[0037] in, Represents the prototype index mapping table; Represents a cross-layer prototype feature set The prototype number in, and ; Indicates the first The encoder hierarchy from which each prototype feature originates; Indicates the first The position of each prototype feature in the three-dimensional space window of the corresponding encoder level.

[0038] (1.5) Prototype memory generation: The cross-layer prototype feature set obtained in sub-step (1.4) is generated. and index mapping table Combined, construct a prototype memory:

[0039] in, Represents the prototype memory; Represents a searchable set of cross-layer prototype features; This represents a prototype index mapping table between prototype features and their source hierarchy and spatial window position. This prototype memory is used to provide searchable cross-layer semantic prototypes during the decoding phase and to provide hierarchical and spatial indexing for subsequent fine-grained local window backtracking.

[0040] (2) Coarse-grained attention generation: Query features are generated based on the current layer features of the decoder. The query features are then matched with the cross-layer prototype features in the prototype memory obtained in step (1) to generate coarse-grained attention weights. These weights are then used to weight and aggregate the value features to obtain the coarse-grained attention result. The coarse-grained attention result is used to model the correlation between the current layer features of the decoder and the multi-scale prototype features of the encoder from a global semantic perspective, providing a semantic filtering basis for subsequent fine-grained local window backtracking. Specifically, it includes the following sub-steps: (2.1) Query feature generation: Obtain the decoder's first feature. Layer features: .

[0041] in, Indicates the decoder level number. Indicates the decoder's first Number of layer feature channels , , These represent the dimensions of the feature in the three spatial directions. For the decoder... Layer features Perform feature mapping to obtain the query features:

[0042] in, This represents a three-dimensional average pooling operation, used to aggregate the features of the current layer of the decoder to a preset spatial scale; This represents a 3D convolution mapping operation used to map feature channels to a uniform feature dimension. ; This indicates a flattening operation, used to convert a three-dimensional spatial dimension into a one-dimensional sequence; This indicates the transpose operation, used to adjust the arrangement of the feature matrix; The representation layer normalization operation is used to stabilize the distribution of query features.

[0043] After the above processing, the query features are represented as follows: .

[0044] in, Indicates the decoder's first The query feature matrix corresponding to the layer; This represents the number of spatial locations after average pooling of 3D features, and the spatial size after pooling is... ; This represents a unified feature dimension.

[0045] (2.2) Prototype feature key-value mapping: Based on the prototype memory obtained in step (1) Take the cross-layer prototype feature set: .

[0046] in, Represents a cross-layer prototype feature set; This represents the total number of cross-layer prototype features in the prototype memory; This represents a unified feature dimension.

[0047] For the cross-layer prototype feature set Perform linear mappings separately to obtain the key feature matrix and the value feature matrix:

[0048] in, Represents a set of cross-layer prototype features The key feature matrix obtained from the mapping is used to map the key feature matrix to the decoder. The layer queries the feature matrix to perform similarity matching; Represents a set of cross-layer prototype features The value feature matrix obtained from the mapping is used to generate coarse-grained attention results based on the coarse-grained attention weights; Indicates the number used for decoder. Layer coarse-grained matching of key or value features linear mapping operation.

[0049] (2.3) Matching query features with prototype features: Calculate the query features obtained in sub-step (2.1). The key feature matrix obtained in sub-step (2.2) The similarity between them is used to obtain the relevance score matrix:

[0050] in, Indicates the decoder's first The correlation score matrix between layer query features and cross-layer prototype features. Indicates the first The query position and the first Correlation scores among cross-layer prototype features; Indicates the query location number, and ; Indicates the cross-layer prototype feature number, and ; Indicates the decoder's first The number of query locations formed after layer features are processed by three-dimensional average pooling; This represents the total number of cross-layer prototype features in the prototype memory. This indicates the transpose operation. Used for scaling the dot product similarity results.

[0051] (2.4) Coarse-grained attention weight generation: Based on the relevance score obtained in sub-step (2.3) Normalization is performed to generate coarse-grained attention weights. In this embodiment, the normalization process uses the Softmax operation:

[0052] in, , Indicates the decoder's first The coarse-grained attention weights corresponding to the layers, superscript Indicates coarse-grainedness; This indicates that Softmax normalization is performed along the dimension of the number of cross-layer prototype features, so that each query position is within the range of... The weights on each cross-layer prototype feature sum to 1. The coarse-grained attention weights are used to characterize the semantic correlation between different query positions in the current layer of the decoder and each cross-layer prototype feature in the prototype memory.

[0053] (2.5) Coarse-grained attention feature generation: Based on the coarse-grained attention weights obtained in sub-step (2.4) The eigenvalue matrix obtained in sub-step (2.2) We perform weighted aggregation to obtain a coarse-grained attention feature matrix:

[0054] in: This represents the coarse-grained attention feature matrix obtained by weighted aggregation of cross-layer prototype features; , represents the coarse-grained attention weight matrix; , representing the characteristic matrix.

[0055] Subsequently, the coarse-grained attention feature matrix is ​​spatially rearranged and mapped to the decoder's corresponding feature matrix through 3D convolution and feature mapping operations. By considering the spatial scale and channel dimension corresponding to the layer, coarse-grained attention results are obtained:

[0056] in, This indicates a feature rearrangement operation, used to... Rearranged into a three-dimensional feature form: ; This represents a 3D convolution mapping operation used to convert the number of feature channels from... Mapped to decoder Number of channels corresponding to the layer ,get ; This represents a 3D interpolation upsampling operation, used to transform the spatial size from... Adjust the spatial size to the j-th layer of the decoder ,get ; Indicates the decoder's first The coarse-grained attention results corresponding to the layer are used to aggregate global context information related to the current decoded semantics from cross-layer semantic prototypes in the prototype memory.

[0057] (3) Fine-grained attention generation: Based on the coarse-grained attention weights obtained in step (2), candidate prototypes related to the semantic features of the current layer of the decoder are selected from the prototype memory; then, based on the hierarchical index and spatial index corresponding to the candidate prototypes, the three-dimensional local window features in the corresponding encoder layer are traced back, and fine-grained attention results are generated based on the three-dimensional local window features. The fine-grained attention results are used to supplement the local structural details that are not fully expressed in the coarse-grained semantic matching. Specifically, it includes the following sub-steps: (3.1) Candidate prototype selection: Based on the coarse-grained attention weights obtained in step (2): .

[0058] in, This represents the number of spatial locations of the query features after three-dimensional average pooling. This represents the total number of cross-layer prototype features in the prototype memory; , Indicates the decoder's first The coarse-grained attention weights corresponding to the layers, superscript Indicates coarse-grainedness; Indicates the first The query position and the first Coarse-grained attention weights between prototype features.

[0059] The coarse-grained attention weights are used to describe the semantic relevance between features at different spatial locations in the current layer of the decoder and cross-layer prototype features in the prototype memory. Since the same cross-layer prototype feature may correspond to multiple query locations, the coarse-grained attention weights need to be aggregated along the query location dimension to obtain an overall relevance score at the prototype level.

[0060] Specifically, for coarse-grained attention weights The relevance score for each prototype is obtained by averaging along the query location dimension:

[0061] in, , Indicates the first The first cross-layer prototype feature and decoder Overall correlation score between layer features; Indicates the query location number; Indicates the cross-layer prototype feature number, and .

[0062] Based on the correlation score Select the previous prototype from the prototype memory. A set of candidate prototype indices is obtained by identifying 10 highly relevant candidate prototypes:

[0063] in, Indicates the candidate prototype in the cross-layer prototype feature set The set of numbers in; This indicates the number of candidate prototypes.

[0064] (3.2) Candidate prototype index query: Based on the candidate prototype index set obtained in sub-step (3.1) The index mapping table constructed in sub-step (1.4) In the query, the encoder level and 3D space window position corresponding to each candidate prototype are obtained: in, This represents the index mapping table in the prototype memory; Indicates the candidate prototype number. This indicates the encoder level from which the candidate prototype originated. This indicates the position of the candidate prototype in the three-dimensional space window at the corresponding encoder level.

[0065] (3.3) Encoder local window backtracking: Based on the encoder level obtained in sub-step (3.2) and spatial window position The 3D local window features are extracted by backtracking from the corresponding encoder feature layer. Let the th... The characteristics of a layer encoder are: .

[0066] in, The encoder-level feature representing the source of the k-th candidate prototype; This indicates the number of feature channels in this layer; , , These represent the dimensions of the layer's features in the three spatial directions.

[0067] Because the encoder features are given a preset window size in step (1) Perform non-overlapping 3D window partitioning, therefore spatial window index There is a one-to-one correspondence between the spatial index and the position of the 3D local window in this encoder layer. The mapping from spatial index to 3D window coordinates is denoted as:

[0068] in, Indicates the first The mapping function from spatial window index to 3D window coordinates in the layer encoder features. , , These represent the window coordinates of the candidate window in the three spatial directions, respectively.

[0069] Furthermore, the spatial region of this local window within the encoder feature layer is determined based on the window coordinates:

[0070] in, Indicates the first The three-dimensional local window region corresponding to each candidate prototype.

[0071] Therefore, from the first Extracting the corresponding encoder local window features from the layer encoder features:

[0072] in, Indicates from encoder features Extracting spatial regions Local features within, Indicates the decoder's first The corresponding layer Local window features of candidate encoders, satisfying: .

[0073] For candidate prototype index set Perform the above local window backtracking operation on each candidate prototype to obtain the candidate encoder local window feature set:

[0074] in, This represents the set of local window features of the candidate encoder corresponding to the j-th layer of the decoder.

[0075] (3.4) Local window feature mapping and fine-grained modeling: Since the candidate prototypes may come from different encoder levels and have different numbers of channels, the encoder local window features obtained in sub-step (3.3) are subjected to three-dimensional convolution mapping to map their channel dimensions to a unified feature dimension. :

[0076] in, This represents the candidate local window features after unifying the feature dimensions. This represents the fine-grained feature mapping operation corresponding to the j-th layer of the decoder; This represents a unified feature dimension.

[0077] Furthermore, fine-grained attention modeling is performed on the candidate local window features to obtain fine-grained local window features:

[0078] in, This represents a multi-head self-attention operation that incorporates spatial location information; This represents the candidate local window features after unifying the feature dimensions. This represents the three-dimensional window coordinates corresponding to the candidate window; This represents the local window features after fine-grained attention modeling, and satisfies: .

[0079] Fine-grained feature backfilling and count accumulation: In order to restore the fine-grained features in the candidate local window to their spatial locations at the corresponding encoder level, at each encoder level... Construct fine-grained feature accumulation containers and counting containers:

[0080]

[0081] in, Indicates the decoder's first The corresponding layer A fine-grained feature accumulation container for the layer encoder level; This represents the corresponding counting container, used to record the number of times each spatial location is covered by the candidate window; subscript Indicates coarse-grainedness; Indicates a unified feature dimension; , , They represent the first The dimensions of the layer encoder features in three spatial directions.

[0082] definition This is a local window backfill operator used to backfill candidate prototypes. The corresponding local window features were backfilled to the first... Regions in encoder feature space And pad with zeros in other positions. Then for the ... The layer, whose fine-grained feature accumulation container is represented as:

[0083] The corresponding counting container is represented as:

[0084] in, Indicates the decoder's first The set of candidate prototype indices corresponding to the layer; Indicates the candidate prototype number; Indicates candidate prototype The source encoder level; Indicates candidate prototype The corresponding three-dimensional local window area; Indicates candidate prototype The corresponding fine-grained local window features; Indicates the space size as The all-one characteristic matrix.

[0085] Count normalization: Perform count normalization on the fine-grained feature accumulation container to obtain the first... Fine-grained features after layer normalization:

[0086] in, This means counting the items in the container that are less than [a certain value]. Position set to This is to avoid feature bias caused by repeated accumulation in overlapping areas of local windows and to improve the stability of local structure recovery. Indicates the decoder's first The corresponding layer A fine-grained feature accumulation container for the layer encoder level; Indicates the decoder's first The corresponding layer The fine-grained features after layer normalization, and satisfying: .

[0087] Cross-layer fine-grained feature aggregation: Since candidate prototypes may originate from multiple encoder layers, it is necessary to unify the fine-grained features obtained from different encoder layers to the decoder. The spatial scale corresponding to each layer is determined, and cross-layer aggregation is performed. For the first layer... The fine-grained features after layer normalization are subjected to 3D interpolation to obtain scale-aligned fine-grained features:

[0088] in, Indicates by the first Scale-aligned features obtained by fine-grained feature interpolation at the layer encoder level; This represents a 3D interpolation operation used to unify fine-grained features from different encoder levels to the decoder. Layer spatial dimensions This ensures that local details generated by different encoder levels can be fused at a unified scale.

[0089] Furthermore, the scale-aligned features from different encoder levels are accumulated to obtain cross-layer fine-grained attention features:

[0090] in, Represents fine-grained attention features after cross-layer aggregation; superscript Indicates fine granularity; This indicates the number of encoder levels involved in the construction of the prototype memory.

[0091] Fine-grained attention result generation: Perform 3D convolution mapping on the cross-layer fine-grained attention features, changing their channel dimension from... Mapped to decoder Number of channels corresponding to the layer The fine-grained attention results are obtained:

[0092] in, Indicates the decoder's first The fine-grained attention results corresponding to the layer are used to supplement the local structural details required in the current decoding stage. (Superscript) Indicates fine granularity; Represents fine-grained attention features across layers; Indicates the decoder's first Fine-grained 3D convolution mapping operations corresponding to the layers; , , , These represent the decoder's first... The layer corresponds to the number of channels and the dimensions in the three spatial directions of the skip connection feature.

[0093] (4) Skip Connection Feature Enhancement and Segmentation Output: The coarse-grained attention obtained in step (2) and the fine-grained attention obtained in step (3) are fused to obtain a two-stage attention result. The skip connection feature is then weighted and enhanced using the two-stage attention result to make the global semantic information and local structural information complementary. This result is then input into the decoder to obtain the three-dimensional medical image segmentation result. Specifically, the following sub-steps are included: (4.1) Two-stage attention fusion: The coarse-grained attention results obtained in step (2) are fused together. Compared with the fine-grained attention results obtained in step (3) By fusing the results, a two-stage attention process is obtained:

[0094] in, Indicates the decoder's first The two-stage attention results corresponding to the layer; Indicates the decoder's first The coarse-grained attention results corresponding to the layer; Indicates the decoder's first The fine-grained attention results corresponding to the layer; This indicates an attention fusion operation.

[0095] In this embodiment, the fusion method can be convolutional fusion after splicing:

[0096] in, This indicates an operation that concatenates features along the channel dimension; Indicates the decoder's first The attention fusion 3D convolution operation corresponding to the layer is used to map the concatenated coarse and fine attention features into a two-stage attention result that matches the current skip connection features; This represents the instance normalization operation, used to normalize the features after fusion from 3D convolutions. After fusion, we have: .

[0097] (4.2) Skip connection feature weighting enhancement: Let the decoder be the first... The encoder features that perform skip connections and fusion across layers are: The two-stage attention results obtained in sub-step (4.1) The encoder skip connection feature Element-wise multiplication or channel-wise weighting yields the enhanced skip connectivity features:

[0098] Among them, due to and Having the same number of channels and spatial dimensions, they can be weighted element-wise or channel-wise. , which represents the enhanced skip connection feature; ⊙ represents the channel-wise weighted operation.

[0099] (4.3) Enhanced skip connection features and decoder features: The enhanced skip connection features obtained in sub-step (4.2) are fused together. With decoder upsampling features The features are then fused to obtain the output features of the current layer decoder:

[0100] in, Indicates the decoder's first Upsampling features of the layer; This represents the enhanced skip connection features; This represents a feature concatenation operation along the channel dimension; Indicates the decoder's first Feature fusion operations within a layer may include 3D convolution, normalization, and non-linear activation operations. This indicates the output features of the current layer decoder.

[0101] (4.4) Step-by-step decoding and segmentation output: Repeat steps (2) to sub-step (4.3), perform coarse and fine two-stage attention enhancement on the skip connection features of each decoding stage, and restore the spatial resolution step by step to obtain the final decoded features. Subsequently, the final decoded features are subjected to 3D convolutional mapping and classification processing to obtain the voxel-level segmentation results of the 3D medical image:

[0102] in, Indicates the final decoded features; This represents the segmentation output operation, which may include 3D convolutional mapping and classification activation operations; This represents the voxel-level segmentation results of a three-dimensional medical image.

[0103] During the model training phase, the voxel-level segmentation results are... The results are compared with the corresponding manually labeled results, and the mapping parameters of the encoder, decoder, prototype memory and the coarse and fine two-stage attention generation parameters are optimized based on Dice loss, cross-entropy loss or a combination of Dice loss and cross-entropy loss.

[0104] Example 1 An embodiment of the invention was implemented on a workstation equipped with an NVIDIA RTX 4090 graphics processor. Multimodal brain magnetic resonance images were used as input data, including T1-weighted images, contrast-enhanced T1-weighted images, T2-weighted images, and fluid-attenuated inversion recovery sequence images. After registration, resampling, intensity normalization, and cropping, the input images were fed into a 3D encoder-decoder segmentation network for voxel-level automatic segmentation of brain tumor regions.

[0105] like Figure 1 As shown, the three-dimensional medical image segmentation method of the present invention comprises four interrelated parts: prototype memory construction, coarse-grained attention generation, fine-grained attention generation, and skip connection enhancement. The multimodal three-dimensional medical image to be segmented is processed by an encoder to obtain the... Layer encoder features; these encoder features are passed as traditional skip connection features to the corresponding layer encoder features. Layer decoders, on the other hand, are used to build prototype memories.

[0106] In the prototype memory construction section, the first... After the layer encoder features are divided into three-dimensional windows, multiple encoder local window features are formed. Prototype features are then generated from these encoder local window features to obtain cross-layer prototype features. These cross-layer prototype features are stored in a prototype memory, along with corresponding prototype index mapping information, which includes a hierarchical index and a spatial index. The hierarchical index represents the encoder hierarchy from which the prototype feature originates, and the spatial index represents the three-dimensional spatial window position of the prototype feature within the corresponding encoder features.

[0107] In the coarse-grained attention generation part, according to the first... Layer decoder feature generation query feature And map the cross-layer prototype features in the prototype memory to key features and value features, that is Query characteristics Key features After similarity matching, coarse-grained attention weights are obtained, and these coarse-grained attention weights are further used to adjust the value features. Weighted aggregation is performed to generate coarse-grained attention results. These coarse-grained attention results are used to model the correlation between the current layer features of the decoder and the cross-layer prototype features of the encoder at the global semantic level.

[0108] In the fine-grained attention generation part, the prototype features in the prototype memory are sorted according to the coarse-grained attention matching results, and then... Candidate prototypes are retrieved to obtain those relevant to the current decoding semantics. Subsequently, based on the hierarchical and spatial indices corresponding to these candidate prototypes, encoder local window backtracking is performed to extract encoder local window features from the corresponding encoder levels. The extracted encoder local window features are then processed by fine-grained attention generation to obtain fine-grained attention results. These fine-grained attention results are used to supplement local spatial structure details that were not fully expressed during the coarse-grained semantic matching process.

[0109] In the skip connection enhancement section, the coarse-grained attention results and fine-grained attention results are fused to obtain a two-stage attention result, and the two-stage attention result is used to enhance the skip connection. The skip connection features corresponding to the layer encoder features are enhanced. The enhanced skip connection features are compared with those of the first layer encoder. The features of the layer decoder are fused to obtain the first layer. Layer decoder output features. Figure 1 Solid arrows in the diagram represent skip connections or feature transfer paths, while dashed arrows represent two-stage attention guidance paths. Specifically, red dashed arrows indicate encoder local window backtracking based on prototype index mapping information, and bottom dashed arrows indicate encoder features being transferred to the decoder via skip connections. Through this process, the present invention can simultaneously utilize cross-layer global semantic information and three-dimensional local structural details during the decoding stage, thereby enhancing the feature representation capability of skip connections between the encoder and decoder.

[0110] like Figure 2 As shown, the 3D medical image segmentation network model of this invention adopts an encoder-decoder structure to achieve voxel-level segmentation of multimodal 3D medical images. After the multimodal 3D medical image is input into the network, it first passes through the encoder to extract 3D features layer by layer. The encoder consists of multiple residual blocks and downsampling operations, wherein the residual blocks include 3D convolution, instance normalization, and Leaky ReLU activation operations to extract local spatial structure features; the downsampling operation is used to reduce the feature space resolution and enhance high-level semantic expression.

[0111] The decoder consists of upsampling operations, residual blocks, and an output mapping layer, used to progressively restore the feature space resolution. Features at the same scale are transferred between the encoder and decoder via skip connections to supplement the spatial detail information required during decoding. Unlike traditional skip connection methods that directly concatenate or add features, this invention introduces a two-stage coarse-fine attention module at the skip connections from layer 2 to layer 4. This module enhances the encoder skip connection features at the corresponding layers before fusing them with the decoder's upsampled features.

[0112] Figure 2 In the diagram, blue modules represent residual blocks, and green modules represent... 3D convolution, instance normalization, and Leaky ReLU activation operations are used for the input feature map; the purple module represents... 3D convolution and Softmax operations are used to generate voxel-level classification probabilities and obtain the final segmentation results; red triangles represent downsampling operations, and green triangles represent upsampling operations. Black dashed arrows represent skip connections between the encoder and decoder, orange dashed arrows represent the transmission path of prototype features in the network, yellow modules represent prototype memory modules, orange modules represent coarse and fine two-stage attention modules, and yellow triangles represent the two-stage attention results output by the coarse and fine two-stage attention modules.

[0113] Specifically, features from layers 2 to 4 of the encoder are used to construct a prototype memory module and provide cross-layer prototype features to corresponding skip connections during the decoding phase. During the progressive restoration of spatial resolution, the decoder queries the prototype memory based on the current layer's decoded features to generate coarse-grained attention results, and further backtracks to the encoder's local window features to generate fine-grained attention results. Subsequently, the coarse-grained and fine-grained attention results are fused to obtain a two-stage attention result, which is used to enhance the skip connection features at the corresponding layers. The enhanced skip connection features are then fused with the decoder's upsampled features and continue to participate in subsequent decoding processes. Finally, the network... The 3D convolution and Softmax classification operations output voxel-level segmentation results for 3D medical images.

[0114] like Figure 3 As shown, this figure is Figure 1 The specific content of the prototype memory construction phase is also Figure 2 The specific implementation process of the prototype memory module. This process is based on the first... The layer encoder features are used as input, and are sequentially processed through three-dimensional window partitioning, channel mapping, in-window average pooling, and cross-layer concatenation to obtain the prototype memory library, and a prototype index mapping table is constructed simultaneously.

[0115] Specifically, let the first Layer encoder features are represented as Its characteristic size is ,in, Indicates the first The number of channels in the layer encoder feature. , , These represent the dimensions of the feature in the three spatial directions. First, for the... Layer encoder features A three-dimensional window is partitioned, which is achieved through a three-dimensional convolution operation, i.e., using... The encoder features are divided into multiple non-overlapping 3D local windows. Represents 3D convolution; Represents the kernel size; This represents the convolution sliding stride.

[0116] After dividing the window into three dimensions, the resulting size is... The set of local window features, where, Indicates the first The number of 3D local windows obtained from feature segmentation by the layer encoder. This represents the spatial dimensions of each three-dimensional local window.

[0117] Subsequently, channel mapping is performed on the aforementioned local window features. This channel mapping is achieved through a 3D convolution operation with a kernel size of 1 and a stride of 1, i.e., using... Map window features from different encoder levels to a unified feature dimension. After channel mapping, the first... The local window feature corresponding to the layer is represented as follows ,in This represents the unified prototype feature dimension.

[0118] Furthermore, in-window average pooling is performed on each 3D local window after channel mapping, and each window with a size of [missing information] is [missing information]. The local window features are aggregated into one Dimensional prototype features. Therefore, the first... Layer encoder feature correspondence generation Each prototype feature has a size represented as follows: The in-window average pooling is used to compress the spatial dimension within the local window, enabling each prototype feature to represent the overall semantic information of the corresponding three-dimensional local window.

[0119] The aforementioned 3D window partitioning, channel mapping, and in-window average pooling operations are performed on multiple encoder levels involved in the prototype memory construction to obtain prototype feature sets at different levels. Then, the prototype features obtained from each encoder level are concatenated across layers along the prototype quantity dimension to form a cross-layer prototype feature set, i.e., the prototype memory. The size of the prototype memory is represented as... ,in, Represents the total number of cross-layer prototype features, satisfying:

[0120] in, Indicates the first The number of prototype features corresponding to the layer encoder features. This represents the summation of the number of prototypes at each encoder level involved in the construction of the prototype memory.

[0121] While constructing the prototype memory, a prototype index mapping table corresponding to each prototype feature is recorded. This prototype index mapping table includes a hierarchical index and a spatial index. The hierarchical index indicates which encoder level the prototype feature originates from, and the spatial index indicates the three-dimensional local window position of the prototype feature within the corresponding encoder level. For the first... Layer encoder features, whose spatial index range can be represented as 0 to Therefore, the prototype index mapping table can record the spatial index range corresponding to different levels. For example, the spatial index range corresponding to level 0 is 0 to... , No. The spatial index range corresponding to the layer is 0 to The same applies to the other levels.

[0122] Figure 3 The module shown corresponds to the content of step (1), where the prototype memory is denoted as The prototype index mapping table is denoted as The searchable set of cross-layer prototype features is denoted as .

[0123] Through the above process, the prototype memory not only contains cross-layer prototype features generated from multi-layer encoder features, but also the hierarchical index and spatial index corresponding to each prototype feature. The cross-layer prototype features are used for similarity matching with decoder query features during the coarse-grained attention generation stage, and the prototype index mapping table is used to backtrack the three-dimensional local window features in the corresponding encoder level based on candidate prototypes during the fine-grained attention generation stage. This provides a global semantic retrieval foundation and a local window backtracking basis for subsequent skip connection feature enhancement.

[0124] like Figure 4 As shown, Figure 4 This is a schematic diagram of the specific structure of the coarse and fine two-stage attention module of the present invention, and its position is as follows: Figure 2 The orange module is shown. This module is the first... Layer encoder features, first The layer decoder features and prototype memory are used as inputs, including a coarse-grained attention generation module, a fine-grained attention generation module, and a two-stage attention fusion module, which are used to generate two-stage attention results that enhance skip connection features.

[0125] Specifically, no. The layer encoder features are used to construct the prototype memory and serve as the feature source for subsequent fine-grained attention generation modules to backtrack the encoder's local windows. The prototype memory stores cross-layer prototype features generated from the encoder's multi-scale features, and also maintains a prototype index mapping table. The prototype index mapping table includes a hierarchical index and a spatial index, where the hierarchical index is used to determine the encoder level corresponding to the candidate prototype, and the spatial index is used to determine the 3D local window position corresponding to the candidate prototype.

[0126] In the coarse-grained attention generation module, the first... The layer decoder features first undergo AvgPool3d, Conv3d, Flatten, Transpose, and IN operations in sequence to generate query features. Here, AvgPool3d represents the three-dimensional average pooling operation, used for spatial aggregation of decoder features; Conv3d represents the three-dimensional convolution operation, used for channel mapping; Flatten represents the flattening operation, used to convert three-dimensional spatial features into sequence features; Transpose represents the transpose operation, used to adjust the dimension of the feature matrix; and IN represents the layer normalization operation, used to stabilize the distribution of query features.

[0127] Meanwhile, the cross-layer prototype features in the prototype memory are processed by Linear operations to generate value features. Bond features Where Linear represents a linear mapping operation. Used with query features Perform similarity matching. This is used for weighted aggregation based on attention weights. Then, the query features are... Key features Matrix multiplication (MatMul) is performed to obtain the correlation score between the current layer features of the decoder and the cross-layer prototype features in the prototype memory; then the correlation score is normalized by the Softmax operation to obtain the coarse-grained attention weight.

[0128] Based on the coarse-grained attention weights, on the one hand, they are compared with the value features. Perform matrix multiplication (MatMul) again to calculate coarse-grained attention weights and value features. The dot product similarity between features yields coarse-grained attention features formed by weighted aggregation of cross-layer prototype features. These features are then sequentially processed through Reshape, Conv3d, and feature mapping operations to generate the coarse-grained attention result. Reshape represents a feature rearrangement operation, used to restore the sequence-based features to a three-dimensional feature form; Conv3d represents a three-dimensional convolution operation, used to adjust the channels and fuse spatial features of the coarse-grained attention features; and feature mapping is used to adjust the coarse-grained attention features to a feature size that matches the current skip connection features. Furthermore, the coarse-grained attention weights also guide the selection of candidate prototypes in the fine-grained attention generation module.

[0129] In the fine-grained attention generation module, the coarse-grained attention weights first undergo a mean operation to average and aggregate the attention weights at different query positions, obtaining a prototype-level relevance score; then, a Top-k operation is used to select the top... A number of highly relevant candidate prototype features. Based on the previous... The candidate prototype features are indexed by hierarchical and spatial indices in the prototype index mapping table. The encoder local window backtracking is performed to extract local window features from the corresponding encoder hierarchy and corresponding spatial location.

[0130] The backtracked local window features are sequentially processed through Conv3d, Reshape, Pos Embed, MHSA, and feature mapping operations to generate fine-grained attention results. Specifically, Conv3d represents a 3D convolution operation, used to map local window features to a unified feature dimension; Reshape represents a feature rearrangement operation, used to adapt to subsequent attention calculations; PosEmbed represents a position embedding operation, used to introduce 3D spatial position information into the local window features; MHSA represents a multi-head self-attention operation, used to model the fine-grained spatial structural relationships within the local window; subsequent Reshape is used to restore the attention-modeled sequence features to the 3D local window feature form; and feature mapping is used to adjust the fine-grained local window features to the spatial scale and channel dimension corresponding to the current decoding layer. Through these processes, the fine-grained attention generation module can backtrack the encoder's local window based on coarse-grained candidate prototypes and supplement local structural details.

[0131] In the two-stage attention fusion module, coarse-grained attention results and fine-grained attention results are fused to obtain a two-stage attention result. Specifically, the coarse-grained and fine-grained attention results are concatenated or merged along the channel dimension, and then sequentially processed through Conv3d and IN operations to obtain the fused two-stage attention result. Here, Conv3d represents the attention fusion 3D convolution operation, used to fuse coarse-grained semantic information and fine-grained local structural information; IN represents the instance normalization operation, used to normalize the fused 3D features.

[0132] Subsequently, the fused two-stage attention results are weighted and fused with the original skip connection features to obtain the enhanced skip connection output features, which are then passed to the corresponding decoder layer. Figure 4 The term "output" in this context refers to the output of the coarse-grained and fine-grained attention modules after enhancing the skip connection features. Through this structure, the coarse-grained attention generation module retrieves globally semantically relevant prototype features from the prototype memory, the fine-grained attention generation module backtracks the encoder's local window based on candidate prototypes and models local structural details, and the two-stage attention fusion module unifies and fuses the coarse-grained semantic information and the fine-grained structural information, thereby achieving adaptive enhancement of the encoder-decoder skip connection features.

[0133] therefore, Figure 4 The module shown corresponds to the specific implementation of steps (2) to sub-steps (4.3), where the coarse-grained attention result is denoted as Fine-grained attention results are denoted as The result of the two-stage attention fusion is denoted as .

[0134] Figure 5 This is a visualization comparison of the segmentation performance of the method of the present invention and various comparative examples on the BraTS 2020 adult glioma dataset. Figure 6 This is a visualization comparison of the segmentation performance of the method of this invention and various comparative examples on the BraTS-PEDs 2023 pediatric glioma dataset. Each image uses two-dimensional slices from several typical cases to visually compare the segmentation performance of different methods on glioma regions in three-dimensional medical images.

[0135] Figure 5 and Figure 6Each row in the figure represents a case or a slice example, and each column represents a segmentation method or a manually labeled result. Ground Truth represents the manually labeled result; UNet++, SegResNet (SegResNet), nnU-Net (nnU-Net), U-Net based on Transformer encoder (UNETR), U-Net based on SwinTransformer encoder (Swin UNETR), Attention U-Net, and Segmamba (Segmamba-based medical image segmentation network) represent the segmentation results of proportional methods. The colored areas superimposed on the MRI image represent the segmentation results of different tumor sub-regions. Red areas represent necrotic and non-enhancing tumor core areas, green areas represent peritumoral edema areas, and blue areas represent enhancing tumor areas. Based on these three types of tumor sub-regions, three evaluation regions are further combined: the overall tumor region (WT) is composed of red, green, and blue areas; the tumor core region (TC) is composed of red and blue areas; and the enhancing tumor region (ET) corresponds to the blue area.

[0136] Based on the above method, actual three-dimensional medical image segmentation processing is performed, specifically including the following steps: (1) The BraTS 2020 dataset published by the Brain Tumor Segmentation Challenge (BraTS) was used as the experimental data for the adult glioma segmentation task. This dataset contains 369 adult glioma training samples, each case containing four multimodal magnetic resonance imaging sequences: T1-weighted imaging, contrast-enhanced T1-weighted imaging, T2-weighted imaging, and liquid attenuation inversion recovery sequence. The dataset provides voxel-level manual annotation results, with annotation categories including background region, necrotic and non-enhanced tumor core region, peritumoral edema region, and enhanced tumor region. Based on the above annotation results, three target regions were constructed for performance evaluation: the overall tumor region (WT), the tumor core region (TC), and the enhanced tumor region (ET). Among them, the overall tumor region (WT) includes the necrotic and non-enhanced tumor core region, the peritumoral edema region, and the enhanced tumor region; the tumor core region (TC) includes the necrotic and non-enhanced tumor core region and the enhanced tumor region; and the enhanced tumor region (ET) represents the enhanced tumor region.

[0137] (2) To further verify the applicability of the method of the present invention in the segmentation of pediatric gliomas, the Pediatric Brain Tumor Segmentation Challenge dataset (BraTS-PEDs 2023) published by the Brain Tumor Segmentation Challenge in 2023 was used as the experimental data for the pediatric glioma segmentation task. This dataset contains 99 training samples of pediatric gliomas, and each case also includes four multimodal magnetic resonance imaging sequences: T1-weighted imaging, contrast-enhanced T1-weighted imaging, T2-weighted imaging, and liquid-attenuated inversion recovery sequence. The above-mentioned BraTS series datasets come from different clinical institutions, scanning equipment, and imaging protocols, and have the characteristics of multi-center image data, which can be used to evaluate the segmentation performance of the three-dimensional medical image segmentation model in a heterogeneous clinical environment.

[0138] (3) In the data preprocessing stage, the original size is first... The three-dimensional magnetic resonance images were uniformly resampled to To reduce computational overhead and standardize the network input scale, Z-score normalization was then applied to each modality image to standardize its intensity, transforming it to a zero-mean, unit-variance space. This reduced the intensity distribution differences between different cases and modalities, thereby improving the stability of model training.

[0139] (4) The coarse and fine attention jump connection enhanced three-dimensional medical image segmentation model proposed in this invention is trained using the above training data. The optimal model parameters in the verification stage are loaded and applied to the test data. The voxel-level prediction results of the glioma region in the three-dimensional medical image are output. Finally, the segmentation results are quantitatively evaluated.

[0140] Furthermore, to verify the technical effectiveness of the method of the present invention, the following comparative examples were designed. Each comparative example was trained and tested using the same data partitioning, input image preprocessing method, learning rate, number of iterations, optimizer parameters, and evaluation metrics as the method of the present invention.

[0141] Comparative Example 1 This paper describes how to segment glioma regions in 3D medical images using 3D-UNet. 3D-UNet extends the convolution operations in 2D U-Net to 3D convolution operations, enabling dense segmentation of 3D volumetric data.

[0142] Comparative Example 2 This paper uses SegResNet to segment glioma regions in 3D medical images. SegResNet employs a residual encoder-decoder structure to extract features from 3D medical images and outputs voxel-level segmentation results.

[0143] Comparative Example 3 This paper uses nnU-Net to segment glioma regions in 3D medical images. nnU-Net can automatically configure preprocessing methods, network structure, training strategies, and post-processing methods based on dataset features, and is a representative benchmark method in medical image segmentation tasks.

[0144] Comparative Example 4 UNet++ was used to segment glioma regions in 3D medical images. UNet++ employs a nested skip connection structure to reduce the semantic differences between encoder and decoder features.

[0145] Comparative Example 5 Attention U-Net is used to segment glioma regions in 3D medical images. Attention U-Net introduces an attention gating mechanism in skip connections to highlight target-related regions and suppress irrelevant background responses.

[0146] Comparative Example 6 This paper uses UNETR to segment glioma regions in 3D medical images. UNETR introduces the Transformer architecture into 3D medical image segmentation to model global contextual information.

[0147] Comparative Example 7 This paper uses Swin UNETR to segment glioma regions in 3D medical images. Swin UNETR is based on a window self-attention mechanism for 3D feature modeling to capture local and global contextual relationships.

[0148] Comparative Example 8 This paper describes how to segment glioma regions in 3D medical images using SegMamba. SegMamba is based on a state-space model structure for long-range dependency modeling, which is used for 3D medical image segmentation.

[0149] Results Analysis: To demonstrate the technical effectiveness of the method of this invention, the segmentation results of this invention are compared with those of Comparative Examples 1 to 8. Quantitative comparison uses the Dice coefficient and Hausdorff distance as evaluation indicators. The Dice coefficient measures the degree of overlap between the predicted segmented region and the manually labeled region; a larger Dice coefficient indicates higher segmentation accuracy. The Hausdorff distance measures the distance difference between the predicted segmentation boundary and the manually labeled boundary; a smaller Hausdorff distance indicates that the predicted boundary is closer to the manually labeled boundary.

[0150] The Dice coefficient is defined as follows:

[0151] in, This represents the segmented region predicted by the model. Indicates manually labeled areas. This indicates the number of intersection elements between the predicted segmented region and the manually labeled region. This indicates the number of voxels in the predicted segmentation region. This indicates the number of voxels in the manually labeled area.

[0152] Hausdorff distance is defined as follows:

[0153] in, This represents the set of boundary points of the segmented region predicted by the model. This represents the set of boundary points of the manually labeled region. This represents the points in the predicted boundary point set. This represents the points in the manually labeled boundary point set. This represents the Euclidean distance between two points. Indicates the supremacy. Indicates the infimum.

[0154] Furthermore, to quantitatively compare the improvement of the method of the present invention relative to the optimal comparative example, the Dice improvement value and the Hausdorff distance reduction value are defined as follows:

[0155]

[0156] in, This represents the Dice coefficient obtained by the method of the present invention. This represents the optimal Dice coefficient obtained in each pair of proportions. This represents the Dice improvement value of the method of the present invention relative to the optimal comparative example; This represents the Hausdorff distance obtained by the method of the present invention. This represents the optimal Hausdorff distance obtained in each pair of proportions. This indicates the reduction in Hausdorff distance by the method of the present invention relative to the optimal comparative example.

[0157] Table 1 lists the segmentation results of the method of the present invention and Comparative Examples 1 to 8 on the BraTS 2020 adult glioma dataset. As can be seen from Table 1, the Dice coefficients of the method of the present invention for the overall tumor region (WT), tumor core region (TC), enhanced tumor region (ET), and the mean of the three regions (Mean) are 0.9172, 0.8719, 0.8016, and 0.8634, respectively; and the Hausdorff distances for WT, TC, ET, and Mean are 3.45, 3.11, 3.34, and 3.30, respectively. Compared with Comparative Examples 1 to 8, the method of the present invention achieves the best results in Dice coefficients for WT, TC, ET, and Mean, and also achieves the best results in Hausdorff distances for WT, TC, ET, and Mean. Compared to Comparative Example 4, which had the best average Dice coefficient, the average Dice coefficient of the method of this invention increased from 0.8449 to 0.8634, an improvement of 0.0185; compared to Comparative Example 3, which had the best average Hausdorff distance, the average Hausdorff distance of the method of this invention decreased from 5.67 to 3.30, a reduction of 2.37. These results demonstrate that the method of this invention can achieve higher segmentation accuracy and better boundary localization in adult glioma segmentation tasks.

[0158] Table 2 lists the segmentation results of the method of the present invention and Comparative Examples 1 to 8 on the BraTS-PEDs 2023 pediatric glioma dataset. As can be seen from Table 2, the Dice coefficients of the method of the present invention on WT, TC, ET, and Mean are 0.8880, 0.8333, 0.6036, and 0.7749, respectively; and the Hausdorff distances on WT, TC, ET, and Mean are 4.07, 4.43, 3.74, and 4.08, respectively. Compared with Comparative Examples 1 to 8, the method of the present invention achieves the highest average Dice coefficient and the lowest average Hausdorff distance. Compared with Comparative Example 7, which has the best average Dice coefficient, the average Dice coefficient of the method of the present invention increases from 0.7504 to 0.7749, an increase of 0.0245; compared with Comparative Example 8, which has the best average Hausdorff distance, the average Hausdorff distance of the method of the present invention decreases from 5.83 to 4.08, a decrease of 1.75. Furthermore, the Dice coefficients of the method of the present invention are higher in the WT and TC regions than those of the respective pairs, and the Hausdorf distances in the WT, TC and ET regions are lower than those of the respective pairs, indicating that the method of the present invention can improve the overall accuracy and boundary localization ability of pediatric glioma segmentation.

[0159] Table 1: Comparison of segmentation results on BraTS 2020 adult data

[0160] Table 2: Comparison of segmentation results on BraTS-PEDs 2023 child data

[0161] The experimental results above demonstrate that this invention, by constructing a cross-layer prototype memory and utilizing coarse-grained attention for global semantic candidate selection and fine-grained attention for encoder local window backtracking during the decoding stage, can establish effective cross-layer semantic associations between the encoder and decoder, and supplement local structural details in 3D space. Compared to traditional skip connection methods that use direct concatenation or addition, this invention reduces the impact of redundant and semantically inconsistent features on the decoding process, improves the quality of skip connection feature fusion, and thus achieves higher segmentation accuracy and better boundary localization results in 3D medical image segmentation tasks such as glioma.

[0162] See Figure 7 The present invention provides a three-dimensional medical image segmentation apparatus with enhanced coarse and fine attention skip connections, comprising one or more processors for implementing a three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections as described in the above embodiments.

[0163] The embodiments of the three-dimensional medical image segmentation device enhanced by coarse and fine attention skip connections of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities, in which the coarse-fine attention skip connection enhancement 3D medical image segmentation device of the present invention is located. (Except for...) Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0164] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0165] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0166] Accordingly, embodiments of the present invention also provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections described in the above embodiments.

[0167] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0169] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A three-dimensional medical image segmentation method enhanced with coarse and fine attention skip connections, characterized in that, Includes the following steps: (1) Input the multimodal three-dimensional medical image to be segmented into the encoder, perform window division and feature embedding on the multi-scale three-dimensional features output by the encoder, generate cross-layer prototype features, and record the hierarchical index and spatial index corresponding to the cross-layer prototype features to construct a prototype memory bank. (2) Generate query features based on the current layer features of the decoder, perform similarity matching between the query features and the cross-layer prototype features in the prototype memory obtained in step (1) to obtain coarse-grained attention weights, and generate coarse-grained attention results based on the coarse-grained attention weights. (3) Based on the coarse-grained attention weights obtained in step (2), select candidate prototypes from the prototype memory, backtrack the corresponding encoder local window features based on the hierarchical index and spatial index of the candidate prototypes, and generate fine-grained attention results based on the encoder local window features. (4) The coarse-grained attention result obtained in step (2) and the fine-grained attention result obtained in step (3) are fused to obtain a two-stage attention result. The two-stage attention result is used to weight and enhance the skip connection feature, and then input into the decoder to obtain the three-dimensional medical image segmentation result.

2. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 1, characterized in that, Step (1) includes the following sub-steps: (1.1) Input the multimodal 3D medical image to be segmented into the encoder, and extract 3D encoder features with different spatial resolutions and different semantic levels through multi-layer 3D convolution, normalization and nonlinear activation operations; (1.2) For the first Layer encoder features According to the preset window size The three-dimensional window is divided to obtain multiple three-dimensional local window features. , Indicates the number of the 3D local window; (1.3) Regarding the three-dimensional local window features Perform feature embedding mapping and in-window aggregation to obtain prototype features. ; (1.4) Map the prototype features obtained from different encoder levels to a unified feature dimension and perform cross-layer concatenation to form a cross-layer prototype feature set. Record the encoder level corresponding to each cross-layer prototype feature. and the position of the three-dimensional space window Build an index mapping table , Indicates the prototype number in the cross-layer prototype feature set; (1.5) The cross-layer prototype feature set and index mapping table Combine these elements to build a prototype memory.

3. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 1, characterized in that, Step (2) includes the following sub-steps: (2.1) Obtain the decoder's first... Layer features and for the decoder the first Layer features Perform feature mapping to obtain the query features; (2.2) Perform linear mapping on the cross-layer prototype feature set in the prototype memory to obtain the key feature matrix and the value feature matrix; (2.3) Calculate the similarity between the query features and the key feature matrix to obtain the relevance score matrix; (2.4) Normalize the correlation score matrix to obtain coarse-grained attention weights, and use the coarse-grained attention weights to perform weighted aggregation on the value feature matrix to generate coarse-grained attention results.

4. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 3, characterized in that, In sub-step (2.3), the similarity between the query feature and the key feature matrix is ​​calculated using dot product similarity.

5. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 3, characterized in that, In sub-step (2.4), the normalization process uses the Softmax operation.

6. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 1, characterized in that, Step (3) includes the following sub-steps: (3.1) The coarse-grained attention weights are averaged and aggregated along the query position dimension to obtain the relevance score corresponding to each prototype; based on the relevance score, the top prototypes are selected from the prototype memory. A set of candidate prototype indices is obtained by identifying highly relevant candidate prototypes. (3.2) Based on the candidate prototype index set, query the encoder level and three-dimensional space window position corresponding to each candidate prototype in the index mapping table; (3.3) Based on the encoder level and the three-dimensional space window position, backtrack from the corresponding encoder feature layer and extract the encoder local window features; (3.4) Perform feature mapping and fine-grained modeling on the local window features of the encoder, and backfill the obtained fine-grained local window features to the spatial position of the corresponding encoder level. After count normalization, scale alignment and cross-layer aggregation, generate fine-grained attention results.

7. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 1, characterized in that, Step (4) includes the following sub-steps: (4.1) The coarse-grained attention results are spliced ​​and fused with the fine-grained attention results, and the two-stage attention results are obtained through three-dimensional convolution and normalization operations; (4.2) Let the decoder be the first... The encoder skip connection feature corresponding to the layer is The two-stage attention results are compared with... Element-by-element addition yields the enhanced skip connection features; (4.3) The enhanced skip connection features are concatenated and convolved with the decoder upsampled features to obtain the current layer decoder output features; (4.4) Repeat step (2) to sub-step (4.3) to perform coarse and fine two-stage attention enhancement on the skip connection features of each decoding stage and restore the spatial resolution step by step to obtain the final decoding features; The final decoded features are subjected to three-dimensional convolutional mapping and classification processing to obtain voxel-level segmentation results of the three-dimensional medical image.

8. The three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections according to claim 7, characterized in that, During the model training phase, the voxel-level segmentation results obtained in sub-step (4.4) are compared with the corresponding manually labeled results. Based on Dice loss, cross-entropy loss, or a combination of Dice loss and cross-entropy loss, the relevant mapping parameters of the encoder, decoder, prototype memory, and coarse and fine attention generation parameters are optimized.

9. A three-dimensional medical image segmentation apparatus with enhanced coarse and fine attention skip connections, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the three-dimensional medical image segmentation method with enhanced coarse and fine attention skip connections as described in any one of claims 1-8.

Images