Medical image segmentation method and system based on pixel-by-pixel adaptive feature fusion

Abstract

The invention discloses a medical image segmentation method and system based on pixel-by-pixel adaptive feature fusion, and the method employs a double-branch feature extraction structure of a CNN branch and a Mama branch based on a U-shaped network, and designs a pixel-by-pixel adaptive feature fusion mode in a segmentation process. And carrying out refined fusion on the multi-scale feature maps from different feature extraction paths. The method comprises the following steps: firstly, carrying out spatial alignment on a multi-source feature map in an interpolation or resampling mode; generating a corresponding feature fusion weight for each pixel position based on the local context information of the fusion features; differentiation weighting is carried out on features of different sources at the same spatial position, and fusion features are output in a residual mode, so that the stability of the feature fusion process is improved. According to the method, local detail expression can be enhanced in a fine structure area, the overall structure continuity can be kept in a trunk area and a branch area, disconnection and false detection phenomena in medical image segmentation are effectively reduced, the continuity and reliability of a segmentation result are improved, and the method is suitable for automatic segmentation processing of blood vessel type medical images.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and deep learning technology, and in particular to a method and system for segmenting coronary artery X-ray angiography images based on a pixel-wise adaptive feature fusion network. Background Technology

[0002] With increasing emphasis on the diagnosis and prevention of cardiovascular diseases, digital subtraction angiography (DSA) and X-ray coronary angiography are gaining more attention as accurate diagnostic tools. Automatically extracting a clear and complete vascular tree structure from coronary angiography images plays a significant role in the study of coronary artery disease: high-precision vessel segmentation results help calculate quantitative data such as coronary artery stenosis length, lumen diameter, and blood flow velocity in scientific research, and also facilitate subsequent 3D modeling and hemodynamic analysis of the affected area; in clinical applications, it helps physicians accurately diagnose the lesion area and develop personalized interventional treatment or surgical plans. Therefore, automated and precise segmentation of coronary arteries is of significant clinical importance for the quantitative assessment, diagnosis, and treatment planning of coronary artery disease.

[0003] Currently, coronary artery segmentation typically relies on manual annotation by medical experts, which is not only time-consuming and costly but also suffers from significant subjective differences and inter-observer variability, making it difficult to meet the needs of clinical practice and large-scale data processing. Traditional segmentation methods based on thresholding, region growing, and edge detection are highly sensitive to image quality and cannot effectively handle complex situations such as noise, low contrast, and overlapping vessels in coronary angiography images, resulting in limited segmentation accuracy and robustness.

[0004] While deep learning methods have made significant progress, existing models still have key technical problems. U-Net and its variants mainly rely on convolutional operations, which have limited local receptive fields and are insufficient in modeling the long-range topological structure of blood vessels. Although attention-based U-Net and SA-UNet have enhanced spatial attention, they still struggle to capture the overall continuity and global morphology of slender blood vessels. Summary of the Invention

[0005] Purpose of the Invention: The purpose of this invention is to provide a medical image segmentation method and system based on pixel-wise adaptive feature fusion. Existing medical image segmentation methods typically use fixed weights or global weights to fuse features from different feature extraction paths, making it difficult to adaptively adjust the feature fusion strategy to accommodate local structural differences at different spatial locations in medical images. This is especially problematic in areas with complex tissue boundaries or high requirements for structural continuity, where inaccurate segmentation boundaries or structural breaks are prone to occur. This invention proposes a medical image segmentation method, system, and computer-readable storage medium based on pixel-wise adaptive feature fusion to improve the accuracy and stability of medical image segmentation results in complex regions.

[0006] Technical solution: This invention provides a medical image segmentation method based on pixel-wise adaptive feature fusion, which includes the following steps.

[0007] First, the medical image is input into the segmentation network. The medical image is then processed by at least two different feature extraction paths set in the encoder of the segmentation network. These different feature extraction paths have different focuses in feature representation, so as to extract local spatial detail features and overall structural continuity features of the medical image respectively, thereby obtaining multi-scale feature maps from different feature extraction paths.

[0008] Subsequently, a pixel-wise adaptive feature fusion operation is performed on feature maps from different feature extraction paths but at the same scale. In this pixel-wise adaptive feature fusion operation, the feature maps at the same scale are first spatially aligned to eliminate spatial resolution inconsistencies caused by differences in the encoding structures of different feature extraction paths or downsampling operations. After spatial alignment, fused features for weight calculation are constructed based on feature maps from different feature extraction paths. Combined with the contextual information of these fused features in their local spatial neighborhood, feature fusion weights corresponding to different feature extraction paths are generated for each pixel location.

[0009] During feature fusion, at the same pixel location, feature maps from different feature extraction paths are differentially weighted according to the feature fusion weights to obtain a fused feature map. The feature fusion weights can be normalized before being used for feature weighting to ensure that multiple feature fusion weights corresponding to the same pixel location meet predetermined constraints, thereby improving the numerical stability of the pixel-wise adaptive feature fusion process. Furthermore, the fused feature map can be further residual-fused with a baseline feature generated based on the spatially aligned feature map to enhance feature representation and improve the stability of the feature fusion process.

[0010] Furthermore, the pixel-wise adaptive feature fusion operation can be performed at multiple feature levels of the segmentation network encoder, and the feature maps performing the pixel-wise adaptive feature fusion operation at different feature levels have different spatial resolutions. The segmentation network is an encoder-decoder structure segmentation network, where the encoder and decoder transfer features through skip connections. The pixel-wise adaptive feature fusion operation is set between the encoder output and the decoder input to replace or enhance the feature fusion process in the skip connections.

[0011] Finally, based on the fused feature map, the segmentation result of the medical image is output through decoding and reconstruction processing.

[0012] This invention also provides a medical image segmentation system based on pixel-wise adaptive feature fusion, including a feature extraction module, a pixel-wise adaptive feature fusion module, and a decoding and reconstruction module. The feature extraction module is used to extract features from the medical image through at least two different feature extraction paths to generate multi-scale feature maps from different feature extraction paths. The pixel-wise adaptive feature fusion module is used to perform pixel-wise adaptive feature fusion operation on the same-scale feature maps from different feature extraction paths. The decoding and reconstruction module is used to generate the segmentation result of the medical image based on the fused feature map.

[0013] In addition, the present invention also provides a computer system and a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described medical image segmentation method based on pixel-by-pixel adaptive feature fusion.

[0014] Beneficial effects: Compared with the prior art, the present invention has at least the following beneficial effects.

[0015] By performing pixel-wise adaptive feature fusion based on local spatial context information on the same scale feature map, the feature fusion strategy can be flexibly adjusted according to the local structural characteristics of different spatial locations, thereby effectively improving the accuracy of medical image segmentation in complex boundary regions.

[0016] By performing differential weighted fusion of features from different feature extraction paths, the continuity of the overall structure is enhanced while preserving local details, thus reducing structural breaks in the segmentation results.

[0017] The pixel-wise adaptive feature fusion operation can be set between the encoder and decoder of the segmentation network, and is compatible with various encoder-decoder structures of segmentation networks. It has good versatility and scalability, and is easy to deploy and apply in different medical image segmentation tasks. Attached Figure Description

[0018] Figure 1This is a flowchart illustrating the implementation of this invention;

[0019] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0020] Figure 3 This is a schematic diagram of the attention-gated fusion module structure;

[0021] Figure 4 This is a graph showing the comparison between the present invention and existing methods on a dataset. Detailed Implementation

[0022] The technical solution and effects of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0023] This invention discloses a medical image segmentation method based on pixel-wise adaptive feature fusion, the implementation flowchart of which is shown below. Figure 1 As shown, the specific steps include the following:

[0024] Step 1: Image Preprocessing: Color space conversion and normalization are performed on the coronary angiography images to unify the input images into RGB format. Medical image-specific data augmentation strategies are applied to the training set images, including elastic deformation to simulate the natural curvature of blood vessels, CLAHE adaptive histogram equalization to enhance the visibility of low-contrast blood vessels, as well as geometric transformations, intensity transformations, and noise reduction. Only size normalization is performed on the validation and test sets to maintain the realism of the evaluation.

[0025] Step 2: Construct a pixel-wise adaptive feature fusion segmentation network: First, construct a U-shaped network framework. In the encoder stage, establish a parallel structure with two branches of CNN and Mamba. The two branches extract local detail features and global continuity features of blood vessels, respectively. Introduce a hierarchical adaptive gated fusion module (AGF) to replace the simple skip connections of the original U-shaped network. Based on the semantic complexity of features at different levels, adopt fusion strategies with different complexities to intelligently weight and fuse the features of the two branches. The image spatial resolution is gradually restored through the decoder, and the blood vessel segmentation result is finally output.

[0026] Specifically, in this embodiment, the pixel-wise adaptive feature fusion segmentation network includes a dual-branch encoder, a hierarchical adaptive gated fusion module (AGF), a decoder, and a segmentation head. The dual-branch encoder includes a CNN branch and a Mamba branch that work in parallel. The CNN branch uses ResNet50 as the backbone to extract local detail features from the image, while the Mamba branch extracts global continuity features from the image through an improved two-dimensional Mamba layer. The features at different scales output by the two branches are intelligently weighted and fused in the hierarchical adaptive gated fusion module according to different strategies based on semantic complexity, generating fused features that combine local detail and global continuity. The fused features are then fed into the decoder and organically combined with the feature map in the decoder through skip connections and convolutions to fully extract the spatial positional relationship and topological continuity information of blood vessels.

[0027] like Figure 2 As shown, in this embodiment, the dual-branch encoder contains five layers. The CNN branch uses ResNet50 as the backbone to generate feature layers of five scales through convolution and pooling operations. The Mamba branch generates corresponding feature layers of five scales through an improved two-dimensional Mamba layer. The decoder contains four layers, each corresponding to the encoder, and the layers are generated through upsampling operations. The dual-branch encoder and decoder are connected through a hierarchical adaptive gating fusion module. The fusion module adopts a fusion strategy of different complexity according to the feature level, and fuses the corresponding layer features of the CNN branch and the Mamba branch at the five levels before connecting them to the decoder.

[0028] The hierarchical adaptive gating fusion module mainly involves adaptively weighted fusion of CNN local features and Mamba global features, employing a progressive fusion strategy based on the semantic complexity of the feature levels. Simple fusion is used in layers 0 and 1, with fast fusion through channel concatenation and convolution; medium-complexity fusion or attention-gated fusion is used in layers 2 and 3, adaptively adjusting the fusion weights through channel attention or dynamic gating mechanisms; and medium-complexity fusion is used in layer 4 to process high-semantic features. The fused features are then combined with features from different levels of the decoder in skip connections, enabling the auxiliary network to simultaneously focus on local details and global continuity during segmentation, thus strengthening the constraint on the integrity of the vascular structure.

[0029] Specifically, for the attention-gated fusion module, CNN features are... With Mamba characteristics Separating the data along the channel dimension, semantic features are extracted using depthwise separable convolution and CBAM attention mechanism. After dimensionality reduction to 2 channels via 1×1 convolution, pixel-wise dynamic gating weights are generated through Softmax normalization. and The gating weights are multiplied element-wise by their corresponding features, concatenated, and then convolved to obtain the fused features. Simultaneously, the spliced ​​features are processed through 1×1 convolution and batch normalization to obtain baseline features. The final output feature is The feature representation capability is enhanced through residual connections.

[0030] This process is represented by the following formula:

[0031]

[0032]

[0033]

[0034] in Representing CNN branch features, Indicates Mamba branching characteristics, This indicates a depthwise separable convolution operation. This represents the channel and spatial attention mechanism. This indicates element-wise multiplication. This indicates a channel splicing operation.

[0035] like Figure 3 As shown, the Attention Gated Fusion (AGF) module consists of an attention gate generator, a feature weighted fusion unit, and a residual connection unit. The attention gate generator extracts semantic features by performing depthwise separable convolutions on the concatenated CNN and Mamba features. After processing by the CBAM attention mechanism, the features are reduced to 2 channels via 1×1 convolutions, and then normalized using Softmax with a temperature parameter to obtain a pixel-wise dynamic gated weight map. The gated weight map includes local feature weights and global feature weights, and its size is consistent with the input feature layer.

[0036] The attention-gated fusion module is connected to the dual-branch encoder and decoder. The input is the features extracted by the CNN branch and the Mamba branch at different levels, and the output is a feature layer with local-global adaptive fusion information.

[0037] The fused features are obtained by element-wise weighting of the CNN features and Mamba features with local and global feature weights respectively, followed by concatenation. These fused features contain both local details and global continuity of blood vessels. Simultaneously, the concatenated features are processed through 1×1 convolutions and batch normalization to obtain baseline features. The fused features are then added to the baseline features to obtain an output feature layer with residual connections. These feature layers are ultimately fed into the decoder.

[0038] The attention-gated fusion module is shown in the following figure:

[0039]

[0040]

[0041]

[0042] In the formula For the generated output feature layer, For the feature layer of the CNN branch, For the characteristic layer of the Mamba branch, For depthwise separable convolution operations, For channel and spatial attention mechanisms, and For the generated dynamic gating weights, This is an element-wise multiplication operation. This is for channel splicing operations. This is the normalized activation function.

[0043] Step 3: Train the network: Divide the data processed in Step 1 into training, testing, and validation sets. Use the data to train the network model to obtain a converged network model. During training, a composite loss function is used for end-to-end training of the network, including Focal loss, Dice loss, clDice loss, and TV regularization loss.

[0044] The composite loss function calculation process is as follows: The input image and the corresponding segmentation label are input into the network. The dual-branch encoder in the network is responsible for extracting local detail features and global continuous features respectively. After being fused by the hierarchical adaptive gating fusion module, the images are sent to the decoder to gradually restore the spatial resolution. Finally, the segmentation label predicted after training is obtained through the Sigmoid activation function.

[0045] Focal loss is used to handle foreground / background class imbalance:

[0046]

[0047] in It is a prediction probability. It is a balancing factor. It is the focusing parameter.

[0048] Dice loss is used to optimize region overlap:

[0049]

[0050] in It is the predicted probability value. It's a real label. It is the sample size. It is a smoothing term.

[0051] The clDice loss is used to constrain the continuity of the vessel centerline: first, the predicted segmentation results and the ground truth labels are skeletonized to obtain the centerline. and Then calculate the Dice coefficient for the centerline:

[0052]

[0053] TV regularization loss is used to smooth the gated weight graph:

[0054]

[0055] in Let $i$ be the value of the gated weight graph at position $(i,j)$.

[0056] The total loss function is:

[0057]

[0058] in The dynamic weights for the clDice loss.

[0059] Step 4: Coronary angiography image segmentation: Input the coronary angiography image to be processed into the network for fully automatic coronary artery segmentation.

[0060] The performance of the network model is evaluated using the metrics Precision, Recall, F1-score, Intersection over Union (IoU), Dice-Score, and clDice.

[0061] Precision represents the percentage of samples correctly predicted (with a true positive label) out of all samples predicted as positive by the model, and it represents the accuracy of the model during segmentation.

[0062] Recall represents the percentage of samples with positive true labels that are predicted; that is, for blood vessels in the true label, how many of the samples are classified as blood vessels in the model segmentation results.

[0063] The F1 score is the harmonic mean of precision and recall, which takes into account the effects of both and provides a balanced evaluation metric. It is suitable for situations where there is a trade-off between precision and recall.

[0064] Intersection over Union (IOU) is the intersection-over-union ratio of predicted samples and actual samples. Dice-Score is a metric used to evaluate the similarity between predicted segmentation maps and ground truth labels in image segmentation tasks. It is defined as twice the intersection of two sets divided by the sum of the two sets.

[0065] clDice is an index specifically designed to assess the continuity of the vessel centerline. It is calculated by extracting the vessel centerline using a skeletonization algorithm, which can more accurately reflect the integrity and continuity of vessel segmentation.

[0066] To objectively demonstrate the performance of the proposed method in coronary angiography image segmentation, the network is compared and evaluated with other representative medical segmentation models, such as U-Net, Attention U-Net, SA-UNet, and Swin-Unet. Experimental data for the network models are shown in Table 1.

[0067] Table 1 Comparison of Evaluation Metrics for Algorithm Segmentation Results

[0068] Model IOU Dice Precision Recall F1 clDice U-Net 0.827 0.689 0.917 0.878 0.895 0.896 Attention-UNet 0.835 0.704 0.929 0.896 0.909 0.899 SA-UNet 0.834 0.698 0.927 0.883 0.898 0.912 Swin-UNet 0.837 0.705 0.930 0.881 0.898 0.923 This method 0.845 0.721 0.928 0.915 0.908 0.927

[0069] As shown in Table 1, Figure 4 As stated above, in comparison tests with other methods on the JMA dataset, the method of the present invention achieved the best performance.

[0070] This invention also discloses a medical image segmentation system based on pixel-wise adaptive feature fusion, comprising: a data preprocessing module for performing color space conversion and normalization on coronary X-ray angiography images, and applying medical image-specific data augmentation strategies to the training set, including elastic deformation, CLAHE adaptive histogram equalization, geometric transformation, intensity transformation, and noise processing;

[0071] The network model construction module is used to build a pixel-wise adaptive feature fusion and segmentation network based on a U-shaped network framework. In the encoder stage, a parallel structure of CNN branches and Mamba branches is established, and a hierarchical adaptive gated fusion module (AGF) replaces the simple skip connection module of the original U-shaped network. The CNN branch extracts local detail features through the ResNet50 backbone, and the Mamba branch extracts global continuity features through an improved two-dimensional Mamba layer. The hierarchical adaptive gated fusion module intelligently weights and fuses the features of the two branches according to the semantic complexity of the feature hierarchy, generating adaptive fused features that combine local detail and global continuity. The loss function calculation module is used to calculate the composite loss function, including a weighted combination of Focal loss, Dice loss, clDice loss, and TV regularization loss, where the clDice loss constrains the continuity of the vessel centerline through a skeletonization algorithm.

[0072] The network training and testing module is used to train the network using a two-stage training strategy, including a backbone freezing stage and a backbone unfreezing stage. It is combined with a cosine annealing learning rate scheduler, gradient pruning, and mixed precision training to obtain a converged network model. The trained network model is then input into a test coronary angiography image to obtain the segmentation result.

[0073] For specific implementation details of each module, please refer to the above method implementation examples, which will not be repeated here.

[0074] This invention also discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the medical image segmentation method based on pixel-by-pixel adaptive feature fusion.

[0075] This invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the medical image segmentation method based on pixel-by-pixel adaptive feature fusion.

Claims

1. A medical image segmentation method based on pixel-wise adaptive feature fusion, characterized in that, Includes the following steps: A medical image is input into a segmentation network, and features are extracted from the medical image through at least two different feature extraction paths in the segmentation network encoder to obtain multi-scale feature maps from different feature extraction paths. For feature maps of the same scale from different feature extraction paths, a pixel-wise adaptive feature fusion operation is performed. The pixel-wise adaptive feature fusion operation includes: spatially aligning the feature maps of the same scale; constructing fusion features for weight calculation based on the spatially aligned feature maps, and generating corresponding feature fusion weights for each pixel position according to the context information of the fusion features in the local spatial neighborhood; and differentially weighting the feature maps from different feature extraction paths according to the feature fusion weights at the same pixel position to obtain the fused feature map. Based on the fused feature map, the segmentation result of the medical image is output through decoding and reconstruction.

2. The medical image segmentation method based on pixel-wise adaptive feature fusion according to claim 1, characterized in that, The at least two different feature extraction paths refer to at least one feature extraction path for extracting local spatial detail features of the medical image and another feature extraction path for extracting the overall structural continuity features of the medical image. The multi-scale feature maps include feature maps from different levels of the segmentation network encoder, with different levels having different spatial resolutions.

3. The medical image segmentation method based on pixel-wise adaptive feature fusion according to claim 1, characterized in that, The spatial alignment of feature maps at the same scale includes: when feature maps from different feature extraction paths have inconsistent spatial resolutions, adjusting the feature maps to the same spatial resolution through interpolation or resampling; the fusion features used for weight calculation are generated from the spatially aligned feature maps from different feature extraction paths according to a predetermined feature combination method; the feature combination method includes at least one of feature concatenation, feature mapping, or linear transformation. Generating corresponding feature fusion weights for each pixel position includes: based on the context information of the fusion features in the local spatial neighborhood, and considering the correlation of pixels in both the spatial and feature dimensions, generating at least two feature fusion weights for the corresponding pixel position, each feature fusion weight corresponding to a feature map from a different feature extraction path.

4. The medical image segmentation method based on pixel-wise adaptive feature fusion according to claim 1, characterized in that, The feature fusion weights are normalized before being used for differential weighting of the feature map, ensuring that multiple feature fusion weights corresponding to the same pixel position meet predetermined constraints. The differential weighting includes: multiplying the feature fusion weights pixel-by-pixel with the feature map of the corresponding feature extraction path at the same pixel position, and then combining the weighted features to obtain the fused feature map. Before output, the fused feature map is further residual-fused with a baseline feature generated based on the spatially aligned feature map to enhance feature representation and improve the stability of the pixel-by-pixel adaptive feature fusion process.

5. The medical image segmentation method based on pixel-wise adaptive feature fusion according to claim 1, characterized in that, The pixel-wise adaptive feature fusion operation is performed at multiple feature levels of the segmentation network encoder, with different feature map spatial resolutions at different feature levels. The segmentation network is an encoder-decoder structure, where the encoder and decoder transfer features through skip connections. The pixel-wise adaptive feature fusion operation is positioned between the encoder output and the decoder input, acting to replace or enhance the feature fusion process in the skip connections.

6. A medical image segmentation system based on pixel-wise adaptive feature fusion, characterized in that, include: The feature extraction module is used to extract features from medical images through at least two different feature extraction paths, and generate multi-scale feature maps from different feature extraction paths. A pixel-wise adaptive feature fusion module is used to perform a pixel-wise adaptive feature fusion operation on feature maps of the same scale from different feature extraction paths. The pixel-wise adaptive feature fusion module includes a spatial alignment unit, a weight generation unit, and a differential weighting unit. The decoding and reconstruction module is used to generate segmentation results for medical images based on the fused feature maps.

7. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the medical image segmentation method based on pixel-wise adaptive feature fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the medical image segmentation method based on pixel-wise adaptive feature fusion as described in any one of claims 1 to 5.

Images