Medical image segmentation method based on connectivity
By introducing a connectivity-aware gated fusion network, the problem of connectivity neglect in medical image segmentation is solved, higher precision segmentation results and better topological integrity are achieved, and the shortcomings in the prior art are solved.
Patent Information
- Application Number
- CN202510327603.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-19
- Publication Date
- 2025-07-18
AI Technical Summary
Existing deep learning methods ignore connectivity between pixels in medical image segmentation, resulting in lack of continuity and rough edges in segmentation results, and the risks of overfitting and large demand for labeled samples.
Connectivity-aware gated fusion network (CAGFNet) is used to enhance the connectivity of image features by building connectivity-aware enhancement module (CAAM), dynamic shift convolution (SDConv) and gated fusion module (GFFM). Combining topological ideas, eight-channel connectivity masks are generated and segmentation results are optimized using multiple loss functions.
It improves the accuracy of medical image segmentation and the internal topological consistency of segmentation results, reduces the problem of edge roughness, and significantly improves the segmentation effect.
Smart Images
Figure CN120339298A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical image processing, and more specifically, relates to a medical image segmentation method based on connectivity. Background Art
[0002] Medical image segmentation is an important problem in the fields of computer vision and medical imaging research. Its goal is to accurately extract regions of interest from complex medical images, such as cells, tissues, organs, or lesion regions, etc. The segmentation accuracy of medical images directly affects the accuracy of clinical diagnosis and is also an important tool for exploring biological science knowledge and understanding disease mechanisms.
[0003] However, medical image segmentation faces many challenges. First, the complexity of medical images is very high, including the complexity of cell structures, the heterogeneity of human tissues, and the diversity of disease states. Second, the quality of medical images is also affected by many factors, such as radiation noise, differences in imaging devices, and the physiological movements of patients. These factors make medical images usually have characteristics such as low contrast, high noise, and uncertain resolution, bringing great difficulties to image segmentation. In addition, accurate medical image segmentation requires a lot of clinical experience and professional knowledge, and these knowledge are often difficult to be fully reflected in formalized and automated processing.
[0004] To solve the above problems, many medical image segmentation methods based on statistics, computational geometry, and machine learning have been proposed. Under the leadership of deep learning, convolutional neural networks (CNNs), especially variants such as fully convolutional networks (FCNs) and U-Nets, have shown excellent performance in medical image segmentation. Such methods can extract image features at multiple scales and can learn a large number of useful segmentation features.
[0005] However, most of the existing deep learning methods focus on local segmentation performance and ignore the fact that there is often a strong connectivity between spatially adjacent pixels. Especially when dealing with problems that require considering pixel connectivity, such as blood vessels, nerve fibers, tumor boundaries, etc., the current methods often fail to meet the requirements. In addition, the existing methods also have problems such as a large risk of overfitting, the need for a large number of labeled samples, and the lack of continuity in segmentation results. Summary of the Invention
[0006] Based on the idea of connectivity in topology, the present invention proposes a medical image segmentation method that can accurately segment medical tissues or lesion regions, and on this basis, while ensuring the internal topology integrity of the segmentation result, solve the problem of rough edges.
[0007] To achieve the above object, the present invention is implemented by the following technical solutions: The method includes:
[0008] Step 1: Collect a medical image dataset and perform preprocessing, dividing it into a training set and a validation set;
[0009] Step 2: Perform data augmentation on the training set and the validation set. The data augmentation includes random rotation, elastic deformation, and color adjustment, and generate an eight-channel connectivity mask for the mask of the training set. The connectivity mask reflects the class consistency of pixels with their adjacent pixels in eight directions;
[0010] Step 3: Construct a Connectivity-Aware Gated Fusion Network (CAGFNet), including:
[0011] A ResNet34 backbone network for extracting multi-layer feature maps of the input image;
[0012] A Connectivity-Aware Enhancement Module (CAAM) that divides the feature map output by the backbone network into eight
[0013] groups and performs directional shifting, and adjusts the feature weights through channel attention and spatial attention;
[0014] Dynamic Shift Convolution (SDConv) that optimizes feature extraction through parallel convolution and shifting strategies in the decoding stage;
[0015] Step 4: Input the training set data into the constructed Connectivity-Aware Gated Fusion Network for training;
[0016] Step 5: Input the preprocessed medical image into the trained Connectivity-Aware Gated Fusion Network to obtain the segmentation result.
[0017] In one scheme, the method for generating the connectivity mask in Step 2 includes:
[0018] For the original mask of the training set, calculate the pixel connectivity in eight directions, with each direction corresponding to one channel;
[0019] For each direction channel, if the current pixel and its adjacent pixel in that direction belong to the same category, mark the corresponding position in that channel with the corresponding category label (e.g., if both are label 1, then it is 1), otherwise mark it as 0;
[0020] Concatenate the marking results of the eight direction channels into an eight-channel connectivity mask.
[0021] In one scheme, the execution steps of the CAAM module include:
[0022] Evenly divide the input feature map into eight groups according to channels, and each group performs a single-pixel shift along a corresponding direction;
[0023] For each group of shifted features, channel attention and spatial attention are sequentially applied. Among them, channel attention generates channel weights through global feature compression and a fully connected layer, and spatial attention generates spatial weights through convolution;
[0024] The eight groups of weighted features are concatenated and added to the original features to output the fused features.
[0025] In one solution, the method for generating the channel attention includes:
[0026] Global average pooling is performed on the input feature map to obtain a channel description vector;
[0027] The vector is input into two layers of fully connected layers. The number of neurons in the first layer is one-eighth of the number of input channels, and the number of neurons in the second layer is restored to the original number of channels;
[0028] A channel weight matrix is generated through an activation function and multiplied with the input features channel by channel.
[0029] In one solution, the implementation of the dynamic shift convolution (SDConv) includes:
[0030] The following three operations are simultaneously performed on the input feature map: standard convolution, convolution after shifting, and convolution with dynamic weight adjustment;
[0031] The output feature maps of the three operations are weighted and summed according to a preset ratio to obtain the final features.
[0032] In one solution, the loss function consists of three parts:
[0033] Dice loss based on the overlap degree between the prediction result and the true mask;
[0034] Cross-entropy loss based on the class probability distribution;
[0035] Connectivity loss based on the prediction error of the eight-direction connectivity mask;
[0036] The final loss is the weighted sum of the three, where the weight of the Dice loss is 1, the weight of the cross-entropy loss is 0.5, and the weight of the connectivity loss is 0.3.
[0037] In one solution, in the decoding stage, multi-scale features are fused through a gated fusion module, which specifically includes:
[0038] The high-level features are upsampled and concatenated with the low-level features, and a gated weight map is generated through 1×1 convolution;
[0039] The gated weight map is used to perform weighted screening on the low-level features to suppress the noise regions;
[0040] The weighted features are input into the dynamic shift convolution to complete the fusion.
[0041] In one solution, the post - processing method of the segmentation result includes:
[0042] Count the number of consistent directions of each pixel in the eight - direction connectivity prediction. If more than half of the directions are determined to be connected, retain the segmentation result of the pixel;
[0043] Perform morphological closing operation filtering on the isolated regions in the segmentation result to ensure the topological continuity of the target region.
[0044] Advantages of the present invention:
[0045] Improve the accuracy of the segmentation result: By introducing connectivity and enhancing the connectivity significantly and performing feature fusion in the algorithm, the present invention can additionally enhance the image features from the perspective of connectivity, overcoming the limitation of insufficient utilization of connectivity information in the prior art.
[0046] While ensuring the internal topological consistency of the edge result and achieving edge lubrication, the present invention reduces the problems of internal topological loss and edge roughness caused by the lack of connectivity features through the introduced connectivity enhancement related module. As shown in the figure, compared with other algorithms, the segmentation result of the algorithm proposed by the present invention has better internal topological integrity and edge lubrication degree than the prediction results of other algorithms. Description of the drawings
[0047] Figure 1 Is the flowchart of the method of the present invention; Figure 2 Is the perception gating fusion network diagram of the present invention; Figure 3 Is the connectivity perception enhancement module diagram of the present invention; Figure 4 Is the dynamic shift convolution module diagram of the present invention; Figure 5 Is the output diagram of the present invention where the low - level feature map is multiplied element - by - element and then residual with the high - level feature map; Figure 6 Is the direction convolution module diagram of the present invention; Figure 7 Is the segmentation effect diagram of the present invention compared with other algorithms; Figure 8 Is the segmentation effect diagram of the present invention compared with other algorithms; Figure 9 Is the segmentation effect diagram of the present invention compared with other algorithms; Figure 10 Is the image, ground truth, and prediction result diagram of the present invention for the BUSI, CVC - ClinicDB, and Kvasir - SEG three datasets. Detailed implementation manners
[0048] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant accompanying drawings. Typical embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.
[0049] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as understood by those skilled in the technical field to which the present invention pertains. The terms used in the description of the present invention in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant accompanying drawings. Typical embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.
[0050] As Figure 1 shown, the present invention provides a connectivity-based medical image segmentation method, which is specifically as follows:
[0051] Step 1: Collect different data sets, preprocess the data sets, and obtain corresponding training set and validation set groupings.
[0052] Step 2: Perform simple data augmentation methods on the training set and test set, including random rotation, elastic deformation, and color adjustment. For the masks in the training set, connectivity masks are additionally generated for the network to train.
[0053] The connectivity mask reflects whether the pixel and the pixel in a specific direction are of the same specific category and can be obtained through a simple multiplication operation.
[0054]
[0055] where M C represents the connectivity mask, M represents the mask, where i represents eight different directions (also different channels). a and b have values of 0 or ±1 corresponding to different directions.
[0056] As Figure 2 shown, Step 3: Construct a connectivity-aware gating fusion network (CAGFNet) based on connectivity, including:
[0057] 1) ResNet34 backbone network: For the initial input image, a pre-trained ResNet34 model is used as the backbone network for feature extraction. In the backbone network, an image of 3×H×W is input, and the multi-layer backbone network outputs e1, e2, e3, e4, and e5 are obtained.
[0058] 2) Connectivity-Aware Enhancement Module CAAM: Enhance the feature information of the multi-layer feature maps of the backbone network. CAAM realizes the optimization and fusion of various features by introducing a direction-aware connectivity constraint and a dual attention cooperation mechanism. CAAM inputs the outputs E of different layers of the backbone network, divides E into 8 groups for different shifting operations, and then uses 8 connectivity channel attention blocks and connectivity spatial attention blocks with shared weights to adjust the attention weights of different channels and different positions, and multiplies them with the input features to obtain the output Out.
[0059] E i = shift i (split(E))
[0060] F = cat(E i ⊙ att C (E i ))
[0061] Out = F⊙ att S (F)
[0062] The module structure is as Figure 3 shown, att C corresponds to the connectivity channel attention mechanism with shared weights, and att S corresponds to the connectivity spatial attention mechanism. In shift i (split(E)), we divide the feature map E into eight groups by channel, and perform a one-pixel shifting operation in each group in eight directions of connectivity.
[0063] 3) Dynamic Shift Convolution SDConv: Extract features from the input feature map of each layer during the decoding process. SDConv increases the model complexity without increasing the network depth or width through the convolution kernel attention mechanism and the shifting strategy. For the input feature Figure X , calculate the attention weights α of the convolution kernel in the kernel space for the number of channels and convolution kernels in parallel ni , α ci . Multiply with multiple saved convolution kernels W i to obtain the final convolution kernel, and then perform grouped shifting on the input feature Figure X and perform the final convolution operation to obtain the module output Out.
[0064] α ci , α ni = attc (att o X), att n (att o X)
[0065]
[0066] The module structure is as Figure 4 shown, where att o is the global attention calculation, corresponding to the calculation of α in the image.
[0067] 4) Gated Feature Fusion Module (GFFM): It fuses the high-level feature map from the encoder and the low-level feature map from the decoder. GFFM uses the high-level feature map as a gating signal by introducing attention features, dynamically controlling the information flow at different stages of the network, thus significantly optimizing the feature representation in the medical image segmentation task. The input of GFFM is the high-level feature Figure X 1 and the low-level feature Figure X 2. The gating attention weights obtained by using different convolution operations are fused by addition, and then through a 1×1 convolution operation, the attention coefficient W rich in global context relationship is generated through the activation function. After multiplying element-wise with the low-level feature map and taking the residual with the high-level feature map, the final output Out is obtained, as Figure 5 shown.
[0068] W = G att (X1, X2)
[0069]
[0070] 5) Directional Convolution (PConv): It converts the output of the highest layer of the model into a connectivity output Y C . PConv improves the result of the connectivity output through the idea of grouping and shifting. PConv groups and shifts the input feature Figure X , performs convolution operations using exactly the same convolution kernel within each group, and after connecting the results for encoding, realizes the empirical verification of the interpretability of the connectivity output, as Figure 6 shown.
[0071] Y C = Conv 1*1 (ReLU(BN(cat(Conv 3*3 (shift(X))))))
[0072] 6) Bilateral Voting and Channel Aggregation Module: It converts the connectivity output of the model into a bilateral output and the final prediction result. Bilateral voting refers to the relevant settings of connectivity, multiplies the values of two corresponding pixels in the pixel connectivity relationship, and obtains the classification result at the corresponding position.
[0073]
[0074] Among them, Y B is a bilateral output, corresponding to the final result of bilateral voting. i represents eight different directions (also different channels). a and b have values of 0 or ±1 corresponding to different directions.
[0075] Channel aggregation converts the bilateral output into the final single-channel probability output to obtain the final prediction result. The max method is used for aggregation of 8 channels to extract the most likely class of the target pixel.
[0076]
[0077] Step 4: Input the training set data into the constructed connectivity-aware gating fusion network for training, and adjust the parameters on the validation set until the optimal model and its corresponding parameters are obtained. The loss function during the training process is calculated as follows.
[0078] L = L MainBCE + 0.2 * L BBCE + 0.8 * L CBCE
[0079] L MainBCE = L BCE (Y, M) = Y * M + (1 - Y) * (1 - M)
[0080] L BBCE = L BCE (Y B , M C ) = Y B * M C + (1 - Y B ) * (1 - M C )
[0081] L CBCE = L BCE (Y C , M C ) = Y C * M C + (1 - Y C ) * (1 - M C )
[0082] Among them, L is the final loss output, and Y, Y B , Y C , M C , Y C have been described accordingly in the previous text.
[0083] Step 5: Input the preprocessed medical image into the trained connectivity-aware gating fusion network to obtain the segmentation result.
[0084] Verification and comparison experiments:
[0085] The present invention proposes a connectivity-based medical image prediction algorithm - CAGFNet, which significantly improves the accuracy of medical image segmentation results. Compared with the existing technologies, the present invention has the following beneficial effects.
[0086] (1) Improving the accuracy of segmentation results: By introducing connectivity and enhancing the connectivity significantly and performing feature fusion in the algorithm, the present invention can additionally enhance the image features from the perspective of connectivity, overcoming the limitation of insufficient utilization of connectivity information in the existing technologies. The experimental results show that the algorithm of the present invention shows significant performance improvement compared with the existing technologies on different datasets and evaluation metrics as follows.
[0087]
[0088] (2) While ensuring the internal topological consistency of the edge results, achieving edge lubrication. The present invention reduces the problems of internal topological loss and edge roughness caused by the lack of connectivity features through the introduced connectivity enhancement related modules. As shown in the figure, compared with other algorithms, the segmentation results of the algorithm proposed by the present invention have better internal topological integrity and edge lubrication than the prediction results of other algorithms, as Figures 7 - 9 shown.
[0089] Embodiment:
[0090] Step 1: Collect the dataset and divide the dataset into a training set, a validation set, and a test set according to a ratio of 6:2:2.
[0091] Step 2: Uniformly adjust the image resolution to a size of 224*224, and process the mask, setting the background area to 0 and the target area to 1. For the training set, set the transform to perform image flipping and elastic deformation operations with a certain probability.
[0092] Step 3: Set up the connectivity-aware gating fusion network
[0093] The network structure is as shown in the figure (already used in the previous section). Set the pre-trained resnet34 as the backbone network, and the corresponding number of channels for E1 - E5 is 64, 64, 128, 256, 512, and the sizes are 112*112, 56*56, 28*28, 14*14, 7*7.
[0094] Correspondingly, 5 different CAAM modules are set up to receive image inputs with 64, 64, 128, 256, and 512 different channels. The channel attention receives inputs and outputs with 8, 8, 16, 32, and 64 channels. The number of channels for the two 1*1 convolutions of the multi-layer perceptron is set to (X, X / 2) and (X / 2, X) in sequence. The spatial attention receives inputs and outputs with 64, 64, 128, 256, and 512 channels. The number of channels for the convolution operation is set to (2,1) and a 7*7 convolution kernel is used.
[0095] Correspondingly, 5 SDConv operations are set up. The number of convolution kernels is set to 4, and the convolution kernel size is set to 3*3. The input and output channel numbers for different operations are (64, 64), (64, 64), (128, 64), (256, 128), and (512, 256) in sequence.
[0096] Correspondingly, 4 GFFM modules are set up. The input channels, intermediate channels, and output channels are set to (64, 64, 32), (64, 64, 32), (128, 128, 64), and (256, 256, 128) respectively, and the convolution kernel size is set to 3*3.
[0097] Set up the PConv operation to receive an input of 64 channels and output a connectivity output of 8 * the number of classification categories channels.
[0098] Step 4: Input the training set and the validation set into the connectivity-aware gating fusion network for training to obtain the best results.
[0099] Step 5: Input the test set into the best model to obtain the final segmentation result.
[0100] The results are as follows Figure 10 , which are the images, ground truths, and prediction results of the present invention for the BUSI, CVC-ClinicDB, and Kvasir-SEG datasets in sequence.
[0101] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The said program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above various methods. Among them, the said storage medium can be a magnetic disk, an optical disc, a read-only memory (ROM), or a random access memory (RAM), etc.
[0102] It should be understood that the detailed description of the technical solutions of the present invention by means of the preferred embodiments above is illustrative rather than restrictive. Those of ordinary skill in the art can modify the technical solutions described in each embodiment on the basis of reading the specification of the present invention, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.
Claims
1. A medical image segmentation method based on connectivity, characterized in that: The method described above includes: Step 1: Collect a medical image dataset and perform preprocessing, and divide it into a training set and a validation set; Step 2: Perform data augmentation on the training set and the validation set. The data augmentation includes random rotation, elastic deformation, and color adjustment, and generate an eight-channel connectivity mask for the mask of the training set. The connectivity mask reflects the class consistency between pixels and their adjacent pixels in eight directions; Step 3: Construct a connectivity-aware gated fusion network, including: A ResNet34 backbone network for extracting multi-layer feature maps of the input image; A connectivity-aware enhancement module CAAM that divides the feature maps output by the backbone network into eight groups and performs directional shifting, and adjusts the feature weights through channel attention and spatial attention; Dynamic shift convolution (SDConv), which optimizes feature extraction through parallel convolution and shifting strategies in the decoding stage; Step 4: Input the training set data into the constructed connectivity-aware gated fusion network for training; Step 5: Input the preprocessed medical image into the trained connectivity-aware gated fusion network to obtain a segmentation result.
2. The method for segmenting medical images based on connectivity according to claim 1, wherein: The method for generating the connectivity mask in Step 2 includes: For the original mask of the training set, calculate the pixel connectivity in eight directions respectively, and each direction corresponds to one channel; For each direction channel, if the current pixel and its adjacent pixel in this direction belong to the same category, mark the corresponding position in this channel with the corresponding category label (such as 1 if both are 1), otherwise mark it as 0; Concatenate the marking results of the eight direction channels into an eight-channel connectivity mask.
3. A method for medical image segmentation based on connectivity according to claim 1, characterized in that: The execution steps of the connectivity-aware enhancement module include: Evenly divide the input feature map into eight groups according to channels, and each group performs single-pixel shifting along a corresponding direction; For each shifted group of features, sequentially apply channel attention and spatial attention. Among them, channel attention generates channel weights through global feature compression and fully connected layers, and spatial attention generates spatial weights through convolution; Concatenate the eight groups of weighted features and add them to the original features, and output the fused features.
4. A method for medical image segmentation based on connectivity according to claim 1, characterized in that: The method for generating the channel attention includes: Perform global average pooling on the input feature map to obtain a channel description vector; Input the vector into two layers of fully connected layers. The number of neurons in the first layer is one-eighth of the number of input channels, and the second layer restores to the original number of channels; Generate a channel weight matrix through an activation function and multiply it with the input features channel by channel.
5. A method for medical image segmentation based on connectivity according to claim 1, characterized in that: The implementation of the dynamic shift convolution (SDConv) includes: Simultaneously perform the following three operations on the input feature map: standard convolution, convolution after shifting, and convolution with dynamic weight adjustment; Weightedly sum the output feature maps of the three operations according to a preset ratio to obtain the final features.
6. The method for segmenting medical images based on connectivity according to claim 1, characterized in that: The loss function consists of three parts: Dice loss based on the overlap degree between the prediction result and the true mask; Cross-entropy loss based on the class probability distribution; Connectivity loss based on the prediction error of the eight-direction connectivity mask; The final loss is the weighted sum of the three, where the weight of the Dice loss is 1, the weight of the cross-entropy loss is 0.5, and the weight of the connectivity loss is 0.
3.
7. A method for medical image segmentation based on connectivity according to claim 1, wherein: In the decoding stage, multi-scale features are fused through a gated fusion module, specifically including: Upsample the high-level features and concatenate them with the low-level features, and generate a gating weight map through 1×1 convolution; Use the gating weight map to perform weighted screening on the low-level features to suppress the noise regions; Input the weighted features into the dynamic shift convolution to complete the fusion.
8. A method for medical image segmentation based on connectivity according to claim 1, characterized in that: The post-processing method of the segmentation result includes: Count the number of consistent directions of each pixel in the eight-direction connectivity prediction. If more than half of the directions are determined to be connected, keep the segmentation result of the pixel; Perform morphological closing operation filtering on the isolated regions in the segmentation result to ensure the topological continuity of the target region.