A medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism

Abstract

This invention discloses a medical image segmentation method based on dynamic deformable convolution and a sliding window adaptive complementary attention mechanism, which improves the perception of small lesions and highly deformed targets in medical images, as well as the ability to distinguish between segmented targets and the background. Dynamic deformable convolution, through task-adaptive learning, can flexibly change weight coefficients and deformation biases, enhancing the ability to express local image features and achieving adaptive extraction of spatial features. The sliding window adaptive complementary attention mechanism achieves cross-dimensional global modeling of medical images through a self-attention branch of adaptively learned weight coefficients. This mechanism overcomes the shortcomings of conventional methods in modeling cross-dimensional relationships between space and channels, and can capture long-distance cross-dimensional correlation features in images. Furthermore, the parallel interactive approach combines local and global features at different resolutions to enhance representation learning, maximizing the preservation of both local and global features in medical images.

Description

Technical Field

[0001] This invention belongs to the fields of image processing technology and pattern recognition, specifically relating to a medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism. Background Technology

[0002] Medical image segmentation plays a crucial role in the field of medical image processing and is one of the core technologies of computer-aided diagnosis and treatment systems. Traditional methods rely on experienced physicians manually annotating and segmenting large amounts of medical image data, which is not only time-consuming and labor-intensive but also susceptible to subjective influences. In recent years, with the rapid development of artificial intelligence and computer technology, researchers have developed many new automatic medical image segmentation algorithms based on extensive experimentation. Existing medical image segmentation methods are mainly based on deep learning, and these methods can be broadly categorized into two types: those based on CNNs and those based on Transformer networks.

[0003] Deep learning-based algorithms can learn high-dimensional feature information of medical images through multi-layer network structures. Among various deep learning networks related to medical image segmentation, Convolutional Neural Networks (CNNs) have performed exceptionally well. CNNs are capable of effectively learning discriminative features and extracting prior knowledge from large-scale medical datasets, making them an important component of modern intelligent medical image analysis systems. In 2015, inspired by FCN networks, Ronnebreger et al. designed the first end-to-end network for medical image segmentation, U-Net, in the ISBI cell tracking challenge. U-Net is a symmetric encoder-decoder structure, and its unique design can fully utilize the local detail information of medical images, reducing the network's dependence on the training dataset. Therefore, even with small datasets, U-Net can still achieve good medical image segmentation results. Alom et al. designed R2U-Net by combining U-Net, ResNet, and Recurrent Neural Networks (RCNN), which achieved excellent performance on multiple medical image segmentation datasets, including those for blood vessels and the retina. Subsequently, Gu et al. introduced dynamic convolution into U-Net, proposing CA-Net. Experiments on medical datasets have demonstrated that CA-Net not only improves the segmentation accuracy of medical images but also reduces model training time. Building upon U-Net, Yang et al., drawing on the ideas of residual connections and deformable convolutions, added a residual deformable convolution to U-Net, proposing DCU-Net. DCU-Net demonstrated more advanced segmentation results than U-Net on the DRIVE medical dataset. Lei et al. designed SGU-Net based on U-Net; their proposed ultra-lightweight convolutional module and additional adversarial shape-constraint significantly improve the segmentation accuracy of abdominal medical images through self-supervised training. While CNNs have made significant progress in network architecture, their success is mainly attributed to invariance across different scales and inductive bias in local modeling. This fixed receptive field, while improving the computational efficiency of CNNs, limits their ability to capture relationships between distant pixels in medical images and lacks the ability to model long-range medical images.

[0004] In 2017, Vaswani et al. proposed the first Transformer network. Its unique design allows for variable-length inputs, long-range dependency modeling, and the capture of global information. The success of the Transformer is primarily attributed to its self-attention mechanism (SA), which captures long-range dependencies. Following the excellent performance of the Transformer in NLP tasks, ViT applied it to image processing for the first time, using multiple cascaded Transformer layers to capture the global contextual information of the input image, leading to significant success in image classification. Subsequently, Chen et al. proposed TransUNet, which ushered in a new era for the application of Transformers in medical image segmentation. However, TransUNet directly applied the Transformer model from NLP to image segmentation, resulting in fixed-size input image patches and high computational cost. Valanarasu et al. addressed the shortcomings of TransUNet by proposing MedT, which incorporates a gating mechanism. This mechanism allows the model to automatically adjust gating parameters when training on medical datasets of different sizes, obtaining location embedding weights suitable for the current dataset. Because images are more diverse and have higher resolution than text, Cao et al. proposed Swin-Unet, a pure Transformer network for medical image segmentation, combining shifted window multi-head self-attention (SW-MSA) from the SwinTransformer. Swin-Unet achieved state-of-the-art segmentation performance on the Synapse and ACDC multi-organ segmentation datasets at the time. To better utilize the Transformer for processing skin lesion image data, Wang et al. designed the BAT network by incorporating edge detection. The proposed boundary-wise attention gate (BAG) can fully utilize prior knowledge of image boundaries to capture more detailed information from medical images. This enabled BAT to achieve remarkable segmentation results on skin lesion datasets, surpassing many state-of-the-art medical image segmentation networks.

[0005] Despite the significant success these networks have achieved in medical image segmentation, they still suffer from two main problems: First, they neglect organ deformation and lesion irregularities when modeling local image features, resulting in weak representation of local features for organs with significant deformation and lesions. Second, they ignore the correlation between feature map space and channels when modeling global image features, leading to insufficient self-attention representation. Summary of the Invention

[0006] To address the problems in existing technologies, this invention provides a medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism. By using dynamic deformable convolution and sliding window adaptive complementary attention mechanism based on a fusion architecture of CNNs and Transformer, the method explores the geometric features, local detail features, and global semantic features of target organs from limited human medical annotation data. Furthermore, by closely combining local and global features at different resolutions through parallel interaction of CNNs and Transformer, the method enhances representation learning and maximizes the preservation of local and global features in medical images.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0008] 1) Load the medical image dataset and perform preprocessing;

[0009] 2) Construct and train a fusion network of CNNs and Transformers. The fusion network consists of parallel interaction of two-branch networks. Dynamically deformable convolution operations are performed within the CNN branch, while the sliding window adaptive complementary attention mechanism is performed within the Transformer branch. The dynamically deformable convolution can adaptively learn the kernel deformation offset and kernel weight coefficients according to specific medical image segmentation tasks and data distributions. At the same time, end-to-end training is achieved through backpropagation of the network, thereby realizing dual changes in kernel shape and weights. The sliding window adaptive complementary attention mechanism has a Transformer self-attention branch with four learnable parameters, which can extract global image information in space and capture long-distance cross-dimensional correlation features between space and channels.

[0010] 3) Output the segmentation prediction results of the CNNs branch and the Transformer branch respectively; perform fusion and discrimination on the segmentation prediction results, and output the final optimized segmentation result.

[0011] Furthermore, the preprocessing includes: uniformly adjusting the original image size of the dataset to 224×224; randomly scaling the images in the original dataset with a scaling ratio between 0.9 and 1.5; and randomly flipping the images in the original dataset vertically, horizontally, rotating them 90°, or rotating them 270° at a ratio of 0.5.

[0012] Furthermore, the shape change of the convolution kernel in the dynamically deformable convolution is based on the network's learning of deformation offset. The network first samples the input feature map X using a square convolution kernel S, and then performs a weighted summation with the weight matrix M; the 3×3 convolution kernel in the learning of deformation offset is represented as follows:

[0013] S={(0,0),(0,1),(0,2),…,(2,1),(2,2)}

[0014] At this point, output the coordinates of each position in the feature map Y. Represented as:

[0015]

[0016] When deformation offset is introduced into the weight matrix M Then, N is the total length of S, then:

[0017]

[0018] Through network learning, an offset matrix of the same size as the input feature map is finally obtained, with the matrix dimension being twice that of the input feature map.

[0019] Furthermore, the changes in the kernel weights in the dynamically deformable convolution are determined by introducing weight coefficients. For learning these weight coefficients, the feature map output of a conventional convolution is represented as follows:

[0020] y = σ(W × x)

[0021] Where σ is the activation function and W is the convolution kernel weight matrix, after introducing the weight coefficients, the output of the feature map after dynamic deformable convolution is:

[0022] y = σ*(α1×W1+…+α n ×W n )×x)

[0023] n is the number of weighting coefficients, α n These are weight coefficients with learnable parameters.

[0024] Furthermore, the sliding window adaptive complementary attention mechanism uses a sliding window calculation method to calculate self-attention only within a local window, while setting a compact convolutional projection, including: first, reducing the local size of the medical image through a sliding window operation, then compressing the channel dimension of the medical image through compact convolutional projection, and finally calculating self-attention.

[0025] Furthermore, among the four Transformer self-attention branches, two branches capture the correlation between channels and space, respectively, while the other two branches capture the correlation between channel dimension C and spatial dimension H, and between channel dimension C and spatial dimension W, respectively.

[0026] Furthermore, after employing the shifted window partitioning method in the sliding window adaptive complementary attention mechanism, the calculation process of consecutive Transformer blocks is as follows:

[0027]

[0028]

[0029]

[0030]

[0031] in and T l , respectively, represent the output features of sliding window adaptive complementary attention and compact convolutional projection, W-ACAM represents window adaptive complementary attention, SW-ACAM represents sliding window adaptive complementary attention, and LPM represents compact convolutional projection.

[0032] Furthermore, the attention calculation process for each Transformer self-attention branch is as follows:

[0033]

[0034] The relative position offset These are query, key, and value matrices, respectively; C / 8 represents the dimensions of the query and key; M 2 Represents the number of blocks;

[0035] After computation through four parallel Transformer self-attention branches Out1, Out2, Out3, and Out4, the final feature fusion output is as follows:

[0036] Out=λ1×Out1+λ2×Out2+λ3×Out3+λ4×Out4

[0037] λ1, λ2, λ3 and λ4 are learnable parameters used to adaptively control the importance of each attention branch for spatial and channel information in a specific segmentation task.

[0038] Furthermore, the CNNs and Transformer fusion network is configured with three loss functions, namely the overall loss L... CiT Loss L of CNN branches CNNs The loss L of the Transformer branch Trans :

[0039]

[0040]

[0041]

[0042] Among them, L MSE (·) represents the mean squared error loss, L Dice (·) represents Dice loss. and Representing the input image x respectively i The corresponding methods output the prediction map, the prediction map output by the CNNs branch, the prediction map output by the Transformer branch, and the label map.

[0043] Furthermore, the final loss function of the fusion network of CNNs and Transformer is expressed as:

[0044] L Total =λ×L CiT +((1-λ) / 2)L CNNs +((1-λ) / 2)L Trans

[0045] Where λ=δe -5(1-k)2 λ is a Gaussian rising curve, and k represents the total number of training rounds.

[0046] Compared with existing technologies, this invention proposes a novel dynamically deformable convolution. Through task-adaptive learning, dynamically deformable convolution can flexibly change its own weight coefficients and deformation biases. It also overcomes the problems of fixed receptive fields and shared kernel parameters in standard convolution and its variants (e.g., dilated convolution, involute convolution), while enhancing the ability to express local features of medical images, achieving adaptive extraction of spatial features, and improving the perception of minute lesions and highly deformed targets in medical images.

[0047] This invention proposes a novel sliding window adaptive complementary attention mechanism, which achieves cross-dimensional global modeling of medical images through four Transformer self-attention branches with adaptively learned weight coefficients. Compared to currently popular attention mechanisms (e.g., CBAM, Non-Local), the sliding window adaptive complementary attention mechanism effectively compensates for the shortcomings of conventional attention mechanisms in modeling cross-dimensional relationships between space and channels. It can capture long-distance cross-dimensional correlation features in medical images, enhancing the separability between segmented targets and backgrounds in medical images. By combining sliding window operations and compact convolutional projection, the sliding window adaptive complementary attention mechanism significantly reduces the overall number of parameters and computational cost compared to CNNs and the original Transformer method, while simultaneously capturing global features of medical images.

[0048] This invention presents a novel parallel method based on dynamically adaptive CNNs and a cross-dimensional feature fusion Transformer for medical image segmentation. Compared to currently popular medical image segmentation methods, the proposed method enhances representation learning by leveraging the parallel interaction of CNNs and Transformers to tightly integrate local and global features at different resolutions, maximizing the preservation of both local and global features in medical images. Notably, this method not only eliminates pre-training but also boasts significantly fewer parameters and computational cost, at 11.58M and 4.53 GFLOPs, respectively. Clearly, the proposed method, based on a CNNs+Transformer dual-branch architecture with dynamically deformable convolution and a sliding window adaptive complementary attention mechanism, provides an efficient and accurate solution for image segmentation tasks involving human liver and heart organs, demonstrating significant application value and broad clinical application prospects. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention;

[0050] Figure 2 This is a structural diagram of the present invention;

[0051] Figure 3 This is a schematic diagram of the dynamically deformable convolution structure of the present invention;

[0052] Figure 4 This is a schematic diagram of the sliding window adaptive complementary attention mechanism of the present invention;

[0053] Figure 5 This is a segmentation result diagram of an embodiment of the present invention on the publicly available liver dataset LiTS;

[0054] Figure 6This paper compares the segmentation performance of the proposed method with currently popular methods on the publicly available liver dataset LiTS. This means that the model was pre-trained using the ImageNet21K dataset to pre-train the network weight parameters;

[0055] Figure 7 This paper compares the segmentation performance of the proposed method with currently popular methods on the publicly available cardiac dataset ACDC. Detailed Implementation

[0056] The present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] This invention relates to theoretical knowledge of deep learning technology, convolutional neural network technology, Transformer network technology, and network adaptive learning technology. This invention can be effectively applied to the automatic segmentation of medical images such as human liver and heart organs, and is applicable to both CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) image data. It better solves the problems of existing segmentation networks' poor perception of small lesions and highly deformed targets in medical images, as well as their poor ability to distinguish between segmented targets and background. This invention provides new research ideas and technical means for the automatic segmentation of medical images such as human liver and heart.

[0058] In real-world medical image segmentation tasks, organs such as the liver, heart, kidneys, and spleen exhibit high complexity, with significant shape differences between them, and their close proximity and indistinguishable boundaries make them difficult to differentiate. Existing organ segmentation methods firstly neglect the ability to perceive minute lesions and objects with large deformations in medical images. Secondly, because medical images typically have low contrast, blurred edges, and limited labeled data, existing methods often lack the ability to learn the separability between the segmentation target and the background during training. This invention first addresses minute lesions and objects with large scale variations. How to segment such samples more efficiently and accurately is key to improving the overall segmentation accuracy and effect of the method, and is crucial to its overall application value. Minor lesions often contain more information about the patient's condition, enabling early detection and treatment, significantly shortening the recovery period. Simultaneously, objects with large scale variations represent the organ's importance within the body; therefore, it is important to pay attention to changes in its shape and condition, which are closely related to the patient's health. Secondly, regarding the separability between the segmentation target and the background, significantly improving the difference between the segmentation target and the background will significantly improve the efficiency and accuracy of human organ segmentation. Since medical data is usually labeled in small quantities, using a small amount of data to learn accurate information about organ morphology and location will significantly improve the efficiency and quality of disease diagnosis.

[0059] This invention proposes a method based on a CNNs+Transformer dual-branch architecture with dynamic deformable convolution and a sliding window adaptive complementary attention mechanism, aiming to perform faster, more accurate, automatic, and universal segmentation of two important human organs: the liver and the heart. Compared with existing methods, this invention first proposes a novel dynamic deformable convolution, which, through task-adaptive learning, can flexibly change its own weight coefficients and deformation bias. This enhances the ability to express local features of medical images, achieves adaptive extraction of spatial features, and improves the perception of small lesions and targets with large deformations in medical images. Second, a novel sliding window adaptive complementary attention mechanism is proposed, which achieves cross-dimensional global modeling of medical images through four Transformer self-attention branches with adaptively learned weight coefficients. It can capture long-distance cross-dimensional correlation features in medical images, enhancing the separability between segmented targets and backgrounds in medical images. The sliding window adaptive complementary attention mechanism, by combining sliding window operation and compact convolutional projection, significantly reduces the number of network parameters and computational cost, while simultaneously capturing global features of medical images. Finally, this invention enhances representation learning by leveraging the parallel interaction of CNNs and Transformers to tightly integrate local and global features at different resolutions, maximizing the preservation of both local and global features in medical images. This invention also requires fewer parameters and less computation. Clearly, this invention provides an efficient and accurate solution for image segmentation tasks involving human liver and heart organs, possessing significant application value and broad clinical application prospects.

[0060] See details Figure 1 The method flow of the present invention includes:

[0061] 1) Load the medical image dataset and perform preprocessing;

[0062] 2) Construct and train a fusion network of CNNs and Transformers;

[0063] 3) Output the segmentation prediction results of the CNNs branch and the Transformer branch respectively; perform fusion and discrimination on the segmentation prediction results, and output the final optimized segmentation result.

[0064] See Figure 2The CNNs and Transformer fusion network architecture of this invention consists of a parallel interaction of two-branch networks. Dynamically deformable convolution operations are performed within the CNNs branch, while the sliding window adaptive complementary attention mechanism is performed within the Transformer branch. The dynamically deformable convolution can adaptively learn the kernel deformation offset and kernel weight coefficients according to specific medical image segmentation tasks and data distributions. Simultaneously, end-to-end training is achieved through backpropagation of the network, thus realizing dual changes in kernel shape and weights. The sliding window adaptive complementary attention mechanism has a Transformer self-attention branch with four learnable parameters, capable of extracting global image information in space and capturing long-distance cross-dimensional correlation features between space and channels. This invention's method, based on a CNNs+Transformer dual-branch architecture with dynamically deformable convolution and a sliding window adaptive complementary attention mechanism, consists of a dynamically adaptive convolutional neural network and a cross-dimensional feature fusion Transformer dual-branch interaction. The dynamically deformable convolution in the CNNs branch can adaptively change the weight coefficients and deformation offset of the convolution itself, thereby improving the segmentation accuracy of irregular targets in medical images. The sliding window adaptive complementary attention in the Transformer branch can capture long-range dependencies across dimensions in medical images, improving the separability of segmented objects and backgrounds. Compact convolutional projection effectively reduces the parameters and computational cost of the original Transformer network by using the Ghost strategy.

[0065] The present invention will be illustrated below using a liver and heart dataset as an example. It addresses the problems of existing human liver and heart segmentation methods neglecting the perception of minute lesions and targets with large scale variations, and lacking the ability to learn the separability between the segmentation target and the background during training. Specifically, it includes:

[0066] (1) Liver and Heart Dataset Sources and Preprocessing: The liver dataset comes from the Liver and Tumor Segmentation Dataset (LiTS) of the Society for Medical Image Computing and Computer-Aided Diagnosis, which contains 131 3D CT liver scans. 100 scans were used for training in this invention, and the remaining 31 scans were used for testing. The heart dataset comes from the Automated Cardiac Diagnosis Dataset (ACDC) of Dijon University Hospital (France), which contains short-axis MRI data from 100 patients. This dataset includes healthy patients, patients with a history of myocardial infarction, dilated cardiomyopathy, hypertrophic cardiomyopathy, and right ventricular abnormalities, with 20 scans in each group. These data were acquired over 6 years using MRI scanners with two magnetic intensities (1.5T and 3.0T). To improve the computational efficiency, generalization ability, and robustness of the proposed network, the following processing operations were performed on the dataset: ① The original image size of the dataset was uniformly adjusted to 224×224; ② The images in the original dataset were randomly scaled between 0.9 and 1.5; ③ The images in the original dataset were randomly flipped vertically, horizontally, rotated 90°, and rotated 270° at a ratio of 0.5. These operations only apply to the images in the training dataset, performing real-time random augmentation during model training to enhance the model's generalization performance; no augmentation was performed on the images in the test dataset.

[0067] (2) Training process of the dynamically deformable convolution method proposed in this invention: The dynamically deformable convolution adaptively learns the kernel deformation offset and kernel weight coefficients according to specific medical image segmentation tasks and data distributions. Simultaneously, end-to-end training is achieved through backpropagation of the network, thus realizing dual changes in kernel shape and weights. Therefore, it can effectively address the problems of large differences in data distribution and large target deformation among different targets in medical image segmentation tasks. Furthermore, the dynamically deformable convolution is a plug-and-play module that can be embedded in any network structure. See also Figure 3 Compared to currently popular convolutional strategies, dynamically deformable convolution can dynamically adjust the weight coefficients and deformation biases of the convolution itself during the network training process, which is beneficial for feature capture and extraction of irregular targets in medical images. α and β represent the different weight coefficients of the dynamically deformable convolution under different states.

[0068] (a) In dynamically deformable convolutions, the shape change of the convolution kernel is based on the network's learning of deformation offsets. The segmentation network first samples the input feature map X using a square convolution kernel S, and then performs a weighted summation with the weight matrix M. The square convolution kernel S determines the range of the receptive field; for example, a 3×3 convolution kernel can be represented as:

[0069] S={(0,0),(0,1),(0,2),…,(2,1),(2,2)}

[0070] (b) At this point, output the coordinates of each position in the feature map Y. It can be represented as:

[0071]

[0072] (c) When a deformation offset is introduced into the weight matrix M Then, N is the total length of S, and the above formula becomes:

[0073]

[0074] (d) Through network learning, an offset matrix of the same size as the input feature map can be obtained, with the matrix dimension being twice that of the input feature map.

[0075] (e) In dynamically deformable convolutions, the kernel weights are varied by introducing weight coefficients. The feature map output of a regular convolution can be represented as:

[0076] y = σ(W × x)

[0077] Where σ is the activation function and W is the convolution kernel weight matrix.

[0078] (f) After introducing weight coefficients, the output of the feature map after dynamic deformable convolution is:

[0079] y=σ((α1×W1+…+α n ×W n )×x)

[0080] n is the number of weighting coefficients, α n These are weight coefficients with learnable parameters. This method achieves dynamic adjustment of the convolution kernel weights by linearly combining different weight matrices according to their corresponding weight coefficients before entering the convolution operation.

[0081] (3) Training process of the sliding window adaptive complementary attention mechanism proposed in this invention: The sliding window adaptive complementary attention mechanism has four learnable Transformer self-attention branches, which can not only fully extract global image information in space, but also fully capture long-distance cross-dimensional correlation features between space and channels. The four Transformer self-attention branches complement each other, maximizing the preservation of the information integrity of the segmentation target, enhancing the separability between foreground and background in medical images, and improving the segmentation accuracy of the target. See also Figure 4Unlike traditional self-attention mechanisms, the sliding window adaptive complementary attention mechanism not only possesses the advantages of spatial and channel attention mechanisms but also captures long-distance correlation features across spatial and channel dimensions. Through the sliding window operation, the spatial size of the image is significantly reduced, and through the compact convolutional projection operation, the channel dimension of the image is significantly reduced, thereby lowering the overall computational cost and complexity of the method. λ1, λ2, λ3, and λ4 are learnable weight parameters.

[0082] (a) The standard Transformer architecture employs global self-attention computation, calculating the relationship between one block and all other blocks. This computation is complex, and the computational cost typically grows exponentially. To improve computational efficiency, this method uses a sliding window approach, calculating self-attention only within a local window. Furthermore, this method employs compact convolutional projection. First, the sliding window operation significantly reduces the local size of the medical image. Then, the compact convolutional projection compresses the channel dimension of the medical image, and finally, self-attention is calculated. It is worth noting that this approach not only better captures the global high-dimensional information of the medical image but also significantly reduces the computational cost of the module. Assuming an image contains h×w windows, and each window contains M×M patches, the complexity comparison of the sliding window adaptive complementary attention mechanism, the global MSA in the original Transformer, and the (S)W-MSA in the Swing Transformer is as follows:

[0083] Ω(MSA)=4hwC 2 +2(hw) 2 C

[0084] Ω((S)WMSA)=4hwC 2 +2M 2 hwC

[0085]

[0086] If the first term is a quadratic function of the number of patches hw, and the second term is linear when M is fixed (usually 7), then the computational cost of the sliding window adaptive complementary attention mechanism is negligible compared to MSA and (S)W-MSA.

[0087] (b) In the four parallel Transformer self-attention branches of the sliding window adaptive complementary attention mechanism, two branches capture the correlation between channels and space, respectively, while the other two branches capture the correlation between channel dimension C and spatial dimension H, and between channel dimension C and spatial dimension W. After adopting the shift window partitioning method, the computation process of consecutive Transformer blocks is as follows:

[0088]

[0089]

[0090]

[0091]

[0092] in and T l , respectively, represent the output features of sliding window adaptive complementary attention and compact convolutional projection, W-ACAM represents window adaptive complementary attention, SW-ACAM represents sliding window adaptive complementary attention, and LPM represents compact convolutional projection.

[0093] (c) The attention calculation process for each branch is as follows:

[0094]

[0095] The relative position offset These are query, key, and value matrices. C / 8 represents the dimensions of the query and key, and M... 2 Represents the number of blocks.

[0096] (d) After computation through four parallel Transformer self-attention branches Out1, Out2, Out3, and Out4, the final feature fusion output is as follows:

[0097] Out=λ1×Out1+λ2×Out2+λ3×Out3+λ4×Out4

[0098] λ1, λ2, λ3, and λ4 are learnable parameters used to adaptively control the importance of each attention branch for spatial and channel information in a specific segmentation task.

[0099] (4) Setting of loss functions in this invention: A total of three loss functions are set in this invention, namely the overall loss L CiT The loss L of CNN branches CnN The loss L of the Transformer branch Trans .

[0100]

[0101]

[0102]

[0103] Among them, L MSE (·) represents the mean squared error loss, L Dice(·) represents Dice loss. and Representing the input image x respectively i The corresponding methods output the final prediction map, the prediction map output by the CNN branch, the prediction map output by the Transformer branch, and the label map. The final loss function of this method can be expressed as:

[0104] L Total =λ×L CiT +((1-λ) / 2)L CNNs +((1-λ) / 2)L Trans

[0105] Where λ=δe -5(1-k)2 λ is a Gaussian rising curve, and k represents the total number of training rounds.

[0106] The final result is as follows Figure 5 The images shown are segmentation results of the present invention on the publicly available liver dataset LiTS. (a) is the predicted image output by the CNN branch, (b) is the predicted image output by the Transformer branch, (c) is the final predicted image output by the present invention, (d) is the label image, and (e) is the original image of the dataset.

[0107] The effects of this invention can be further illustrated by the following experiments.

[0108] To verify the accuracy and superiority of the proposed method for segmenting human liver and heart, the hardware experimental platform consisted of an Intel Core i9-9900X 3.5GHz CPU, an NVIDIA GeForce RTX 3090Ti GPU with 24GB of VRAM; the software experimental platform used open-source software and frameworks such as PyTorch, Python, OpenCV, and NumPy. This invention employs a simple Stochastic Gradient Descent (SGD) optimization algorithm to train the model, with momentum set to 0.9, weight_decay set to 0.001, and an initial learning rate of 0.0001. The learning rate is adjusted exponentially every 2500 iterations, with a decay rate of 0.1. The algorithm performance is evaluated by calculating the following metrics:

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115] Where DI represents set similarity, TP, FP, and FN represent true positives, false positives, and false negatives, respectively, ASD represents the average symmetric surface distance, and HD represents the similarity between two sets of points. A and B represent the label map and the prediction result, respectively. S(A) and S(B) represent a set of surface voxels corresponding to A and B, respectively.

[0116] To demonstrate the accuracy and computational efficiency of this invention for liver and heart segmentation, a fair and comprehensive experimental comparison was conducted with 11 currently mainstream methods. These comparison methods include U-Net, R2UNet, AttentionUNet, CENet, 3D UNet, V-Net, Swin-UNet, TransUNet, CvT, PVT, and CrossForm. The same experimental environment and settings were used throughout the experiments. Table 1 compares the segmentation performance of this invention's method with currently popular methods on the publicly available liver dataset LiTS, showcasing a quantitative comparative analysis of the results. Experimental results demonstrate that this invention's method has significant advantages in liver image segmentation, further validating the complete preservation of local and global features in medical images. Notably, with the fewest model parameters and computational cost, this invention's method achieves a DI (Discretionary Interpretation) metric that is 1.20%, 1.03%, and 1.01% higher than pre-trained Swin-UNet, TransUNet, and CvT networks, respectively.

[0117] In terms of other evaluation metrics, this method also significantly outperforms other comparative methods. Table 2 presents the quantitative analysis of the results of this method and current mainstream methods on cardiac datasets. The experimental results show that this method still exhibits significant advantages in MRI-type cardiac segmentation datasets. The method proposed in this invention demonstrates state-of-the-art segmentation results for the left ventricle (LV), right ventricle (RV), and left ventricular myocardium (MYO). The segmentation effect for LV is the best, while the segmentation effect for MYO is relatively poor. Compared with the latest CvT, PVT, and CrossForm methods, the average segmentation performance of this method is improved by 0.98%, 1.45%, and 1.38%, respectively, while the 95HD performance is reduced by an average of 0.72%, 0.78%, and 0.74%, respectively. This also demonstrates that this method has strong generalization performance on different datasets and can be flexibly applied to liver and cardiac image segmentation tasks in different modalities and acquisition environments. Extensive experiments on the above two types of datasets fully demonstrate the practicality and effectiveness of the two modules proposed in this invention.

[0118] This invention proposes a novel method for segmenting human liver and heart images using a fusion architecture of CNNs and Transformers, incorporating dynamic deformable convolution and a sliding window adaptive complementary attention mechanism. This method integrates the unique advantages of both CNNs and Transformers, maximizing the preservation of local detail features and global semantic features in medical images through local relation modeling and long-range dependency modeling. The proposed dynamic deformable convolution strategy overcomes the limitations of conventional convolutions, such as fixed receptive fields and poor generalization ability, enhancing the expressive power of local features in medical images and achieving adaptive extraction of spatial features. The proposed sliding window adaptive complementary attention mechanism effectively captures the cross-dimensional correlation between medical image channels and space, adaptively learning the relevant information between space and channels. Furthermore, this method replaces the traditional MLP in Transformers with a compact convolutional projection lightweight strategy, significantly reducing the number of parameters in traditional CNNs and Transformers networks, eliminating the network's dependence on pre-training, and addressing the problems of limited labeled data and network overfitting in medical images. Compared to current mainstream CNNs and Transformers for medical image segmentation, this invention demonstrates significant advantages in both operational efficiency and segmentation results. This provides a general and efficient solution for image segmentation of the human liver and heart, with broad application prospects in smart medical systems and actual clinical analysis.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism, characterized in that, include: 1) Load the medical image dataset and perform preprocessing; 2) Construct and train a fusion network of CNNs and Transformers. The fusion network of CNNs and Transformers consists of parallel interaction of two-branch networks. The dynamic deformable convolution operation is in the CNNs branch, and the sliding window adaptive complementary attention mechanism is in the Transformer branch. Dynamically deformable convolution can adaptively learn the kernel deformation offset and kernel weight coefficients according to specific medical image segmentation tasks and data distributions. At the same time, it achieves end-to-end training through backpropagation of the network, thereby realizing dual changes in kernel shape and weights. The sliding window adaptive complementary attention mechanism has a Transformer self-attention branch with four learnable parameters, which can extract global image information in space and capture long-distance cross-dimensional correlation features between space and channels. The shape change of the convolution kernel in the dynamically deformable convolution is based on the network's learning of deformation offset. The network first uses a square convolution kernel. For the input feature map Sampling is performed, followed by comparison with the weight matrix. Perform weighted summation; in the learning of deformation offset. The convolution kernel is represented as: At this point, the feature map is output. Each position coordinate in Represented as: When in the weight matrix Introducing deformation offset back, for The total length is then: Through network learning, an offset matrix of the same size as the input feature map is finally obtained, but the matrix dimension is twice that of the input feature map. In the dynamically deformable convolution, the kernel weights are changed by introducing weight coefficients. For learning these weight coefficients, the feature map output of a regular convolution is represented as follows: in For activation function, Given the convolution kernel weight matrix, after introducing the weight coefficients, the output of the feature map after dynamic deformable convolution is: The number of weighting coefficients. These are weight coefficients with learnable parameters; Of the four Transformer self-attention branches, two branches capture the correlation between channels and space, respectively, while the other two branches capture the channel dimension. With spatial dimension and channel dimensions With spatial dimension The correlation between them; The calculation process of consecutive Transformer blocks in the sliding window adaptive complementary attention mechanism after adopting the shifted window partitioning method is as follows: in and , representing the output features of sliding window adaptive complementary attention and compact convolutional projection, respectively. W-ACAM represents window adaptive complementary attention, SW-ACAM represents sliding window adaptive complementary attention, and LPM represents compact convolutional projection. 3) Output the segmentation prediction results of the CNNs branch and the Transformer branch respectively; perform fusion and discrimination on the segmentation prediction results, and output the final optimized segmentation result.

2. The medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism according to claim 1, characterized in that, The preprocessing includes: uniformly adjusting the original image size of the dataset to 224×224; randomly scaling the images in the original dataset with a scaling ratio between 0.9 and 1.5; and randomly flipping the images in the original dataset vertically, horizontally, rotating them 90°, or rotating them 270° at a ratio of 0.

5.

3. The medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism according to claim 1, characterized in that, The sliding window adaptive complementary attention mechanism uses a sliding window calculation method to calculate self-attention only within a local window, while setting a compact convolutional projection, including: first, reducing the local size of the medical image through a sliding window operation, then compressing the channel dimension of the medical image through compact convolutional projection, and finally calculating self-attention.

4. The medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism according to claim 1, characterized in that, The attention calculation process for each Transformer self-attention branch is as follows: The relative position offset ; They are query , key and value matrix; Dimensions representing query and key; Represents the number of blocks; After four parallel Transformer self-attention branches , , and After calculation, the final feature fusion output is as follows: , , and These are learnable parameters used to adaptively control the importance of each attention branch for spatial and channel information in a specific segmentation task.

5. The medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism according to claim 1, characterized in that, The CNNs and Transformer fusion network uses three loss functions, namely the overall loss. Loss of CNN branches Loss of Transformer branches : in, Represents the mean squared error loss. Represents Dice's loss. , , and Representing the input image respectively The corresponding methods output the prediction map, the prediction map output by the CNNs branch, the prediction map output by the Transformer branch, and the label map.

6. The medical image segmentation method based on dynamic deformable convolution and sliding window adaptive complementary attention mechanism according to claim 5, characterized in that, The final loss function of the CNNs and Transformer fusion network is expressed as: in, , It is a Gaussian rising curve. This represents the total number of training rounds.

Images