Dual-encoder collaborative fusion medical image segmentation method

Abstract

The application provides a double-encoder collaborative fusion medical image segmentation method, which sets a cross-branch fusion module between corresponding levels of a convolutional encoding branch and a Transformer encoding branch, aligns, interacts and selectively fuses two-way heterogeneous features; meanwhile, a context state vector is generated according to cross-branch fusion features and double-branch interaction features output by a current level, and the context state vector of the previous level is transmitted to the next level; in addition, the expression of a spatial guidance map is further enhanced through an HCCA module from aspects of channel attention, coordinate attention and local detail perception enhancement. In summary, the medical image segmentation method provided by the application effectively solves the problems of insufficient fusion of local detail features and global semantic features, limited interaction ability of heterogeneous encoding features and lack of effective semantic guidance between different level fusion processes in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a dual encoder collaborative fusion medical image segmentation method. Background Technology

[0002] Medical image segmentation is an important technique in medical image processing and computer-aided diagnosis, playing a crucial role in skin lesion identification, colon polyp detection, and other lesion-assisted diagnosis. However, accurate segmentation of lesion regions in medical images remains challenging due to their characteristics, including significant scale variations, irregular shapes, blurred boundaries, low contrast between the target area and surrounding tissues, and strong background interference.

[0003] With the development of deep learning technology, encoder-decoder structures, represented by U-Net, have been widely used in medical image segmentation. U-Net uses skip connections to pass shallow spatial detail features extracted by the encoder to the decoder, and then fuses them with deep semantic features to gradually restore the spatial resolution of the segmentation result. Building on this, UNet++ reduces the semantic difference between the encoder and decoder through nested dense connections and combines deep supervision to improve image segmentation performance; ResUNet introduces residual structures into the U-Net network to improve feature transfer and gradient propagation in deep networks; AttentionU-Net introduces an attention mechanism in skip connections to enhance the response of the target region and suppress irrelevant background information.

[0004] However, convolutional neural networks typically use fixed-size convolutional kernels for local computation, and their effective receptive field is limited by the network structure, making it difficult to fully model the dependencies between distant regions in an image. Furthermore, the continuous downsampling process in the encoder can easily lead to the gradual loss of edge contours, texture details, and small object information. This makes existing convolutional neural networks prone to problems such as missing target regions, broken boundaries, incomplete contours, and missegmentation of the background when processing small lesions, elongated structures, and blurred boundary regions.

[0005] To address the insufficient global information modeling capabilities of convolutional neural networks, existing technologies have begun to apply Transformers to medical image segmentation. Among these, TransUNet introduces Transformers into the encoding stage to extract global contextual information from images and combines it with a U-shaped decoding structure to recover spatial details; Swin Transformer employs a window-based multi-head self-attention and shift window mechanism to construct hierarchical feature representations while controlling computational complexity; and Swin-Unet further constructs a Transformer-based U-shaped encoder-decoder network.

[0006] Transformers can establish long-distance dependencies between different regions of an image using self-attention mechanisms, giving them an advantage in global semantic modeling. However, lesion boundaries in medical images are typically determined by fine-grained texture, local gray-level variations, and weak contrast. Pure Transformer networks or networks using image patches and feature labels as basic processing units may still have shortcomings in preserving local texture, extracting fine-grained structures, and characterizing high-frequency edges. Furthermore, existing Transformer-based medical image segmentation methods still have room for improvement in the collaborative modeling of global semantic information and local texture information, effective feature selection in complex backgrounds, and restoration of blurred boundaries.

[0007] To simultaneously leverage the local feature extraction capabilities of convolutional neural networks (CNNs) and the global context modeling capabilities of Transformers, existing techniques have further proposed hybrid segmentation networks that combine CNNs and Transformers. These methods typically extract local texture and edge detail features through convolutional branches and extract global semantic and long-range dependency features through Transformer branches, then fuse these two types of features to enhance the network's ability to represent complex lesion regions.

[0008] However, while hybrid segmentation networks combining CNN and Transformer can take into account both local texture features and global semantic information, the two types of features differ in distribution, channel semantics, and spatial response. Some methods mainly use feature concatenation, element-wise addition, or fixed weight fusion, which makes it difficult to achieve full interaction and adaptive selection of heterogeneous features. At the same time, there is a lack of effective contextual transmission and feedback guidance between different layers, and the ability to collaboratively model multi-scale features is insufficient. As a result, the network is still prone to problems such as incomplete feature representation, inaccurate target localization, missed segmentation, or over-segmentation under conditions of small targets, blurred boundaries, and complex backgrounds. Summary of the Invention

[0009] To address the shortcomings and deficiencies of existing technologies, a dual-encoder collaborative fusion medical image segmentation method is provided. This method can solve the problems of insufficient fusion of local detail features and global semantic features, limited interaction capabilities of heterogeneous coding features, and lack of effective semantic guidance between different levels of fusion processes in existing medical image segmentation methods.

[0010] A dual-encoder collaborative fusion medical image segmentation method provided to achieve the purpose of this invention includes the following steps:

[0011] S1: Input the preprocessed image into both the CNN encoding branch and the Transformer encoding branch simultaneously, and select four corresponding CNN branch features from each of the two encoding branches. Transformer branch features Where n = 1, 2, 3, 4;

[0012] S2: Input the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer into the first fusion module for processing to obtain the first final fusion output F1 and the first context feedback vector G1;

[0013] S3: The CNN branch features C corresponding to the i-th level. i Transformer branch feature T i and the (i-1)th context feedback vector G i-1 The i-th input is processed by the synchronous fusion module to obtain the i-th final fusion output F. i and the i-th context feedback vector G i Where i = 2, 3, 4; where the resolution of the first final fused output F1 > the resolution of the second final fused output F2 > the resolution of the third final fused output F3 > the resolution of the fourth final fused output F4;

[0014] S4: Input the fourth final fusion output F4 into the fourth decoder for double upsampling to obtain the fourth decoding feature; where the resolution of the fourth decoding feature is equal to the resolution of the third final fusion output F3;

[0015] S5: Fuse the i-th decoded feature and the (i-1)-th final output F i-1 The (i-1)th decoder is synchronously input and concatenated along the channel dimension. Channel compression and feature reconstruction are then performed through convolution operations to obtain the (i-1)th high-resolution decoded feature. The resolution of the first decoded feature is greater than that of the second decoded feature, which is greater than that of the third decoded feature, which is greater than that of the fourth decoded feature.

[0016] S6: Input the highest resolution first decoded feature into the segmentation prediction layer, map the number of feature channels to the number of target categories through convolution operation, and further perform double upsampling to restore the prediction result to the same spatial size as the original input image, thus obtaining the target region segmentation result in the medical image.

[0017] As a further improvement to the above scheme, the CNN coding branch in S1 adopts EfficientNet-B3, and the Transformer coding branch adopts Swing Transformer.

[0018] As a further improvement to the above scheme, the CNN branch feature of the nth corresponding level in S1 is denoted as... Let the Transformer branch feature of the nth corresponding level be denoted as Where B represents the batch size. and These represent the number of channels for the two encoded features, respectively. and This indicates the spatial dimensions of the current level.

[0019] As a further improvement to the above scheme, the specific steps in S2 for inputting the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer into the first fusion module for processing are as follows:

[0020] S2.1: Normalize the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer to reduce the difference in numerical distribution and response range between the two branch features; where:

[0021] ;

[0022] S2.2: The CNN branch feature C1 and Transformer branch feature T1, which were normalized in S2.1, are mapped to the same preset number of channels using a 1×1 convolution. Then, element-wise multiplication is performed on the two aligned features to extract the common response and correlation information of the two features at the same spatial location, resulting in the first dual-branch interaction feature B1, which is represented as follows:

[0023] = ;

[0024] S2.3: The CNN branch features C1 and Transformer branch features T1, which have been normalized in S2.1, are input into the multi-scale enhancement module MSC for processing. The multi-scale enhancement module MSC consists of three parallel convolutional branches: 1×1, 3×3, and 5×5. All three convolutional branches maintain the same number of input and output channels and extract feature responses under different receptive fields, as shown in the following formula:

[0025] ;

[0026] This yields the multi-scale enhancement features of CNN branches. Multi-scale augmentation features of the Transformer branch ;

[0027] S2.4: Multi-scale enhancement features for each CNN branch Multi-scale augmentation features of the Transformer branch Spatial descriptive features are obtained by performing max pooling and average pooling along the channel dimension. Then, the spatial descriptive features of the CNN branch and the Transformer branch are cross-concatenated in different orders. The cross-concatenated features are then input into two independent 7×7 convolutions, and spatial guidance maps corresponding to the CNN branch and the Transformer branch are generated using the Sigmoid function. This allows the CNN branch to perceive the global target region that the Transformer branch focuses on, while simultaneously enabling the Transformer branch to perceive the local edges, textures, and fine-grained locations emphasized by the CNN branch. Ultimately, this achieves bidirectional spatial information transfer between the two types of heterogeneous features. The specific process is as follows:

[0028] ;

[0029] S2.5: Apply residual multiplication to the corresponding multi-scale augmentation features using the CNN branch space guidance map and the Transformer branch space guidance map respectively to obtain the CNN branch space augmentation features and the Transformer branch space augmentation features; then input the two spatial augmentation features into the HCCA attention module for joint calibration to obtain the CNN branch calibration features. Transformer branching characteristics ;

[0030] S2.6: Labeling features of CNN branches The calibration characteristics of Transformer branches The first bi-branch interactive feature B1 obtained from S2.2 is spliced ​​along the channel dimension, and the spliced ​​feature is processed by the residual aggregation unit Resdiual to obtain the first final fusion feature F1.

[0031] S2.7: Perform global average pooling on the first final fused feature F1 and the first bi-branch interaction feature B1 obtained from S2.2, respectively. Then, the first context feedback vector G1 is formed by concatenating the vectors along the channel dimension. G1 contains the overall fusion semantics of the first stage, the interaction information between the CNN branch and the Transformer branch encoded features.

[0032] As a further improvement to the above scheme, in S3, the CNN branch feature C of the i-th corresponding level is... i Transformer branch feature T i and the (i-1)th context feedback vector G i-1 The specific steps for processing the i-th synchronous input fusion module are as follows:

[0033] S3.1: For the CNN branch feature C of the i-th corresponding level i Transformer branch feature T i Normalization is performed to reduce the differences in numerical distribution and response range between the two branch features; where:

[0034] ;

[0035] At the same time, the context feedback vector G passed from the previous level will be... i-1 Input to a multilayer perceptron and generate CNN branch gating vectors using the sigmoid function. Transformer branch gating vector and interaction branch gating vector The specific formula is as follows:

[0036] ;

[0037] ,

[0038] ;

[0039] S3.2: The CNN branch features C, which were normalized in S3.1, are processed by 1×1 convolution. i Transformer branch feature T i Mapping to the same preset number of channels, then performing element-wise multiplication interaction on the two aligned features, extracting the common response and correlation information of the two features at the same spatial location, and obtaining the i-th bi-branch interaction feature B. i It is represented as:

[0040] ,

[0041] ;

[0042] S3.3: Process the CNN branch features C after the normalization process in S3.1. i Transformer branch feature T i The input is processed by the multi-scale enhancement module (MSC), which consists of three parallel convolutional branches: 1×1, 3×3, and 5×5. Each of the three branches maintains the same number of input and output channels and extracts feature responses from different receptive fields, as shown in the following formula:

[0043] ;

[0044] This yields the multi-scale enhancement features of CNN branches. Multi-scale augmentation features of the Transformer branch ;

[0045] S3.4: Multi-scale feature enhancement for each CNN branch Multi-scale augmentation features of the Transformer branch Spatial descriptive features are obtained by performing max pooling and average pooling along the channel dimension. Then, the spatial descriptive features of the CNN branch and the Transformer branch are cross-concatenated in different orders. The cross-concatenated features are then input into two independent 7×7 convolutions, and spatial guidance maps corresponding to the CNN branch and the Transformer branch are generated using the Sigmoid function. This allows the CNN branch to perceive the global target region that the Transformer branch focuses on, while simultaneously enabling the Transformer branch to perceive the local edges, textures, and fine-grained locations emphasized by the CNN branch. Ultimately, this achieves bidirectional spatial information transfer between the two types of heterogeneous features. The specific process is as follows:

[0046] ;

[0047] S3.5: Apply residual multiplication to the corresponding multi-scale augmentation features using the CNN branch space guidance map and the Transformer branch space guidance map respectively to obtain the CNN branch space augmentation features and the Transformer branch space augmentation features; then input the two spatial augmentation features into the HCCA attention module for joint calibration to obtain the CNN branch calibration features. Transformer branching characteristics ;

[0048] S3.6: Labeling features of CNN branches The calibration characteristics of Transformer branches And the i-th bi-branch interaction feature B obtained from S3.2 i The features are concatenated along the channel dimension, and the concatenated features are processed by the residual aggregation unit (Residual) to obtain the i-th final fused feature F. i ;

[0049] S3.7: For the i-th final fused feature F respectively i The i-th bi-branch interaction feature B obtained from S3.2 i Perform global average pooling Then, the vectors are concatenated along the channel dimension to form the i-th context feedback vector G. i Among them, G iIt also includes the overall fusion semantics of stage i, the interaction information between the CNN branch and the Transformer branch encoded features.

[0050] As a further improvement to the above scheme, the HCCA attention module includes a channel attention branch, a coordinate attention branch, and a local detail awareness branch. The channel attention branch models the inter-channel dependencies through global average pooling and one-dimensional convolution to highlight the effective channel responses related to the lesion region. The coordinate attention branch aggregates spatial information along the horizontal and vertical directions, respectively, to enhance the model's ability to perceive the directional structure of the target region while preserving positional information. The local detail awareness branch generates a spatial detail weight map through 3×3 convolution, 1×1 convolution, and 7×7 convolution to enhance the response of edge transitions, texture changes, and fine-grained structural regions.

[0051] The beneficial effects of this invention are:

[0052] Compared with existing technologies, the dual-encoder collaborative fusion medical image segmentation method provided by this invention has the following advantages:

[0053] 1. Spatial-guided multi-scale cross-branch fusion

[0054] A cross-branch fusion module is set up between corresponding layers of the convolutional coding branch and the Transformer coding branch to align, interact, and selectively fuse the two heterogeneous features. This mainly involves mapping CNN branch features and Transformer branch features to a unified feature space, dynamically adjusting the contributions of the two branch features through learnable weights, introducing element-wise interaction information between the two branch features, extracting features with different receptive fields using 1×1, 3×3, and 5×5 convolutions respectively, concatenating features of different scales along the channel dimension, and performing channel adjustment and cross-scale recombination through 1×1 convolutions. Spatial guidance information is generated based on the average pooling and max pooling results of the two branch features, and the salient region information of one branch is used to guide the other branch for enhancement.

[0055] 2. Context-feedback driven layer-by-layer fusion mechanism

[0056] A context state vector is generated based on the cross-branch fusion features and dual-branch interaction features output from the current level, and the context state vector of the previous level is passed to the next level. The next level generates convolutional branch gating coefficients, Transformer branch gating coefficients and interaction branch gating coefficients based on the context state vector of the previous level, respectively, to modulate the convolutional features, Transformer features and dual-branch interaction features of the next fusion level, so that the fusion process of the next level is dynamically guided by the fusion semantics of the previous level, rather than each level fusing features independently.

[0057] 3. Multiple attention joint mechanism

[0058] The HCCA module further enhances the spatial guidance map by improving channel attention, coordinate attention, and local detail perception.

[0059] In summary, the medical image segmentation method provided by this invention effectively solves the problems of insufficient fusion of local detail features and global semantic features, limited interaction capabilities of heterogeneous coding features, and lack of effective semantic guidance between different levels of fusion processes in existing methods. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0061] Figure 2 This is a schematic diagram of the CNN coding branch structure in this invention;

[0062] Figure 3 This is a schematic diagram of the Transformer encoding branch in this invention;

[0063] Figure 4 This is a schematic diagram of the structure of the first fusion module in this invention;

[0064] Figure 5 This is a schematic diagram of the structure of the i-th fusion module in this invention;

[0065] Figure 6 This is a schematic diagram of the structure of the multi-scale enhancement module MSC in this invention;

[0066] Figure 7 This is a schematic diagram of the attention module HCCA in this invention.

[0067] In the diagram: CNN Block is the CNN coding branch, Transformer Block is the Transformer coding branch, SG-MCF is the fusion module, and CNN Decoder is the decoder. Detailed Implementation

[0068] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings:

[0069] Figure 1 In the CNN, CNN Blocks and Transformer Blocks at the same height form a layer, which are divided into layers 1 through n from top to bottom. The SG-MCF is divided into fusion modules 1 through 4 from top to bottom, and the CNN Decoder is divided into decoders 1 through 4 from top to bottom.

[0070] like Figures 1-6 As shown, the present invention provides a dual-encoder collaborative fusion medical image segmentation method, which includes the following steps:

[0071] S1: Input the preprocessed image into both the CNN encoding branch and the Transformer encoding branch simultaneously, and select four corresponding CNN branch features from each of the two encoding branches. Transformer branch features Where n = 1, 2, 3, 4; the CNN encoding branch uses EfficientNet-B3, and the Transformer encoding branch uses Swin Transformer; the feature of the CNN branch corresponding to the nth layer is denoted as... Let the Transformer branch feature of the nth corresponding level be denoted as Where B represents the batch size. and These represent the number of channels for the two encoded features, respectively. and This indicates the spatial dimensions of the current level.

[0072] S2: Input the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer into the first fusion module for processing, to obtain the first final fusion output F1 and the first context feedback vector G1. The specific steps for inputting the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer into the first fusion module for processing are as follows:

[0073] S2.1: Normalize the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer to reduce the difference in numerical distribution and response range between the two branch features; where:

[0074] ;

[0075] S2.2: The CNN branch feature C1 and Transformer branch feature T1, which were normalized in S2.1, are mapped to the same preset number of channels using a 1×1 convolution. Then, element-wise multiplication is performed on the two aligned features to extract the common response and correlation information of the two features at the same spatial location, resulting in the first dual-branch interaction feature B1, which is represented as follows:

[0076] = ;

[0077] S2.3: The CNN branch features C1 and Transformer branch features T1, which have been normalized in S2.1, are input into the multi-scale enhancement module MSC for processing. The multi-scale enhancement module MSC consists of three parallel convolutional branches: 1×1, 3×3, and 5×5. All three convolutional branches maintain the same number of input and output channels and extract feature responses under different receptive fields, as shown in the following formula:

[0078] ;

[0079] This yields the multi-scale enhancement features of CNN branches. Multi-scale augmentation features of the Transformer branch ;

[0080] S2.4: Multi-scale enhancement features for each CNN branch Multi-scale augmentation features of the Transformer branch Spatial descriptive features are obtained by performing max pooling and average pooling along the channel dimension. Then, the spatial descriptive features of the CNN branch and the Transformer branch are cross-concatenated in different orders. The cross-concatenated features are then input into two independent 7×7 convolutions, and spatial guidance maps corresponding to the CNN branch and the Transformer branch are generated using the Sigmoid function. This allows the CNN branch to perceive the global target region that the Transformer branch focuses on, while simultaneously enabling the Transformer branch to perceive the local edges, textures, and fine-grained locations emphasized by the CNN branch. Ultimately, this achieves bidirectional spatial information transfer between the two types of heterogeneous features. The specific process is as follows:

[0081] ;

[0082] S2.5: Apply residual multiplication to the corresponding multi-scale augmentation features using the CNN branch space guidance map and the Transformer branch space guidance map respectively to obtain the CNN branch space augmentation features and the Transformer branch space augmentation features; then input the two spatial augmentation features into the HCCA attention module for joint calibration to obtain the CNN branch calibration features. Transformer branching characteristics ;

[0083] S2.6: Labeling features of CNN branches The calibration characteristics of Transformer branches The first bi-branch interactive feature B1 obtained from S2.2 is spliced ​​along the channel dimension, and the spliced ​​feature is processed by the residual aggregation unit Resdiual to obtain the first final fusion feature F1.

[0084] S2.7: Perform global average pooling on the first final fused feature F1 and the first bi-branch interaction feature B1 obtained from S2.2, respectively. Then, the first context feedback vector G1 is formed by concatenating the vectors along the channel dimension. G1 contains the overall fusion semantics of the first stage, the interaction information between the CNN branch and the Transformer branch encoded features.

[0085] S3: The CNN branch features C corresponding to the i-th level. i Transformer branch feature T i and the (i-1)th context feedback vector G i-1 The i-th input is processed by the synchronous fusion module to obtain the i-th final fusion output F. i and the i-th context feedback vector G i Where i = 2, 3, 4; where the resolution of the first final fused output F1 is greater than the resolution of the second final fused output F2, which is greater than the resolution of the third final fused output F3, which is greater than the resolution of the fourth final fused output F4.

[0086] Wherein, the CNN branch feature C of the i-th corresponding level i Transformer branch feature T i and the (i-1)th context feedback vector G i-1 The specific steps for processing the i-th synchronous input fusion module are as follows:

[0087] S3.1: For the CNN branch feature C of the i-th corresponding level i Transformer branch feature T i Normalization is performed to reduce the differences in numerical distribution and response range between the two branch features; where:

[0088] ;

[0089] At the same time, the context feedback vector G passed from the previous level will be... i-1 Input to a multilayer perceptron and generate CNN branch gating vectors using the sigmoid function. Transformer branch gating vector and interaction branch gating vector The specific formula is as follows:

[0090] ;

[0091] ,

[0092] ;

[0093] S3.2: The CNN branch features C, which were normalized in S3.1, are processed by 1×1 convolution. i Transformer branch feature T i Mapping to the same preset number of channels, then performing element-wise multiplication interaction on the two aligned features, extracting the common response and correlation information of the two features at the same spatial location, and obtaining the i-th bi-branch interaction feature B. i It is represented as:

[0094] ,

[0095] ;

[0096] S3.3: Process the CNN branch features C after the normalization process in S3.1. i Transformer branch feature T i The input is processed by the multi-scale enhancement module (MSC), which consists of three parallel convolutional branches: 1×1, 3×3, and 5×5. Each of the three branches maintains the same number of input and output channels and extracts feature responses from different receptive fields, as shown in the following formula:

[0097] ;

[0098] This yields the multi-scale enhancement features of CNN branches. Multi-scale augmentation features of the Transformer branch ;

[0099] S3.4: Multi-scale feature enhancement for each CNN branch Multi-scale augmentation features of the Transformer branch Spatial descriptive features are obtained by performing max pooling and average pooling along the channel dimension. Then, the spatial descriptive features of the CNN branch and the Transformer branch are cross-concatenated in different orders. The cross-concatenated features are then input into two independent 7×7 convolutions, and spatial guidance maps corresponding to the CNN branch and the Transformer branch are generated using the Sigmoid function. This allows the CNN branch to perceive the global target region that the Transformer branch focuses on, while simultaneously enabling the Transformer branch to perceive the local edges, textures, and fine-grained locations emphasized by the CNN branch. Ultimately, this achieves bidirectional spatial information transfer between the two types of heterogeneous features. The specific process is as follows:

[0100] ;

[0101] S3.5: Apply residual multiplication to the corresponding multi-scale augmentation features using the CNN branch space guidance map and the Transformer branch space guidance map respectively to obtain the CNN branch space augmentation features and the Transformer branch space augmentation features; then input the two spatial augmentation features into the HCCA attention module for joint calibration to obtain the CNN branch calibration features. Transformer branching characteristics ;

[0102] S3.6: Labeling features of CNN branches The calibration characteristics of Transformer branches And the i-th bi-branch interaction feature B obtained from S3.2 i The features are concatenated along the channel dimension, and the concatenated features are processed by the residual aggregation unit (Residual) to obtain the i-th final fused feature F. i ;

[0103] S3.7: For the i-th final fused feature F respectively i The i-th bi-branch interaction feature B obtained from S3.2 i Perform global average pooling Then, the vectors are concatenated along the channel dimension to form the i-th context feedback vector G. i Among them, G i It also includes the overall fusion semantics of stage i, the interaction information between the CNN branch and the Transformer branch encoded features.

[0104] S4: Input the fourth final fusion output F4 into the fourth decoder for double upsampling to obtain the fourth decoding feature; where the resolution of the fourth decoding feature is equal to the resolution of the third final fusion output F3;

[0105] S5: Fuse the i-th decoded feature and the (i-1)-th final output F i-1 The (i-1)th decoder is synchronously input and concatenated along the channel dimension. Channel compression and feature reconstruction are then performed through convolution operations to obtain the (i-1)th high-resolution decoded feature. The resolution of the first decoded feature is greater than that of the second decoded feature, which is greater than that of the third decoded feature, which is greater than that of the fourth decoded feature.

[0106] S6: Input the highest resolution first decoded feature into the segmentation prediction layer, map the number of feature channels to the number of target categories through convolution operation, and further perform double upsampling to restore the prediction result to the same spatial size as the original input image, thus obtaining the target region segmentation result in the medical image.

[0107] In addition, such as Figure 7 As shown, the HCCA attention module includes a channel attention branch, a coordinate attention branch, and a local detail awareness branch. The channel attention branch models the inter-channel dependencies through global average pooling and one-dimensional convolution to highlight effective channel responses related to the lesion region. The coordinate attention branch aggregates spatial information along the horizontal and vertical directions, enhancing the model's ability to perceive the directional structure of the target region while preserving positional information. The local detail awareness branch generates spatial detail weight maps through 3×3 convolution, 1×1 convolution, and 7×7 convolution to enhance the response of edge transitions, texture changes, and fine-grained structural regions.

[0108] The method of this invention is compared with ten advanced medical image segmentation networks, including U-Net, U-Net++, TransUNet, TransFuse, Polyp-PVT, PraNet, PVT-CASCADE, FAT-Net, I2U-Net, and CSAP-Unet.

[0109] The Adam optimizer was used during training. The batch size was set to 16. The initial learning rate was set to 1e-4, and the training process consisted of 80 epochs. All experiments were conducted in PyTorch 1.11.0 and accelerated using an NVIDIA DTX3080-10GB. We primarily used the Dice coefficient and five metrics to evaluate their segmentation performance, including Dice coefficient (Dice), mean intersection-over-union ratio (mIoU), accuracy (Acc), sensitivity (Sen), and specificity (Spe). Experiments were conducted using ISIC2018, CVC-300, CVC-ClinicDB, Kvasir-SEG, and ETIS-LaribPolypDB to validate the model's effectiveness.

[0110] The experimental results are shown in Tables 1-3. The comparison results in each table show that, compared with existing medical image segmentation methods, the method proposed in this invention achieves better segmentation performance in both skin lesion segmentation and polyp segmentation tasks. On the ISIC2018 dataset, the Dice coefficient, mIoU, and accuracy of the method proposed in this invention reach 90.56%, 84.74%, and 93.52%, respectively, all achieving optimal results, indicating that this invention can effectively handle problems such as irregular morphology, blurred boundaries, and uneven color distribution of skin lesions. On the Kvasir-SEG dataset, the Dice coefficient and mIoU of the method proposed in this invention reach 92.04% and 87.07%, respectively, with overall performance approaching the best level among the compared methods. On the CVC-ClinicDB dataset, the Dice coefficient and mIoU of the method proposed in this invention reach 93.66% and 89.12%, respectively, both achieving optimal results, verifying the localization ability and segmentation integrity of the target region in typical polyp images.

[0111] Table 1. Quantitative comparison results on the ISIC2018 dataset.

[0112] U-Net 0.8644 0.7818 0.8368 0.8419 0.9204 U-Net++ 0.8732 0.7963 0.8652 0.8521 0.9231 TransUnet 0.9032 0.8317 0.8614 0.9282 0.9320 TransFuse 0.8934 0.8286 0.8492 0.9454 0.9268 Polyp-PVT 0.9021 0.8412 0.8436 0.9317 0.9332 I2U-Net 0.8869 0.8212 0.8440 0.8835 0.9195 CSAP-UNet 0.9011 0.8410 0.8728 0.9401 0.9347 FATnet 0.8924 0.8259 0.8180 0.9403 0.9243 SGMCFNet 0.9056 0.8474 0.8874 0.9278 0.9352

[0113] Table 2 Quantitative comparison results on the Kvasir-SEG dataset

[0114] U-Net 0.7874 0.6821 0.8414 0.0622 U-Net++ 0.8617 0.7521 0.8646 0.0411 TransUnet 0.9127 0.8682 0.9034 0.0273 TransFuse 0.9083 0.8541 0.8733 0.0306 Polyp-PVT 0.9223 0.8709 0.9153 0.0223 PraNet 0.9112 0.8470 0.89947 0.0271 PVT-CASCADE 0.9237 0.8768 0.9208 0.0227 FAT-Net 0.9090 0.8511 0.8908 0.0310 SGMCFNet 0.9204 0.8707 0.9061 0.0262

[0115] Table 3 Quantitative comparison results on the CVC-ClinicDB dataset

[0116] U-Net 0.8271 0.7596 0.8675 0.0223 U-Net++ 0.8743 0.8075 0.8949 0.0120 TransUnet 0.9174 0.8618 0.9540 0.0111 TransFuse 0.9207 0.8684 0.9441 0.0132 Polyp-PVT 0.9181 0.8672 0.9363 0.0119 PraNet 0.9223 0.8733 0.9321 0.0091 PVT-CASCADE 0.9253 0.8762 0.9445 0.0073 FAT-Net 0.9155 0.8692 0.9465 0.0123 SGMCFNet 0.9366 0.8912 0.9679 0.0086

[0117] On external test datasets not used for training, the method of this invention also demonstrates good generalization ability. As shown in Tables 1-3, on the CVC-300 dataset, the Dice coefficient and mIoU of the method of this invention reach 89.21% and 82.16%, respectively, and the sensitivity reaches 95.40%, indicating that the method of this invention can identify more real polyp regions, which is beneficial to reducing the risk of missing segmentation of small target polyps. On the ETIS-LaribPolypDB dataset, the Dice coefficient and mIoU of the method of this invention reach 78.89% and 71.91%, respectively, with a mean absolute error of 1.45%. The method of this invention achieves the best results in terms of Dice coefficient, mIoU, and mean absolute error, indicating that the method of this invention still has good segmentation accuracy and cross-dataset adaptability under conditions of significant lesion scale changes, low target-background contrast, and complex background interference.

[0118] Table 4 Quantitative comparison results on the CVC-300 dataset

[0119] U-Net 0.7095 0.6086 0.8497 0.0211 U-Net++ 0.7518 0.6649 0.8138 0.0142 TransUnet 0.8816 0.8213 0.9527 0.0091 TransFuse 0.8839 0.8071 0.9522 0.0105 Polyp-PVT 0.9011 0.8350 0.9513 0.0071 PraNet 0.8790 0.8093 0.9432 0.0085 PVT-CASCADE 0.8890 0.8162 0.9451 0.0085 FAT-Net 0.8833 0.8131 0.9446 0.0095 SGMCFNet 0.8921 0.8216 0.9540 0.0101

[0120] Table 5. Quantitative comparison results on the ETIS-LaribPolypDB dataset.

[0121] U-Net 0.4244 0.3356 0.5334 0.0464 U-Net++ 0.4872 0.4245 0.5245 0.0275 TransUnet 0.7681 0.6872 0.8434 0.0287 TransFuse 0.7042 0.6210 0.8574 0.0265 Polyp-PVT 0.7749 0.6955 0.8622 0.0194 PraNet 0.6758 0.6077 0.7199 0.0179 PVT-CASCADE 0.7415 0.6809 0.8613 0.0163 FAT-Net 0.7409 0.6556 0.8436 0.0273 SGMCFNet 0.7889 0.7191 0.8436 0.0145

[0122] The above embodiments are not limited to the technical solutions of the embodiments themselves, and the embodiments can be combined with each other to form new embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of the technical solutions of the present invention.

Claims

1. A dual-encoder collaborative fusion method for medical image segmentation, characterized in that: Includes the following steps S1: Input the preprocessed image into both the CNN encoding branch and the Transformer encoding branch simultaneously, and select four corresponding CNN branch features from each of the two encoding branches. Transformer branch features Where n = 1, 2, 3, 4; S2: Input the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer into the first fusion module for processing to obtain the first final fusion output F1 and the first context feedback vector G1; S3: The CNN branch features C corresponding to the i-th level. i Transformer branch feature T i and the (i-1)th context feedback vector G i-1 The i-th input is processed by the synchronous fusion module to obtain the i-th final fusion output F. i and the i-th context feedback vector G i Where i = 2, 3, 4; The resolution of the first final fused output F1 is greater than the resolution of the second final fused output F2, which is greater than the resolution of the third final fused output F3, which is greater than the resolution of the fourth final fused output F4. S4: Input the fourth final fused output F4 into the fourth decoder for double upsampling processing to obtain the fourth decoding feature; The resolution of the fourth decoding feature is equal to the resolution of the third final fused output F3; S5: Fuse the i-th decoded feature and the (i-1)-th final output F i-1 The (i-1)th decoder is synchronously input and spliced ​​along the channel dimension. Channel compression and feature reconstruction are completed through convolution operations to obtain high-resolution (i-1)th decoded features. The resolution of the first decoding feature is greater than the resolution of the second decoding feature, which is greater than the resolution of the third decoding feature, which is greater than the resolution of the fourth decoding feature. S6: Input the highest resolution first decoded feature into the segmentation prediction layer, map the number of feature channels to the number of target categories through convolution operation, and further perform double upsampling to restore the prediction result to the same spatial size as the original input image, thus obtaining the target region segmentation result in the medical image.

2. The medical image segmentation method based on dual encoder collaborative fusion according to claim 1, characterized in that: The CNN coding branch in S1 uses EfficientNet-B3, and the Transformer coding branch uses Swing Transformer.

3. The medical image segmentation method based on dual encoder collaborative fusion according to claim 2, characterized in that: The CNN branch feature of the nth corresponding level in S1 is denoted as... Let the Transformer branch feature of the nth corresponding level be denoted as Where B represents the batch size. and These represent the number of channels for the two encoded features, respectively. and This indicates the spatial dimensions of the current level.

4. The medical image segmentation method based on dual encoder collaborative fusion according to claim 1, characterized in that: The specific steps in S2 for inputting the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer into the first fusion module for processing are as follows: S2.1: Normalize the CNN branch feature C1 and Transformer branch feature T1 of the first corresponding layer to reduce the difference in numerical distribution and response range between the two branch features; where: ； S2.2: The CNN branch feature C1 and Transformer branch feature T1, which were normalized in S2.1, are mapped to the same preset number of channels using a 1×1 convolution. Then, element-wise multiplication is performed on the two aligned features to extract the common response and correlation information of the two features at the same spatial location, resulting in the first dual-branch interaction feature B1, which is represented as follows: = ； S2.3: The CNN branch features C1 and Transformer branch features T1, which have been normalized in S2.1, are input into the multi-scale enhancement module MSC for processing. The multi-scale enhancement module MSC consists of three parallel convolutional branches: 1×1, 3×3, and 5×5. All three convolutional branches maintain the same number of input and output channels and extract feature responses under different receptive fields, as shown in the following formula: ； This yields the multi-scale enhancement features of CNN branches. Multi-scale augmentation features of the Transformer branch ; S2.4: Multi-scale enhancement features for each CNN branch Multi-scale augmentation features of the Transformer branch Spatial descriptive features are obtained by performing max pooling and average pooling along the channel dimension. Then, the spatial descriptive features of the CNN branch and the Transformer branch are cross-concatenated in different orders. The cross-concatenated features are then input into two independent 7×7 convolutions, and spatial guidance maps corresponding to the CNN branch and the Transformer branch are generated using the Sigmoid function. This allows the CNN branch to perceive the global target region that the Transformer branch focuses on, while simultaneously enabling the Transformer branch to perceive the local edges, textures, and fine-grained locations emphasized by the CNN branch. Ultimately, this achieves bidirectional spatial information transfer between the two types of heterogeneous features. The specific process is as follows: ； S2.5: Apply residual multiplication to the corresponding multi-scale augmentation features using the CNN branch space guidance map and the Transformer branch space guidance map respectively to obtain the CNN branch space augmentation features and the Transformer branch space augmentation features; then input the two spatial augmentation features into the HCCA attention module for joint calibration to obtain the CNN branch calibration features. Transformer branching characteristics ; S2.6: Labeling features of CNN branches The calibration characteristics of Transformer branches The first bi-branch interactive feature B1 obtained from S2.2 is spliced ​​along the channel dimension, and the spliced ​​feature is processed by the residual aggregation unit Resdiual to obtain the first final fusion feature F1. S2.7: Perform global average pooling on the first final fused feature F1 and the first bi-branch interaction feature B1 obtained from S2.2, respectively. Then, the first context feedback vector G1 is formed by concatenating the vectors along the channel dimension. G1 contains the overall fusion semantics of the first stage, the interaction information between the CNN branch and the Transformer branch encoded features.

5. The medical image segmentation method based on dual encoder collaborative fusion according to claim 4, characterized in that: In S3, the CNN branch feature C of the i-th corresponding level is... i Transformer branch feature T i and the (i-1)th context feedback vector G i-1 The specific steps for processing the i-th synchronous input fusion module are as follows: S3.1: For the CNN branch feature C of the i-th corresponding level i Transformer branch feature T i Normalization is performed to reduce the differences in numerical distribution and response range between the two branch features; where: ； At the same time, the context feedback vector G passed from the previous level will be... i-1 Input to a multilayer perceptron and generate CNN branch gating vectors using the sigmoid function. Transformer branch gating vector and interaction branch gating vector The specific formula is as follows: ； ， ； S3.2: The CNN branch features C, which were normalized in S3.1, are processed by 1×1 convolution. i Transformer branch feature T i Mapping to the same preset number of channels, then performing element-wise multiplication interaction on the two aligned features, extracting the common response and correlation information of the two features at the same spatial location, and obtaining the i-th bi-branch interaction feature B. i It is represented as: ， ； S3.3: Process the CNN branch features C after the normalization process in S3.

1. i Transformer branch feature T i The input is processed by the multi-scale enhancement module (MSC), which consists of three parallel convolutional branches: 1×1, 3×3, and 5×5. Each of the three branches maintains the same number of input and output channels and extracts feature responses from different receptive fields, as shown in the following formula: ； This yields the multi-scale enhancement features of CNN branches. Multi-scale augmentation features of the Transformer branch ; S3.4: Multi-scale feature enhancement for each CNN branch Multi-scale augmentation features of the Transformer branch Spatial descriptive features are obtained by performing max pooling and average pooling along the channel dimension. Then, the spatial descriptive features of the CNN branch and the Transformer branch are cross-concatenated in different orders. The cross-concatenated features are then input into two independent 7×7 convolutions, and spatial guidance maps corresponding to the CNN branch and the Transformer branch are generated using the Sigmoid function. This allows the CNN branch to perceive the global target region that the Transformer branch focuses on, while simultaneously enabling the Transformer branch to perceive the local edges, textures, and fine-grained locations emphasized by the CNN branch. Ultimately, this achieves bidirectional spatial information transfer between the two types of heterogeneous features. The specific process is as follows: ； S3.5: Apply residual multiplication to the corresponding multi-scale augmentation features using the CNN branch space guidance map and the Transformer branch space guidance map respectively to obtain the CNN branch space augmentation features and the Transformer branch space augmentation features; then input the two spatial augmentation features into the HCCA attention module for joint calibration to obtain the CNN branch calibration features. Transformer branching characteristics ; S3.6: Labeling features of CNN branches The calibration characteristics of Transformer branches And the i-th bi-branch interaction feature B obtained from S3.2 i The features are concatenated along the channel dimension, and the concatenated features are processed by the residual aggregation unit (Residual) to obtain the i-th final fused feature F. i ; S3.7: For the i-th final fused feature F respectively i The i-th bi-branch interaction feature B obtained from S3.2 i Perform global average pooling Then, the vectors are concatenated along the channel dimension to form the i-th context feedback vector G. i Among them, G i It also includes the overall fusion semantics of stage i, the interaction information between the CNN branch and the Transformer branch encoded features.

6. The medical image segmentation method based on dual encoder collaborative fusion according to claim 5, characterized in that: The HCCA attention module includes a channel attention branch, a coordinate attention branch, and a local detail awareness branch. The channel attention branch models the inter-channel dependencies through global average pooling and one-dimensional convolution to highlight effective channel responses related to the lesion region. The coordinate attention branch aggregates spatial information along the horizontal and vertical directions, respectively, to enhance the model's ability to perceive the directional structure of the target region while preserving positional information. The local detail awareness branch generates spatial detail weight maps through 3×3 convolution, 1×1 convolution, and 7×7 convolution to enhance the response of edge transitions, texture changes, and fine-grained structural regions.

Images