Double-bridge normal form U-shaped medical image segmentation network

Abstract

The invention discloses a double-bridge normal type U-shaped medical image segmentation network, which comprises an encoder, a double-bridge framework and a decoder, the encoder adopts a backbone network, the double-bridge architecture comprises a multi-scale bridge and a single-scale bridge, the multi-scale bridge adopts an SCAM, the single-scale bridge adopts a CAM, and the decoder adopts an RMSCAM; the encoder extracts multi-stage image features from the medical image of the single-channel gray scale, and the features extracted by the encoder in four stages are fed into the double-bridge architecture; the multi-scale bridge receives features extracted from the four stages of the encoder and captures global features; the single-scale bridge receives features extracted by the encoder in a single stage and captures local features; the global features and the local features are added and fused and then sent to a multi-stage decoder; the decoder restores the feature map to obtain a final prediction map; according to the method, the problems of semantic difference between different stage features extracted by the encoder and semantic difference between the encoder and the decoder can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a U-shaped medical image segmentation network of a double-bridge paradigm. Background Art

[0002] With the continuous advancement of medical imaging technology, medical image segmentation has been widely used in organ localization, lesion detection and disease diagnosis. However, medical images often have problems such as noise, low contrast and blur, which makes the medical image segmentation task face many challenges. For example, different types of tissues or lesions appear differently in medical images, resulting in the inhomogeneity of the image area and the fuzziness of the organ boundaries, making it difficult for traditional image segmentation methods to accurately extract the region of interest.

[0003] In recent years, Convolutional Neural Networks (CNNs) have emerged and have been widely used in various visual tasks, achieving better performance than traditional methods. For medical image segmentation tasks, the U-type network composed of CNNs has achieved a lot of research results, such as U-Net, 3D U-Net, UNet++, UNet 3+, etc. However, the convolution operation has limitations in modeling global dependencies. To solve this problem, Vision Transformer uses the Self-Attention Mechanism to capture the long-range dependencies between pixels by learning the correlation between input patches, showing great potential. Some scholars introduce it into the U-type architecture to improve the performance of medical image segmentation.

[0004] For example, Swin-Unet introduces Swin Transformer to capture global dependencies, MISSFormer builds an enhanced Transformer block by redesigning the feedforward network, and DAE-Former redesigns a new self-attention mechanism to capture spatial and channel relationships at the same time. Although Transformer can model long-range dependencies well, it is difficult to learn the spatial relationship between local pixels. The reason is that Transformer reshapes the image into a one-dimensional sequence, and because the spatial information is ignored, it is impossible to strongly learn the local features of the image.

[0005] As we all know, there is a semantic gap between the encoder and the decoder. To bridge this semantic gap, most methods based on the U-shaped architecture use skip connections between them to improve the performance of the network. These existing skip connection methods can be summarized into three categories, such as Figure 1 (a) Figure 1 (b) Figure 1 (c) as shown. The first category is Figure 1 As shown in (a), the traditional UNet uses a direct jump connection method to directly pass the feature map of the corresponding layer in the encoder to the decoder, retaining the early spatial features. This jump connection method is widely used due to its simple structure. Figure 1 As shown in (b), MultiResUNet adopts a single-scale feature interactive connection method, using residual paths (Res Path) to replace the simple skip connection in traditional U-Net. The motivation is that although the skip connection mechanism of U-Net can help restore details, there is a semantic gap between the features of the encoder and decoder, which affects the model performance. To make up for this deficiency, MultiResUNet introduces convolutional layers and residual connections in the skip connection to reduce the semantic gap and enhance feature fusion. The third category is Figure 1 As shown in (c), UCTransNet constructs a multi-scale channel cross-fusion (CTrans) module to effectively fuse multi-scale channel features that may have semantic gaps in a collaborative learning manner. The motivation is that existing methods are insufficient in capturing full-scale information and bridging the semantic gap between the encoder and decoder. To address this problem, UCTransNet establishes a connection between the encoder and decoder by exploring multi-scale global contexts, which can also effectively bridge the semantic gap between the encoder and decoder.

[0006] Figure 1 (a) Figure 1 (b) Figure 1 Methods in (c) all use a single path to transfer local and global features, which places high demands on the model’s expressive power and is difficult to effectively solve the following two semantic gap problems: (1) the semantic gap between features extracted at different stages by the encoder; and (2) the semantic gap between the encoder and decoder. Summary of the invention

[0007] The present invention provides a U-shaped medical image segmentation network of a double bridge paradigm to solve the problem that the prior art is difficult to effectively solve the semantic gap between features of different stages extracted by the encoder and the semantic gap between the encoder and the decoder.

[0008] The present invention provides a U-shaped medical image segmentation network of a dual-bridge paradigm, comprising: an encoder, a dual-bridge architecture and a decoder; the encoder adopts a backbone network, the dual-bridge architecture comprises a multi-scale bridge and a single-scale bridge, the multi-scale bridge adopts SCAM, the single-scale bridge adopts CAM, and the decoder adopts RMSCAM;

[0009] The encoder is used to extract multi-level image features from a single-channel grayscale medical image, and the features extracted by the four stages of the encoder are fed into a dual-bridge architecture;

[0010] The dual-bridge architecture is used to receive features extracted from four stages of the encoder through a multi-scale bridge to capture global features; receive features extracted from a single stage of the encoder through a single-scale bridge to capture local features; add and fuse the global features with the local features to obtain a feature map, and send the feature map to a multi-stage decoder;

[0011] The decoder is used to restore the feature map to obtain a final prediction map.

[0012] Furthermore, the single-scale bridge adopts a double-branch structure, including branch one and branch two;

[0013] The branch 1 is used to obtain q, k, v from the input x, and pass them to a block consisting of a deep convolution with a kernel size of 3×3, batch normalization, and an activation function after splicing along the channel dimension; by using 3×3 convolution, local details are aggregated from q, k, v, where q is a table query, k represents a key, and v represents a value; point-by-point convolution with batch normalization is used to compress the channel dimension to the input dimension C and generate detail features;

[0014] The second branch is used to perform global semantic extraction through axial attention, and then multiply and fuse the detail features obtained from the first branch with the features extracted by the axial attention of the second branch.

[0015] Furthermore, the formula of the branch one is as follows:

[0016] F0=concat(q(x),k(x),v(x)) (1)

[0017] F1=BN(PWConv(ReLU(BN(DWConv(F0))))) (2)

[0018] Among them, F0 is the concatenated feature, concat represents feature concatenation, x is the input, q represents query, k represents the key, v represents the value, F1 is the detail feature obtained by branch one, BN represents batch normalization, PWConv represents point-by-point convolution, ReLU represents the activation function, and DWConv represents depth convolution.

[0019] Furthermore, the formula of the second branch is as follows:

[0020]

[0021] Among them, 1 W is a vector of size (W,1) with all 1s. H is a vector of size (H, 1) with all 1s, T represents transpose, Q h The width W of the feature map q(x) is compressed to 1 by horizontal compression, Q wThe height H of the feature map q(x) is compressed to 1 through longitudinal compression; F2 is the feature extracted by axial attention in branch 2; the above compression method is used to compress k and v to obtain K h 、V h , K w and V w .

[0022] Furthermore, the formula used to multiply and fuse the detail features obtained from branch one with the features extracted by the axial attention of branch two is as follows:

[0023]

[0024] Among them, F out It is the multiplication and fusion result of F1 and F2. F1 is the detail feature obtained by branch one, and F2 is the feature extracted by axial attention in branch two. Represents the multiplication of corresponding matrix elements.

[0025] Furthermore, the method for capturing global features by the multi-scale bridge includes:

[0026] First, we get image patches from the four stages of the encoder, given encoder features at different scales Use image blocks of different sizes: H is the width, W is the height, and C is the number of channels; the pooling size and stride are both P i The adaptive average pooling method is used to extract image patches, and then a deep convolution with a kernel size of 1×1 is used on the flat 2D image patches, while the number of feature map channels remains unchanged.

[0027] For T i ∈N×Ci(i=1,2,3), perform layer normalization and concatenation along the channel dimension; use the concatenated vector Use each tag as both a query and a key as values; use depthwise convolution and pointwise convolutional projection on query, key, and value:

[0028] Q=σ(BN(PW(DW(T Σ )))) (6)

[0029] K=σ(BN(PW(DW(T Σ )))) (7)

[0030] V i =σ(BN(PW(DW(T i )))) (8)

[0031] Where Q, K, Vi are the query, key, and value of the projection respectively; DW and PW represent depthwise convolution and pointwise convolution respectively; BN represents batch normalization; σ represents the ReLu activation function; T i (i=1,2,3,4) refers to the four-level features from the encoder, T Σ It refers to the new features obtained by concatenating and reducing the dimension of the above four-level features; SCA is expressed as:

[0032]

[0033] Among them, d k is the scaling factor; for the long case h is the number of heads;

[0034] The outputs of SCAM are projected using depthwise convolution and pointwise convolution to form the final output; layer normalization and GeLU are applied to these outputs; finally, the multiple outputs of SCA are connected to the corresponding decoder part through an upsampling layer, followed by a depthwise convolution with a kernel size of 1×1, batch normalization, and a ReLU activation function.

[0035] Furthermore, the decoder includes an Attention module, two normalization layers, and a residual feedforward network;

[0036] The Attention module includes two 1×1 convolutions, GeLU activation function and multi-scale convolution attention module. The output of MSCAN at the i-th level of the decoder is expressed as:

[0037] F′ i =Attention(LayerNorm(X i ))+X i (10)

[0038] F i =FFN(LayerNorm(F′ i ))+F′ i (11)

[0039] Among them, X i is the feature of input Attention, F' i is the attention output, F i is the output feature of MSCAN, LayerNorm is layer normalization, MSCAN is the multi-scale convolutional attention network, and FFN is the feed-forward network;

[0040] The multi-scale convolutional attention module consists of three parts: a 5×5 deep convolution for aggregating local information, a multi-branch deep convolution for capturing multi-scale context, and a 1×1 convolution for building relationships between different channels. The output of the 1×1 convolution is used as the attention weight to directly weight the input of MSCA, where the formula of MSCA is:

[0041] Attention(x)=Linear(MSCA(σ(Linear(x))) (12)

[0042]

[0043] Among them, Linear is the linear projection layer, MSCA represents multi-scale convolutional attention, σ is the ReLu activation function, x represents the input feature, It is the multiplication operation of element matrix, DWConv means deep convolution; Scale i , i∈0,1,2,3;

[0044] Next, in each branch, depthwise convolution with two strip-shaped convolution kernels is used to approximate the standard depthwise convolution with a large kernel, and the kernel size of each branch is set to 7, 11, and 21 respectively;

[0045] The feed-forward network FFN is expressed as:

[0046] FFN(x f )=Linear(DWConv 3×3 (σ(Linear(x f )))) (14)

[0047] Among them, x f is the feed-forward network FFN feature input, Linear is the linear projection layer, σ is the ReLu activation function, DWConv 3×3 For depthwise convolution with a kernel size of 3×3, the expansion ratio of the first linear layer in FFN is set to 4 by default;

[0048] Residual FFN is expressed as:

[0049] y=BN(σ(Conv 1×1 (x r ))) (15)

[0050] x out =BN(Conv 1×1 (BN(σ(DWConv 3×3 (y)+y))))(16)

[0051] Among them, Residual FFN is the residual feedforward network, BN is batch normalization, σ is the ReLu activation function, and x r Residual FFN is the feature input of the residual feedforward network, Conv 1×1 and Conv 3×3 Respectively represent convolutions with kernel sizes of 1×1 and 3×3, x out represents the output of the residual feed-forward network.

[0052] The present invention has the following beneficial effects: The U-shaped medical image segmentation network of the double-bridge paradigm of the present invention has two novel bridging structures. The multi-scale bridge realizes interactive learning between multi-stage features from a global perspective, bridging the long-range contextual semantic gap between the encoder and the decoder. The single-scale bridge uses a series of nonlinear operations such as convolutional attention from a local perspective to alleviate the semantic gap between the features of the encoder and the decoder at different stages. This double-bridge structure can effectively extract the interdependence of long-range contextual information and detail information in the multi-scale encoder features, solving the problem of the semantic gap between the encoder and the decoder. The attention block of the decoder uses multi-scale convolution to awaken the spatial attention features, and the redesigned feedforward network can more effectively aggregate multi-scale feature channel information, further improving the model's expressiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 Schematic diagram of the comparison between the DBNet skip connection method and other methods, including (a) UNet; (b) MultiResUNet; (c) UCTransNet; (d) DBNet proposed by the present invention;

[0055] Figure 2 The overall architecture of the DBNet method;

[0056] Figure 3 It is a single-scale bridge;

[0057] Figure 4 It is a multi-scale bridge;

[0058] Figure 5 It is a residual multi-scale convolutional attention model;

[0059] Figure 6 is the feed-forward layer;

[0060] Figure 7Visual comparison of the Synapse dataset and SOTA methods, including (a) GT; (b) DBNet; (c) MISSFormer; (d) PVT-CASCADE; (e) Swin-UNet;

[0061] Figure 8 Visual comparison of the ACDC dataset and SOTA methods, including: (a) GT; (b) DBNet; (c) MISSFormer; (d) PVT-CASCADE; (e) Swin-UNet;

[0062] Fig. 9 Visual comparison of the ACDC dataset and SOTA methods, including (a) GT; (b) DBNet; (c) MISSFormer; (d) FR-UNet;

[0063] Fig.10 Visualization of feature maps, including: (a) feature map of backbone stage one; (b) single-scale bridge feature map of the first stage; (c) multi-scale bridge feature map of the first stage; (d) feature map after double bridge feature concatenation; (e) feature map after double bridge feature fusion; (f) output feature map of the decoder in the last stage; (g) true value. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The technical solutions provided by the embodiments of the present invention are described in detail below in conjunction with the drawings.

[0065] Medical image segmentation faces challenges such as complex organ structures and blurred boundaries, which makes it very difficult to accurately segment structures or regions of interest. In view of the problems of segmentation errors and boundary burrs in existing medical image segmentation methods, in order to improve the accuracy and robustness of segmentation, this paper proposes a medical image segmentation network with dual-bridge parallel information propagation of semantics and details, called DBNet (Dual-Bridge Network). Figure 1(d) shows the dual-bridge connection mode, i.e., single-scale feature interaction and multi-scale feature interaction learning. Specifically, the present invention designs a novel dual-bridge structure: a multi-scale bridge and a single-scale bridge. The network has two novel bridging structures, including a key multi-scale bridge (Multi-scale Bridge), which uses spatial cross-attention to effectively compress the token vector into a global semantic with lower complexity for multi-scale information interaction and achieve rough organ positioning; the other constructed single-scale bridge (Single-scale Bridge) learns finer local pixel-level details through convolutional attention to achieve accurate boundary segmentation. The multi-scale bridge and the single-scale bridge propagate enhanced attention information in parallel. The improved decoder feedforward network can more effectively aggregate semantic and detail information, further improving the model's expressiveness. The present invention has demonstrated the superior performance of DBNet in terms of performance, effectiveness, and robustness through experimental verification on multi-organ segmentation, heart segmentation, and retinal vascular segmentation tasks.

[0066] See also Figure 2 The present invention provides a U-shaped medical image segmentation network of a dual-bridge paradigm, comprising: an encoder, a dual-bridge architecture and a decoder; the encoder adopts a backbone network, the dual-bridge architecture comprises a multi-scale bridge and a single-scale bridge, the multi-scale bridge adopts a spatial cross-attention module (Spatial Cross-Attention Module, SCAM), the single-scale bridge adopts a convolutional attention module (Convolution Attention Module, CAM), and the decoder adopts a residual multi-scale convolutional attention module (Residual Multi-Scale Convolutional Attention Module, RMSCAM).

[0067] The encoder is used to extract multi-level image features from single-channel grayscale medical images, and feed the features extracted by the encoder in four stages into the dual-bridge architecture; the dual-bridge architecture is used to receive the features extracted from the encoder in four stages through a multi-scale bridge to capture global features; the features extracted by the encoder in a single stage are received through a single-scale bridge to capture local features; the global features are added and fused with the local features to obtain a feature map, and the feature map is fed into a multi-stage decoder; the decoder is used to restore the feature map to obtain the final prediction map.

[0068] In order to ensure that the network has excellent generalization and the ability to handle multi-scale features, different backbone networks can be used, such as Dual-ViT and PVTv2. For example, Dual-ViT uses the parallel dual-stream Transformer structure to capture local and global features. PVTv2 uses convolution operations instead of slice embedding in traditional Transformer blocks to supplement spatial information.

[0069] Specifically, the single-scale bridge uses a convolution-based attention method combined with axial attention guidance to enhance spatial details. In multiple stages, local features are captured by single-scale bridges. The multi-scale bridge mainly relies on the patch embedding layer (Patch Embedding) and spatial cross-attention (Spatial Cross-Attention, SCA). The patch embedding layer is responsible for compressing large-sized feature maps such as 56×56 into small highly semantic token blocks (7×7 size) through an aggressive downsampling strategy. The spatial cross-attention mechanism refines the global spatial context by deeply exploring the spatial interdependencies between highly semantic token blocks in four different stages. SCA can cleverly capture rich global context information in the spatial dimension, thereby effectively extracting long-distance context information.

[0070] The single-scale bridge uses CAM to enhance spatial details, such as Figure 3 Specifically, the single-scale bridge adopts a double-branch structure, including branch one and branch two;

[0071] Branch 1 is used to obtain q, k, v from the input x, concatenate along the channel dimension and pass it to a block consisting of a deep convolution with a kernel size of 3×3, batch normalization and activation function; aggregate local details from q, k, v by using 3×3 convolution, where q is a table query, k represents the key, and v represents the value; use point-by-point convolution with batch normalization to compress the channel dimension to the input dimension C and generate detailed features;

[0072] Branch 2 is used to extract global semantics through axial attention, and then multiply and fuse the detail features obtained from branch 1 with the features extracted by the axial attention of branch 2.

[0073] In order to achieve more efficient computation and aggregate global information at the same time, this paper adopts an aggressive compression strategy, such as Figure 3 As shown in branch 2 in . First, horizontal compression is achieved by averaging the query feature map in the horizontal direction, and the compression operation like q is repeated for k and v. The compressed q, k, and v are then fed into Multi-Head Attention. Similarly, vertical squeezing is performed and multiple attentions are used. The squeezing operation retains global information on a single axis, greatly reducing the computational overhead of global semantic extraction. Secondly, the two sets of axial features are restored to the original image size by broadcasting, and then the horizontally compressed features and the vertically compressed features are added. Finally, it is passed to a block consisting of point-by-point convolution (PWConv), batch normalization, and activation function to obtain the axial attention feature map.

[0074] The formula for branch one is as follows:

[0075] F0=concat(q(x),k(x),v(x)) (1)

[0076] F1=BN(PWConv(ReLU(BN(DWConv(F0))))) (2)

[0077] Among them, F0 is the concatenated feature, concat represents feature concatenation, x is the input, q represents query, k represents the key, v represents the value, F1 is the detail feature obtained by branch one, BN represents batch normalization, PWConv represents point-by-point convolution, ReLU represents the activation function, and DWConv represents depth convolution.

[0078] The formula for branch 2 is as follows:

[0079]

[0080] Among them, 1 W is a vector of size (W,1) with all 1s. H is a vector of size (H, 1) with all 1s, T represents transpose, Q h The width W of the feature map q(x) is compressed to 1 by horizontal compression, Q w The height H of the feature map q(x) is compressed to 1 through longitudinal compression; F2 is the feature extracted by axial attention in branch 2; the above compression method is used to compress k and v to obtain K h 、V h , K w and V w .

[0081] The formula used to multiply and fuse the detail features obtained from branch one with the features extracted from the axial attention of branch two is as follows:

[0082]

[0083] Among them, F out It is the multiplication and fusion result of F1 and F2. F1 is the detail feature obtained by branch one, and F2 is the feature extracted by axial attention in branch two. Represents the multiplication of corresponding matrix elements.

[0084] The present invention adopts the cross-attention method to capture multi-scale semantic information, such as Figure 4 As shown. The methods of multi-scale bridges to capture global features include:

[0085] First, we get image patches from the four stages of the encoder, given encoder features at different scales Use image blocks of different sizes: H is the width, W is the height, and C is the number of channels; the pooling size and stride are both P i The adaptive average pooling method is used to extract image patches, and then a deep convolution with a kernel size of 1×1 is used on the flat 2D image patches, while the number of feature map channels remains unchanged.

[0086] For T i ∈N×Ci(i=1,2,3), perform layer normalization and concatenation along the channel dimension; use the concatenated vector Use each tag as both query and key as values; use depthwise convolution and pointwise convolutional projection on query, key, and value:

[0087] Q=σ(BN(PW(DW(T Σ )))) (6)

[0088] K=σ(BN(PW(DW(T Σ )))) (7)

[0089] V i =σ(BN(PW(DW(T i )))) (8)

[0090] Where Q, K, Vi are the query, key, and value of the projection respectively; DW and PW represent depthwise convolution and pointwise convolution respectively; BN represents batch normalization; σ represents the ReLu activation function; T i (i=1,2,3,4) refers to the four-level features from the encoder, T Σ It refers to the new features obtained by concatenating and reducing the dimension of the above four-level features; SCA is expressed as:

[0091]

[0092] Among them, d k is the scaling factor; for the long case h is the number of heads;

[0093] The output of SCAM is projected using depthwise convolution and pointwise convolution to form the final output; layer normalization and GeLU are applied to these outputs; finally, the multiple outputs of SCA are connected to the corresponding decoder part through an upsampling layer, followed by a depthwise convolution with a kernel size of 1×1, batch normalization, and ReLU activation function. Among them, the main difference between cross attention and self-attention is that cross attention creates an attention map by fusing multi-scale encoder features together instead of utilizing each stage separately, which enables the cross attention mechanism to capture the long-range dependencies between different scales of the encoder stage.

[0094] The present invention refers to SegNext's Multi-Scale Convolutional Attention Network (MSCAN) to extract features. On this basis, the feedforward network is improved to make the decoder channel fusion ability more powerful. The improved residual multi-scale convolutional attention module (RMSCAM) consists of an attention module to build long-distance dependencies, two normalization layers and a redesigned residual feedforward network (Residual FFN), such as Figure 5 As shown. Note that each RMSCAM has 2 inputs, and after concatenating them in the channel dimension, we get the feature X i .

[0095] The Attention module includes two 1×1 convolutions, GeLU activation function and multi-scale convolution attention module. The output of MSCAN at the i-th level of the decoder is expressed as:

[0096] F′ i =Attention(LayerNorm(X i ))+X i (10)

[0097] F i =FFN(LayerNorm(F′ i ))+F′ i (11)

[0098] Among them, X i is the feature of input Attention, F' i is the attention output, F i is the output feature of MSCAN, LayerNorm is layer normalization, MSCAN is the multi-scale convolutional attention network, and FFN is the feed-forward network;

[0099] The multi-scale convolutional attention module consists of three parts: a 5×5 deep convolution for aggregating local information, a multi-branch deep convolution for capturing multi-scale context, and a 1×1 convolution for building relationships between different channels. The output of the 1×1 convolution is used as the attention weight to directly weight the input of MSCA, where the formula of MSCA is:

[0100] Attention(x)=Linear(MSCA(σ(Linear(x))) (12)

[0101]

[0102] Among them, Linear is the linear projection layer, MSCA represents multi-scale convolutional attention, σ is the ReLu activation function, x represents the input feature, It is the multiplication operation of element matrix, DWConv means deep convolution; Scale i , i∈0,1,2,3;

[0103] Next, in each branch, depthwise convolution with two strip-shaped kernels is used to approximate the standard depthwise convolution with a large kernel, and the kernel size of each branch is set to 7, 11, and 21, respectively.

[0104] The main reason for choosing a strip convolution kernel in the present invention is that it is lightweight and has fewer parameters. For example, to simulate a standard 2D convolution with a convolution kernel size of 7×7, the present invention only needs to connect in series a pair of convolutions with kernel sizes of 7×1 and 1×7.

[0105] FFN Figure 6 As shown in (a), the feedforward network FFN is expressed as:

[0106] FFN(x f )=Linear(DWConv 3×3 (σ(Linear(x f )))) (14)

[0107] Among them, x f is the feed-forward network FFN feature input, Linear is the linear projection layer, σ is the ReLu activation function, DWConv 3×3 For depthwise convolution with a kernel size of 3×3, the expansion ratio of the first linear layer in FFN is set to 4 by default.

[0108] The role of the feedforward layer is to fuse channel information after the previous layer of Attention performs global interaction, further enhancing the model's expressiveness. The motivation for improving the feedforward layer in this invention is that through experiments, this invention found that the simple feedforward layer lacks the channel fusion capability after extracting multi-scale features. This invention attempts to add more activations, normalization, and an additional residual connection to improve the traditional FFN layer.

[0109] Residual FFN is expressed as:

[0110] y=BN(σ(Conv 1×1 (x r ))) (15)

[0111] x out =BN(Conv 1×1 (BN(σ(DWConv3×3 (y)+y))))(16)

[0112] Among them, Residual FFN is the residual feedforward network, BN is batch normalization, σ is the ReLu activation function, and x r Residual FFN is the feature input of the residual feedforward network, Conv 1×1 and Conv 3×3 Respectively represent convolutions with kernel sizes of 1×1 and 3×3, x out represents the output of the residual feed-forward network.

[0113] Unlike traditional FFN, the present invention also adds batch normalization after the activation layer and after the last 1×1 convolution. Deep convolution is used to extract local information with almost no additional computational cost. The motivation for inserting jump connections is similar to that of classic residual networks, which can promote the ability of gradients to propagate between layers. Such jump connections help the network converge quickly to obtain better results. The ablation experiment in Table 8 proves that the improved Residual FFN is a better choice.

[0114] The following briefly introduces the experimental dataset and implementation details, and then compares with some state-of-the-art methods to prove the superiority of the technical solution of the present invention. Finally, an ablation experiment is conducted to verify the effectiveness of each module in DBNet.

[0115] 1. Dataset

[0116] Experiments were conducted on three different types of datasets: the Synapse Multi-Organ Segmentation Dataset (Synapse), the Automatic Cardiac Diagnosis Challenge Dataset (ACDC), and the Digital Retinal Image Dataset for Blood Vessel Extraction (DRIVE).

[0117] (1) The Synapse dataset contains 30 abdominal CT scans, a total of 3779 axial abdominal clinical CT images. The present invention follows the data partitioning method of TransUNet and MISSFormer and divides these data into two groups: 18 cases for training and 12 cases for testing. The present invention evaluates 8 abdominal organs (aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas and stomach), and the evaluation indicators use the average Dice similarity coefficient (DSC) and the average Hausdorff distance (HD).

[0118] (2) The ACDC dataset contains 100 MRI scans from different patients, each of which divides the heart into three components: left ventricle (LV), right ventricle (RV), and myocardium (MYO). Consistent with the previous methods TransUNet and MISSForme, 70 cases were used for training, 10 for validation, and 20 for testing, and the average Dice similarity coefficient was used to evaluate the performance of each method.

[0119] (3) The DRIVE dataset comes from a diabetic retinopathy screening project in the Netherlands. The dataset contains 40 color fundus images, each with a resolution of 565×584 pixels. The entire dataset is divided into 20 images for training and 20 for testing, and the first manual annotation is used as the true standard for evaluation. The present invention uses sensitivity (Sen), accuracy (Acc) and area under the ROC curve (AUC) to evaluate the performance of each method in the DRIVE dataset.

[0120] Implementation Details

[0121] DBNet is implemented based on Python 3.6 and Pytorch 1.8.0, and the model training is performed on an Nvidia RTX4090 GPU with 24GB of memory. For all training examples, the present invention uses data enhancement techniques such as flipping, rotation, scaling, Gaussian noise, and contrast transformation to increase data diversity. For both the Synapse and ACDC datasets, the input image size is set to 224×224 or 256×256, and the maximum number of training rounds is 400 rounds. The training batch size of the Synapse dataset is 24, the initial learning rate is 0.05, and the learning rate is gradually reduced using the cosine annealing algorithm. The training batch size of the ACDC dataset is 12, and the initial learning rate is 0.0001. For the DRIVE dataset, a sliding window of 64×64 size with a step size of 6 is used to extract image patches from vascular images to increase the amount of training data. In the training phase, the model parameters are initialized using weights pre-trained on ImageNet, and the popular SGD optimizer is used with momentum set to 0.9 and weight decay set to 1e-4 to optimize the model of the present invention for back propagation.

[0122] 3. Comparison of Methods

[0123] 1) Objective evaluation

[0124] DBNet is compared with the latest medical image segmentation methods. SOTA methods include UNet, TransUNet, Swin-Unet, MISSFormer, CASCADE, G-CASCADE, H2Former, etc. In order to make a fair comparison, this paper uses the indicators provided by the author's paper. It should be noted that if there are multiple versions of the comparison method, this paper will choose to compare with the version with the same training strategy. The indicators are described as follows: DSC (Dice Similariy Coefficient), AUC (Area Under Curve), Sen (Sensitivity), ACC (Accuracy).

[0125] Compared with these models, the method of the present invention achieves the best balance between performance and model complexity. Under the input of 224×224 image size, the method of the present invention achieves the best results in two objective evaluation indicators: average Dice score and HD95. Except for the gallbladder and pancreas, the other methods of the present invention are the best for the segmentation of 8 organ types. Under the input of 256×256 image size, the present invention achieves considerable results with less parameters and floating-point operations. Note that the parameters of CASCADE MERIT and MERIT-CASCADE are 148M and 133M, and the FLOPs are 33M and 26M, and the throughput of both is about 4 times that of the method of the present invention. Table 1 compares the test results of various methods on the multi-organ segmentation dataset (Synapse).

[0126] Table 1 Comparison of the proposed method and the SOTA method on the Synapse dataset

[0127]

[0128] Table 2 shows the test results of these methods on the heart segmentation dataset (ACDC). The method of the present invention achieves the best results with input images of 224×224 and 256×256 sizes. For input images of 256×256 size, the number of parameters and FLOPs of CASCADE MERIT and MERIT-CASCADE is very large, and H2Former achieves a DSC of 92.4 with large-size input images. The method of the present invention achieves the highest DSC with less computation and FLOPs and smaller input size.

[0129] Table 2 Comparison of the proposed method and the SOTA method on the ACDC dataset

[0130]

[0131] Table 3 shows the experimental results of the retinal blood vessel segmentation task (DRIVE). The method of the present invention failed to achieve the optimal performance because FR-UNet is an algorithm designed specifically for the eyeball dataset that is sensitive to thin bloodshot and has a better segmentation effect on strips. The method of the present invention and MISSFormer are both algorithms that are applicable to multi-classification and binary classification and do not perform personalized adaptation to specific datasets. However, it is worth noting that the method of the present invention achieves the optimal performance in terms of Sen value, and perhaps Sen value is more sensitive to the method of the present invention.

[0132] Table 3 Comparison of the proposed method and the SOTA method on the DRIVE dataset

[0133]

[0134] 2) Subjective evaluation

[0135] Figure 7 The segmentation results of a single CT slice and a single 3D image in the Synapse dataset are shown, and a variety of classic medical image segmentation algorithms are compared. Figure 7 (a) GT; Figure 7 (b) DBNet; Figure 7 (c) MISSFormer; Figure 7 (d) PVT-CASCADE; Figure 7 (e) Swin-UNet. By observing the visualization results, it can be seen that the method of the present invention exhibits good segmentation effects. For large organs (such as the left and right kidneys), the present invention can effectively capture the entire organ without confusing the positions of different organs. Even in the face of difficult small organs and heteromorphic organs (such as the aorta and pancreas), the DBNet of the present invention can still accurately identify them. This is due to the double-bridge structure of DBNet, which can encode local features and long-range pixel-level dependencies, making the prediction of the present invention more accurate and superior than other models.

[0136] Figure 8 This is a single segmentation result display in the ACDC dataset. This paper compares a variety of classic medical image segmentation algorithms. Among them, Figure 8 (a) GT; Figure 8 (b) DBNet; Figure 8 (c) MISSFormer; Figure 8 (d) PVT-CASCADE; Figure 8(e) Swin-UNet. By observing the visualization graph, the method of the present invention exhibits excellent recognition ability, indicating that it has outstanding generalization and robustness. Compared with other methods, the method of the present invention can greatly reduce the number of false positive predictions. A typical example is the second row on the ACDC dataset. It can be seen that Swin-UNet produces a large number of erroneous right ventricle pixels outside the myocardium. In contrast, the method of the present invention does not produce predictions for the right ventricle outside the myocardium, which shows that DBNet is more discriminative for heart segmentation than Swin-UNet. Fig. 9 The segmentation results in the DRIVE dataset are shown, showing the performance of DBNet in the binary segmentation task. Fig. 9 (a) GT; Fig. 9 (b) DBNet; Fig. 9 (c) MISSFormer; Fig. 9 (d) FR-UNet. The method of the present invention achieves very competitive performance compared with other methods. DBNet can focus on more details of retinal vascular images, thanks to the single-scale bridge for local feature extraction. Fig.10 This is the visualization result of some feature maps of the model on the Synapse dataset. Through the continuous changes in the feature maps, it can be seen that the single-scale bridge retains the organ boundary information, and the multi-scale bridge obtains a large receptive field for organ localization through aggressive downsampling and multi-scale feature interaction. In this way, the bridge module integrates local and global feature information, and then passes it to the decoder for subsequent processing.

[0137] Fig.10 Feature map visualization. Fig.10 (a) Feature map of backbone stage 1; Fig.10 (b) Single-scale bridge feature map of the first stage; Fig.10 (c) Multi-scale bridge feature map in the first stage; Fig.10 (d) Feature map after double bridge feature splicing; Fig.10 (e) Feature map after dual-bridge feature fusion; Fig.10 (f) Output feature map of the decoder at the last stage; Fig.10 (g) True value.

[0138] 4. Ablation Experiment

[0139] In order to verify the effectiveness of the basic components in the method of the present invention, a series of ablation experiments were conducted on the Synapse and ACDC datasets. The present invention uses a simple U-shaped network without a bridge module based on the Dual-ViT-S backbone as a baseline method. For fair comparison, the decoder block of the present invention is the basic block MSCAN of SegNext, and the number of feature extraction blocks per stage is set to 4. Consistent with other methods, DSC in the evaluation index is the most important indicator of the present invention. All experiments of the present invention are carried out under the same hyperparameter settings.

[0140] 1) Double bridge ablation experiment

[0141] The effects of multi-scale bridges and single-scale bridges on model performance are shown in Table 4. On all datasets, the combination of SCAM and CAM generally outperforms other baselines, which illustrates the effectiveness of the combination of the two modules. The research results of the present invention show that the fusion of multi-scale local features and global features is very important for improving segmentation performance in segmentation tasks.

[0142] Table 4 The impact of multi-scale bridges and single-scale bridges on model performance. S-Bridge represents a single-scale bridge, and M-Bridge represents a multi-scale bridge.

[0143]

[0144] 2) Ablation experiment of multi-scale bridge

[0145] In order to verify the effectiveness of the multi-scale bridge designed by the present invention. The present invention studies the application of multi-scale bridges in feature maps of different scales, as shown in Table 5, from the deepest 4th layer (size 7×7) to the shallowest first layer (56×56), and the impact of the size of the patch on the performance of the entire model. Experiments show that when the patch size is 7 and all stages (Stage1 to Stage4) use the SCA module for global semantic feature extraction, the model achieves optimal performance. It can be seen that the joint learning of features from all stages is beneficial to the model.

[0146] Table 5 Effects of different multi-scale bridge structures on model performance

[0147]

[0148] 3) Ablation experiment of single-scale bridge

[0149] In this part of the present invention, the present invention ablates two different branches in the proposed single-scale bridge, including branch one of the present invention: detail enhancement convolution kernel (DEK) and branch two: compressed axial attention (SAA).

[0150] The present invention experiments with several settings, including using only detail enhancement convolution kernels, using only compressed axial attention, and a combination of the two. The impact of different single-scale bridge structures on model performance is shown in Table 6. The performance of using only detail enhancement convolution kernels or compressed axial attention is relatively poor, while combining compressed axial attention with detail enhancement kernels can improve performance. The results show that enhancing the global semantic features from compressed axial attention to guide convolution to obtain local details can optimize the model's ability to extract detail features.

[0151] Table 6 Effects of different single-scale bridge structures on model performance

[0152]

[0153] 4) Decoder Ablation Experiment

[0154] Regarding the selection of the number of decoder stacking layers, the present invention designs decoders with different numbers of layers. Experiments have found that when the number of layers is small, the model is shallow and it is difficult to see useful features; when the number of layers is large, the model is deep, the convergence speed is slow, and training is difficult. In order to balance performance and parameter quantity, the present invention selects an MSCA block stacked 4 times as a decoder stage. The decoder of the present invention consists of 3 stages in total. Table 7 shows the impact of different numbers of layers on the performance of the entire model in the Synapse and ACDC datasets. Table 8 shows the impact of different feedforward networks on the performance of the entire model in the Synapse and ACDC datasets. The Residual FFN redesigned by the present invention has a DSC increase from 84.03 to 84.77 on the Synapse dataset with almost no increase in the number of parameters. It can be seen that the method of the present invention is better than the original FFN in aggregating channel features.

[0155] Table 7 Effect of different decoder layers on model performance

[0156]

[0157] Table 8 Effect of different feedforward layers on model performance

[0158]

[0159] 5) Ablation experiment of backbone network

[0160] In order to verify the adaptability and robustness of the dual-bridge architecture and decoder of the present invention to different backbone networks, the present invention conducted two sets of experiments, using PVTv2_b2 and Dual-ViT-s as backbone networks respectively. The experimental results are shown in Table 9. As can be seen from Table 9, the network of the present invention can adapt to different types of backbone networks, and achieves considerable performance based on these two backbone networks.

[0161] Table 9 The impact of different backbone networks on model performance

[0162]

[0163] V. Conclusion

[0164] The present invention proposes a medical image segmentation algorithm called DBNet. The dual-bridge structure proposed in the present invention is more effective than the bridging structure of previous methods while maintaining a small number of parameters. The multi-scale bridge uses spatial cross-attention to effectively compress the token vector into a global semantic with lower complexity for multi-scale information interaction and achieve rough organ positioning; the single-scale bridge learns finer local pixel-level details through convolution to achieve accurate boundary segmentation. The parallel dual-bridge structure propagates semantic information and detail information respectively, and both types of information are indispensable for the segmentation task. The redesigned feedforward layer in the decoder can more effectively aggregate channel information and enhance the feature representation of attention. The present invention evaluated the method of the present invention on three data sets, and the excellent results demonstrated the effectiveness and robustness of DBNet.

[0165] The above-described embodiments of the present invention do not limit the protection scope of the present invention.

Claims

1. A U-shaped medical image segmentation network with a double bridge paradigm, characterized in that: include: An encoder, a dual-bridge architecture, and a decoder; the encoder adopts a backbone network, the dual-bridge architecture includes a multi-scale bridge and a single-scale bridge, the multi-scale bridge adopts SCAM, the single-scale bridge adopts CAM, and the decoder adopts RMSCAM; The encoder is used to extract multi-level image features from a single-channel grayscale medical image, and the features extracted by the four stages of the encoder are fed into a dual-bridge architecture; The dual-bridge architecture is used to receive features extracted from four stages of the encoder through a multi-scale bridge to capture global features; The features extracted by a single stage of the encoder are received through a single-scale bridge to capture local features; Adding and fusing the global features with the local features to obtain a feature map, and sending the feature map to a multi-stage decoder; The decoder is used to restore the feature map to obtain a final prediction map.

2. A U-shaped medical image segmentation network of double bridge paradigm as claimed in claim 1, characterized in that: The single-scale bridge adopts a double-branch structure, including branch one and branch two; The branch 1 is used to obtain q, k, v from the input x, and pass them to a block consisting of a deep convolution with a kernel size of 3×3, batch normalization, and an activation function after splicing along the channel dimension; by using 3×3 convolution, local details are aggregated from q, k, v, where q is a table query, k represents a key, and v represents a value; point-by-point convolution with batch normalization is used to compress the channel dimension to the input dimension C and generate detail features; The second branch is used to perform global semantic extraction through axial attention, and then multiply and fuse the detail features obtained from the first branch with the features extracted by the axial attention of the second branch.

3. A U-shaped medical image segmentation network of double bridge paradigm as claimed in claim 2, characterized in that: The formula of the branch one is as follows: F0=concat(q(x),k(x),v(x)) (1) F1=BN(PWConv(ReLU(BN(DWConv(F0))))) (2) Among them, F0 is the concatenated feature, concat represents feature concatenation, x is the input, q represents query, k represents the key, v represents the value, F1 is the detail feature obtained by branch one, BN represents batch normalization, PWConv represents point-by-point convolution, ReLU represents the activation function, and DWConv represents depth convolution.

4. A U-shaped medical image segmentation network of double bridge paradigm as claimed in claim 3, characterized in that: The formula of the second branch is as follows: Among them, 1 W is a vector of size (W,1) with all 1s. H is a vector of size (H, 1) with all 1s, T represents transpose, Q h The width W of the feature map q(x) is compressed to 1 by horizontal compression, Q w The height H of the feature map q(x) is compressed to 1 through longitudinal compression; F2 is the feature extracted by axial attention in branch 2; the above compression method is used to compress k and v to obtain K h 、V h , K w and V w .

5. A U-shaped medical image segmentation network of double bridge paradigm as claimed in claim 4, characterized in that: The formula used to multiply and fuse the detail features obtained from branch one with the features extracted from the axial attention of branch two is as follows: Among them, F out It is the multiplication and fusion result of F1 and F2. F1 is the detail feature obtained by branch one, and F2 is the feature extracted by axial attention in branch two. Represents the multiplication of corresponding matrix elements.

6. A U-shaped medical image segmentation network of double bridge paradigm as claimed in claim 1, characterized in that: The method for capturing global features by the multi-scale bridge includes: First, we get image patches from the four stages of the encoder, given encoder features at different scales Use image blocks of different sizes: H is the width, W is the height, and C is the number of channels; the pooling size and stride are both P i The adaptive average pooling method is used to extract image patches, and then a deep convolution with a kernel size of 1×1 is used on the flat 2D image patches, while the number of feature map channels remains unchanged. For T i ∈N×Ci(i=1,2,3), perform layer normalization and concatenation along the channel dimension; use the concatenated vector Use each tag as both query and key as values; use depthwise convolution and pointwise convolutional projection on query, key, and value: Q=σ(BN(PW(DW(T Σ )))) (6) K=σ(BN(PW(DW(T Σ )))) (7) V i =σ(BN(PW(DW(T i )))) (8) Where Q, K, Vi are the query, key, and value of the projection respectively; DW and PW represent depthwise convolution and pointwise convolution respectively; BN represents batch normalization; σ represents the ReLu activation function; T i (i=1,2,3,4) refers to the four-level features from the encoder, T Σ It refers to the new features obtained by concatenating and reducing the dimension of the above four-level features; SCA is expressed as: Among them, d k is the scaling factor; for the long case h is the number of heads; The outputs of SCAM are projected using depthwise convolution and pointwise convolution to form the final output; layer normalization and GeLU are applied to these outputs; finally, the multiple outputs of SCA are connected to the corresponding decoder part through an upsampling layer, followed by a depthwise convolution with a kernel size of 1×1, batch normalization, and a ReLU activation function.

7. A U-shaped medical image segmentation network of double bridge paradigm as claimed in claim 1, characterized in that: The decoder includes an Attention module, two normalization layers, and a residual feedforward network; The Attention module includes two 1×1 convolutions, GeLU activation function and multi-scale convolution attention module. The output of MSCAN at the i-th level of the decoder is expressed as: F′ i =Attention(LayerNorm(X i ))+X i (10) F i =FFN(LayerNorm(F′ i ))+F′ i (11) Among them, X i is the feature of input Attention, F' i is the attention output, F i is the output feature of MSCAN, LayerNorm is layer normalization, MSCAN is the multi-scale convolutional attention network, and FFN is the feed-forward network; The multi-scale convolutional attention module consists of three parts: a 5×5 deep convolution for aggregating local information, a multi-branch deep convolution for capturing multi-scale context, and a 1×1 convolution for building relationships between different channels. The output of the 1×1 convolution is used as the attention weight to directly weight the input of MSCA, where the formula of MSCA is: Attention(x)=Linear(MSCA(σ(Linear(x))) (12) Among them, Linear is the linear projection layer, MSCA represents multi-scale convolutional attention, σ is the ReLu activation function, x represents the input feature, It is the multiplication operation of element matrix, DWConv means deep convolution; Scale i , i∈0,1,2,3; Next, in each branch, depthwise convolution with two strip-shaped convolution kernels is used to approximate the standard depthwise convolution with a large kernel, and the kernel size of each branch is set to 7, 11, and 21 respectively; The feed-forward network FFN is expressed as: FFN(x f )=Linear(DWConv 3×3 (σ(Linear(x f )))) (14) Among them, x f is the feed-forward network FFN feature input, Linear is the linear projection layer, σ is the ReLu activation function, DWConv 3×3 For depthwise convolution with a kernel size of 3×3, the expansion ratio of the first linear layer in FFN is set to 4 by default; Residual FFN is expressed as: and=BN(σ(Conv 1×1 (x r ))) (15) x out =BN(Conv 1×1 (BN(σ(DWConv 3×3 (y)+y))))(16) Among them, Residual FFN is the residual feedforward network, BN is batch normalization, σ is the ReLu activation function, and x r Residual FFN is the feature input of the residual feedforward network, Conv 1×1 and Conv 3×3 Respectively represent convolutions with kernel sizes of 1×1 and 3×3, x out represents the output of the residual feed-forward network.

Images