Method for segmenting medical image based on multi-branch attention network

Abstract

The invention provides a method for segmenting a medical image based on a multi-branch attention network. The method comprises the following steps: acquiring a medical image data set; dividing the preprocessed data set into a training set and a test set; building a multi-branch attention network model; the model comprises an encoder part, a Feature Fusion Bridge part and a decoder part, each layer of encoder is composed of N combination units which are connected in series and a Coordinate Attention; each layer of decoder is composed of two Efficiency Transformer Blocks and a Patch Expanding, wherein the two Efficiency Transformer Blocks and the Patch Expanding are connected in series in sequence; inputting the training set into the model for training; and inputting the test set into the trained model to obtain a segmentation result. According to the segmentation method, global and local feature fusion is enhanced, the calculation complexity is effectively reduced, and the robustness of the model is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a method for segmenting medical images based on a multi-branch attention network. Background Art

[0002] Medical image segmentation is a key task in medical image analysis. It plays an important role in auxiliary diagnosis, surgical navigation, radiotherapy and other fields. Traditional medical image segmentation methods mainly rely on classical image processing techniques based on threshold, region growing or edge detection. These methods usually require deep involvement of experts, are difficult to adapt to different types of medical images and complex pathological features, and have poor robustness to noise and deformation.

[0003] With the rapid development of deep learning technology, convolutional neural network (CNN) has become one of the mainstream methods for medical image segmentation. U-Net is a typical CNN architecture that uses an encoder-decoder structure and connects the encoder and decoder through skip connections, achieving good segmentation results. However, the CNN architecture is limited by the local receptive field and is difficult to effectively capture long-distance dependencies, resulting in decreased segmentation accuracy in complex scenarios.

[0004] In order to solve the above problems caused by the defects of CNN architecture, Transformer-based models have gradually emerged in medical image segmentation in recent years. Models such as Vision Transformer (ViT) capture the global information of the image through the self-attention mechanism, making up for the shortcomings of CNN. In medical image segmentation, such models can effectively identify the boundaries and regions of the target. However, the Transformer structure usually requires a large amount of data for training, and due to the lack of local feature extraction capabilities, it may not be able to accurately capture detailed information, especially when the medical data set is small and complex, and the performance is subject to certain limitations.

[0005] In addition, some hybrid models try to combine the advantages of CNN and Transformer to achieve the fusion of global and local information through different strategies. For example, Swin Transformer introduces a hierarchical structure and extracts features at different scales through a sliding window attention mechanism. However, the performance of its model on small sample data sets still needs to be improved.

[0006] The above existing technologies have made some progress in medical image segmentation, but they still have the following limitations:

[0007] (1) Insufficient fusion of global and local information: Existing methods usually focus on the extraction of global or local information, resulting in insufficient feature fusion and affecting segmentation accuracy.

[0008] (2) High computational complexity: Especially in Transformer-based methods, the computational requirements of the attention mechanism significantly increase the model complexity and resource consumption.

[0009] (3) Insufficient adaptability to small sample datasets: Many existing methods perform poorly on small or complex medical datasets because the lack of training data makes it difficult for the model to fully learn features.

[0010] (4) Lack of robustness: Existing methods have poor segmentation effects on noise, image deformation, and irregular targets, which limits their performance in practical applications. Summary of the invention

[0011] The present invention aims to solve the problems of insufficient fusion of global and local information, high computational complexity, lack of robustness, etc. in existing medical image segmentation technology.

[0012] To this end, the purpose of the present invention is to propose a method for segmenting medical images based on a multi-branch attention network.

[0013] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a method for segmenting medical images based on a multi-branch attention network, the method comprising: step S1: obtaining a medical image dataset; step S2: preprocessing the medical image dataset; step S3: dividing the preprocessed medical image dataset into a training set and a test set according to a preset ratio; step S4: building a multi-branch attention network model; wherein the multi-branch attention network model is a U-shaped architecture, and the multi-branch attention network model comprises: an encoder part, a Feature Fusion Bridge, and a decoder part; the encoder part and the decoder part are both four-layered; the encoder part also includes a PatchEmbedding module; the Feature Fusion Bridge is arranged between the encoder part and the decoder part; the output end of each layer of the encoder is respectively connected to the input end of the Feature Fusion Bridge; the output end of each layer of the encoder of the first layer, the second layer and the third layer is respectively connected to the input end of the encoder of the corresponding next layer; the FeatureFusion The output end of the Bridge is connected to the input end of each layer of the decoder; the output end of each layer of the decoder of the first layer, the second layer and the third layer is respectively connected to the input end of the decoder of the corresponding next layer; the output end of the fourth layer of the decoder outputs the segmentation result corresponding to the training set or the test set; each layer of the encoder is composed of N combination units connected in series and a CoordinateAttention, and the combination unit includes a VFT Block and a ConvFNN connected in series; each layer of the decoder is composed of a first Efficient Transformer Block, a second Efficient Transformer Block, and a Patch Expanding connected in series; the Feature Fusion Bridge includes: a first DoubleBridge, a Mix Block, and a second Double Bridge connected in series; the first Double Bridge includes: a first Efficient Transformer Block and a second Efficient Transformer Block connected in series; the second DoubleBridge includes: a third Efficient Transformer Block and a fourth Efficient Transformer Block connected in series; the Mix Block is used to perform feature fusion on the outputs of the first Efficient Transformer Block and the second Efficient Transformer Block;The Patch Embedding module is used to divide the received preprocessed training set or test set into a number of patches of equal size, and then input the patches of equal size into the first layer encoder; the first layer encoder is used to extract features and downsample the input patches of equal size, output the feature map obtained after downsampling, and then input the feature map obtained after downsampling into the second layer encoder; the second and third layer encoders are used to receive the feature map output by the previous layer encoder, perform feature learning and downsampling on the received feature map, output the feature map obtained after downsampling, and then input the feature map obtained after downsampling into the encoder of the next layer; the fourth layer encoder is used to receive the feature map sent by the encoder of the previous layer, perform feature learning and downsampling on the received feature map, and output the feature map obtained after downsampling; the Feature Fusion Bridge is used to perform multi-scale feature fusion on the feature maps output by the encoders of each layer, output the fused feature map, and then input the fused feature map into each layer decoder;

[0014] The first layer decoder is used to receive the fused feature map, upsample the fused feature map to restore the resolution of the image, and output the upsampled feature map; the second, third, and fourth layer decoders are used to upsample the feature map output by the decoder of the previous layer to restore the resolution of the image, and concatenate and fuse it with the feature map output by the corresponding Feature Fusion Bridge to obtain the corresponding feature map; wherein the fourth layer decoder outputs the segmentation result corresponding to the training set or the test set through a Segmentation Head; step S5: input the training set into the multi-branch attention network model for training; step S6: input the test set into the trained multi-branch attention network model to obtain the segmentation result corresponding to the test set.

[0015] Preferably, each VFT Block includes: a Laynorm layer, a Maxpool Global Branch, an Avgpool Global Branch, and a Local Branch; the Laynorm layer is used to perform layer normalization operation on its own input features to output feature C1, and then obtain tensors Q, K and V through the fully connected layer FC; the MaxpoolGlobal Branch, Avgpool Global Branch, and Local Branch are used to perform feature extraction respectively to obtain features Attn respectively. maxglobal , FeatureAttn avgglobal , and feature X local; Wherein, the expression of the calculation process of attention performed by the Maxpool Global Branch is as follows:

[0016] (Q,K,V)=FC(X in )

[0017] K MG =MaxPool(K)

[0018] V MG =MaxPool(V)

[0019]

[0020] In formula (1), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents the maximum pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Maxpool Global Branch maxglobal ;

[0021] The expression of the attention calculation process of the Avgpool Global Branch is as follows:

[0022] (Q,K,V)=FC(X in )

[0023] K AG =AvgPool(K)

[0024] V AG =MaxPool(V)

[0025]

[0026] In formula (2), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents maximum pooling; AvgPool represents average pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Avgpool Global Branch avgglobal ;

[0027] The expression of the Local Branch's attention calculation process is as follows:

[0028] (Q,K,V)=FC(X in )

[0029] V l =DWconv(V)

[0030] Q l =DWconv(Q)

[0031] K l =DWconv(K)

[0032]

[0033] Attn″=FC(Attn′)

[0034]

[0035] In formula (3), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; DWConv represents a depth-separable convolution operation; represents Hardmard multiplication; Laynorm represents layer normalization operation; d represents the number of dimensions of the tensor K; Tanh represents the Tanh activation function;

[0036] After the Maxpool Global Branch, Avgpool Global Branch and Local Branch complete feature extraction, the feature Attn maxglobal , FeatureAttn avgglobal , and feature X local Make stitching.

[0037] Preferably, the ConvFNN is used to fuse the diversified features extracted by the VFT Block, and to perform downsampling when necessary, and to perform downsampling by adjusting the convolution step size; when downsampling is not required, the ConvFNN specifically includes: a first normalization layer, a first convolution, a GELU activation function, a first depth-separable convolution, and a second convolution connected in sequence; wherein the input end of the first normalization layer LN and the output end of the second convolution are connected through a residual connection to obtain the feature map output by the ConvFNN; when downsampling is required, the ConvFNN specifically includes: a main branch and a residual branch; the main branch includes: a second normalization layer, a third convolution, a GELU activation function, a second depth-separable convolution, and a fourth convolution connected in sequence; the residual branch includes: a third depth-separable convolution, a GELU activation function, and a fifth convolution connected in sequence; wherein the input ends of the second normalization layer and the third depth-separable convolution are connected, and the output ends of the fourth convolution and the fifth convolution are connected through a residual connection to obtain the feature map output by the ConvFNN.

[0038] Preferably, the Coordinate Attention is used to extract the position information of the input feature; and the Coordinate Attention is specifically used to: perform average pooling operations on the input features along the horizontal direction X and the vertical direction Y, respectively, and generate feature maps of dimensions C×H×1 and C×1×W in the horizontal direction X and the vertical direction Y, respectively. and Among them, the feature map and The expressions are as follows:

[0039]

[0040] In formula (4), It means summing the hth row on the cth channel; It means summing the wth column on the cth channel;

[0041] Mapping the feature and Perform a concatenation operation to generate a feature map of size C×(H+W)×1; use the sixth convolution to reduce the number of channels C to C / r, and obtain a feature map f of dimension C / r×(H+W)×1; wherein the expression of the feature map f is as follows:

[0042]

[0043] In formula (5), conv represents the convolution operation of the sixth convolution; concat represents the concatenation operation;

[0044] Perform spatial segmentation on the feature map f to obtain f h and f w , the process is expressed as follows:

[0045] [f h ,f w ]=Split(f)(6)

[0046] In formula (6), Split represents the split operation; f h ∈R C / r×H×1 ;f w ∈R C / r×1×W ;

[0047] f h and f w Apply 1×1 convolution to restore the channel dimension C / r to C, and use the sigmoid activation function to get g h and g w , the process is expressed as follows:

[0048] [g h ,g w ])=δ(conv2[f h ,f w ])(7)

[0049] In formula (7), δ represents the sigmoid activation function; g h ∈R C×H×1 ; g w ∈R C×1×W ;

[0050] Using g h and g w The input feature x is weighted, and the process is expressed as follows:

[0051]

[0052] In formula (8), x c (i,j) indicates the input feature x in the i-th row and j-th column of the c-th channel; Indicates g h At the i-th position of the c-th channel; Indicates g w At the jth position of the cth channel.

[0053] Preferably, the process of multi-scale feature fusion performed by the Feature Fusion Bridge specifically includes: flattening and reshaping the outputs F1, F2, F3, and F4 of the four-layer encoder into a feature tensor X with a dimension of (B, -1, C) input , the process is expressed as follows:

[0054] X input =Reshape(Concate(F1,F2,F3,F4),[B,-1,C])(9)

[0055] In formula (9), Reshape represents tensor reshaping; concat represents concatenation operation; B represents the size of each batch; C represents the number of channels of the feature; F1, F2, F3, and F4 represent the features output by the four-layer encoder respectively;

[0056] The feature tensor X input Input to the first Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0057] Attn1 = Efficient_Attn1 (X input )(10)

[0058] In formula (10), Attn1 represents the feature map output by the first Efficient Transformer Block;

[0059] Attn1 is input to the second Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0060] Attn2=Efficient_Attn2(Attn1)(11)

[0061] In formula (11), Attn2 represents the feature map output by the second Efficient Transformer Block;

[0062] The features of Attn1 and Attn2 are fused by the Mix Block, and the process is shown as follows:

[0063] Attn mix =Mix_block(Attn1,Attn2)(12)

[0064] In formula (12), Attn mix Represents the feature map output by Mix Block;

[0065] Attn mix Input to the third Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0066] Attn3=Efficient_Attn3(Attn mix )(13)

[0067] In formula (13), Attn3 represents the feature map output by the third Efficient Transformer Block;

[0068] Attn3 is input to the fourth Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0069] X output =Efficient_Attn4(Attn3)(14)

[0070] In formula (14), X output Represents the feature map output by the fourth Efficient Transformer Block.

[0071] Preferably, the process of performing attention calculation by the first Efficient Transformer Block, the second Efficient Transformer Block, the third Efficient Transformer Block, and the fourth Efficient Transformer Block all includes: reshaping the dimension of the tensor K output by the fully connected layer FC into N / r×rC, and then projecting it into rC×C through a linear layer. The process is expressed as follows:

[0072]

[0073] The dimension of the tensor V output by the fully connected layer FC is reshaped into N / r×rC, and then projected into rC×C through the linear layer. The process is expressed as follows:

[0074]

[0075] In formula (15) and formula (16), X in Represents input features; FC represents a fully connected layer; Reshape represents a reshaping function; Project represents a linear layer.

[0076] Preferably, the Mix Block includes: a concatenation module Concat, a feedforward neural network FNN, a third normalization layer and an activation function GELU connected in series in sequence; the concatenation module Concat is used to perform feature concatenation on the outputs of the first Efficient Transformer Block and the second Efficient Transformer Block along the channel dimension to obtain a feature map T1 with a channel dimension of 2C; the feedforward neural network FNN is used to use a linear layer to project the number of channels of the feature map T to C to obtain a feature map T2; the third normalization layer is used to perform a normalization operation on the feature map T2, so that the feature map after the normalization operation can be subsequently nonlinearly enhanced through the activation function GELU.

[0077] Preferably, the step S2 specifically includes: step S2.1: adjusting the pixel size of each medical image in the medical image data set to 224×224; step S2.2: normalizing each medical image after the pixel size is adjusted; step S2.3: performing data enhancement on each medical image after the normalization.

[0078] Beneficial effects of the present invention:

[0079] (1) The method for segmenting medical images based on a multi-branch attention network provided by the present invention is a method for segmenting medical images based on an improved Transformer architecture. Specifically, through the strategy of combining an improved multi-branch attention module VFT Block and a feature fusion bridge Feature Fusion Bridge, global information and local details are effectively extracted and fused, thereby achieving the purpose of enhancing the fusion of global and local features, thereby greatly improving the segmentation accuracy of the network model.

[0080] (2) The method for segmenting medical images based on a multi-branch attention network provided by the present invention achieves the purpose of reducing the demand for computing resources by optimizing the attention mechanism of Transformer and downsampling the tensors K and V, thereby effectively reducing the computational complexity.

[0081] (3) The method for medical image segmentation based on a multi-branch attention network provided by the present invention enhances the robustness of the model through data enhancement strategies for image noise, deformation, and irregular targets, thereby greatly improving the robustness of the network model.

[0082] Additional aspects and advantages of the invention will become apparent from the following description, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic flow chart of a method for segmenting a medical image based on a multi-branch attention network according to an embodiment of the present invention is shown;

[0084] Figure 2 A schematic diagram showing the overall structure of a multi-branch attention network model according to an embodiment of the present invention is shown;

[0085] Figure 3 A schematic diagram showing the structure of a VFT Block according to an embodiment of the present invention is shown;

[0086] Figure 4 (a) shows a schematic diagram of the structure of a ConvFNN when downsampling is not required according to an embodiment of the present invention;

[0087] Figure 4 (b) shows a schematic diagram of the structure of a ConvFNN when downsampling is required according to an embodiment of the present invention;

[0088] Figure 5 A schematic diagram showing the structure of Coordinate Attention according to an embodiment of the present invention is shown;

[0089] Figure 6 A schematic diagram showing the structure of a Feature Fusion Bridge according to an embodiment of the present invention is shown;

[0090] Figure 7 A schematic diagram showing the structure of a first Efficient transformer according to an embodiment of the present invention is shown;

[0091] Figure 8 A schematic diagram showing the comparison between the segmentation results and the true labels on the test set of the Synapse dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0093] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0094] Figure 1 FIG. 1 is a schematic flow chart of a method for segmenting medical images based on a multi-branch attention network according to an embodiment of the present invention. Figure 1 As shown, the method for segmenting medical images based on a multi-branch attention network includes:

[0095] Step S1: Obtain a medical image dataset;

[0096] Step S2: preprocessing the medical image dataset;

[0097] Step S3: Divide the preprocessed medical image dataset into a training set and a test set according to a preset ratio;

[0098] Step S4: Build a multi-branch attention network model;

[0099] Step S5: input the training set into the multi-branch attention network model for training;

[0100] Step S6: Input the test set into the trained multi-branch attention network model to obtain the segmentation result corresponding to the test set.

[0101] In this embodiment, if Figure 2As shown, the multi-branch attention network model is a U-shaped architecture, and the multi-branch attention network model includes: an encoder part, a Feature Fusion Bridge, and a decoder part; the encoder part and the decoder part are both four layers; the encoder part also includes a Patch Embedding module; the Feature Fusion Bridge is arranged between the encoder part and the decoder part; the output end of each layer of the encoder is respectively connected to the input end of the Feature Fusion Bridge; the output end of each layer of the encoder of the first layer, the second layer and the third layer is respectively connected to the input end of the encoder of the corresponding next layer; the output end of the Feature Fusion Bridge is connected to the input end of each layer of the decoder; the output end of each layer of the decoder of the first layer, the second layer and the third layer is respectively connected to the input end of the decoder of the corresponding next layer; the output end of the fourth layer of the decoder outputs the segmentation result corresponding to the training set or the test set; each layer of the encoder is composed of N combination units connected in series and a Coordinate Attention, and the combination unit includes a VFT Block and a ConvFNN connected in series; each layer of the decoder is composed of a first Efficient Transformer Block, a second Efficient Transformer Block, and a Patch Expanding connected in series; the Feature Fusion The Bridge includes: a first Double Bridge, a Mix Block, and a second Double Bridge connected in series in sequence; the first Double Bridge includes: a first Efficient Transformer Block and a second Efficient Transformer Block connected in series; the second Double Bridge includes: a third Efficient Transformer Block and a fourth Efficient Transformer Block connected in series; the Mix Block is used to perform feature fusion on the outputs of the first Efficient Transformer Block and the second Efficient Transformer Block.

[0102] In this embodiment, the Patch Embedding module is used to divide the received preprocessed training set or test set into a number of patches of equal size, and then input the patches of equal size into the first layer encoder; the first layer encoder is used to extract features and downsample the input patches of equal size, output the feature map obtained after downsampling, and then input the feature map obtained after downsampling into the second layer encoder; the second and third layer encoders are used to receive the feature map output by the previous layer encoder, and perform feature learning and downsampling on the received feature map, output the feature map obtained after downsampling, and then input the feature map obtained after downsampling into the next layer encoder; the fourth layer encoder is used to receive the feature map sent by the previous layer encoder, and perform feature learning and downsampling on the received feature map, and output the feature map obtained after downsampling; Feature Fusion Bridge, used to perform multi-scale feature fusion on the feature maps output by encoders of each layer, output the fused feature maps, and then input the fused feature maps to each layer of decoders; the first layer decoder, used to receive the fused feature maps, upsample the fused feature maps to restore the resolution of the image, and output the upsampled feature maps; the second, third, and fourth layer decoders, used to upsample the feature maps output by the decoder of the previous layer to restore the resolution of the image, and concatenate and fuse them with the feature maps output by the corresponding Feature Fusion Bridge to obtain the corresponding feature maps; wherein the fourth layer decoder outputs the segmentation results corresponding to the training set or the test set through a Segmentation Head.

[0103] In a specific embodiment, the medical image dataset is the Synapse dataset or the ISIC 2018 dataset. Among them, the Synapse dataset is a widely used and highly authoritative multi-organ segmentation dataset in the field of medical image segmentation, covering multiple organs in the abdomen, including the left kidney, right kidney, gallbladder, pancreas, liver, aorta, spleen and stomach. The Synapse dataset contains abdominal CT scan data of 30 patients, which is highly diverse, and each image is manually annotated by authoritative imaging experts. The ISIC 2018 dataset is widely used in the detection and diagnosis of skin lesions, focusing on skin lesion segmentation tasks. The ISIC 2018 dataset consists of thousands of skin lesion images, each of which provides detailed manual annotations, covering a variety of types of skin lesions.

[0104] In one embodiment of the present invention, Figure 3As shown, each VFT Block includes: a Laynorm layer, a Maxpool Global Branch, an Avgpool Global Branch, and a Local Branch;

[0105] The Laynorm layer is used to perform layer normalization operation on its own input features to output feature C1, and then obtain tensors Q, K and V through the fully connected layer FC;

[0106] The Maxpool Global Branch, Avgpool Global Branch, and Local Branch are used to extract features respectively to obtain features Attn maxglobal , FeatureAttn avgglobal , and feature X local ;

[0107] The expression of the attention calculation process of the Maxpool Global Branch is as follows:

[0108] (Q,K,V)=FC(X in )

[0109] K MG =MaxPool(K)

[0110] V MG =MaxPool(V)

[0111]

[0112] In formula (1), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents the maximum pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Maxpool Global Branch maxglobal ;

[0113] The expression of the attention calculation process of the Avgpool Global Branch is as follows:

[0114] (Q,K,V)=FC(X in )

[0115] K AG=AvgPool(K)

[0116] V AG =MaxPool(V)

[0117]

[0118] In formula (2), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents maximum pooling; AvgPool represents average pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Avgpool Global Branch avgglobal ;

[0119] The expression of the Local Branch's attention calculation process is as follows:

[0120] (Q,K,V)=FC(X in )

[0121] V l =DWconv(V)

[0122] Q l =DWconv(Q)

[0123] K l =DWconv(K)

[0124]

[0125] Attn″=FC(Attn′)

[0126]

[0127] In formula (3), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; DWConv represents a depth-separable convolution operation; represents Hardmard multiplication; Laynorm represents layer normalization operation; d represents the number of dimensions of the tensor K; Tanh represents the Tanh activation function;

[0128] After the Maxpool Global Branch, Avgpool Global Branch and Local Branch complete feature extraction, the feature Attnmaxglobal , FeatureAttn avgglobal , and feature X local Make stitching.

[0129] In one embodiment of the present invention, the ConvFNN is used to fuse the diverse features extracted by the VFT Block and perform downsampling when necessary by adjusting the convolution step size; Figure 4 As shown in (a), when downsampling is not required, the ConvFNN specifically includes: a first normalization layer, a first convolution, a GELU activation function, a first depth-separable convolution, and a second convolution connected in sequence; wherein the input end of the first normalization layer LN and the output end of the second convolution are connected through a residual connection to obtain a feature map output by the ConvFNN; when downsampling is required, as Figure 4 As shown in (b), the ConvFNN specifically includes: a main branch and a residual branch; the main branch includes: a second normalized layer, a third convolution, a GELU activation function, a second depth-wise separable convolution, and a fourth convolution connected in sequence; the residual branch includes: a third depth-wise separable convolution, a GELU activation function, and a fifth convolution connected in sequence; wherein the input ends of the second normalized layer and the third depth-wise separable convolution are connected, and the output ends of the fourth convolution and the fifth convolution are connected through a residual connection to obtain a feature map output by the ConvFNN.

[0130] In one embodiment of the present invention, Figure 5 As shown, the Coordinate Attention is used to extract the position information of the input feature; and the Coordinate Attention is specifically used to: perform average pooling operations on the input features along the horizontal direction X and the vertical direction Y, respectively, and generate feature maps of dimensions C×H×1 and C×1×W in the horizontal direction X and the vertical direction Y, respectively. and Among them, the feature map and The expressions are as follows:

[0131]

[0132] In formula (4), It means summing the hth row on the cth channel; It means summing the wth column on the cth channel;

[0133] Mapping the feature and Perform a concatenation operation to generate a feature map of size C×(H+W)×1;

[0134] The sixth convolution is used to reduce the number of channels C to C / r, and a feature map f with a dimension of C / r×(H+W)×1 is obtained; wherein the expression of the feature map f is as follows:

[0135]

[0136] In formula (5), conv represents the convolution operation of the sixth convolution; concat represents the concatenation operation;

[0137] Perform spatial segmentation on the feature map f to obtain f h and f w , the process is expressed as follows:

[0138] [f h ,f w ]=Split(f)(6)

[0139] In formula (6), Split represents the split operation; f h ∈R C / r×H×1 ;f w ∈R C / r×1×W ;

[0140] f h and f w Apply 1×1 convolution to restore the channel dimension C / r to C, and use the sigmoid activation function to get g h and g w , the process is expressed as follows:

[0141] [g h ,g w ])=δ(conv2[f h ,f w ])(7)

[0142] In formula (7), δ represents the sigmoid activation function; g h ∈R C×H×1 ; g w ∈R C×1×W ;

[0143] Using g h and g w The input feature x is weighted, and the process is expressed as follows:

[0144]

[0145] In formula (8), x c (i,j) indicates the input feature x in the i-th row and j-th column of the c-th channel; Indicates g h At the i-th position of the c-th channel; Indicates g w At the jth position of the cth channel.

[0146] In one embodiment of the present invention, Figure 6 As shown in FIG. 1 , the process of multi-scale feature fusion performed by the Feature Fusion Bridge specifically includes: flattening and reshaping the outputs F1, F2, F3, and F4 of the four-layer encoder into a feature tensor X with a dimension of (B, -1, C) input , the process is expressed as follows:

[0147] X input =Reshape(Concate(F1,F2,F3,F4),[B,-1,C])(9)

[0148] In formula (9), Reshape represents tensor reshaping; concat represents concatenation operation; B represents the size of each batch; C represents the number of channels of the feature; F1, F2, F3, and F4 represent the features output by the four-layer encoder respectively;

[0149] The feature tensor X input Input to the first Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0150] Attn1 = Efficient_Attn1 (X input )(10)

[0151] In formula (10), Attn1 represents the feature map output by the first Efficient Transformer Block;

[0152] Attn1 is input to the second Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0153] Attn2=Efficient_Attn2(Attn1)(11)

[0154] In formula (11), Attn2 represents the feature map output by the second Efficient Transformer Block;

[0155] The features of Attn1 and Attn2 are fused by the Mix Block, and the process is shown as follows:

[0156] Attn mix =Mix_block(Attn1,Attn2)(12)

[0157] In formula (12), Attn mix Represents the feature map output by Mix Block;

[0158] Attn mix Input to the third Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0159] Attn3=Efficient_Attn3(Attn mix )(13)

[0160] In formula (13), Attn3 represents the feature map output by the third Efficient Transformer Block;

[0161] Attn3 is input to the fourth Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0162] X output =Efficient_Attn4(Attn3)(14)

[0163] In formula (14), X output Represents the feature map output by the fourth Efficient Transformer Block.

[0164] In one embodiment of the present invention, Figure 7 As shown, the process of attention calculation of the first Efficient Transformer Block includes: reshaping the dimension of the tensor K output by the fully connected layer FC into N / r×rC, and then projecting it into rC×C through the linear layer. The process is expressed as follows:

[0165]

[0166] The dimension of the tensor V output by the fully connected layer FC is reshaped into N / r×rC, and then projected into rC×C through the linear layer. The process is expressed as follows:

[0167]

[0168] In formula (15) and formula (16), X in Represents input features; FC represents a fully connected layer; Reshape represents a reshaping function; Project represents a linear layer.

[0169] In this embodiment, the process of performing attention calculation by the second Efficient Transformer Block, the third Efficient Transformer Block, and the fourth Efficient Transformer Block is the same as the process of performing attention calculation by the first Efficient Transformer Block.

[0170] In one embodiment of the present invention, Figure 6 As shown, the Mix Block includes: a concatenation Concat, a feedforward neural network FNN, a third normalization layer and an activation function GELU connected in series in sequence; the concatenation module Concat is used to perform feature concatenation on the outputs of the first Efficient Transformer Block and the second Efficient Transformer Block along the channel dimension to obtain a feature map T1 with a channel dimension of 2C; the feedforward neural network FNN is used to use a linear layer to project the number of channels of the feature map T to C to obtain a feature map T2; the third normalization layer is used to perform a normalization operation on the feature map T2, so that the feature map after the normalization operation can be subsequently enhanced nonlinearly through the activation function GELU.

[0171] In one embodiment of the present invention, the step S2 specifically includes: step S2.1: adjusting the pixel size of each medical image in the medical image dataset to 224×224; step S2.2: normalizing each medical image after the pixel size is adjusted; step S2.3: performing data enhancement on each medical image after the normalization.

[0172] In a specific embodiment, the medical image dataset is the Synapse dataset or the ISIC 2018 dataset. In the preprocessing process of the ISIC 2018 dataset, each image is first resized to 224×224 pixels, then normalized, and finally data augmentation is performed on each normalized image. In the data augmentation process, we increase the diversity of the data through rotation and cropping operations so that the model can better adapt to image changes in the real environment. When dividing the data, we extract the first 1815 images for training and 520 samples for testing as in other studies.

[0173] The technical solution of the present invention will be demonstrated below with a specific embodiment. The specific implementation steps of a method for segmenting medical images based on a multi-branch attention network provided by the present invention are as follows:

[0174] (1) Obtain a medical image dataset.

[0175] The medical image dataset is the Synapse dataset. The Synapse dataset is a widely used and authoritative multi-organ segmentation dataset in the field of medical image segmentation, covering multiple organs in the abdomen, including the left kidney, right kidney, gallbladder, pancreas, liver, aorta, spleen, and stomach. The dataset contains abdominal CT scan data of 30 patients, which is highly diverse, and each image is manually annotated by authoritative imaging experts.

[0176] (2) Preprocess the Synapse dataset.

[0177] The steps for preprocessing the Synapse dataset include: First, we adjust the pixel size of each medical image (each slice image) in the Synapse dataset to 224×224 to standardize the input size and adapt to the input requirements of the model. Secondly, we normalize each medical image after adjusting the pixel size so that the pixel values ​​are distributed in the same scale range, thereby improving the stability and performance of model training. Finally, in order to enhance the robustness of the model and improve its generalization ability on the test data, we use a series of data augmentation techniques on each normalized medical image. These data augmentation techniques include: random up and down flipping, left and right flipping, Gaussian noise addition, Gaussian blur, affine transformation, and translation.

[0178] (3) The preprocessed medical image dataset is divided into training set and test set in a ratio of 3:2.

[0179] When dividing the data, we randomly divide the data set into a training set and a test set. For the Synapse data set, the data of 18 patients are used for training, and the data of the remaining 12 patients are used for testing. This ensures the diversity and independence of the training and testing data, which helps to objectively evaluate the performance of the model.

[0180] (4) Build a multi-branch attention network model.

[0181] like Figure 2As shown, the multi-branch attention network model is a U-shaped architecture, and the multi-branch attention network model includes: an encoder part, a Feature Fusion Bridge, and a decoder part; the encoder part and the decoder part are both four layers; the encoder part also includes a Patch Embedding module; a Feature Fusion Bridge is arranged between the encoder part and the decoder part; the output end of each layer of the encoder is respectively connected to the input end of the Feature Fusion Bridge; the output end of each layer of the encoder of the first layer, the second layer and the third layer is respectively connected to the input end of the encoder of the corresponding next layer; the output end of the Feature Fusion Bridge is connected to the input end of each layer of the decoder; the output end of each layer of the decoder of the first layer, the second layer and the third layer is respectively connected to the input end of the decoder of the corresponding next layer; the output end of the fourth layer of the decoder outputs the segmentation result corresponding to the training set or the test set;

[0182] Each layer of encoder consists of N combination units connected in series and a Coordinate Attention, and the combination unit includes a VFT Block and a ConvFNN connected in series; each layer of decoder consists of a first Efficient Transformer Block, a second Efficient Transformer Block, and a PatchExpanding connected in series; the Feature Fusion Bridge includes: a first Double Bridge, a MixBlock, and a second Double Bridge connected in series; the first Double Bridge includes: a first Efficient Transformer Block and a second Efficient Transformer Block connected in series; the second Double Bridge includes: a third Efficient Transformer Block and a fourth Efficient Transformer Block connected in series; the Mix Block is used to perform feature fusion on the outputs of the first Efficient Transformer Block and the second Efficient TransformerBlock. The number of combination units of each layer of encoder is the same or different, that is, the value of the number N of combination units of each layer of encoder is the same or different.

[0183] Patch Embedding module, used to divide the received preprocessed training set or test set into several patches of equal size, and then input the several patches of equal size into the first layer encoder; the first layer encoder, used to extract features and downsample the input several patches of equal size, output the feature map obtained after downsampling, and then input the feature map obtained after downsampling into the second layer encoder; the second and third layer encoders, used to receive the feature map output by the previous layer encoder, and perform feature learning and downsampling on the received feature map, output the feature map obtained after downsampling, and then input the feature map obtained after downsampling into the next layer encoder; the fourth layer encoder, used to receive the feature map sent by the previous layer encoder, and perform feature learning and downsampling on the received feature map, and output the feature map obtained after downsampling; Feature Fusion Bridge, used to perform multi-scale feature fusion on the feature maps output by encoders of each layer, output the fused feature maps, and then input the fused feature maps to each layer of decoders; the first layer decoder, used to receive the fused feature maps, upsample the fused feature maps to restore the resolution of the image, and output the upsampled feature maps; the second, third, and fourth layer decoders, used to upsample the feature maps output by the decoder of the previous layer to restore the resolution of the image, and concatenate and fuse them with the feature maps output by the corresponding Feature Fusion Bridge to obtain the corresponding feature maps; wherein the fourth layer decoder outputs the segmentation results corresponding to the training set or the test set through a Segmentation Head.

[0184] The model we proposed is a U-shaped network, which consists of an encoder part, a Feature Fusion Bridge, and a decoder part. The encoder and decoder parts adopt a symmetrical four-layer structure. For an image of size H×W×C, the model first divides it into patches of equal size using the Patch Embedding module. These small patches are then input into the hierarchical encoder to generate features of different scales. The bridge connection splices the multi-scale features generated by each layer of the encoder to achieve multi-scale feature fusion, and the fused features are sent to the decoder to generate the segmentation result.

[0185] Specifically, Figure 3 As shown, each VFT Block includes: a Laynorm layer, a Maxpool GlobalBranch, an Avgpool Global Branch, and a Local Branch;

[0186] The Laynorm layer is used to perform layer normalization operation on its own input features to output feature C1, and then obtain tensors Q, K and V through the fully connected layer FC;

[0187] The Maxpool Global Branch, Avgpool Global Branch, and Local Branch are used to extract features respectively to obtain features Attn maxglobal , FeatureAttn avgglobal , and feature X local ;

[0188] The expression of the attention calculation process of the Maxpool Global Branch is as follows:

[0189] (Q,K,V)=FC(X in )

[0190] K MG =MaxPool(K)

[0191] V MG =MaxPool(V)

[0192]

[0193] In formula (1), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents the maximum pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Maxpool Global Branch maxglobal ;

[0194] The expression of the attention calculation process of the Avgpool Global Branch is as follows:

[0195] (Q,K,V)=FC(X in )

[0196] K AG =AvgPool(K)

[0197] V AG =MaxPool(V)

[0198]

[0199] In formula (2), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents maximum pooling; AvgPool represents average pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Avgpool Global Branch avgglobal ;

[0200] The expression of the Local Branch's attention calculation process is as follows:

[0201] (Q,K,V)=FC(X in )

[0202] V l =DWconv(V)

[0203] Q l =DWconv(Q)

[0204] K l =DWconv(K)

[0205]

[0206] Attn″=FC(Attn′)

[0207]

[0208] In formula (3), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; DWConv represents a depth-separable convolution operation; represents Hardmard multiplication; Laynorm represents layer normalization operation; d represents the number of dimensions of the tensor K; Tanh represents the Tanh activation function;

[0209] After the Maxpool Global Branch, Avgpool Global Branch and Local Branch complete feature extraction, the feature Attn maxglobal , FeatureAttn avgglobal , and feature X local Make stitching.

[0210] Various Features Transformer Block (VFT Block for short) is an innovative multi-branch self-attention module that effectively integrates global and local feature information to improve the richness and accuracy of feature expression. Figure 3 As shown: Mul represents ordinary matrix multiplication, which is usually omitted in the above description. For example, A multiplied by B can be written as AB; HardMul represents Hardmard multiplication, such as the expression in formula (3) The VFT Block consists of three independent branches, each of which is designed to extract and process feature information of different scales. The design of three independent branches not only optimizes the computational complexity, but also improves the feature extraction capability of the model, which is an important innovation of the present invention.

[0211] Specifically, Maxpool Global Branch is mainly responsible for capturing key global features in the image. Before attention calculation, Maxpool Global Branch uses maximum pooling (Maxpool) to downsample the key (K) and value (V). This operation not only reduces the amount of calculation, but also extracts the most significant global features in the image. Through this pooling method, Maxpool Global Branch can efficiently retain the most significant features of the image, especially the extraction of edge and contour information, while reducing the model's demand for resources. Avgpool Global Branch focuses on capturing more comprehensive global background information. Similar to Maxpool Global Branch, this branch uses average pooling (Avgpool) to process K and V to retain subtle global information in the scene. This method is suitable for scenes that need to maintain image background and details, such as background analysis in complex environments. Local Branch is mainly responsible for capturing local features in the image. Unlike the first two global branches, it focuses on the extraction of high-frequency details and local information. This branch aggregates local information through deep separable convolution (DWConv). This design can retain more spatial details, making Local Branch more advantageous in processing high-frequency features and providing rich context-aware capabilities for the entire feature map. After the feature extraction of the three branches is completed, the VFT Block integrates multiple features into a unified feature map through feature splicing. This process not only enhances the diversity of feature maps, but also ensures the effective combination of global and local features, and finally forms a feature map with rich semantics and details.

[0212] In each encoder layer, we connect VFT Block and ConvFNN in series to form a combined unit and repeat this unit n times for feature extraction. At the end of each encoder, we use Coordinate Attention to extract the position information of the input features.

[0213] Specifically, ConvFNN is used to fuse the diverse features extracted by the VFT Block and perform downsampling when necessary by adjusting the convolution step size; Figure 4 As shown in (a), when downsampling is not required, the ConvFNN specifically includes: a first normalization layer, a first convolution, a GELU activation function, a first depth-separable convolution, and a second convolution connected in sequence; wherein the input end of the first normalization layer LN and the output end of the second convolution are connected through a residual connection to obtain a feature map output by the ConvFNN; as shown in Figure 4 As shown in (b), when downsampling is required, the ConvFNN specifically includes: a main branch and a residual branch; the main branch includes: a second normalized layer, a third convolution, a GELU activation function, a second depth-separable convolution, and a fourth convolution connected in sequence; the residual branch includes: a third depth-separable convolution, a GELU activation function, and a fifth convolution connected in sequence; wherein the input ends of the second normalized layer and the third depth-separable convolution are connected, and the output ends of the fourth convolution and the fifth convolution are connected through a residual connection to obtain a feature map output by the ConvFNN.

[0214] The main difference between ConvFNN and traditional FNN is that the hidden layer in traditional FNN is replaced by deep separable convolution (DWConv). The purpose of this is to better aggregate local information and accelerate the fusion of multi-scale features. Secondly, ConvFNN can also perform downsampling by controlling the step size of DWConv.

[0215] Specifically, Coordinate Attention is used to extract the location information of the input features; Figure 5 As shown, Mul represents ordinary matrix multiplication, which is usually omitted in the above description. For example, A times B can be written as AB; BatchNorm represents batch normalization, and Silu represents Silu activation function.

[0216] Coordinate Attention is used for:

[0217] The input features are average pooled along the horizontal direction X and the vertical direction Y, respectively, and feature maps of dimensions C×H×1 and C×1×W are generated in the horizontal direction X and the vertical direction Y, respectively. and Among them, the feature map and The expressions are as follows:

[0218]

[0219] In formula (4), It means summing the hth row on the cth channel; It means summing the wth column on the cth channel;

[0220] Mapping the feature and Perform a concatenation operation to generate a feature map of size C×(H+W)×1;

[0221] The sixth convolution is used to reduce the number of channels C to C / r, and a feature map f with a dimension of C / r×(H+W)×1 is obtained; wherein the expression of the feature map f is as follows:

[0222]

[0223] In formula (5), conv represents the convolution operation of the sixth convolution; concat represents the concatenation operation;

[0224] Perform spatial segmentation on the feature map f to obtain f h and f w , the process is expressed as follows:

[0225] [f h ,f w ]=Split(f)(6)

[0226] In formula (6), Split represents the split operation; f h ∈R C / r×H×1 ;f w ∈R C / r×1×W ;

[0227] f h and f w Apply 1×1 convolution to restore the channel dimension C / r to C, and use the sigmoid activation function to get g h and g w , the process is expressed as follows:

[0228] [g h ,g w ])=δ(conv2[f h ,f w ])(7)

[0229] In formula (7), δ represents the sigmoid activation function; g h ∈RC×H×1 ; g w ∈R C×1×W ;

[0230] Using g h and g w The input feature x is weighted, and the process is expressed as follows:

[0231]

[0232] In formula (8), x c (i,j) indicates the input feature x in the i-th row and j-th column of the c-th channel; Indicates g h At the i-th position of the c-th channel; Indicates g w At the jth position of the cth channel.

[0233] Furthermore, we designed a Feature Fusion Bridge to replace the jump connection in the U-shaped network. This design can fuse the multi-scale features output by the encoder at different stages, improving the efficiency of feature fusion, which is an important innovation of the present invention.

[0234] like Figure 6 As shown in FIG. 1 , the process of multi-scale feature fusion performed by the Feature Fusion Bridge specifically includes:

[0235] The outputs F1, F2, F3, and F4 of the four-layer encoder are flattened and reshaped into a feature tensor X of dimension (B, -1, C) input , the process is expressed as follows:

[0236] X input =Reshape(Concate(F1,F2,F3,F4),[B,-1,C])(9)

[0237] In formula (9), Reshape represents tensor reshaping; concat represents concatenation operation; B represents the size of each batch; C represents the number of channels of the feature; F1, F2, F3, and F4 represent the features output by the four-layer encoder respectively;

[0238] Similar to SegFormer, X input It is sent to the serially connected Transformer Block for feature fusion.

[0239] Specifically, the feature tensor X input Input to the first Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0240] Attn1 = Efficient_Attn1 (X input )(10)

[0241] In formula (10), Attn1 represents the feature map output by the first Efficient Transformer Block;

[0242] Attn1 is input to the second Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0243] Attn2=Efficient_Attn2(Attn1)(11)

[0244] In formula (11), Attn2 represents the feature map output by the second Efficient Transformer Block;

[0245] The features of Attn1 and Attn2 are fused by the Mix Block, and the process is shown as follows:

[0246] Attn mix =Mix_block(Attn1,Attn2)(12)

[0247] In formula (12), Attn mix Represents the feature map output by Mix Block;

[0248] Attn mix Input to the third Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0249] Attn3=Efficient_Attn3(Attn mix )(13)

[0250] In formula (13), Attn3 represents the feature map output by the third Efficient Transformer Block;

[0251] Attn3 is input to the fourth Efficient Transformer Block for attention calculation. The process is expressed as follows:

[0252] X output =Efficient_Attn4(Attn3)(14)

[0253] In formula (14), X outputRepresents the feature map output by the fourth Efficient Transformer Block.

[0254] Specifically, Figure 7 As shown, Mul represents ordinary matrix multiplication, which is usually omitted in the above description. For example, A multiplied by B can be written as AB. The process of performing attention calculation in the first Efficient Transformer Block, the second Efficient Transformer Block, the third Efficient Transformer Block, and the fourth Efficient Transformer Block includes:

[0255] The dimension of the tensor K output by the fully connected layer FC is reshaped into N / r×rC, and then projected into rC×C through the linear layer. The process is expressed as follows:

[0256]

[0257] The dimension of the tensor V output by the fully connected layer FC is reshaped into N / r×rC, and then projected into rC×C through the linear layer. The process is expressed as follows:

[0258]

[0259] In formula (15) and formula (16), X in Represents input features; FC represents a fully connected layer; Reshape represents a reshaping function; Project represents a linear layer.

[0260] Furthermore, different from SegFormer, we further concatenate the features obtained from the first and second layers of Feature Fusion Bridge and use Mix Block to enhance the further fusion of features.

[0261] In Feature Fusion Bridge, we designed a module called Mix Block, which is used to fuse the output of the first layer and the second layer of Feature Fusion Bridge. Its structure is as follows: Figure 6 As shown in Figure 1, we first connect the output features of the first and second layers of the feature fusion bridge along the channel dimension to obtain a feature map T with a channel dimension of 2C. Then, a linear layer is used to project the number of channels of T to C, followed by normalization and activation function to enhance the nonlinearity of the features.

[0262] Specifically, Figure 6As shown, Mix Block includes: a concatenation Concat, a feedforward neural network FNN, a third normalization layer and an activation function GELU connected in series in sequence; the concatenation module Concat is used to perform feature concatenation on the outputs of the first Efficient Transformer Block and the second Efficient Transformer Block along the channel dimension to obtain a feature map T1 with a channel dimension of 2C; the feedforward neural network FNN is used to use a linear layer to project the number of channels of the feature map T to C to obtain a feature map T2; the third normalization layer is used to perform a normalization operation on the feature map T2, so that the feature map after the normalization operation can be subsequently enhanced nonlinearly through the activation function GELU.

[0263] Furthermore, in the design of the decoder, each decoder consists of two Efficient Transformer Blocks and a Patch Expanding module. These decoders restore the resolution of the input feature map through feature learning, while improving the global expression ability of the features. The four decoders are connected in series to form the decoder part, which gradually upsamples and processes the multi-scale features extracted by the encoder, and finally restores the resolution of the original image and generates the segmentation result.

[0264] (5) Input the training set into the multi-branch attention network model for training.

[0265] During training, we set the batch size to 24 and performed 400 rounds of training. The optimizer selected SGD and adopted a learning rate decay strategy with an initial learning rate of 0.05, a momentum of 0.9, and an initial weight decay of 0.0001. In terms of loss function, we combined cross entropy loss and Dice loss to balance the accuracy between different categories, with the weight of cross entropy loss accounting for 0.4 and the weight of Dice loss accounting for 0.6. Training was performed in an environment equipped with an NVIDIA4080Ti graphics card.

[0266] (6) Input the test set into the trained multi-branch attention network model to obtain the segmentation result corresponding to the test set.

[0267] The test set is input into the trained multi-branch attention network model to verify the feasibility of the method for segmenting medical images based on the multi-branch attention network of the present invention. The trained model is applied to the test set in the Synapse dataset, and the preprocessed CT images are input to generate the predicted segmentation results. Figure 8 As shown in the figure, by comparing with the annotation results of authoritative experts, it is verified that the model has high robustness on different cases and can provide more accurate segmentation results.

[0268] It turns out that:

[0269] (1) The multi-branch attention network model constructed by the present invention can effectively perform feature extraction and fusion. Specifically, the present invention significantly enhances the model's ability to extract multi-scale features in medical images by using the innovative design of VFT Block. VFT Block combines three different branches, enabling the model to capture local features in the image while retaining global texture information and background information. This multi-branch self-attention mechanism improves the model's feature expression ability and provides more accurate analysis for complex medical images.

[0270] (2) The multi-branch attention network model constructed by the present invention can perform accurate positioning and segmentation. Specifically, by introducing the Coordinate Attention module, the present invention performs well in encoding the position information of the feature map. This module enables the model to accurately locate the target area by capturing the correlation between channels and emphasizing the position information. This is particularly critical in medical image segmentation, because accurate segmentation has a direct impact on diagnosis and treatment, and can help medical professionals make judgments more quickly and accurately.

[0271] (3) The multi-branch attention network model constructed by the present invention can perform efficient calculation and resource optimization. Specifically, the present invention fully considers the computational efficiency and resource consumption in its design, and greatly reduces the computational complexity through the Efficient Attention mechanism. Compared with traditional models, Mixformer can significantly improve the processing speed while maintaining high accuracy. This high efficiency not only reduces the hardware requirements, but also makes the model more suitable for large-scale deployment and use in practical application scenarios.

[0272] Therefore, the method for segmenting medical images based on a multi-branch attention network of the present invention provides practical guidance for clinicians in segmenting medical images, and has unsurpassed technical advantages in related technologies of medical image segmentation.

[0273] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for segmenting medical images based on a multi-branch attention network, characterized in that: include: Step S1: Obtain a medical image dataset; Step S2: preprocessing the medical image dataset; Step S3: Divide the preprocessed medical image dataset into a training set and a test set according to a preset ratio; Step S4: building a multi-branch attention network model; wherein the multi-branch attention network model is a U-shaped architecture, and the multi-branch attention network model includes: an encoder part, a Feature Fusion Bridge, and a decoder part; the encoder part and the decoder part are both four-layer; the encoder part also includes a Patch Embedding module; the Feature Fusion Bridge is arranged between the encoder part and the decoder part; the output end of each layer of the encoder is respectively connected to the input end of the Feature Fusion Bridge; the output end of each layer of the encoder of the first layer, the second layer and the third layer is respectively connected to the input end of the encoder of the corresponding next layer; the output end of the Feature Fusion Bridge is connected to the input end of each layer of the decoder; the output end of each layer of the decoder of the first layer, the second layer and the third layer is respectively connected to the input end of the decoder of the corresponding next layer; the output end of the fourth layer of the decoder outputs the segmentation result corresponding to the training set or the test set; Each layer of encoder is composed of N combination units connected in series and a Coordinate Attention, and the combination unit includes a VFT Block and a ConvFNN connected in series; each layer of decoder is composed of a first Efficient Transformer Block, a second Efficient Transformer Block, and a PatchExpanding connected in series; the Feature Fusion Bridge includes: a first Double Bridge, a MixBlock, and a second Double Bridge connected in series; the first Double Bridge includes: a first Efficient Transformer Block and a second Efficient Transformer Block connected in series; the second Double Bridge includes: a third Efficient Transformer Block and a fourth Efficient Transformer Block connected in series; the Mix Block is used to perform feature fusion on the outputs of the first Efficient Transformer Block and the second Efficient TransformerBlock; The Patch Embedding module is used to divide the received preprocessed training set or test set into a plurality of patches of equal size, and then input the plurality of patches of equal size into the first layer encoder; The first layer encoder is used to extract features and downsample the inputted patches of equal size, output a feature map obtained after downsampling, and then input the feature map obtained after downsampling to the second layer encoder; The second and third encoders are used to receive the feature map output by the encoder of the previous layer, perform feature learning and down-sampling on the received feature map, output the feature map obtained after down-sampling, and then input the feature map obtained after down-sampling to the encoder of the next layer; The fourth layer encoder is used to receive the feature map sent by the encoder of the previous layer, perform feature learning and down-sampling on the received feature map, and output the feature map obtained after down-sampling; The Feature Fusion Bridge is used to perform multi-scale feature fusion on the feature maps output by the encoders of each layer, output the fused feature maps, and then input the fused feature maps to the decoders of each layer; A first layer decoder is used to receive the fused feature map, perform an upsampling operation on the fused feature map to restore the resolution of the image, and output the feature map obtained after the upsampling; The second, third, and fourth decoders are used to upsample the feature maps output by the decoder of the previous layer to restore the resolution of the image, and concatenate and fuse them with the feature maps output by the corresponding Feature Fusion Bridge to obtain the corresponding feature maps; the fourth decoder outputs the segmentation results corresponding to the training set or the test set through a Segmentation Head; Step S5: inputting the training set into the multi-branch attention network model for training; Step S6: Input the test set into the trained multi-branch attention network model to obtain the segmentation result corresponding to the test set.

2. The method for segmenting medical images based on a multi-branch attention network according to claim 1, characterized in that: Each VFT Block includes: a Laynorm layer, a Maxpool Global Branch, an AvgpoolGlobal Branch, and a Local Branch; The Laynorm layer is used to perform layer normalization operation on its own input features to output feature C1, and then obtain tensors Q, K and V through the fully connected layer FC; The Maxpool Global Branch, Avgpool Global Branch, and Local Branch are used to extract features respectively to obtain features Attn maxglobal , FeatureAttn avgglobal , and feature X local ; The expression of the attention calculation process of the Maxpool Global Branch is as follows: (Q,K,V)=FC(X in ) K MG =MaxPool(K) V MG =MaxPool(V) In formula (1), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents the maximum pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Maxpool Global Branch maxglobal ; The expression of the attention calculation process of the Avgpool Global Branch is as follows: (Q,K,V)=FC(X in ) K AG =AvgPool(K) V AG =MaxPool(V) In formula (2), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; MaxPool represents maximum pooling; AvgPool represents average pooling; d k represents the number of dimensions of the tensor K; The calculation result is expressed as a dot product attention score; Softmax is used to normalize the dot product attention score to convert the dot product attention score into the feature Attn output by the Avgpool Global Branch avgglobal ; The expression of the Local Branch's attention calculation process is as follows: In formula (3), FC represents the fully connected layer; X in represents the input feature of FC, which is the feature map C1; DWConv represents a depth-wise separable convolution operation; represents Hardmard multiplication; Laynorm represents layer normalization operation; d represents the number of dimensions of the tensor K; Tanh represents the Tanh activation function; After the Maxpool Global Branch, Avgpool Global Branch and Local Branch complete feature extraction, the feature Attn maxglobal , FeatureAttn avgglobal , and feature X local Make stitching.

3. The method for segmenting medical images based on a multi-branch attention network according to claim 1, characterized in that: The ConvFNN is used to fuse the diverse features extracted by the VFT Block and perform downsampling when necessary by adjusting the convolution step size; When downsampling is not required, the ConvFNN specifically includes: a first normalization layer, a first convolution, a GELU activation function, a first depth-separable convolution, and a second convolution connected in sequence; wherein the input end of the first normalization layer LN and the output end of the second convolution are connected through a residual connection to obtain a feature map output by the ConvFNN; When downsampling is required, the ConvFNN specifically includes: a main branch and a residual branch; the main branch includes: a second normalized layer, a third convolution, a GELU activation function, a second depth-separable convolution, and a fourth convolution connected in sequence; the residual branch includes: a third depth-separable convolution, a GELU activation function, and a fifth convolution connected in sequence; wherein the input ends of the second normalized layer and the third depth-separable convolution are connected, and the output ends of the fourth convolution and the fifth convolution are connected through a residual connection to obtain a feature map output by the ConvFNN.

4. The method for segmenting medical images based on a multi-branch attention network according to claim 1, characterized in that: The Coordinate Attention is used to extract the position information of the input feature; and The Coordinate Attention is specifically used for: The input features are average pooled along the horizontal direction X and the vertical direction Y, respectively, and feature maps of dimensions C×H×1 and C×1×W are generated in the horizontal direction X and the vertical direction Y, respectively. and Among them, the feature map and The expressions are as follows: In formula (4), It means summing the hth row on the cth channel; It means summing the wth column on the cth channel; The feature map and Perform a concatenation operation to generate a feature map of size C×(H+W)×1; The sixth convolution is used to reduce the number of channels C to C / r, and a feature map f with a dimension of C / r×(H+W)×1 is obtained; wherein the expression of the feature map f is as follows: In formula (5), conv represents the convolution operation of the sixth convolution; concat represents the concatenation operation; Perform spatial segmentation on the feature map f to obtain f h and f w , the process is expressed as follows: [f h ,f w ]=Split(f)(6) In formula (6), Split represents the split operation; f h ∈R C / r×H×1 ;f w ∈R C / r×1×W ; f h and f w Apply 1×1 convolution to restore the channel dimension C / r to C, and use the sigmoid activation function to get g h and g w , the process is expressed as follows: [g h ,g w ])=δ(conv2[f h ,f w ])(7) In formula (7), δ represents the sigmoid activation function; g h ∈R C×H×1 ; g w ∈R C×1×W ; Using g h and g w The input feature x is weighted, and the process is expressed as follows: In formula (8), x c (i,j) represents the i-th row and j-th column of the input feature x in the c-th channel; Indicates g h At the i-th position of the c-th channel; Indicates g w At the jth position of the cth channel.

5. The method for segmenting medical images based on a multi-branch attention network according to claim 1, characterized in that: The process of multi-scale feature fusion performed by the Feature Fusion Bridge specifically includes: The outputs F1, F2, F3, and F4 of the four-layer encoder are flattened and reshaped into a feature tensor X of dimension (B, -1, C) input , the process is expressed as follows: X input =Reshape(Concate(F1,F2,F3,F4),[B,-1,C])(9) In formula (9), Reshape represents tensor reshaping; concat represents concatenation operation; B represents the size of each batch; C represents the number of channels of the feature; F1, F2, F3, and F4 represent the features output by the four-layer encoder respectively; The feature tensor X input Input to the first Efficient Transformer Block for attention calculation. The process is expressed as follows: Attn1=Efficient_Attn1(X input )(10) In formula (10), Attn1 represents the feature map output by the first Efficient Transformer Block; Attn1 is input to the second Efficient Transformer Block for attention calculation. The process is expressed as follows: Attn2=Efficient_Attn2(Attn1)(11) In formula (11), Attn2 represents the feature map output by the second Efficient Transformer Block; The Mix Block is used to fuse the features of Attn1 and Attn2. The process is as follows: Attn mix =Mix_block(Attn1,Attn2)(12) In formula (12), Attn mix Represents the feature map output by Mix Block; Attn mix Input to the third Efficient Transformer Block for attention calculation. The process is expressed as follows: Attn3=Efficient_Attn3(Attn mix )(13) In formula (13), Attn3 represents the feature map output by the third Efficient Transformer Block; Attn3 is input to the fourth Efficient Transformer Block for attention calculation. The process is expressed as follows: X output =Efficient_Attn4(Attn3)(14) In formula (14), X output Represents the feature map output by the fourth Efficient Transformer Block.

6. The method for segmenting medical images based on a multi-branch attention network according to claim 1, characterized in that: The process of attention calculation of the first Efficient Transformer Block, the second Efficient Transformer Block, the third Efficient Transformer Block, and the fourth Efficient Transformer Block includes: The dimension of the tensor K output by the fully connected layer FC is reshaped into N / r×rC, and then projected into rC×C through the linear layer. The process is expressed as follows: The dimension of the tensor V output by the fully connected layer FC is reshaped into N / r×rC, and then projected into rC×C through the linear layer. The process is expressed as follows: In formula (15) and formula (16), X in Represents input features; FC represents a fully connected layer; Reshape represents a reshaping function; Project represents a linear layer.

7. The method for segmenting medical images based on a multi-branch attention network according to claim 6, characterized in that: The Mix Block includes: a concatenation Concat, a feedforward neural network FNN, a third normalization layer and an activation function GELU connected in series in sequence; The concatenation module Concat is used to concatenate the outputs of the first Efficient Transformer Block and the second Efficient Transformer Block along the channel dimension to obtain a feature map T1 with a channel dimension of 2C; The feedforward neural network FNN is used to project the number of channels of the feature map T to C using a linear layer to obtain a feature map T2; The third normalization layer is used to perform a normalization operation on the feature map T2 so as to perform nonlinear feature enhancement on the feature map after the normalization operation through the activation function GELU.

8. The method for segmenting medical images based on a multi-branch attention network according to any one of claims 1 to 7, characterized in that: The step S2 specifically includes: Step S2.1: adjusting the pixel size of each medical image in the medical image dataset to 224×224; Step S2.2: normalizing each medical image after adjusting the pixel size; Step S2.3: Perform data enhancement on each normalized medical image.

Images