A medical image segmentation method based on a hybrid convolutional neural network and a transformer

Abstract

The application relates to a medical image segmentation method based on a mixed convolutional neural network and a transformer, a channel feature correlation matrix is constructed based on channel features between local feature maps and global feature maps, channel information between the global feature maps and the local feature maps is interactively fused based on the channel feature correlation matrix, and the global feature maps and the local feature maps after the channel information interaction and fusion are interacted and fused in spatial information, so that the local feature maps and the spatial feature maps both have local feature information and global feature information, the image segmentation model's ability to capture the global feature information and the local feature information is enhanced, and the image segmentation model's ability to reconstruct a mask with high precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, and particularly relates to a medical image segmentation method based on a hybrid convolutional neural network and a transformer. BACKGROUND

[0002] In the past few decades, deep learning-based semantic segmentation techniques have attracted extensive attention from researchers due to their higher efficiency than manual annotation. The essence of semantic segmentation is to classify pixel values, thereby achieving pixel-level annotation of complex lesion regions in medical images such as brain tumors, melanomas, and various cancerous regions (Azad et al., 2024; Asgari Taghanaki et al., 2021).

[0003] Deep convolutional neural network-based semantic segmentation models have been widely applied to various visual tasks, among which the U-shaped structure is particularly popular in the medical field. The U-shaped structure is usually composed of an encoder and a decoder, where the encoder captures semantic and contextual information through consecutive convolutional layers and downsampling; the decoder then reconstructs the output mask through step-by-step upsampling (Zhou et al., 2019). Although deeper convolutional layers and more downsampling can expand the receptive field of the model, they may lead to the loss of contextual information. However, the U-shaped model can recover the lost contextual information through skip connections. However, due to the convolution mechanism of convolutional layers, the receptive field is limited and long-distance dependencies cannot be modeled (Yuan et al., 2023; Heidari et al., 2023).

[0004] Vision Transformer (ViTs) improves the global receptive field of the model by dividing the image into small pixel blocks (patches) and modeling the relationship between small pixel blocks to extract abstract relationships. However, ViTs still has deficiencies in capturing low-level features (Heidari et al., 2023).

[0005] In summary, convolutional neural networks (CNNs) contribute to medical image segmentation through their lightweight design and efficient capture of local features. However, because CNNs extract features using a sliding window, they cannot consider information relevance outside the sliding window. Therefore, CNNs cannot model global information in medical image segmentation. Transformers, on the other hand, model feature blocks through a self-attention mechanism, resulting in excellent global information capture capabilities. However, this same mechanism prevents transformers from modeling local information within features. Therefore, relying solely on either technique makes it difficult to simultaneously model both local and global information of features for effective medical image segmentation. Summary of the Invention

[0006] The technical problem this invention aims to solve is how to improve the ability to capture local and global feature information in order to achieve accurate medical image segmentation.

[0007] This invention provides a medical image segmentation method based on a hybrid convolutional neural network and a transformer, comprising:

[0008] Step 1. Obtain the medical image to be segmented;

[0009] Step 2. Construct an image segmentation model by inputting the medical image to be segmented into the image segmentation model, which includes a preprocessing layer, a hybrid encoder layer, and a decoder layer.

[0010] The preprocessing layer is used for image segmentation and local feature extraction of medical images;

[0011] The hybrid encoder layer is connected to the preprocessing layer and is used to extract global feature maps and local feature maps from the segmented image and local features. Based on the channel features of the global feature maps and local feature maps, a channel feature correlation matrix is ​​constructed. Based on the channel feature correlation matrix, the channel information of the global feature maps and local feature maps is interactively fused. Then, the spatial information of the interactively fused global feature maps and local feature maps is interactively fused.

[0012] The decoder layer is used to stitch and upsample the local features output by the preprocessing layer and the features after spatial information interaction and fusion by the hybrid encoder layer to output the target segmentation image.

[0013] Compared with the prior art, the present application has the following advantages: the present application constructs a channel feature correlation matrix based on the channel features between the local feature map and the global feature map, and interacts and fuses the channel information between the local feature map and the global feature map based on the channel feature correlation matrix, then interacts and fuses the global feature map and the local feature map after interaction and fusion in the spatial information, so that the local feature map and the spatial feature map both have local feature information and global feature information, thereby enhancing the image segmentation model's ability to capture global feature information and local feature information, and improving the image segmentation model's ability to reconstruct the mask with high precision.

[0014] In a possible implementation, the mixed encoder layer comprises a first fusion layer, a second fusion layer, a third fusion layer and a fourth fusion layer connected in sequence, the decoder layer comprises a fifth decoder layer, a fourth decoder layer, a third decoder layer, a second decoder layer and a first decoder layer connected in sequence, the first fusion layer transmits the interactive fusion of the communication information to the second fusion layer, the second fusion layer transmits the interactive fusion of the channel information to the third fusion layer, the third fusion layer transmits the interactive fusion of the channel information to the fourth fusion layer, the fourth fusion layer performs the interactive fusion of the spatial information and then enters the fifth decoder layer for up-sampling operation, the third fusion layer performs the interactive fusion of the spatial information and then splices the features output by the fifth decoder layer and enters the fourth decoder layer for up-sampling operation, the second fusion layer performs the interactive fusion of the spatial information and then splices the features output by the fourth decoder layer and enters the third decoder layer for up-sampling operation, the first fusion layer performs the interactive fusion of the spatial information and then splices the features output by the third decoder layer and enters the second decoder layer for up-sampling operation, and the local features output by the preprocessing layer are spliced with the features output by the second decoder layer and then enter the first decoder layer for double up-sampling operation to output the target segmentation image.

[0015] Compared with the prior art, although one fusion layer can make the local feature map have global information and the global feature map have local information, in order to facilitate the decoder layer to reconstruct the mask, four fusion layers are used to interact and fuse the channel information of the local feature map and the global feature map and to interact and fuse the spatial information, so that the image segmentation model has more accurate local feature capturing ability and global feature capturing ability, thereby being able to segment low-quality medical images.

[0016] In a possible implementation, the second decoder layer, the third decoder layer, the fourth decoder layer and the fifth decoder layer have the same network structure, and each comprises a first convolutional block, a second convolutional block and a deconvolutional block connected in sequence, and the first decoder layer comprises two CNN decoder layers and a 1x1 convolutional block connected in sequence, and the network structure of the two CNN decoder layers is the same as that of the second decoder layer to the fifth decoder layer.

[0017] Compared with the prior art, the present application jointly performs the up-sampling operation by the double convolutional block and the deconvolutional block of the decoder layer, which is helpful for the recovery of the local feature map and the spatial feature map in the spatial size, and also ensures the lightweight of the image segmentation model.

[0018] In a possible implementation, the network structure of the first convolutional block and the second convolutional block each comprises a 3x3 convolutional block, a BN block, a ReLu function block, a 3x3 convolutional block, a BN block and a ReLu function block connected in sequence to perform twice 3x3 convolution operation, normalization operation and activation operation.

[0019] In a possible implementation, the network structure of the first fusion layer, the second fusion layer, the third fusion layer and the fourth fusion layer is the same, and each comprises a transformer module, a convolutional neural network module, a cross-domain channel attention module and a cross-type spatial feature fusion module.

[0020] The transformer module is configured to extract a global feature map.

[0021] The convolutional neural network module is configured to extract a local feature map.

[0022] The cross-domain channel attention module is connected with the transformer module and the convolutional neural network module respectively, and is configured to construct a channel feature correlation matrix, and perform interactive fusion of channel information based on the channel feature correlation matrix.

[0023] The cross-type spatial feature fusion module is connected with the cross-domain channel attention module, and is configured to perform interactive fusion of spatial information on the global feature map and the local feature map after the interactive fusion.

[0024] In a possible implementation, the cross-domain channel attention module comprises:

[0025] A first branch connected with the transformer module, configured to mine channel information inside the global feature map;

[0026] A second branch connected with the convolutional neural network module, configured to mine channel information inside the local feature map;

[0027] An outer product block is connected with the first branch and the second branch respectively, and a channel correlation matrix is constructed based on channel information inside the global feature map and the local feature map;

[0028] A local softmax block is connected with the outer product block, and is used for adjusting a dimension of local channel attention in the channel correlation matrix;

[0029] A global softmax block is connected with the outer product block, and is used for adjusting a dimension of global channel attention in the channel correlation matrix;

[0030] A global subspace block is connected with the transformer module and the global softmax block respectively, and a global subspace is obtained by performing a tensor product on the global feature map and the global channel attention output by the global softmax block;

[0031] A local subspace block is connected with the convolutional neural network module and the local softmax block respectively, and a local subspace is obtained by performing a tensor product on the local feature map and the local channel attention output by the local softmax block;

[0032] A global feature fusion block is connected with the transformer module and the local subspace block respectively, and is used for fusing local features in the local feature map into the global feature map;

[0033] A local feature fusion block is connected with the convolutional neural network module and the global subspace block respectively, and is used for fusing global features in the global feature map into the local feature map.

[0034] In a possible implementation, the first branch includes a first adaptive average pooling block, a first linear compression block, a first ReLu activation function block, a first linear excitation block and a first Sigmoid compression function block connected in sequence.

[0035] The first adaptive average pooling block compresses the global feature map in a channel feature map by channel feature map manner, and obtains more lightweight global channel level statistical information, and an expression is as follows: , wherein, represents the global feature map, , represents the global channel level statistical information, ;

[0036] The first linear compression block compresses the global channel level statistical information, and maps the global channel level statistical information to ; after nonlinear mapping by the first ReLu activation function block, the first linear excitation block expands to , and finally the compressed global channel-level statistical information is compressed by a first Sigmoid compression function block; the expression of the above process is:

[0037]

[0038] represents a linear compression function of channel-level statistical information, represents a linear excitation function of channel-level statistical information, represents a Sigmoid function function, represents global channel attention;

[0039] The second branch comprises a second adaptive average pooling block, a second linear excitation block, a second ReLu activation function block, a second linear compression block and a second Sigmoid compression function block connected in sequence, wherein the second adaptive average pooling block compresses the local feature map by channel feature map, and the compression is local channel-level statistical information which is more lightweight, and the expression is: , wherein, represents a local feature map, , represents local channel-level statistical information, ;

[0040] The second linear excitation block excites the local channel-level statistical information, and maps the local channel-level statistical information from to ; after the non-linear mapping by the second ReLu activation function block, the second linear compression block maps the local channel-level statistical information from to ; finally, the Sigmoid compression function block is used to compress the mapped local channel-level statistical information to 0-1, to prevent overflow in probability; the expression of the above process is:

[0041] ;

[0042] , wherein, represents a linear compression function of channel-level statistical information, represents a linear excitation function of channel-level statistical information, represents a Sigmoid function function; represents local channel attention.

[0043] ​Compared with the prior art, the cross-domain channel attention module firstly performs spatial transformation on the global feature map and the local feature map, converts the per-channel features into one-dimensional channel feature statistics, then interacts the global channel statistics and the local channel statistics through linear excitation and linear compression, effectively reduces the parameter quantity while mining the internal channel correlation; then constructs the cross-channel correlation between the global channel statistics and the local channel statistics through a module tensor product method, and performs softmax calculation in the convolution dimension and the transformer dimension respectively, finally multiplies the global feature map and the local feature map with the channel correlation matrix respectively, realizes the attenuation and increase of the channel number, and completes the mutual mapping and interaction between the local feature map and the global feature map.

[0044] In a possible implementation, an expression of the outer product block for constructing the channel correlation matrix is:

[0045] , wherein, , T in the formula is the transpose of the matrix;

[0046] An expression of the local softmax block for adjusting the dimension of the local channel attention is:

[0047] ;

[0048] , wherein, represents a subspace of the local feature map; An expression of the global softmax block for adjusting the dimension of the global channel attention is:

[0049]

[0050] ; , wherein,

[0051] represents a subspace of the global feature map; An expression of the global feature fusion block for fusing the local feature in the local feature map into the global feature map is:

[0052] ; wherein,

[0053] represents a global feature fusion map; An expression of the local feature fusion block for fusing the global feature in the global feature map into the local feature map is:

[0054] , wherein,

[0055] represents a local feature fusion map. ​​​

[0056] In a possible implementation, the cross-type spatial feature fusion module comprises a 3x3 convolution block, a 5x5 convolution block, and a 3x3 output convolution block, the output of the 3x3 convolution block is connected to the local feature fusion map through skip connection , the output of the 5x5 convolution block is connected to the global feature fusion map through skip connection , and the 3x3 output convolution block is connected, wherein

[0057] The 3x3 convolution block is connected to the output of the global feature fusion block, and is configured to perform on the global feature fusion map , convert the channel feature dimension of the global feature fusion map from to , and the output of the 3x3 convolution block is connected to the expression of through skip connection ;

[0058] The 5x5 convolution block is connected to the output of the local feature fusion block, and is configured to perform on the local feature fusion map , convert the channel feature dimension of the local feature fusion map from to ; the output of the 5x5 convolution block is connected to the global feature fusion map through skip connection ;

[0059] The 3x3 output convolution block splices the input and in the channel dimension, and the expression is

[0060] ;

[0061] In the formula, represents splicing in the channel dimension; represents a feature splicing map.

[0062] Compared with the prior art, the cross-type feature fusion module is used to perform 5x5 convolution on the local feature fusion map after the cross-domain channel attention module is cross-fused, to capture a larger receptive field, 3x3 convolution is used on the global feature fusion map to capture local features, and the final feature map is constructed through addition and splicing operations; to avoid too many channel features received by the decoder layer and the calculation of redundant information, the cross-type feature fusion module adds a final 3x3 data convolution block to compress the feature channels, and the method realizes the step-by-step fusion of spatial information through twice cross, and effectively reduces the great difference in spatial features.

[0063] In a possible implementation, the medical image segmentation method further comprises step 3. obtaining a plurality of data sets containing medical images, and dividing the data sets into a training set and a test set;

[0064] Step 4. setting a loss function, training the image segmentation model based on the training set and the loss function, and then testing the image segmentation model based on the test set;

[0065] The loss function is a balanced joint loss function, represented as:

[0066] ;

[0067] In the formula, is a weight factor; represents a Dice loss function, represents a cross-entropy loss function. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is a framework diagram of the image segmentation model of the present application;

[0069] Figure 2 is a framework diagram of each fusion layer of the present application;

[0070] Figure 3 is a framework diagram of the second decoder layer or the third decoder layer or the fourth decoder layer or the fifth decoder layer of the present application;

[0071] Figure 4 is a framework diagram of the first decoder layer of the present application;

[0072] Figure 5 is a breast ultrasound image segmentation result graph in experiment D1 of the present specific embodiment;

[0073] Figure 6 is a skin image segmentation result graph in experiment D2 of the present specific embodiment;

[0074] Figure 7 is an intestinal polyp endoscopic image segmentation result graph in experiment D3 of the present specific embodiment;

[0075] Figure 8 is a multi-organ CT image segmentation result graph in experiment D4 of the present specific embodiment;

[0076] Figure 9 is a brain tumor MRI image segmentation result graph in experiment D5 of the present specific embodiment;

[0077] Figure 10 is a GPU resource usage analysis graph in the experiment of the present specific embodiment;

[0078] Figure 11 The average inference speed analysis chart of the embodiment is shown in the following table. DETAILED DESCRIPTION

[0079] First, those skilled in the art should understand that the embodiments are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can adjust them as needed in order to adapt to specific application occasions.

[0080] In the description of the embodiments of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms “connected”, “connected” should be understood in a broad sense, for example, can be fixedly connected, can also be detachably connected, or integrally connected; can be mechanically connected, can also be electrically connected; can be directly connected, or indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0081] In the embodiments of the present application, unless otherwise explicitly specified and limited, the first feature is “on” or “under” the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature can be directly above or obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature can be directly below or obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0082] The present application Figure 1 And Figure 2 The English characters in the above table are translated as follows:

[0083] feature map multiplication represents feature map multiplication; feature map concatenation represents feature map concatenation; pixel-wise addition represents pixel-wise addition; sigmoid represents sigmoid function; conpression represents linear compression; excitation represents linear excitation; adaptive average pooling represents adaptive average pooling; upsampling represents upsampling; skip connection represents skip connection.

[0084] Figure 10 And Figure 11The horizontal coordinate GUP memory uasge in the graph represents CPU memory usage, and the vertical coordinate average dice score represents the average dice score.

[0085] The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] Referring to Figures 1 to 4 The embodiment of the application discloses a medical image segmentation method based on a hybrid convolutional neural network and a transformer, which comprises the following steps:

[0087] Step 1. Obtain a medical image to be segmented; in the specific embodiment, a medical image with a size of 256*256*3 is collected;

[0088] Step 2. Construct an image segmentation model, input the medical image to be segmented into the image segmentation model, and the image segmentation model comprises a preprocessing layer, a hybrid encoder layer and a decoder layer; wherein:

[0089] The preprocessing layer is used for image segmentation processing and local feature extraction of the medical image; in the embodiment, a convolutional neural network (CNN) module is used for local feature extraction, the convolutional neural network module extracts local feature maps with a size of 64*64*256 from the medical image based on a ResNet34 framework, and the image segmentation module is used for image segmentation processing of the input medical image;

[0090] The hybrid encoder layer is connected with the preprocessing layer, is used for extracting global feature maps and local feature maps from the segmentation image and the local features, constructing a channel feature correlation matrix based on the channel features of the global feature maps and the local feature maps, performing interactive fusion of channel information based on the channel feature correlation matrix, and then performing interactive fusion of the global feature maps and the local feature maps in spatial information; the network structure of the hybrid encoder layer comprises a first fusion layer, a second fusion layer, a third fusion layer and a fourth fusion layer connected in sequence;

[0091] ​​The decoder layer is used for splicing and up-sampling operation of the local features output by the preprocessing layer and the features after spatial information interaction fusion of the mixed encoder layer, and outputs a target segmentation image; the decoder layer comprises a fifth decoder layer, a fourth decoder layer, a third decoder layer, a second decoder layer and a first decoder layer connected in sequence; the second decoder layer to the fifth decoder layer have the same network structure, all adopt a CNN decoder layer, and all comprise a first convolution block, a second convolution block and a deconvolution block connected in sequence; the first decoder layer comprises two layers of CNN decoder layers and a 1*1 convolution block connected in sequence, and the network structure of the two layers of CNN decoder layers is the same as that of the second decoder layer to the fifth decoder layer.

[0092] Compared with the prior art, the double convolution block and the deconvolution block of the decoder layer are jointly used for up-sampling operation, which is helpful for the recovery of the local feature map and the spatial feature map in the spatial size, and also ensures the lightweight of the image segmentation model.

[0093] The network structure of the first convolution block and the second convolution block comprises a 3*3 convolution block, a BN block, a ReLu function block, a 3*3 convolution block, a BN block and a ReLu function block connected in sequence, and performs 3*3 convolution operation, normalization operation and activation operation twice.

[0094] The data processing relationship between the first fusion layer to the fourth fusion layer and the first decoder layer to the fifth decoder layer comprises:

[0095] The first fusion layer transmits the communication information after interaction fusion to the second fusion layer, the second fusion layer transmits the channel information after interaction fusion to the third fusion layer, the third fusion layer transmits the channel information after interaction fusion to the fourth fusion layer, and the fourth fusion layer performs interaction fusion of spatial information to obtain a feature map with a size of , and then enters the fifth decoder layer for up-sampling operation; the third fusion layer obtains a feature map with a size of after interaction fusion of spatial information, and then splices the feature output by the fifth decoder layer and enters the fourth decoder layer for up-sampling operation; the second fusion layer obtains a feature splicing map with a size of after interaction fusion of spatial information, and then splices the feature output by the fourth decoder layer and enters the third decoder layer for up-sampling operation; the first fusion layer obtains a feature splicing map with a size of 4 after interaction fusion of spatial information, and then splices the feature output by the third decoder layer and enters the second decoder layer for up-sampling operation; the local features output by the preprocessing layer are spliced with the features output by the second decoder layer, and then enter the first decoder layer for double up-sampling operation to output a target segmentation image.

[0096] The network structures of the first fusion layer, the second fusion layer, the third fusion layer and the fourth fusion layer are consistent, and each includes a transformer module, a convolutional neural network module, a cross-domain channel attention module and a cross-type spatial feature fusion module; specifically including:

[0097] The transformer module is used to extract a global feature map; the cross-domain channel attention module (CFCA module) is connected with the transformer module and the convolutional neural network module respectively, and is used to construct a channel feature correlation matrix and perform interactive fusion of channel information based on the channel feature correlation matrix on the global feature map and the local feature map; the cross-type spatial feature fusion module (XFF module) is connected with the cross-domain channel attention module, and is used to perform interactive fusion of spatial information on the global feature map and the local feature map after interactive fusion.

[0098] The convolutional neural network module (CNN module) is used to extract a local feature map; the convolutional neural network module ResNet34 (He et al., 2016) of the embodiment extracts a local feature map ; , represents the number of channels of the convolutional neural network module, respectively represent the height and width of the feature map;

[0099] The cross-type spatial feature fusion module is connected with the cross-domain channel attention module, and is used to perform interactive fusion of spatial information on the global feature map and the local feature map after interactive fusion.

[0100] The specific network structure of the cross-domain channel attention module and the data processing method thereof specifically include:

[0101] The first branch is connected with the transformer module, and is used to mine the channel information inside the global feature map; the first branch includes a first adaptive average pooling block (Adaptive Average Pooling, AAP), a first linear compression block (Linear), a first ReLu activation function block (ReLu), a first linear excitation block (Linear) and a first Sigmoid compression function block connected in sequence.

[0102] The first adaptive average pooling block (Adaptive Average Pooling, AAP) compresses the global feature map channel by channel, and the compression obtains more lightweight global channel-level statistical information, and the expression is: , in which, represents the global feature map, , represents the global channel-level statistical information, ;

[0103] The first linear compression block (Linear) compresses the global channel-level statistical information, and maps the global channel-level statistical information to ; after non-linear mapping by the first ReLu activation function block, the first linear excitation block maps to , and finally, the first Sigmoid compression function block is used to compress the mapped global channel-level statistical information; the expression of the above process is:

[0104] ;

[0105] represents a linear compression function for channel-level statistical information, represents a linear excitation function for channel-level statistical information, represents a Sigmoid function function, represents global channel attention.

[0106] The second branch is connected with the convolutional neural network module, and is used to mine channel information inside the local feature map; the second branch includes a second adaptive average pooling block (Adaptive Average Pooling, AAP), a second linear excitation block (Linear), a second ReLu activation function block (ReLu), a second linear compression block (Linear), and a second Sigmoid compression function block connected in sequence, wherein the second adaptive average pooling block compresses the local feature map by channel feature map, and compresses it into more lightweight local channel-level statistical information, and the expression is: , wherein represents a local feature map, , represents local channel-level statistical information, ;

[0107] The second linear excitation block excites the local channel-level statistical information, and maps the local channel-level statistical information from to ; after non-linear mapping by the second ReLu activation function block, the second linear compression block maps the local channel-level statistical information from to ; finally, the Sigmoid compression function block is used to compress the mapped local channel-level statistical information to between 0 and 1, to prevent overflow in probability; the expression of the above process is:

[0108] ;

[0109] In the formula, represents a linear compression function of channel-level statistical information, represents a linear excitation function of channel-level statistical information, represents a Sigmoid function function; represents local channel attention.

[0110] An outer product block is connected with the first branch and the second branch respectively, and constructs a channel correlation matrix based on the channel information inside the global feature map and the local feature map. The expression of the channel correlation matrix is:

[0111] In the formula, , T in the formula is the transpose of the matrix;

[0112] A local softmax block is connected with the outer product block, and is used for adjusting the dimension of the local channel attention in the channel correlation matrix;

[0113] A global softmax block is connected with the outer product block, and is used for adjusting the dimension of the global channel attention in the channel correlation matrix;

[0114] A global subspace block is connected with the transformer module and the global softmax block respectively, and performs a one-mode tensor product of the global feature map and the global channel attention output by the global softmax block to obtain a global subspace. The expression is:

[0115] ;

[0116] In the formula, represents a subspace of .

[0117] A local subspace block is connected with the convolutional neural network module and the local softmax block respectively, and performs a one-mode tensor product of the local feature map and the local channel attention output by the local softmax block to obtain a local subspace. The expression is:

[0118] ;

[0119] In the formula, represents a subspace of .

[0120] A global feature fusion block is connected with the transformer module and the local subspace block respectively, and is used for fusing the local features in the local feature map into the global feature map. The expression is:

[0121] wherein, represents a global feature fusion graph.

[0122] The local feature fusion block is connected with the convolutional neural network module and the global subspace block respectively, and is used for fusing the global feature in the global feature graph into the local feature graph; the expression is:

[0123] wherein, represents a local feature fusion graph.

[0124] The specific network structure and image processing flow of the cross-type spatial feature fusion module in the embodiment include:

[0125] The cross-type spatial feature fusion module includes a 3x3 convolution block, a 5x5 convolution block and a 3x3 output convolution block, the output of the 3x3 convolution block is connected with the local feature fusion graph after being connected with the 3x3 output convolution block, the output of the 5x5 convolution block is connected with the global feature fusion graph after being connected with the 3x3 output convolution block, wherein:

[0126] The 3x3 convolution block is connected with the output end of the global feature fusion block, and is used for performing on the global feature fusion graph , converting the channel feature dimension of the global feature fusion graph from to , the output of the 3x3 convolution block is connected with the expression of ; ;

[0127] The 5x5 convolution block is connected with the output end of the local feature fusion block, and is used for performing on the local feature fusion graph , converting the channel feature dimension of the local feature fusion graph from to ; the output of the 5x5 convolution block is connected with the expression of the global feature fusion graph ; ;

[0128] The 3x3 output convolution block performs splicing on the input and in the channel dimension, and the expression is:

[0129] ;

[0130] wherein, denotes concatenation in the channel dimension; denotes a feature concatenation map.

[0131] Step 3. Obtain multiple data sets containing medical images, and divide the data sets into training sets and test sets; In this embodiment, 8 data sets are obtained, including BUSI (Al-Dhabyani et al., 2020), Dataset B (Yap et al., 2017), ISIC2016 (Gutman et al., 2016), PH2 (Mendonça et al., 2013), KvasirSeg (Jha et al., 2020), CVC-ClinicDB (Jha et al., 2019), Synapse multi-organ segmentation data set (Landman et al., 2015) and Brain-MRI (Buda et al., 2019), which cover five modalities: ultrasonic imaging (US), dermoscopy imaging, computed tomography (CT), colonoscopy and magnetic resonance imaging (MRI).

[0132] Step 4. Set the loss function, train the image segmentation model based on the training set and the loss function, and then test the image segmentation model based on the test set.

[0133] The loss function is a balanced joint loss function, denoted as:

[0134] ;

[0135] In the formula, is a weight factor; denotes the Dice loss function, denotes the cross-entropy loss function, in order to balance the accuracy of pixel-level classification and the optimization of global region, this embodiment sets ; in order to ensure that the contributions of and are equal during the training process, so as to avoid the image segmentation model paying too much attention to pixel-level classification; when , the image segmentation model tends to prioritize the consistency of the global segmentation region, which may ignore the fine-grained pixel-level classification; when , the image segmentation model may perform better in pixel-level classification, but fails to fully optimize the consistency of the global segmentation region.

[0136] Experiment

[0137] A. Experimental setting

[0138] To alleviate overfitting and improve the generalization ability of the model, this experiment applies a variety of data augmentation techniques, including: random cropping with a ratio of 0.5, random horizontal flipping with a probability of 0.5, random vertical flipping with a probability of 0.5, and random rotation with a probability of ±15 degrees. We standardized the images with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225]. These data augmentation strategies are applied to all datasets except the Synapse dataset.

[0139] This experiment is conducted under the PyTorch framework, and all models are trained and tested on NVIDIA A5000 GPUs. And set the random seed for all models to 42, including the initialization of the data loader, data extraction, and data division. The total number of epochs for training is set to 130, including 10 preheating epochs and 120 training epochs. Use the AdamW optimizer, weight decay is , betas parameter is (0.9, 0.999). The initial learning rate is set to 0.0003, and the "Poly" learning rate strategy is used, with a power of . Later, the code will be released on Github for further exploration.

[0140] B. Dataset

[0141] Test on 8 datasets, including a total of 5 modalities. The specific data volume for training, validation, testing, and image scaling size is shown in the following Table 1:

[0142] Table 1: Dataset division

[0143]

[0144] C. Evaluation indicators and comparison methods

[0145] In the experiment, a variety of evaluation indicators are used to strictly evaluate the performance of the model under different modalities, including Dice coefficient, Jaccard index and 95 percentile Hausdorff distance (HD95). The specific description of the key evaluation indicators is as follows:

[0146] Dice coefficient: Dice similarity coefficient is used to measure the overlap between predicted segmentation and true segmentation, and is particularly effective in dealing with class imbalance problems. A higher Dice score indicates a higher similarity between predicted segmentation and true segmentation.

[0147] Jaccard Index: Evaluates the similarity between the predicted segmentation and the ground truth segmentation, it penalizes false positives and false negatives more strictly than Dice. A higher Jaccard score indicates better segmentation accuracy.

[0148] HD95: 95th percentile Hausdorff distance (HD95) quantifies the spatial distance between the predicted segmentation and the ground truth segmentation boundaries, focusing on the largest deviations while ignoring extreme outliers. A lower HD95 value indicates that the predicted segmentation boundaries are closer to the ground truth boundaries.

[0149] To evaluate the efficiency of the model in practical applications and its computational resource requirements, the experiment uses GPU memory usage and frames per second (FPS) as the main evaluation indicators. The experiment does not use the number of parameters as a standard indicator because the number of parameters does not directly reflect the actual storage requirements or running efficiency of the model. For example, the sparsity of the weight matrix may result in a higher number of parameters, but the memory occupation may remain insignificant. In addition, although FLOPs is an important indicator to measure computational complexity, it is not necessarily related to the actual inference speed. The actual performance often depends on the degree of model optimization and the support of underlying hardware. Therefore, using GPU memory usage and FPS as evaluation indicators can more intuitively and accurately reflect the performance of the model in practical applications, making this method more convincing and relevant in practical applications.

[0150] To ensure comprehensive benchmarking, the designed image segmentation model of the present application was compared with a variety of state-of-the-art (SOTA) methods in the experiment, covering both CNN-based models and hybrid CNN-Transformer architectures. For all possible CNN models and hybrid CNN-Transformer models, their pre-trained weights were invoked as much as possible to maintain the rigor of the experiment and ensure fair comparison. The CNN-based models include U-Net (Ronneberger et al., 2015), Attention U-Net (Oktay et al., 2018), ResUnet (Diakogiannis et al., 2020), FATnet (Wu et al., 2022), DCSAUnet (Xu et al., 2023), M2Snet (Zhao et al., 2023), CMUNeXt-Large (Tang et al., 2024), and I2U-Net-Large (Dai et al., 2024), while the hybrid CNN-Transformer models include MISSFormer (Huang et al., 2021), TransUnet (Chen et al., 2021), HiFormer (Base) (Heidari et al., 2023), H2Former (He et al., 2023a), and BEFUnet (Manzari et al., 2024).

[0151] D. Experimental Results

[0152] D1. Ultrasound Image Segmentation Challenge

[0153] Breast ultrasound images usually have characteristics such as uniform color intensity distribution, fuzzy boundaries, and irregular tumor morphology, which may indirectly affect the performance of the model (Zhang et al., 2024). Therefore, this poses a major challenge to the model to effectively capture global features.

[0154] Table 2. Quantitative results of breast ultrasound image segmentation

[0155]

[0156] The quantitative results of breast ultrasound image segmentation are shown in Table 2 and Figure 5As shown, the image segmentation model designed by the present application performs best on the Dice coefficient, Jaccard index and HD95 index on the BUSI dataset and Dataset B. As shown in the table, on the BUSI dataset, the image segmentation model of the present application exceeds the SOTA model H2Former (He et al., 2023a) by 1.31% in Dice coefficient, by 1.81% in Jaccard index, and achieves a lower HD95 value of 7.48. At the same time, on the Dataset B dataset, the image segmentation model of the present application exceeds the SOTA model HiFormer-Base (Heidari et al., 2023) by 2.37% in Dice coefficient, by 3.01% in Jaccard index, and by 3.47% in HD95. It is worth noting that the Dataset B dataset is a small dataset, and this evaluation also tests whether the state-of-the-art (SOTA) model can still achieve accurate segmentation in the case of insufficient medical image data.

[0157] In order to evaluate the generalization ability of the model, a domain transfer experiment was also conducted, in which the model was trained on the relatively large BUSI dataset and tested on the Dataset B dataset. The results show that the domain transfer performance of the image segmentation model of the present application exceeds the model trained directly on the Dataset B dataset in all indicators. As shown in Table 2, we observed that M2Snet (Zhao et al., 2023), TransUnet (Chen et al., 2021) and the image segmentation model of the present application all have significant improvements in various indicators, while the performance of other models remains unchanged or decreases. This indicates that there is still a significant difference in data distribution between the two datasets, and other models encounter problems due to the complexity of the model being too high or too low. These problems can significantly limit the application of these models in actual medical image segmentation tasks. In the domain transfer experiment, our model exceeds the SOTA model TransUnet, achieving a Dice coefficient of 89.52, a Jaccard of 81.81, and an HD95 of 4.01.

[0158] D2. SkinScope image segmentation challenge

[0159] Compared to ultrasound images, dermoscopy images have higher resolution and less noise, resulting in better image quality and more distinct color features. In the experiments, we used the relatively large dataset ISIC-2016 (Gutman et al., 2016) and a smaller dataset PH2 (Mendonça et al., 2013) to evaluate the segmentation performance of our model. In this experiment, we continued to evaluate the generalization ability of the model through the domain transfer scenario.

[0160] Although both datasets focus on melanoma segmentation, the PH2 dataset contains more types of non-melanoma samples, such as 80 common nevi and atypical nevi. This setting challenges the generalization ability of the model and its performance in segmenting abnormal data.

[0161] Table 3 Quantitative results of skin image segmentation

[0162]

[0163] As shown in Table 3 and Figure 6 On the ISIC-2016 dataset (Gutman et al., 2016), most models showed strong segmentation performance, indicating low data complexity. After analysis, we observed that CNN-based models and hybrid models performed equally well, indicating that the clear boundaries and distinct color contrasts of this dataset are particularly beneficial to CNN architectures.

[0164] On the ISIC2016 dataset (Gutman et al., 2016), the image segmentation model of the present application achieved the most advanced performance, with Dice, Jaccard, and HD95 scores of 92.20, 86.55, and 3.06, respectively. At the same time, on the PH2 dataset (Mendonça et al., 2013) containing more sample types, the image segmentation model of the present application outperformed existing methods, achieving Dice, Jaccard, and HD95 scores of 95.14, 90.85, and 0.82, respectively. In the domain transfer experiment, the image segmentation model of the present application ranked third on average in all indicators, demonstrating excellent generalization ability and robust segmentation performance in handling abnormal data. This highlights the effectiveness of the model in dealing with cross-domain challenges in medical image segmentation.

[0165] D3. Intestinal polyp endoscopic image segmentation challenge

[0166] The significant variability in the shape, size, color, location, and texture of polyps in the intestinal polyp endoscopic images poses a great challenge for the model to accurately capture semantic features and boundary recognition. In this study, we evaluated the segmentation performance of the model on two datasets, Kvasir-SEG (Jha et al., 2020) and CVC-ClinicDB (Zhou et al., 2019), respectively, where Kvasir-SEG provides about twice the number of samples as CVC-ClinicDB. The comparison and domain transfer results are summarized in Table 4 and Figure 7

[0167] Table 4. Domain transfer results of intestinal polyp endoscopic images

[0168]

[0169] In Table 4, the image segmentation model of the present application performs outstandingly in polyp image segmentation. On the Kvasir-SEG dataset (Jha et al., 2020), we outperform the SOTA model by 1.93% in the Dice index, 2.99% in the Jaccard index, and have the lowest HD95 value of 5.73. On the CVC-ClinicDB dataset, we outperform M2Snet in all indicators, achieving Dice, Jaccard, and HD95 values of 93.86, 88.71, and 1.77, respectively. In the domain transfer experiment, we first train the model on the Kvasir-SEG dataset (Jha et al., 2020) and then test it on the CVC-ClinicDB dataset (Zhou et al., 2019). The results show that the performance of all models in the domain transfer experiment is lower than that of direct training on the CVC-ClinicDB dataset (Zhou et al., 2019), due to the differences between the two datasets. However, the image segmentation model of the present application still maintains SOTA performance even after domain transfer, fully demonstrating its excellent generalization ability.

[0170] D4. Multi-organ CT image segmentation challenge

[0171] The Synapse dataset (Landman et al., 2015) was chosen for this challenge to evaluate the model's performance in multi-class segmentation tasks. The significant morphological differences between organs and tissues, as well as the data coming from 3D scans (not every CT image contains all organs), pose a great challenge for the model to learn spatial relationships and contextual information. Table 5 shows the performance of our model on the 8 organ segmentation tasks on the Synapse dataset, Figure 8 and some visualization results are presented.​

[0172] Table 5 Organ segmentation task data

[0173]

[0174] The results show that the average Dice score of the image segmentation model of the application on 8 organs is 2.03% higher than that of H2Former (He et al., 2023a), and the average HD is 8.90. In the segmentation challenge of 8 organs, the image segmentation model of the application surpasses the SOTA in the segmentation of the right kidney, liver and stomach, and achieves Dice scores of 91.63, 95.41 and 84.96, respectively. In addition, the image segmentation model of the application also achieves the second best performance in the segmentation of the spleen, left kidney, gallbladder and pancreas. The performance of the image segmentation model of the application on the aorta is also highly competitive. Therefore, through the multi-class segmentation challenge, the image segmentation model of the application demonstrates the ability to handle complex variations and shows strong generalization ability. By combining the feature maps of CNN and Transformer, the ability of the model to learn context information is significantly improved.

[0175] D5. Brain tumor MRI image segmentation challenge

[0176] Irregular shape, heterogeneity and low contrast are still significant challenges in brain tumor MRI image segmentation. In this study, we use the brain MRI segmentation dataset to evaluate the ability of the model to capture context and semantic information. The experimental results are shown in Table 6, and some visualization results are presented in Figure 9 .

[0177] The image segmentation model of the application surpasses the existing state-of-the-art technology (SOTA) in multiple indicators, including Dice, Jaccard, recall, pixel accuracy and HD95. Our model is 0.59% higher than HiFormer-Base (Heidari et al., 2023) in Dice score, reaching a score of 88.18, and 3.57% higher than H2Former (He et al., 2023a). In terms of Jaccard index, the image segmentation model of the application exceeds HiFormer-Base (Heidari et al., 2023) by 0.86%. In terms of pixel accuracy and HD95, the image segmentation model of the application also surpasses the existing state-of-the-art technology, achieving scores of 99.53 and 1.89, respectively. The image segmentation model of the application is also highly competitive with other SOTA models in terms of precision and recall.

[0178] Table 6 Brain tumor MRI image segmentation data table

[0179]

[0180] E. Ablation Experiments

[0181] In the ablation experiments, first, the combination of CNN encoder and decoder was evaluated, where the CNN encoder adopted ResNet34 (He et al., 2016) as the backbone network, and its segmentation performance on eight datasets was evaluated. In addition, the performance of using Swin Transformer V2 (Liu et al., 2022) as the encoder paired with the decoder of the present application was also tested. The results show that the convolutional neural network module performs better on the ultrasound dataset than the transformer module (Transformer), while the Transformer performs better in the polyp segmentation task.

[0182] Next, a double-encoder structure was tried, using a simple convolutional layer to fuse feature maps. However, the results show that this method does not perform as well as using a single encoder. This finding highlights the significant differences between convolutional neural network modules and transformer modules (Transformers) in spatial and channel features, and simple convolutional operations cannot effectively eliminate these differences.

[0183] To handle the differences in channel features, especially when the number of feature maps is inconsistent, a selection mechanism was introduced in the experiment to filter and map the features. Specifically, a matrix was designed to map the channel features according to the features of the two encoders. In the model architecture of the present application, the local features extracted by the convolutional neural network module are fused with the global features extracted by the transformer (Transformer) after channel mapping, as the input of the next convolutional neural network module. Similarly, the global feature fusion is fused with the local feature fusion after channel selection, as the input of the next transformer module. This method enables the convolutional neural network module to obtain global features with a larger receptive field, while providing the transformer module with more detailed local features.

[0184] In addition, the present application integrates a cross-type feature fusion module into the model to effectively fuse spatial features. Through iterative convolution operations and feature fusion, the significant differences in spatial features are gradually alleviated. As shown in Table 7, the image segmentation model of the present application has achieved significant improvement in Dice and Jaccard indicators, and has shown high competitiveness in the HD95 indicator. These results strongly verify the effectiveness of the proposed CFCA and XFF modules.

[0185] Table 7 Ablation Experiment Data Table

[0186]

[0187] F. Performance Analysis

[0188] The number of individual parameters is insufficient to fully capture the actual computational load of a model on a GPU. Therefore, we use peak GPU memory usage to provide a more comprehensive perspective of GPU resource consumption. Similarly, while FLOPs (floating-point operations per second) can measure computational complexity, they do not reliably reflect a model's performance in actual inference. For a more accurate assessment, we report FPS (frames per second) to directly represent inference speed.

[0189] As Figure 10 shown, the image segmentation model of the present invention maintains a moderate level of GPU resource usage, significantly lower than the memory requirements of HiFormer-Base (Heidari et al., 2023), CMUNeXtLarge (Tang et al., 2024), DCSAUnet (Xu et al., 2023), BEFUnet (Manzari et al., 2024), and H2Former (He et al., 2023a). Despite lower memory occupancy, the image segmentation model of the present invention still achieves the highest average Dice score on eight datasets compared with other state-of-the-art (SOTA) models, highlighting its efficiency in resource utilization without compromising segmentation accuracy.

[0190] In Figure 11 , upon further observation of the average inference speed of the image segmentation model of the present invention on 1600 images, it surpasses multiple hybrid models, including H2Former (He et al., 2023a), HiFormer-Base (Heidari et al., 2023), TransUnet (Chen et al., 2021), BEFUnet (Manzari et al., 2024), and CNN-based models such as I2U-Net-Large (Dai et al., 2024), DCSAUnet (Xu et al., 2023), ResUnet (Diakogiannis et al., 2020), and CMUNeXtLarge (Tang et al., 2024). Notably, this speed advantage is combined with the highest average Dice score, highlighting the superiority of the image segmentation model of the present invention in terms of segmentation effectiveness and inference efficiency. These findings indicate that the image segmentation model of the present invention achieves the best balance between GPU efficiency and competitive segmentation performance.

[0191] G. Summary

[0192] The application proposes a novel hybrid CNN-Transformer architecture, which introduces a cross-domain channel attention (CFCA) module after the convolutional neural network module and the transformer module. The cross-domain channel attention (CFCA) module utilizes lightweight cross-channel attention calculation to map feature interactions between the convolutional neural network module and the transformer module. The cross-domain channel attention (CFCA) module enables local features to be integrated into global features while ensuring that the convolutional neural network module can access global feature information. In addition, the application also proposes a spatial feature fusion (XFF) module, which efficiently performs two local and global feature fusions to provide key outputs for the skip connection. This design significantly enhances the model's ability to reconstruct masks with high precision. Extensive experimental results on eight datasets and five modalities show that our model performs well in terms of segmentation performance and generalization ability.

[0193] In the description of the embodiments of the present application, it should be noted that in the description of the present application, the terms indicating the direction or position relationship of "in", "out" and the like are based on the direction or position relationship shown in the drawings, which is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0194] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "in this embodiment", "specific examples" or "some examples" and the like means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0195] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0196] Compared with the prior art, the cross-domain channel attention module firstly performs spatial transformation on the global feature map and the local feature map, converts the per-channel features into one-dimensional channel feature statistics, then interacts the global channel statistics and the local channel statistics through linear excitation and linear compression, effectively reduces the parameter quantity while mining the internal channel correlation; then a tensor product method is used to construct the cross-channel correlation between the global channel statistics and the local channel statistics, and the softmax calculation is performed in the convolution dimension and the transformer dimension respectively, finally the global feature map and the local feature map are multiplied with the channel correlation matrix respectively, realizing the attenuation and increase of the channel number, and completing the mutual mapping and interaction between the local feature map and the global feature map.

[0197] In a possible implementation, the second decoder layer, the third decoder layer, the fourth decoder layer and the fifth decoder layer have the same network structure, and each includes a first convolutional block, a second convolutional block and a deconvolutional block connected in sequence, the first decoder layer includes two CNN decoder layers and a 1x1 convolutional block connected in sequence, the network structure of the two CNN decoder layers is the same as that of the second decoder layer to the fifth decoder layer, and the network structure of the first convolutional block and the second convolutional block each includes a 3x3 convolutional block, a BN block, a ReLu function block, a 3x3 convolutional block, a BN block and a ReLu function block connected in sequence, and performs 3x3 convolution operation, normalization operation and activation operation twice.

[0198] Compared with the prior art, the cross-domain channel attention module is cross-fused with the local feature fusion map, 5x5 convolution is performed on the local feature fusion map to capture a larger receptive field, 3x3 convolution is performed on the global feature fusion map to capture local features, and the final feature map is constructed through addition and splicing operation; to avoid too many channel features received by the decoder layer and the calculation of redundant information, the cross-feature fusion module adds a final 3x3 data convolution block to compress the feature channels, the method realizes the gradual fusion of spatial information through twice cross, and effectively reduces the great difference in spatial features.

[0199] In the description of the embodiments of the present application, it should be noted that in the description of the present application, the terms indicating the direction or position relationship are based on the direction or position relationship shown in the drawings, which is only for the convenience of description, and does not indicate or imply that the device or member must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0200] In the description of the application, the description of the terms "one embodiment", "some embodiments", "in this embodiment", "specific example", or "some examples" and the like means that the specific features, mechanisms, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the description, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0201] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for medical image segmentation based on a hybrid convolutional neural network and transformer, characterized in that, The application relates to the technical field of medical image segmentation, and discloses a medical image segmentation method and device. The method comprises the following steps: Step 1, obtaining a medical image to be segmented; Step 2, constructing an image segmentation model, inputting the medical image to be segmented into the image segmentation model, and the image segmentation model comprising a preprocessing layer, a hybrid encoder layer and a decoder layer; The preprocessing layer is used for image segmentation processing and local feature extraction of the medical image; The hybrid encoder layer is connected with the preprocessing layer and is used for extracting a global feature map and a local feature map from the segmented image and the local feature, constructing a channel feature correlation matrix based on channel features of the global feature map and the local feature map, performing interactive fusion of channel information based on the channel feature correlation matrix, and then performing interactive fusion of the global feature map and the local feature map in spatial information; 2. The medical image segmentation method based on a hybrid convolutional neural network and transformer according to claim 1, characterized in that, The decoder layer is used for performing a concatenation and up-sampling operation on the local feature output by the preprocessing layer and the feature after spatial information interactive fusion of the hybrid encoder layer, and outputs a target segmentation image.

3. The medical image segmentation method based on a hybrid convolutional neural network and transformer according to claim 2, characterized in that, The hybrid encoder layer comprises a first fusion layer, a second fusion layer, a third fusion layer and a fourth fusion layer connected in sequence, the decoder layer comprises a fifth decoder layer, a fourth decoder layer, a third decoder layer, a second decoder layer and a first decoder layer connected in sequence, the first fusion layer transmits the interactive fusion result of communication information to the second fusion layer, the second fusion layer transmits the interactive fusion result of channel information to the third fusion layer, the third fusion layer transmits the interactive fusion result of channel information to the fourth fusion layer, the fourth fusion layer performs spatial information interactive fusion and then enters the fifth decoder layer to perform an up-sampling operation, the third fusion layer performs spatial information interactive fusion, is concatenated with the feature output by the fifth decoder layer, and then enters the fourth decoder layer to perform an up-sampling operation, the second fusion layer performs spatial information interactive fusion, is concatenated with the feature output by the fourth decoder layer, and then enters the third decoder layer to perform an up-sampling operation, the first fusion layer performs spatial information interactive fusion, is concatenated with the feature output by the third decoder layer, and then enters the second decoder layer to perform an up-sampling operation, and the local feature output by the preprocessing layer is concatenated with the feature output by the second decoder layer, then enters the first decoder layer to perform a double up-sampling operation, and finally outputs the target segmentation image.

4. The medical image segmentation method based on a hybrid convolutional neural network and transformer according to claim 3, characterized in that, The second decoder layer, the third decoder layer, the fourth decoder layer and the fifth decoder layer have the same network structure and all comprise a first convolution block, a second convolution block and a deconvolution block connected in sequence, the first decoder layer comprises two CNN decoder layers and a 1x1 convolution block connected in sequence, and the network structure of the two CNN decoder layers is the same as that of the second decoder layer to the fifth decoder layer.

5. The medical image segmentation method based on hybrid convolutional neural network and transformer according to claim 2, characterized in that, The network structure of the first convolution block and the second convolution block both comprises a 3x3 convolution block, a BN block, a ReLu function block, a 3x3 convolution block and a BN block connected in sequence. The network structure of the first fusion layer, the second fusion layer, the third fusion layer and the fourth fusion layer is the same and all comprises a transformer module, a convolutional neural network module, a cross-domain channel attention module and a cross-type spatial feature fusion module. The transformer module is configured to extract a global feature map; The convolutional neural network module is configured to extract a local feature map; The cross-domain channel attention module is connected with the transformer module and the convolutional neural network module respectively, configured to construct a channel feature correlation matrix, and perform interactive fusion of channel information based on the channel feature correlation matrix. The cross-domain channel attention module comprises:

6. The medical image segmentation method based on hybrid convolutional neural network and transformer according to claim 5, characterized in that, The first branch is connected with the transformer module, configured to mine channel information inside the global feature map; The second branch is connected with the convolutional neural network module, configured to mine channel information inside the local feature map; The outer product block is connected with the first branch and the second branch respectively, configured to construct a channel correlation matrix based on the channel information inside the global feature map and the local feature map; The local softmax block is connected with the outer product block, configured to adjust the dimension of local channel attention in the channel correlation matrix; The global softmax block is connected with the outer product block, configured to adjust the dimension of global channel attention in the channel correlation matrix; The global subspace block is connected with the transformer module and the global softmax block respectively, configured to obtain a global subspace by performing a tensor product of the global feature map and the global channel attention output by the global softmax block; The local subspace block is connected with the convolutional neural network module and the local softmax block respectively, configured to obtain a local subspace by performing a tensor product of the local feature map and the local channel attention output by the local softmax block; The global feature fusion block is connected with the transformer module and the local subspace block respectively, configured to fuse local features in the local feature map into the global feature map; The local feature fusion block is connected with the convolutional neural network module and the global subspace block respectively, configured to fuse global features in the global feature map into the local feature map. The first branch comprises a first adaptive average pooling block, a first linear compression block, a first ReLu activation function block, a first linear excitation block and a first Sigmoid compression function block connected in sequence; 7. The hybrid convolutional neural network and transformer-based medical image segmentation method of claim 6, wherein, The expression of the outer product block for constructing the channel correlation matrix is: The first adaptive average pooling block compresses the global feature map by channel-by-channel feature map, and compresses the global channel-level statistical information to be more lightweight, and the expression is: , wherein, represents the global feature map, , represents the global channel-level statistical information, ; The first linear compression block compresses global channel-level statistics, reducing the global channel-level statistics... Mapped to After nonlinear mapping by the first ReLU activation function block, the first linear activation block will... Expansion to Finally, the first Sigmoid compression function block is used to compress the mapped global channel-level statistics; the expression for the above process is: ； represents a linear compression function on the channel-level statistics information, represents a linear excitation function on the channel-level statistics information, represents a Sigmoid function function, represents global channel attention; The second branch comprises a second adaptive average pooling block, a second linear excitation block, a second ReLu activation function block, a second linear compression block and a second Sigmoid compression function block connected in sequence, wherein the second adaptive average pooling block compresses the local feature map by channel feature map, and the compression is local channel level statistical information which is more lightweight, and the expression is: , wherein, is expressed as a local feature map, , is expressed as local channel level statistical information, ; The second linear excitation block excites the local channel level statistical information, and maps the local channel level statistical information from to After being subjected to a second ReLu activation function block for nonlinear mapping, the second linear compression block maps the local channel level statistical information from to Finally, the mapped local channel level statistical information is compressed by a Sigmoid compression function block to be between 0 and 1, to prevent overflow in probability; and the expression of the above process is as follows: ； wherein, represents a linear compression function on the channel-level statistics information, represents a linear excitation function on the channel-level statistics information, represents a Sigmoid function function; represents local channel attention.

8. The medical image segmentation method based on a hybrid convolutional neural network and transformer according to claim 7, characterized in that, The expression of the local softmax block for adjusting the dimension of the local channel attention is: wherein T is the transpose of the matrix , T in the equation is the transpose of the matrix ​ The expression of the global softmax block for adjusting the dimension of the global channel attention is: ； In the formula, represents subspace of The expression of the global feature fusion block for fusing the local features in the local feature map into the global feature map is: ； In the formula, represents subspace; The expression of the local feature fusion block for fusing the global features in the global feature map into the local feature map is: ； In the formula, represents a global feature fusion graph; The medical image segmentation method further comprises: ； In the formula, represents a local feature fusion graph.

9. The medical image segmentation method based on a hybrid convolutional neural network and transformer according to claim 8, characterized in that, The crosswise spatial feature fusion module comprises a 3x3 convolution block, a 5x5 convolution block and a 3x3 output convolution block, the output of the 3x3 convolution block is connected to the local feature fusion graph through skip connection The output of the 5x5 convolution block is connected to the global feature fusion graph through skip connection after being connected to the 3x3 output convolution block The output of the 5x5 convolution block is connected to the global feature fusion graph through skip connection after being connected to the 3x3 output convolution block The output end of the 3*3 convolution block is connected with a global feature fusion block, which is used for fusing a global feature fusion graph is performed , the channel feature dimension of the global feature fusion graph is converted from to , and the output of the 3*3 convolution block is jump-connected with The expression of the 3*3 convolution block is as follows: ; The output end of the 5*5 convolution block is connected with the local feature fusion block, and the local feature fusion graph is obtained is performed The channel feature dimension of the local feature fusion graph is converted from to The output of the 5*5 convolution block is jump-connected with the global feature fusion graph The expression is: ； The 3x3 output convolution block is applied to the input and Concatenation is performed in the channel dimension, with the expression being: ； In the formula, denotes that splicing is performed in the channel dimension; denotes a feature splicing map.

10. The medical image segmentation method based on hybrid convolutional neural network and transformer according to claim 1, characterized in that, Step 3, obtaining a plurality of data sets containing medical images, and dividing the data sets into a training set and a test set; Step 4, setting a loss function, training the image segmentation model based on the training set and the loss function, and then testing the image segmentation model based on the test set; The loss function is a balanced joint loss function, expressed as: ​ ； wherein is a weight factor; denotes the Dice loss function, denotes the cross-entropy loss function.

Images