Heart mri segmentation method and system based on u-net and transformer fusion improvement

Abstract

The application provides a kind of based on U-Net and the heart MRI segmentation method and system of fusion improvement of Transformer, carry out data preprocessing, including first to image is carried out zero mean standardization, and carry out data enhancement processing to expand training data;U-Net jump connection is optimized using Transformer, U-Net jump connection is optimized based on multi-granularity feature, U-Net jump connection is optimized based on cross attention mechanism, according to the model of being built to heart MRI data is trained, will be obtained left ventricle, right ventricle, myocardium and background corresponding pixel probability value again according to different pixel value is classified, to reach the segmentation effect of different regions, finally obtain the labeling result of segmentation.This application can reduce the complexity of calculation while capturing more abundant global features, so as to improve the globality of feature fusion between shallow encoder and decoder, realize more abundant feature fusion effect, and finally realize higher precision heart MRI segmentation effect.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision and medical image segmentation. In cardiac magnetic resonance imaging segmentation, it addresses a series of problems in the application of existing image semantic segmentation in the field of medical image segmentation and proposes a cardiac MRI segmentation method based on the fusion of U-Net and Transformer to achieve accurate segmentation of cardiac MRI. Background Technology

[0002] Medical Imaging: Medical images differ significantly from natural images in everyday life. The differences are as follows: (1) Medical images have a variety of formats, each with its own focus. For example, MRI can more efficiently detect soft tissues, while X-rays provide clearer observations of bones. The imaging results output by different models of imaging equipment also vary to some extent. (2) The pixel value range of medical images is very large. For example, CT images are generally over 1,000 pixels. (3) Due to individual differences, imaging principles, and imaging equipment, medical images usually contain more noise. At the same time, noise is separate in terms of spatial constraints and location, so it can be reduced by analyzing the noise distribution. It should be noted that the preservation of image details also needs to be considered. (4) Medical images usually contain artifacts. Artifacts are produced during 3D reconstruction and image registration. In principle, artifacts can only be reduced, not removed.

[0003] Semantic segmentation: Semantic segmentation is a pixel-level classification. In an image containing dogs and cats, the dog's pixels are classified into one category, the cat's pixels into another, and the background pixels are also classified into a third category. This demonstrates that semantic segmentation understands images at the pixel level. Semantic segmentation tasks demand extremely high classification and localization accuracy: not only must the object's outline be accurately located, but the regions within the outline must also be precisely classified to accurately segment the object from the background.

[0004] Convolutional Neural Networks (CNNs): CNNs are primarily used for precise object localization. This function predicts the bounding box of the object to be located, followed by preprocessing, such as image cropping, to reduce computational overhead. The neurons in each layer of a CNN are structured in three dimensions. Two of these dimensions typically include the image's height and width, while the third dimension usually represents the number of channels in the feature map, the size of which is usually determined by the number of convolutional kernels. The basic structure of a CNN consists of an input layer, convolutional layers, activation functions, pooling layers, and fully connected layers.

[0005] U-Net: While U-Net is also a fully convolutional neural network (FCN) with an encoder-decoder structure, it differs from ordinary FCNs. The biggest difference lies in its method of repairing lost details during downsampling: skip connections are used to generate finer segments. U-Net is designed based on the concept of FCN networks, with only convolutional layers and no fully connected layers. In the condensation path of a FCN, the image resolution gradually decreases while the contextual information gradually increases. In the expansion path, upsampling gradually increases the resolution of the feature maps. Simultaneously, to combine the strong location information of the shallow feature maps, the corresponding parts from the condensation path are incorporated into the expansion path. U-Net effectively aggregates high-resolution information (such as precise segmentation and localization information) and low-resolution information (such as object classification and recognition) through multiple skip connections, achieving accurate segmentation. Therefore, it is widely used in image segmentation in the medical field.

[0006] Transformer: The Transformer model is a type of neural network that learns context and meaning by tracking relationships in sequential data (such as words in a sentence). The Transformer model applies a set of evolving mathematical techniques called attention, or self-attention, to detect subtle ways in which distant data elements in a sequence influence and depend on each other. A standard Transformer architecture consists of an encoder and a decoder. The encoder-decoder structure differs slightly from that of a fully convolutional neural network; the decoder includes an encoder-decoder attention layer, a feedforward neural network, and a self-attention layer, while the encoder includes a feedforward neural network and a self-attention layer. The self-attention mechanism is the most crucial element of the Transformer network, where the Q, K, and V matrices are obtained by linearly transforming the output. The Transformer can capture attention scores across multiple dimensions between words because it uses a multi-head attention mechanism.

[0007] Current cardiac MRI segmentation techniques primarily utilize deep learning algorithms to segment different tissues within cardiac MRI images. This technology is commonly used in medical imaging diagnosis and treatment planning, and is of great significance for the treatment of cardiovascular diseases. While numerous cardiac MRI segmentation methods based on deep neural networks exist, the following problems remain: First, due to the complex structure of the heart, blurred substructure boundaries, and motion artifacts and noise generated during image acquisition caused by cardiac pulsation, accurate localization and segmentation of cardiac structures are extremely difficult. Second, the U-Net network, which excels in image segmentation, is limited by the local field of view characteristics of convolutional networks, restricting its ability to extract global contextual information and hindering accurate segmentation. Summary of the Invention

[0008] To address the issues raised above, a medical image segmentation model, CMT-UNet, based on the fusion of U-Net and Transformer, was constructed, which improved the segmentation accuracy of cardiac MRI.

[0009] On the other hand, the present invention provides a cardiac MRI segmentation method based on the fusion of U-Net and Transformer, comprising the following steps:

[0010] Step 1, data preprocessing, includes first normalizing the images to zero mean and then performing data augmentation to augment the training data;

[0011] Step 2: Optimize U-Net skip connections using Transformer, including embedding Transformer into multiple skip connections in the network to form a Transformer skip connection structure. This structure is used to extract semantic features from the encoder output and filter out semantic features and noise features that are irrelevant to the task.

[0012] Step 3: Optimize U-Net skip connections based on multi-granularity features, including improving the Transformer embedded in the Bottleneck layer of U-Net and the Skip1 and Skip2 layers above the Bottleneck layer. Multi-granularity features are constructed through spatial transformation Unfold, allowing the improved Transformer to extract multi-granularity contextual information.

[0013] Step 4: Optimize U-Net skip connections based on cross-attention mechanism. This includes improving the Transformer based on cross-attention mechanism, which is denoted as CA-MGTransformer. The improved Transformer is embedded in Skip3 and Skip4 layers to perform feature fusion between the encoder and decoder. The query input sequence required for calculation in the multi-head attention layer of the added CA-MG Transformer comes from Skip3 and Skip4 layers themselves, while the key-value sequence required for calculation comes from the Bottleneck layer. The remote interaction information captured by the interaction is calculated through cross-attention, and the feature map of the lower-scale Bottleneck layer is introduced as the key-value pair sequence.

[0014] Step 5: Train the cardiac MRI data based on the constructed model, and then classify the corresponding pixel probability values ​​of the left ventricle, right ventricle, myocardium and background according to different pixel values ​​to achieve the segmentation effect of different regions, and finally obtain the segmented and labeled results.

[0015] Furthermore, data augmentation operations on the data include elastic deformation, translation, scaling, and random rotation.

[0016] Furthermore, the implementation method for optimizing U-Net skip connections using Transformer is as follows:

[0017] Step 2.1: Construct an encoder containing five convolution-batch normalization-ReLU activation function modules. Each convolution-batch normalization-ReLU activation function module is followed by a maxpooling operation for downsampling. This stepwise downsampling method encodes the original image into a low-resolution feature map to extract abstract semantic information and obtain rich local contextual information.

[0018] Step 2.2: Construct the Transformer module for high-level semantic feature enhancement;

[0019] Step 2.3 involves constructing a skip connection structure. Before concatenating the feature maps corresponding to the encoder and decoder, the feature map of the encoder is first input into the Transformer to extract global context information, and then fused with the feature map corresponding to the decoder. A Transformer is added to the two skip connections above the Bottleneck layer. The feature maps of the encoder corresponding to these two skip connections are input into the Transformer and then fused with the feature maps of the corresponding positions in the decoder by addition. The last two skip connections are still completed by the original direct concatenation method.

[0020] Step 2.4: Construct a decoder containing four upsampling modules to gradually restore the feature map size through progressive upsampling, thereby obtaining a segmentation result with the same size as the original image in the last layer.

[0021] Furthermore, the implementation method of optimizing U-Net skip connections based on multi-granularity features is as follows:

[0022] Step 3.1: Construct multi-granularity features using Unfold;

[0023] Step 3.2, improve the Transformer based on multi-granularity features, including unfolding the feature map X into a two-dimensional vector through the Flattern operation, then adding position encoding to obtain vector V. Here, the position encoding adopts a learnable position encoding. Then, the vector is reshaped to restore it to a three-dimensional vector V′, at which point the dimensions are (H,W,C), which facilitates the extraction of local information. Then, the local information is extracted through Unfold operations with different window sizes to construct multi-granularity features.

[0024] After the above processing, the surrounding local structure is embedded into the generated token and input into the multi-head attention layer in the Transformer for computation. The Fold operation is equivalent to the inverse of the Unfold operation, converting the token output by the multi-head attention layer back into a three-dimensional tensor F, which is then unfolded into a two-dimensional vector F′ through the Flattern operation. After that, the two-dimensional vector F′ is summed to obtain a new two-dimensional vector Z. Then, the two-dimensional vector Z is normalized by Layer Norm and input into the MLP multilayer perceptron, and then residual connections are added. After the above operations, the output feature vector Z″ is fused with the feature vector of the same layer in the corresponding upsampling.

[0025] Furthermore, the implementation method of optimizing U-Net skip connections based on the cross-attention mechanism is as follows:

[0026] Step 4.1, calculate the Attention mechanism, including calculating weight scores using the key and query input sequence in the key-value pair, and mapping the weight scores to the value of the key-value pair;

[0027] Step 4.2, the cross-attention mechanism uses one sequence as the input to the query Q and another sequence as the key and value input (K,V). Let S1 and S2 be the two embedding sequences for the cross-attention mechanism. The formula for calculating the cross-attention between these two sequences is as follows:

[0028] Attention(S1,S2)=softmax((W Q S1)(W K S2)T W V S2

[0029] First, calculate (K,V) for sequence S2, and simultaneously calculate Q for sequence S1. Then, calculate the attention matrix based on the calculated K and Q, where W is... Q W K W V The weight matrix is ​​represented by V, and Attention represents the final calculated attention result. The attention weights are then calculated using the Softmax function, and the weights are mapped to the corresponding V. Finally, the cross-attention result of the S1 and S2 sequences is output.

[0030] Step 4.3, the calculation process of feature fusion from encoders Enc1 and Enc2 via the improved Transformer to the decoder is as follows:

[0031]

[0032] I = Reshape(Flatten(E) + PE)

[0033] A = MCA(Q) Unfold(I) ,K BN V BN )

[0034] U = add(Flatten(Fold(A)))

[0035] Z = U + MLP(LayerNorm(U))

[0036] Among them, subscript Q represents the encoder Enc1 from Skip4 layer or the encoder Enc2 from Skip3 layer. X represents the feature map input. The feature information is output from the encoder module and then flattened into a two-dimensional sequence. A position-encoded attention vector PE is added, and then the reshape operation restores it into a three-dimensional vector I. The obtained I is transformed into a query input sequence through Unfold, and then input together with the key-value pair sequence obtained from the Bottleneck layer into the multi-head attention layer MCA for attention calculation. The subscript BN is used to identify the Bottleneck layer. A is the result of the attention calculation interaction after improving the Transformer based on the cross-attention mechanism. Then, the obtained attention result is restored through the Fold operation, and then flattened into a vector through the Flatten operation. Then, the vectors are added through the add operation to get the result U. Then, the LayerNorm operation is performed and input into the multilayer perceptron layer MLP. Then, it is added with U to get the output Z. MCA represents the multi-head cross-attention mechanism.

[0037] On the other hand, the present invention also provides a cardiac MRI segmentation system based on the fusion of U-Net and Transformer, which is used to implement the cardiac MRI segmentation method based on the fusion of U-Net and Transformer as described above.

[0038] Moreover, it includes the following modules,

[0039] The first module is used for data preprocessing, which includes first normalizing the images to zero mean and performing data augmentation to augment the training data;

[0040] The second module is used to optimize U-Net skip connections using Transformer, including embedding Transformer into multiple skip connections in the network to form a Transformer skip connection structure. Through this structure, the semantic features output by the encoder are re-extracted, and semantic features and noise features that are irrelevant to the task are filtered out.

[0041] The third module is used to optimize U-Net skip connections based on multi-granularity features. This includes improving the Transformer embedded in the Bottleneck layer of U-Net and the Skip1 and Skip2 layers above the Bottleneck layer. Multi-granularity features are constructed through spatial transformation Unfold, allowing the improved Transformer to extract multi-granularity contextual information.

[0042] The fourth module is used to optimize U-Net skip connections based on the cross-attention mechanism. This includes improving the Transformer based on the cross-attention mechanism, which is denoted as CA-MGTransformer. The improved Transformer is embedded in the Skip3 and Skip4 layers to perform feature fusion between the encoder and decoder. The query input sequence required for computation in the multi-head attention layer of the added CA-MG Transformer comes from the Skip3 and Skip4 layers themselves, while the key-value sequence required for computation comes from the Bottleneck layer. The remote interaction information captured by the interaction is computed through cross-attention, and the feature map of the lower-scale Bottleneck layer is introduced as the key-value pair sequence.

[0043] The fifth module is used to train cardiac MRI data based on the constructed model. The obtained pixel probability values ​​of the left ventricle, right ventricle, myocardium, and background are then classified according to different pixel values ​​to achieve segmentation effects of different regions, and finally obtain the segmented and labeled results.

[0044] Alternatively, it may include a processor and a memory, with the memory used to store program instructions and the processor used to call the stored instructions in the memory to execute a cardiac MRI segmentation method based on the fusion of U-Net and Transformer as described above.

[0045] Alternatively, it may include a readable storage medium storing a computer program that, when executed, implements a cardiac MRI segmentation method based on the fusion of U-Net and Transformer as described above.

[0046] The advantages of this invention are: (1) The Transformer is embedded into multiple skip connections in the network to form a Transformer skip connection structure. This structure can extract semantic features from the encoder output, filter out semantic features and noise features that are irrelevant to the task, and thus promote the fusion of multi-level features on the encoder and decoder. (2) The embedded Transformer is improved to take advantage of the module to establish global dependencies and improve the model’s attention to local information. By constructing features of different granularities through multiple Unfolds of different window sizes, while preserving the integrity of information, the improved Transformer extracts multi-granular context features, which makes up for the interference caused to the convolutional network and thus improves performance. (3) The embedded Transformer is used to construct cross-attention between deep semantic information and shallow detail information of U-Net to reduce the interference of irrelevant semantic information. While capturing richer global features, it can also reduce computational complexity, thereby improving the globality of feature fusion between shallow encoders and decoders and achieving richer feature fusion effects. Finally, higher precision cardiac MRI segmentation effect is achieved. Attached Figure Description

[0047] Figure 1 This is a diagram illustrating the structure of the U-Net skip connection model optimized using Transformer according to an embodiment of the present invention.

[0048] Figure 2 This is a diagram of the overall model structure of the Transformer improved based on multi-granularity features according to the method of this invention.

[0049] Figure 3 This is a cross-attention structure diagram of the method in the embodiment of the present invention.

[0050] Figure 4 This is a diagram of the final improved CMT-UNet model structure of the method in the embodiments of the present invention.

[0051] Figure 5 This is a visualization result of cardiac MRI segmentation using the method described in this embodiment of the invention. Detailed Implementation

[0052] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0053] This invention constructs and trains an improved cardiac MRI segmentation method model based on the fusion of U-Net and Transformer, and then uses the trained model for cardiac MRI segmentation. This invention filters out semantic and noisy features irrelevant to the task by embedding the Transformer into multiple skip connections in the network, thereby promoting the fusion of multi-level features at the encoder-decoder level. By constructing features of different granularities through multiple unfolds with different window sizes, it compensates for interference to the convolutional network, reduces framework redundancy, and thus achieves performance improvement. While capturing richer global features, it also reduces computational complexity, thereby improving the globality of feature fusion between shallow encoders and decoders and achieving richer feature fusion effects. The final model achieves better cardiac MRI segmentation results, with a significant improvement in segmentation accuracy.

[0054] See Figure 4 This invention provides a cardiac MRI segmentation method based on the fusion of U-Net and Transformer, comprising the following steps:

[0055] Step 1: Data Preprocessing

[0056] First, the images are normalized to zero mean, making the input images have a mean of 0 and a variance of 1. Zero-mean normalization confines the data within a certain range, standardizing the data and accelerating training. Zero-mean normalization is often called Z-score normalization. This method uses the mean (μ) and standard deviation (σ) of the initial data to process it into a standardized distribution. After processing, the new data conforms to a standard normal distribution, i.e., a mean of 0 and a standard deviation of 1. The transformation function is:

[0057]

[0058] Where x represents the original image pixels, μ represents the mean of the image pixels, and σ represents the standard deviation of the image pixels.

[0059] Secondly, image data was augmented to improve the model's generalization ability. During the model training phase, all data underwent data augmentation to expand the training data. The data augmentation operations included elastic deformation, translation, scaling, and random rotation. The parameter ranges of α and β for elastic deformation were (0., 350) and (14., 17.), respectively. Translation operations were also performed in the width and height directions of the image, with a range of (-10, 10). The scaling factor ranged from (0.7, 1.3), and the random rotation angle ranged from (-π / 6, π / 6).

[0060] Step 2: Optimize U-Net skip connections using Transformer

[0061] Embedding the Transformer into multiple skip connections in the network constitutes a Transformer skip connection structure. This structure can re-extract semantic features from the encoder output, filtering out semantic features and noise features that are irrelevant to the task, thereby promoting the fusion of multi-level features in the encoder and decoder.

[0062] Step 2.1, Construct the encoder. The encoder structure in the model contains five convolutional-batch normalization-ReLU activation function modules. Each convolutional-batch normalization-ReLU activation function module contains two 2D convolutions, one batch normalization (BN), and one ReLU activation function. Each convolutional-batch normalization-ReLU activation function module is followed by a 2x2 maxpooling operation for downsampling. After each downsampling, the feature map size is halved, while the number of channels is doubled. Thus, the entire encoding structure contains four downsampling operations. This progressive downsampling method encodes the original image into a low-resolution feature map to extract abstract semantic information, while also obtaining rich local contextual information. After the last convolutional-batch normalization-ReLU activation function module, the Bottleneck layer (such as...) Figure 1 The output of the encoder is obtained by adding a residual connection after the output of the Transformer (shown in the diagram) and then adding it to the input of the Transformer.

[0063] Step 2.2: Construct the Transformer module for high-level semantic feature enhancement. For example... Figure 1 As shown in the Transformer module, specifically, for a feature map First, flatten it into a two-dimensional token. Where N = H × W, this process is denoted as M2T, such as Figure 1 The "M2T" module is shown in the diagram. Simultaneously, to encode the image's positional information, a positional encoding, denoted as PE, is added to the obtained tokens. This PE is a learnable parameter matrix of the same size as E, used to represent the positional relationships of elements within each token. The positional encodings are directly added to token E to obtain the final token T: T = E + PE. T is then input into the Transformer. In the introduced Transformer, the Encoder mainly contains two modules: a multi-head self-attention mechanism (MSA) and a multilayer perceptron (MLP). (See diagram...) Figure 1As shown, the entire Transformer process is as follows: For a two-dimensional Token T, the MSA module first uses three sets of linear mappings to obtain Q, K, and V, as shown below:

[0064] Q = TW Q K = TW k V = TW v

[0065] in, These are the learnable parameters of the three linear mapping layers, and d is the size of the second dimension of (Q,K,V). Thus, the subsequent self-attention mechanism (SA) can be expressed as follows:

[0066]

[0067] in, These are the i-th rows of Z and Q, respectively. MSA is an extension of SA, which includes multiple SA operations. Finally, the results are concatenated together and then a linear mapping is added to obtain the MSA result, as shown in the following formula:

[0068] MSA(Z)=[SA1(Z);SA2(Z);...;SA m (Z)]W O

[0069] in m is the number of heads in the MSA. The output of the MSA is then fed into a multilayer perceptron (MLP) layer for transformation, and finally residual connections are added to obtain the final result. The entire process is shown in the following formula:

[0070]

[0071] Layer normalization was applied to the data before it was input into the MSA and MLP modules. Specific details are as follows: Figure 1 As shown, the sequence token obtained after processing by Transformer It needs to be converted into a two-dimensional feature map before it can be fused with the corresponding feature map of the decoder. Here, C2 is the size of the second dimension of the Transformer output, which is equal to the number of channels in the corresponding feature map of the decoder. This processing method is as follows: Figure 1 The “T2M” module is shown in the image.

[0072] Step 2.3: Constructing the skip connection structure. Before concatenating the feature maps corresponding to the encoder and decoder, the encoder's feature map is first input into the Transformer to extract global context information, and then fused with the corresponding feature map of the decoder. Considering that segmentation tasks require sufficient local detail information, such as edge information and spatial information, the shallow feature maps of the encoder often have rich high-resolution spatial detail information. To prevent the Transformer from destroying this local detail information, and considering that the shallow feature maps are large in scale and computationally complex, the skip connection between the shallow feature map encoder and decoder does not include a Transformer, but concatenates directly in the original way. A Transformer is added to the Bottleneck layer and the two skip connections above it (denoted as Skip0, Skip1, and Skip2). The feature maps of the encoder corresponding to these two skip connections are input into the Transformer and then fused with the corresponding feature maps of the decoder by addition. The last two skip connections (denoted as Skip3 and Skip4) are still completed in the original direct concatenation method, and finally the end-to-end segmentation network structure is fully realized.

[0073] Step 2.4, Construct the decoder. After establishing long-range dependencies using Transformer on the Bottleneck layer and Skip1 and Skip2 skip connections, the size of the deep feature maps needs to be restored to the original image size to complete end-to-end segmentation. The decoder mainly consists of four upsampling modules. The first two upsampling modules include an Upsample module, a 1x1 convolution, a batch normalization (BN), and a ReLU activation function. No further convolution operations are added to the first two upsampling modules. In order to preserve the local details of the feature map, the last two upsampling layers (Skip3 and Skip4) directly use skip connections to fuse with the feature map corresponding to the encoder, without being input into the Transformer. For the last two upsampling layers, the decoder feature map is first upsampled to twice its size using the same module as the previous upsampling, while the number of channels is halved. Then it is concatenated with the feature map corresponding to the encoder. Then a convolution module is followed to further fuse local detail features and global semantic features. The feature map size is gradually restored in this step-by-step upsampling manner, so that the segmentation result with the same size as the original image is obtained in the last layer.

[0074] Step 3: Optimize U-Net skip connections based on multi-granularity features. This includes improving the Transformers embedded in the Bottleneck layer of U-Net and the Skip1 and Skip2 layers above the Bottleneck layer. Multi-granularity features are constructed by Unfolding, allowing the improved Transformer to extract multi-granularity contextual information, thereby avoiding interference to the convolutional network. At the same time, the redundancy of the framework is reduced by modeling local structural information, thus improving performance.

[0075] Step 3.1: Construct multi-granularity features using Unfold. Unfold is a data transformation operation, such as... Figure 2 As shown in the MultiGranularity Transformer module, it unfolds a multidimensional tensor into a two-dimensional matrix. The Unfold operation essentially extracts sliding local regions from a batch image, rearranges the vectors of each convolutional window into a column vector, and then concatenates them. Since there should be stronger correlations between surrounding tokens in the feature map, Unfold can fuse locally adjacent tokens into a new token, simulating the local structural information of surrounding tokens. The specific formula for the Unfold operation is shown below:

[0076] unfold(X,kernel_size,stride)i,j

[0077] =X:,(i×stride0):(i×stride0+kernel_size0),(j×stride1):(j×stride1+kernel_size1)

[0078] Here, X is the input tensor with shape (N, C, H, W), where N is the batch size, C is the number of channels, and H and W are the height and width of the input tensor, respectively. kernel_size is a tuple (kernel_size0, kernel_size1) representing the size of the two-dimensional matrix to be expanded, and stride is a tuple (stride0, stride1) representing the stride size during expansion. The final expanded two-dimensional matrix has the following size: Unfold operates on tensor X using a (kernel_size0, kernel_size1) kernel. Similar to a convolution operation, it takes a C×kernel_size0×kernel_size1 block at a time and flattens it into a C×kernel_size0×kernel_size1×1×1 block. This way, Unfold expands the (H,W)-dimensional patch block to the channel dimension without losing feature map information. Multi-granularity features can be constructed by using Unfold operations with different window sizes.

[0079] Step 3.2, improve Transformer based on multi-granularity features.

[0080] Here, the Transformer improved based on multi-granularity features is denoted as MultiGranularityTransformer, such as... Figure 2 The MultiGranularity Transformer module is shown in the diagram. A MultiGranularity Transformer is added to the skip connections of the Bottleneck Layer (Skip0) and the two layers above it (Skip1, Skip2) to fuse downsampled and upsampled features. The feature map X is flattened into a two-dimensional vector using a Flatten operation, and then a positional encoding is added to obtain a vector V. Here, a learnable positional encoding is used. After the above operations, the output two-dimensional vector V has dimensions (N,D). This vector is then reshaped to restore it to a three-dimensional vector V′ with dimensions (H,W,C), facilitating the extraction of local information. Further local information is extracted through Unfold operations with different window sizes to construct multi-granularity features. The surrounding tokens are segmented, flattened, and clustered together to generate new tokens. This iterative process reduces the token length, transforming the spatial structure of the image without losing information. After these steps, the surrounding local structure is embedded into the generated tokens and input into the multi-head attention layer of the Transformer for computation. The Fold operation is essentially the inverse of the Unfold operation. It converts the tokens output from the multi-head attention layer back into a 3D tensor F, then expands it into a 2D vector F′ using the Flattern operation. Next, the 2D vector F′ is summed to obtain a new 2D vector Z. These operations fuse the internal information of each patch block to model local information. The 2D vector Z is then normalized using Layer Norm and input into an MLP (Multilayer Perceptron), with residual connections added. After these operations, the output feature vector Z is... ″It is fused with the feature vector of the same layer in the corresponding upsampling.

[0081] In practice, the following steps can be used:

[0082] Step 3.2.1: Since the input in Vision Transformer is a two-dimensional vector, while the input here is three-dimensional, the feature map X needs to be unfolded into a two-dimensional vector through the Flatten operation, and then a positional encoding is added to obtain the vector V. Here, the positional encoding adopts a learnable positional encoding.

[0083] Step 3.2.2: After the above operations, the output two-dimensional vector V has dimensions (N,D). This vector is then reshaped to a three-dimensional vector V′, with dimensions (H,W,C), facilitating the extraction of local information, such as... Figure 2 The process is shown in the MultiGranularity Transformer module.

[0084] Step 3.2.3 Given the parameter t, the iterative formula...

[0085]

[0086] Step 3.2.4 involves flattening the vector into a two-dimensional vector F′ using the Flatten operation, followed by summing the F′ vector to obtain a new two-dimensional vector Z. This process fuses the internal information of each patch block to model local information. Next, Layer Norm normalization is applied to the two-dimensional vector Z, which is then input into the MLP (Multilayer Perceptron) and a residual connection is added. After these operations, the output feature vector Z″ is fused with the feature vectors from the same layer in the upsampling process, as shown below. Figure 2 The process is shown in the MultiGranularity Transformer module.

[0087] Step 4: Optimize U-Net skip connections based on a cross-attention mechanism. This includes improving the Transformer based on a cross-attention mechanism, denoted as CA-MGTransformer. The improved Transformer is embedded in Skip3 and Skip4 layers for encoder-decoder feature fusion. In the multi-head attention layer of the added CA-MG Transformer, the required query input sequence is calculated from Skip3 and Skip4 layers themselves, while the required key-value sequence is calculated from the Bottleneck layer. Through cross-attention computation, remote interaction information is captured, thereby improving the feature fusion effect of Skip3 and Skip4 layers. At the same time, by introducing the lower-scale Bottleneck layer feature map as the key-value pair sequence, the attention computation overhead is also reduced.

[0088] Step 4.1 involves two steps to calculate the Attention mechanism. The first step calculates weight scores using the Key and Query. This step can be divided into two processes: 1) Calculate the similarity between the Key and Query using a similarity function, returning an attention map; 2) Normalize the scores from the first step. The second step maps the obtained weight scores to the Value.

[0089] Step 4.2, Cross Attention utilizes one sequence as the input to the query Q and another sequence as the key and value input (K, V). The two embedding sequences for Cross Attention are defined as S1 and S2, as follows: Figure 3 As shown. The formula for calculating the cross-attention between these two sequences is as follows:

[0090] Attention(S1,S2)=softmax((W Q S1)(W K S2) T W V S2

[0091] First, calculate (K, V) for sequence S2 and Q for sequence S1. Then, calculate the attention matrix based on the calculated K and Q. Next, calculate the attention weights using the Softmax function, and then map the obtained weights to the corresponding V. Finally, output the cross-attention results (Attention Scores) of sequences S1 and S2. The cross-attention calculation is as follows: Figure 3 As shown.

[0092] Step 4.3, the computation process of feature fusion from the encoder (Enc1, Enc2) to the decoder via the improved Transformer is as follows:

[0093]

[0094] I = Reshape(Flatten(E) + PE)

[0095] A = MCA(Q) Unfold(I) ,K BN V BN )

[0096] U = add(Flatten(Fold(A)))

[0097] Z = U + MLP(LayerNorm(U))

[0098] The attention vector A is the result of the attention calculation interaction after improving the Transformer based on the cross-attention mechanism, and then output. Enc1 and Enc2 represent the skip connections of the first two layers, such as... Figure 4 As shown. MCA represents the multi-head cross-attention mechanism:

[0099] MCA(Q,K,V)=[CA1(Q,K,V);CA2(Q,K,V);…;CA num_heads (Q,K,V)]W o f

[0100] in m is the number of heads in the multi-head cross-attention mechanism (MCA).

[0101] The final model structure diagram is shown in the attached diagram in the instruction manual. Figure 4 .

[0102] Step 5: Train the cardiac MRI data based on the constructed model.

[0103] The model employed the Adam optimization method during training, with a batch size of 4 and a total of 6000 iterations. The kernel size for both downsampling and upsampling convolutional layers was 3×3. During downsampling, the number of kernels in each convolutional module was 32, 64, 128, 256, and 512, respectively. The initial learning rate was set to 1e-4, and it decreased by 1 / 10 every 30 iterations. The weight decay was set to 1e-3.

[0104] The model training uses the Dice Loss function. The Dice coefficient has a threshold value between [0,1], with a best result of 1 and a worst result of 0. The formula for calculating the Dice coefficient is as follows:

[0105]

[0106] Where pred represents the set of predicted values ​​and true represents the set of true values. As can be seen from the formula, the numerator is the intersection between the predicted values ​​and the true values. In order to prevent the repeated calculation of the same elements between the predicted values ​​and the true values, the numerator is multiplied by 2. The denominator is the union between the predicted values ​​and the true values.

[0107] The Dice loss function works by subtracting the Dice coefficient from the Dice coefficient by subtracting 1. The formula is as follows:

[0108]

[0109] Since Dice Loss is better suited for handling cases of severe imbalance between positive and negative samples, the Dice Loss loss function is used as the loss function during training.

[0110] Finally, the left and right ventricles of the heart, myocardium, and background are classified according to different pixel probability values ​​obtained through softmax processing, thereby achieving segmentation of different regions. The final segmented and labeled results are obtained, with green representing myocardium, red representing the left ventricle, and blue representing the right ventricle. The segmentation results are visualized as follows: Figure 5 As shown in the figure. Input represents the input data, Ours represents the segmentation result of the method of this invention, and GT represents the ground truth label.

[0111] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0112] In some possible embodiments, a cardiac MRI segmentation system based on the fusion of U-Net and Transformer is provided, comprising the following modules:

[0113] The first module is used for data preprocessing, which includes first normalizing the images to zero mean and performing data augmentation to augment the training data;

[0114] The second module is used to optimize U-Net skip connections using Transformer, including embedding Transformer into multiple skip connections in the network to form a Transformer skip connection structure. Through this structure, the semantic features output by the encoder are re-extracted, and semantic features and noise features that are irrelevant to the task are filtered out.

[0115] The third module is used to optimize U-Net skip connections based on multi-granularity features. This includes improving the Transformer embedded in the Bottleneck layer of U-Net and the Skip1 and Skip2 layers above the Bottleneck layer. Multi-granularity features are constructed through spatial transformation Unfold, allowing the improved Transformer to extract multi-granularity contextual information.

[0116] The fourth module is used to optimize U-Net skip connections based on the cross-attention mechanism. This includes improving the Transformer based on the cross-attention mechanism, which is denoted as CA-MGTransformer. The improved Transformer is embedded in the Skip3 and Skip4 layers to perform feature fusion between the encoder and decoder. The query input sequence required for computation in the multi-head attention layer of the added CA-MG Transformer comes from the Skip3 and Skip4 layers themselves, while the key-value sequence required for computation comes from the Bottleneck layer. The remote interaction information captured by the interaction is computed through cross-attention, and the feature map of the lower-scale Bottleneck layer is introduced as the key-value pair sequence.

[0117] The fifth module is used to train cardiac MRI data based on the constructed model. The obtained pixel probability values ​​of the left ventricle, right ventricle, myocardium, and background are then classified according to different pixel values ​​to achieve segmentation effects of different regions, and finally obtain the segmented and labeled results.

[0118] In some possible embodiments, a cardiac MRI segmentation system based on the fusion of U-Net and Transformer is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a cardiac MRI segmentation method based on the fusion of U-Net and Transformer as described above.

[0119] In some possible embodiments, a cardiac MRI segmentation system based on the fusion of U-Net and Transformer is provided, including a readable storage medium on which a computer program is stored. When the computer program is executed, it implements the cardiac MRI segmentation method based on the fusion of U-Net and Transformer as described above.

[0120] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art may modify or supplement the described embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A cardiac MRI segmentation method based on U-Net and Transformer fusion improvement, characterized in that, The method comprises the following steps: Step 1, data preprocessing, including first zero-mean standardization of the image, and data enhancement processing to expand the training data; Step 2, optimizing the U-Net skip connection by using the Transformer, including embedding the Transformer into multiple skip connections in the network to form a Transformer skip connection structure, and extracting features from the semantic features output by the encoder through the structure, filtering the semantic features and noise features irrelevant to the task; Step 3, optimizing the U-Net skip connection based on multi-granularity features, including improving the Transformer embedded in the Bottleneck layer and the Skip1 and Skip2 layers above the Bottleneck layer of the U-Net, and constructing multi-granularity features through spatial conversion Unfold to enable the improved Transformer to extract multi-granularity context information; Step 4, optimizing the U-Net skip connection based on the cross-attention mechanism, including improving the Transformer based on the cross-attention mechanism, denoted as CA-MGTransformer, embedding the improved Transformer in the Skip3 and Skip4 layers for encoder-decoder feature fusion, calculating the query input sequence needed in the multi-head attention layer of the added CA-MG Transformer from the Skip3 and Skip4 layers themselves, and calculating the key-value sequence needed from the Bottleneck layer, and calculating the remote interaction information captured through cross-attention, while introducing the feature map of the lower-scale Bottleneck layer as the key-value pair sequence; Step 5, training the heart MRI data according to the constructed model, classifying the corresponding pixel probability values of the left ventricle, right ventricle, myocardium and background of the heart obtained to achieve the segmentation effect of different regions, and finally obtaining the segmented labeling results.

2. The method according to claim 1, wherein the method is characterized in that: The data enhancement operation on the data includes elastic deformation, translation, scaling and random rotation.

3. The method of claim 1, wherein the method is based on a fusion of U-Net and Transformer. The implementation mode of the Transformer optimizing the U-Net skip connection is as follows, Step 2.1, constructing an encoder, containing five convolution-batch normalization-ReLu activation function modules, each convolution-batch normalization-ReLu activation function module is followed by a maxpooling operation for down-sampling, and the original image is encoded into a low-resolution feature map to extract abstract semantic information while obtaining rich local context information by using this step-by-step down-sampling method; Step 2.2, constructing a Transformer module for high-level semantic feature enhancement; Step 2.3, constructing a skip connection structure, including first inputting the feature map of the encoder into the Transformer to extract global context information before splicing the feature map corresponding to the encoder and the decoder, then fusing the feature map corresponding to the decoder, adding the Transformer in the bottleneck layer and the two skip connections above, inputting the feature map in the encoder corresponding to the two skip connections into the Transformer, and then fusing the feature map in the corresponding position of the decoder in the form of addition, and the last two skip connections are still spliced directly; Step 2.4, constructing a decoder, including four up-sampling modules, to gradually restore the size of the feature map in the form of gradual up-sampling, so as to obtain a segmentation result with the same size as the original image in the last layer.

4. The method according to claim 1, wherein the method is characterized in that: The implementation of the U-Net skip connection based on multi-granularity feature optimization is as follows, Step 3.1, constructing multi-granularity features by using Unfold; Step 3.2, improving the Transformer based on multi-granularity features, including expanding the feature map X into a two-dimensional vector by Flattern operation, then adding position encoding to obtain vector V, where the position encoding adopts learnable position encoding, then reshaping the vector to a three-dimensional vector V', which has a dimension of (, W, C) for easy extraction of local information; then extracting local information by Unfold operation with different window sizes to construct multi-granularity features; After the above processing, the surrounding local structure is embedded into the generated Token and input into the multi-head attention layer of the Transformer for calculation; the Fold operation is equivalent to the inverse operation of the Unfold operation, which converts the Token output by the multi-head attention layer back to a three-dimensional tensor F, and then expands it into a two-dimensional vector F' by Flattern operation, and then sums the two-dimensional vector F' to obtain a new two-dimensional vector Z; then the two-dimensional vector Z is normalized by Layer Norm, input into the MLP multi-layer perceptron, and then added with the residual connection; after the above operations, the output feature vector Z" is fused with the feature vector of the same layer in the corresponding up-sampling.

5. The improved cardiac MRI segmentation method based on the fusion of U-Net and Transformer according to any one of claims 1-4, characterized in that: The implementation of the U-Net skip connection based on cross-attention mechanism optimization is as follows, Step 4.1, calculating the Attention mechanism, including calculating the weight score by inputting the key Key and Query in the key-value pair, and mapping the weight score to the value Value of the key-value pair; Step 4.2, the cross-attention mechanism uses one sequence as the input of the query Q, and another sequence as the key and value input (K, V), and defines two embedded sequences S1 and S2 of the cross-attention mechanism, and the formula for calculating the cross-attention of the two sequences is as follows: Attention(S1, S2) = softmax((W Q S1)(W K S2) T )W V S2 First, calculate (K, V) of S2 sequence, and calculate Q of S1 sequence, then calculate attention matrix according to the calculated K and Q, where W Q ,W K ,W V represents a weight matrix, and Attention represents the final calculated attention result; then the weight of the attention is calculated through the Softmax function, and then the obtained weight is mapped to the corresponding V, and finally the cross-attention result of S1 sequence and S2 sequence is output. Step 4.3, the calculation process of feature fusion from the improved Transformer skip connection of the encoder Enc1, Enc2 to the decoder is as follows, I = Reshape(Flatten(E) + PE) A = MCA(Q Unfold(I) ,K BN ,V BN ) U = add(Flatten(Fold(A))) Z = U + MLP(LayerNorm(U)) wherein the subscript represents that Q comes from the encoder Enc1 of the Skip4 layer or the encoder Enc2 of the Skip3 layer, X represents the feature map input, the feature information is output from the encoder through the Encoder module, and is unfolded into a two-dimensional sequence through the Flatten flattening operation, and then the position encoding attention vector PE is added, and then the three-dimensional vector I is restored through the Reshape operation; the obtained I is converted into a query input sequence through the Unfold output, and then is input into the multi-head attention layer MCA together with the key-value pair sequence obtained by the Bottleneck layer to perform attention calculation, the subscript BN is used to identify the Bottleneck layer, A is the result of the attention calculation interaction based on the improved Transformer cross-attention mechanism, then the obtained attention result is restored through the Fold operation, and then is unfolded into a vector through the Flatten flattening operation, then the result U is obtained through the add operation, then the layer normalization LayerNorm operation is performed, and then the output Z is obtained by adding U, and then the output is obtained, and MCA represents the multi-head cross-attention mechanism.

6. A cardiac MRI segmentation system based on a U-Net and Transformer fusion improved, characterized by: The improved cardiac MRI segmentation method based on U-Net and Transformer fusion is used to realize the method of any one of claims 1-5.

7. The cardiac MRI segmentation system based on the fusion of U-Net and Transformer improved according to claim 6, characterized in that: The method comprises the following modules, The first module is used for data preprocessing, which comprises firstly performing zero mean standardization on the image and performing data enhancement processing to expand the training data; The second module is used for optimizing the U-Net skip connection by using the Transformer, which comprises embedding the Transformer into multiple skip connections in the network to form a Transformer skip connection structure, and performing feature re-extraction on the semantic features output by the encoder through the structure to filter the semantic features and noise features irrelevant to the task; The third module is used for optimizing the U-Net skip connection based on multi-granularity features, which comprises improving the Transformer embedded on the Bottleneck layer and Skip1 and Skip2 layers above the Bottleneck layer of the U-Net, and constructing multi-granularity features through spatial conversion Unfold to enable the improved Transformer to extract multi-granularity context information; The fourth module is used for optimizing the U-Net skip connection based on the cross-attention mechanism, which comprises improving the Transformer based on the cross-attention mechanism, recording the Transformer improved based on the cross-attention mechanism as CA-MGTransformer, embedding the improved Transformer in the Skip3 and Skip4 layers to perform feature fusion of the encoder and decoder, calculating the query input sequence required in the multi-head attention layer of the added CA-MG Transformer from the Skip3 and Skip4 layers themselves, and calculating the key-value sequence required from the Bottleneck layer, and calculating the remote interaction information captured through cross-attention, and simultaneously introducing the feature map of the lower scale Bottleneck layer as the key-value pair sequence; The fifth module is used for training the cardiac MRI data according to the constructed model, classifying the pixel probability values of the left ventricle, right ventricle, myocardium and background of the heart obtained, so as to achieve the segmentation effect of different regions, and finally obtaining the segmented labeling result.

8. The cardiac MRI segmentation system based on the fusion of U-Net and Transformer improved according to claim 6, characterized in that: The method comprises a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the storage instructions in the memory to execute the improved cardiac MRI segmentation method based on U-Net and Transformer fusion of any one of claims 1-5.

9. The cardiac MRI segmentation system based on the fusion of U-Net and Transformer improved according to claim 6, characterized in that: The readable storage medium comprises a computer program stored thereon, and the computer program is executed to realize the improved cardiac MRI segmentation method based on U-Net and Transformer fusion of any one of claims 1-5.

Images