MUNet-based coronary artery blood vessel image segmentation method and system

Abstract

The invention provides an MUNet-based coronary artery blood vessel image segmentation method and system. The method comprises the steps of obtaining a coronary artery blood vessel image data set and performing preprocessing; constructing an MUNet neural network model based on self-attention and multi-view feature attention; training an MUNet neural network model by using the preprocessed coronary artery blood vessel image data set; obtaining a coronary artery blood vessel image to be segmented, inputting the coronary artery blood vessel image to the trained MUNet neural network model for segmentation, and obtaining a preliminary segmentation result; smoothing the preliminary segmentation result by using an image morphological algorithm to obtain a final segmentation result; according to the method, the global information and the local information of the coronary artery blood vessel image can be extracted more effectively, more accurate coronary artery blood vessel image segmentation is realized, the generalization ability of the model is improved while the segmentation precision is remarkably improved, and more reliable reference is provided for subsequent diagnosis and treatment.

Description

Technical Field

[0001] The present invention relates to the technical field of CT vascular medical image segmentation, and more specifically, to a coronary artery vascular image segmentation method and system based on MUNet. Background Art

[0002] At present, there are mainly traditional methods and deep learning methods for segmentation of coronary artery three-dimensional vascular medical images (3D CTA). Among the traditional methods, image cutting-based and level set-based techniques are more common. Although these methods can automatically extract coronary arteries, they require manual setting of seed points or calibration of vascular regions, resulting in errors and discontinuities in local areas.

[0003] With the development of deep learning technology, deep learning-based segmentation methods have made significant progress in coronary artery 3D CTA image segmentation. These methods can automatically learn features from data through architectures such as convolutional neural networks (CNNs), thereby achieving more accurate segmentation.

[0004] For example, the existing patent CN111739034A discloses a method and system for segmenting the coronary artery region based on an improved 3D Unet. This method also uses a deep learning method to perform coronary artery segmentation. However, this method optimizes the original network model by using residual connections and deepening the network depth, which may cause the training model to be too complex and produce the risk of overfitting, and is also lacking in the acquisition of global information. Another existing patent CN111476796A discloses a semi-supervised coronary artery segmentation system and segmentation method combining multiple networks. However, this method constructs a new 2D data training set by slicing the original three-dimensional data. This method will lose the connection between features to a certain extent, which will have a certain impact on the segmentation result.

[0005] In addition, in existing deep learning-based segmentation methods, the use of 2D slice segmentation may lead to the loss of key three-dimensional spatial information, especially in curved and intersecting vascular areas; and the use of three-dimensional convolutional neural networks will introduce a lot of computational complexity, limiting its feasibility in practical applications.

[0006] Therefore, existing coronary artery segmentation techniques face challenges such as low accuracy of spatial continuity and three-dimensional morphological information, poor segmentation accuracy, and high computational complexity. Summary of the invention

[0007] In order to overcome the defects of the above-mentioned prior art in terms of low accuracy of spatial continuity and three-dimensional morphological information, poor segmentation accuracy and high computational complexity, the present invention provides a coronary artery image segmentation method and system based on MUNet, by constructing a MUNet network model based on self-attention and multi-view feature attention, and using the model to perform global and local segmentation of coronary artery images, the segmentation accuracy is significantly improved.

[0008] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0009] A coronary artery image segmentation method based on MUNet comprises the following steps:

[0010] S1: Obtain a coronary artery image dataset and perform preprocessing;

[0011] The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels;

[0012] S2: Construct MUNet neural network model based on self-attention and multi-view feature attention;

[0013] The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence;

[0014] The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder;

[0015] S3: inputting the preprocessed coronary artery vascular image dataset into the MUNet neural network model for training, and obtaining a trained MUNet neural network model;

[0016] S4: Obtain the coronary artery image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain a preliminary segmentation result;

[0017] S5: Use image morphology algorithm to smooth the preliminary segmentation result to obtain the final segmentation result and complete the segmentation of the coronary artery.

[0018] Preferably, the preprocessing in step S1 includes any one or more of scale transformation, cropping, random flipping, random rotation, affine transformation and image value normalization.

[0019] Preferably, in step S2, the feature rough extraction module includes: a 3D convolution block, a Convnext feature extraction block 1 and a Convnext feature extraction block 2 connected in sequence;

[0020] The structure of the 3D convolution block includes: a 4×4×4 convolution layer, a batch normalization BN layer, and a ReLU activation layer connected in sequence;

[0021] The structure of each of the Convnext feature extraction blocks includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext feature extraction block also form a residual sum connection through a bypass.

[0022] Preferably, in step S2, the structure of the multi-view feature attention module includes: branches 1, 2 and 3 arranged in parallel, wherein branches 1, 2 and 3 correspond to the coronal view, the sagittal view and the axial view respectively;

[0023] The structure of the branch 1 includes: a convolution block 1 and a mixed attention block 1 connected in sequence; the structure of the branch 2 includes: a convolution block 2 and a mixed attention block 2 connected in sequence; the structure of the branch 3 includes: a convolution block 3 and a mixed attention block 3 connected in sequence;

[0024] Each of the convolution blocks has the same structure, and includes: a 4×4×4 convolution layer, a batch normalization BN layer, a ReLU activation layer, and a 1×1×1 convolution layer connected in sequence;

[0025] Each of the hybrid attention blocks has the same structure, and includes: a spatial domain attention branch and a channel domain attention branch arranged in parallel; the spatial domain attention branch includes: a 1×1×1 convolution layer, a 3×3×3 convolution layer, a 3×3×3 convolution layer, a 1×1×1 convolution layer, and a batch normalization BN layer connected in sequence; the channel domain attention branch includes: an adaptive pooling layer and a one-dimensional convolution layer connected in sequence;

[0026] The outputs of the spatial domain attention branch and the channel domain attention branch are weightedly added as the output of the hybrid attention block;

[0027] The outputs of branches 1, 2 and 3 are concatenated and fused as the output of the multi-view feature attention module.

[0028] Preferably, in step S2, a transition layer is provided before each of the multi-view feature fusion modules, and the transition layer includes: a 1×1×1 convolution layer, a batch normalization BN layer, a ReLU activation layer, a maximum pooling downsampling layer and a Convnext block connected in sequence;

[0029] The Convnext block includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext block also form a residual sum connection through a bypass.

[0030] Preferably, in step S2, the Transformer encoder comprises: an embedding layer and a plurality of Transformer blocks connected in sequence;

[0031] Each of the Transformer blocks has the same structure, and includes: a layer-normalized LN layer 1, a multi-head self-attention layer, a layer-normalized LN layer 2, and a multi-layer perceptron layer connected in sequence; the input of the layer-normalized LN layer 1 also forms a residual sum connection with the output of the multi-head self-attention layer; the input of the layer-normalized LN layer 2 also forms a residual sum connection with the output of the multi-layer perceptron layer.

[0032] Preferably, in step S2, the Transformer encoder includes 4 Transformer blocks, and the outputs of the 4 Transformer blocks are respectively recorded as T1, T2, T3 and T4;

[0033] The number of the multi-view feature attention modules is 3; the output of the feature rough extraction module is recorded as S1, and the outputs of the three multi-view feature attention modules are recorded as M1, M2 and M3 respectively;

[0034] In the feature fusion network, T1 and S1 are fused and output as F1; T2 and M1 are fused and output as F2; ​​T3 and M2 are fused and output as F3; T4 and M3 are fused and output as F4;

[0035] In the decoder, F4 is fused with F3 after upsampling, and the output is C3; C3 is fused with F2 after upsampling, and the output is C2; C2 is fused with F1 after upsampling, and the output is C1; C1 is upsampled and outputs the segmentation prediction result of the coronary artery vascular image as the output of the MUNet neural network model.

[0036] Preferably, in step S3, the loss function used in the MUNet neural network model training is a loss function L based on the similarity Dice coefficient. dice , specifically:

[0037]

[0038] Among them, P represents the prediction result of the MUNet neural network model, and R represents the true label of the preprocessed coronary artery image.

[0039] Preferably, step S5 comprises:

[0040] S5.1: Use morphological opening operation to remove noise and eliminate errors or noise that may be introduced in the image segmentation process; the morphological opening operation is erosion followed by expansion;

[0041] The erosion operation is used to remove a layer of pixels at the edge of the image, thereby reducing the image. The mathematical formula is expressed as:

[0042]

[0043] Among them, S1 represents the image after corrosion, B1 represents the structural element of corrosion, A1 represents the original image before corrosion, and z represents the position of the corroded pixel in the original image. Only when B1 is translated to position z and B1 completely belongs to A1 at this time, then z belongs to the image S1 after corrosion;

[0044] The dilation operation is used to increase the size of an image by adding a layer of pixels to the edge of the image. The mathematical formula is expressed as:

[0045]

[0046] Wherein, S2 represents the image after expansion, B2 represents the expanded structural element, A2 represents the original image before expansion, and z' represents the position of the expanded pixel in the original image. Only when B2 is translated to the position z' and B2 has a common point with A2 at this time, then z' belongs to the expanded image S2;

[0047] S5.2: Use morphological closing operation to connect adjacent blood vessel breaks to enhance the continuity of the blood vessels; the morphological closing operation is to first dilate and then corrode;

[0048] S5.3: repeating steps S5.1 to S5.2 several times, further smoothing the edge of the blood vessel by continuously applying morphological opening and morphological closing operations, and obtaining the final segmentation result;

[0049] S5.4: Map the final segmentation result to the medical image space, store the mapped data in nii format, and complete the segmentation of the coronary artery.

[0050] The present invention also provides a coronary artery image segmentation system based on MUNet, which uses the above-mentioned coronary artery image segmentation method based on MUNet, including:

[0051] Preprocessing unit: used to obtain and preprocess the coronary artery image data set;

[0052] The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels;

[0053] Model building unit: used to build a MUNet neural network model based on self-attention and multi-view feature attention;

[0054] The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence;

[0055] The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder;

[0056] Model training unit: used to input the preprocessed coronary artery vascular image data set into the MUNet neural network model for training, and obtain the trained MUNet neural network model;

[0057] Preliminary segmentation unit: used to obtain the coronary artery vascular image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain the preliminary segmentation result;

[0058] Post-processing unit: used to use image morphology algorithm to smooth the preliminary segmentation results, obtain the final segmentation results, and complete the segmentation of coronary arteries.

[0059] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0060] The present invention provides a coronary artery image segmentation method and system based on MUNet, firstly, a coronary artery image data set is obtained and preprocessed; then a MUNet neural network model is constructed based on self-attention and multi-view feature attention; then the preprocessed coronary artery image data set is input into the MUNet neural network model for training, and a trained MUNet neural network model is obtained; then a coronary artery image to be segmented is obtained and input into the trained MUNet neural network model for segmentation, and a preliminary segmentation result is obtained; finally, an image morphology algorithm is used to smooth the preliminary segmentation result, and a final segmentation result is obtained, thereby completing the segmentation of the coronary artery;

[0061] The present invention constructs a MUNet neural network model based on self-attention and multi-view feature attention, utilizes the ability of Transformer's self-attention mechanism to extract long-distance features, obtains more global information about coronary artery vascular images, and utilizes a multi-view feature attention extraction module to extract feature information of coronary artery vascular images from multiple viewpoints, focusing on feature information in details. Compared with traditional coronary artery vascular image segmentation methods, the present invention can more effectively extract global and local information of coronary artery vascular images, achieve more accurate coronary artery vascular image segmentation, significantly improve the segmentation accuracy, and also improve the generalization ability of the model, providing a more reliable reference for subsequent diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a coronary artery image segmentation method based on MUNet provided in Example 1.

[0063] Figure 2 This is a structural diagram of the MUNet neural network model provided in Example 2.

[0064] Figure 3 This is the Convnext feature extraction block structure diagram provided in Example 2.

[0065] Figure 4 This is a structural diagram of the multi-view feature attention module provided in Example 2.

[0066] Figure 5 This is a structural diagram of the hybrid attention block provided in Example 2.

[0067] Figure 6 This is a structural diagram of a coronary artery image segmentation system based on MUNet provided in Example 3. DETAILED DESCRIPTION

[0068] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0069] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0070] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0071] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0072] Example 1

[0073] like Figure 1As shown, this embodiment provides a coronary artery image segmentation method based on MUNet, comprising the following steps:

[0074] S1: Obtain a coronary artery image dataset and perform preprocessing;

[0075] The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels;

[0076] S2: Construct MUNet neural network model based on self-attention and multi-view feature attention;

[0077] The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence;

[0078] The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder;

[0079] S3: inputting the preprocessed coronary artery vascular image dataset into the MUNet neural network model for training, and obtaining a trained MUNet neural network model;

[0080] S4: Obtain the coronary artery image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain a preliminary segmentation result;

[0081] S5: Use image morphology algorithm to smooth the preliminary segmentation result to obtain the final segmentation result and complete the segmentation of the coronary artery.

[0082] In the specific implementation process, firstly, a coronary artery vascular image dataset is acquired and preprocessed;

[0083] Then, the MUNet neural network model was constructed based on self-attention and multi-view feature attention;

[0084] The MUNet neural network model includes: Transformer encoder, feature extraction network, feature fusion network and decoder; the feature extraction network includes: feature coarse extraction module and several multi-view feature attention modules connected in sequence;

[0085] The Transformer encoder and feature extraction network are set in parallel. The outputs of the Transformer encoder and feature extraction network are connected to the input of the feature fusion network respectively, and the output of the feature fusion network is connected to the decoder.

[0086] Then, the preprocessed coronary artery image data set is input into the MUNet neural network model for training to obtain a trained MUNet neural network model;

[0087] Then, the coronary artery image to be segmented is obtained and input into the trained MUNet neural network model for segmentation to obtain the preliminary segmentation result;

[0088] Finally, the image morphology algorithm is used to smooth the preliminary segmentation results to obtain the final segmentation results and complete the segmentation of the coronary arteries.

[0089] This method constructs a MUNet network model based on self-attention and multi-view feature attention, and uses this model to perform global and local segmentation of coronary artery images, significantly improving its segmentation accuracy.

[0090] Example 2

[0091] This embodiment provides a coronary artery image segmentation method based on MUNet, comprising the following steps:

[0092] S1: Obtain a coronary artery image dataset and perform preprocessing;

[0093] The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels;

[0094] S2: Construct MUNet neural network model based on self-attention and multi-view feature attention;

[0095] The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence;

[0096] The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder;

[0097] S3: inputting the preprocessed coronary artery vascular image dataset into the MUNet neural network model for training, and obtaining a trained MUNet neural network model;

[0098] S4: Obtain the coronary artery image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain a preliminary segmentation result;

[0099] S5: using image morphology algorithm to smooth the preliminary segmentation result, obtaining the final segmentation result, and completing the segmentation of the coronary artery;

[0100] The preprocessing in step S1 includes: any one or more of scale transformation, cropping, random flipping, random rotation, affine transformation and image value normalization;

[0101] In the step S2, the feature rough extraction module includes: a 3D convolution block, a Convnext feature extraction block 1 and a Convnext feature extraction block 2 connected in sequence;

[0102] The structure of the 3D convolution block includes: a 4×4×4 convolution layer, a batch normalization BN layer, and a ReLU activation layer connected in sequence;

[0103] The structure of each of the Convnext feature extraction blocks includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer, and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext feature extraction block also form a residual sum connection through a bypass;

[0104] In step S2, the structure of the multi-view feature attention module includes: branches 1, 2 and 3 arranged in parallel, wherein branches 1, 2 and 3 correspond to the coronal view, the sagittal view and the axial view respectively;

[0105] The structure of the branch 1 includes: a convolution block 1 and a mixed attention block 1 connected in sequence; the structure of the branch 2 includes: a convolution block 2 and a mixed attention block 2 connected in sequence; the structure of the branch 3 includes: a convolution block 3 and a mixed attention block 3 connected in sequence;

[0106] Each of the convolution blocks has the same structure, and includes: a 4×4×4 convolution layer, a batch normalization BN layer, a ReLU activation layer, and a 1×1×1 convolution layer connected in sequence;

[0107] Each of the hybrid attention blocks has the same structure, and includes: a spatial domain attention branch and a channel domain attention branch arranged in parallel; the spatial domain attention branch includes: a 1×1×1 convolution layer, a 3×3×3 convolution layer, a 3×3×3 convolution layer, a 1×1×1 convolution layer, and a batch normalization BN layer connected in sequence; the channel domain attention branch includes: an adaptive pooling layer and a one-dimensional convolution layer connected in sequence;

[0108] The outputs of the spatial domain attention branch and the channel domain attention branch are weightedly added as the output of the hybrid attention block;

[0109] The outputs of branches 1, 2 and 3 are concatenated and fused as the output of the multi-view feature attention module;

[0110] In the step S2, a transition layer is also provided before each of the multi-view feature fusion modules, and the transition layer includes: a 1×1×1 convolution layer, a batch normalization BN layer, a ReLU activation layer, a maximum pooling downsampling layer and a Convnext block connected in sequence;

[0111] The Convnext block includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext block also form a residual sum connection through a bypass;

[0112] In step S2, the Transformer encoder includes: an embedding layer and a plurality of Transformer blocks connected in sequence;

[0113] Each of the Transformer blocks has the same structure, and includes: a layer normalized LN layer 1, a multi-head self-attention layer, a layer normalized LN layer 2, and a multi-layer perceptron layer connected in sequence; the input of the layer normalized LN layer 1 also forms a residual sum connection with the output of the multi-head self-attention layer; the input of the layer normalized LN layer 2 also forms a residual sum connection with the output of the multi-layer perceptron layer;

[0114] In step S2, the Transformer encoder includes four Transformer blocks, and the outputs of the four Transformer blocks are respectively recorded as T1, T2, T3 and T4;

[0115] The number of the multi-view feature attention modules is 3; the output of the feature rough extraction module is recorded as S1, and the outputs of the three multi-view feature attention modules are recorded as M1, M2 and M3 respectively;

[0116] In the feature fusion network, T1 and S1 are fused and output as F1; T2 and M1 are fused and output as F2; ​​T3 and M2 are fused and output as F3; T4 and M3 are fused and output as F4;

[0117] In the decoder, F4 is fused with F3 after upsampling, and the output is C3; C3 is fused with F2 after upsampling, and the output is C2; C2 is fused with F1 after upsampling, and the output is C1; C1 is upsampling and outputs the segmentation prediction result of the coronary artery vascular image as the output of the MUNet neural network model;

[0118] In step S3, the loss function used for MUNet neural network model training is a loss function L based on the similarity Dice coefficient. dice , specifically:

[0119]

[0120] Among them, P represents the prediction result of the MUNet neural network model, and R represents the true label of the preprocessed coronary artery image;

[0121] The step S5 comprises:

[0122] S5.1: Use morphological opening operation to remove noise and eliminate errors or noise that may be introduced in the image segmentation process; the morphological opening operation is erosion followed by expansion;

[0123] The erosion operation is used to remove a layer of pixels at the edge of the image, thereby reducing the image. The mathematical formula is expressed as:

[0124]

[0125] Among them, S1 represents the image after corrosion, B1 represents the structural element of corrosion, A1 represents the original image before corrosion, and z represents the position of the corroded pixel in the original image. Only when B1 is translated to position z and B1 completely belongs to A1 at this time, then z belongs to the image S1 after corrosion;

[0126] The dilation operation is used to increase the size of an image by adding a layer of pixels to the edge of the image. The mathematical formula is expressed as:

[0127]

[0128] Wherein, S2 represents the image after expansion, B2 represents the expanded structural element, A2 represents the original image before expansion, and z' represents the position of the expanded pixel in the original image. Only when B2 is translated to the position z' and B2 has a common point with A2 at this time, then z' belongs to the expanded image S2;

[0129] S5.2: Use morphological closing operation to connect adjacent blood vessel breaks to enhance the continuity of the blood vessels; the morphological closing operation is to first dilate and then corrode;

[0130] S5.3: repeating steps S5.1 to S5.2 several times, further smoothing the edge of the blood vessel by continuously applying morphological opening and morphological closing operations, and obtaining the final segmentation result;

[0131] S5.4: Map the final segmentation result to the medical image space, store the mapped data in nii format, and complete the segmentation of the coronary artery.

[0132] In the specific implementation process, firstly, a coronary artery vascular image dataset is obtained and preprocessed; the preprocessing includes: any one or more of scale transformation, cropping, random flipping, random rotation, affine transformation and image value normalization, and the preprocessed dataset is divided into a training set, a validation set and a test set, so that the network model has a better learning effect;

[0133] Then, the MUNet neural network model was constructed based on self-attention and multi-view feature attention;

[0134] like Figure 2 As shown, the MUNet neural network model includes: Transformer encoder, feature extraction network, feature fusion network and decoder; the feature extraction network includes: feature coarse extraction module and 3 multi-view feature attention modules connected in sequence;

[0135] The Transformer encoder and feature extraction network are set in parallel. The outputs of the Transformer encoder and feature extraction network are connected to the input of the feature fusion network respectively, and the output of the feature fusion network is connected to the decoder.

[0136] exist Figure 2 In the figure, Stem is a rough feature extraction module, which includes: 4×4×4 convolution layer, batch normalization BN layer, ReLU activation layer, Convnext feature extraction block 1 and Convnext feature extraction block 2 connected in sequence;

[0137] like Figure 3 As shown, the structure of each Convnext feature extraction block includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext feature extraction block also form a residual sum connection through a bypass;

[0138] like Figure 4 As shown in FIG. 1 , the structure of the multi-plane hybrid attention module (MPHA) includes: branches 1, 2, and 3 are arranged in parallel, and branches 1, 2, and 3 correspond to the coronal view, sagittal view, and axial view, respectively; the input image is transposed to obtain its coronal view, sagittal view, and axial view images;

[0139] The structure of the branch 1 includes: a convolution block 1 and a mixed attention block 1 connected in sequence; the structure of the branch 2 includes: a convolution block 2 and a mixed attention block 2 connected in sequence; the structure of the branch 3 includes: a convolution block 3 and a mixed attention block 3 connected in sequence;

[0140] Each of the convolution blocks has the same structure, and includes: a 4×4×4 convolution layer, a batch normalization BN layer, a ReLU activation layer, and a 1×1×1 convolution layer connected in sequence;

[0141] like Figure 5 As shown, each of the hybrid attention blocks has the same structure, and includes: a spatial domain attention branch and a channel domain attention branch arranged in parallel; the spatial domain attention branch includes: a 1×1×1 convolution layer, a 3×3×3 convolution layer, a 3×3×3 convolution layer, a 1×1×1 convolution layer, and a batch normalization BN layer connected in sequence; the channel domain attention branch includes: an adaptive pooling layer and a one-dimensional convolution layer connected in sequence;

[0142] The outputs of the spatial domain attention branch and the channel domain attention branch are weightedly added as the output of the hybrid attention block;

[0143] The outputs of branches 1, 2 and 3 are concatenated and fused as the output of the multi-view feature attention module;

[0144] exist Figure 2 In the embodiment, a transition layer (TransitionLayer) is provided before each of the multi-view feature fusion modules, and the transition layer includes: a 1×1×1 convolution layer, a batch normalization BN layer, a ReLU activation layer, a maximum pooling downsampling layer and a Convnext block connected in sequence; the maximum pooling downsampling layer has a dimension of 2×2×2;

[0145] The Convnext block is Figure 3 The structure of is the same as that of , including: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext block also form a residual sum connection through a bypass;

[0146] In this embodiment, the Transformer encoder includes: an embedding layer, and a plurality of Transformer blocks connected in sequence;

[0147] Each of the Transformer blocks has the same structure, and includes: a layer normalized LN layer 1, a multi-head self-attention layer, a layer normalized LN layer 2, and a multi-layer perceptron layer connected in sequence; the input of the layer normalized LN layer 1 also forms a residual sum connection with the output of the multi-head self-attention layer; the input of the layer normalized LN layer 2 also forms a residual sum connection with the output of the multi-layer perceptron layer;

[0148] In this embodiment, the Transformer encoder includes four Transformer blocks, and the outputs of the four Transformer blocks are respectively recorded as T1, T2, T3 and T4;

[0149] The output of the feature coarse extraction module is denoted as S1, and the outputs of the three multi-view feature attention modules are denoted as M1, M2 and M3 respectively;

[0150] In the feature fusion network, T1 and S1 are fused and output as F1; T2 and M1 are fused and output as F2; ​​T3 and M2 are fused and output as F3; T4 and M3 are fused and output as F4;

[0151] In the decoder, F4 is fused with F3 after upsampling, and the output is C3; C3 is fused with F2 after upsampling, and the output is C2; C2 is fused with F1 after upsampling, and the output is C1; C1 is upsampling and outputs the segmentation prediction result of the coronary artery vascular image as the output of the MUNet neural network model;

[0152] After building the MUNet neural network model, the training set is used to train the MUNet neural network model, and the loss function L based on the similarity Dice coefficient is used. dice , specifically:

[0153]

[0154] Among them, P represents the prediction result of the MUNet neural network model, and R represents the true label of the preprocessed coronary artery image;

[0155] The validation set is then used to verify the model performance, and the loss function L is also used. dice For reference, compare each verification result with the previous round of verification results, and automatically save the network model with the best effect in the set epoch;

[0156] Finally, the performance of the model is tested with the test set, and the trained MUNet neural network model is finally obtained;

[0157] Then, the coronary artery image to be segmented is obtained and input into the trained MUNet neural network model for segmentation to obtain the preliminary segmentation result;

[0158] Finally, the image morphological algorithm is used to smooth the preliminary segmentation results, which specifically includes the following steps:

[0159] S5.1: Use morphological opening operation to remove noise and eliminate errors or noise that may be introduced in the image segmentation process; the morphological opening operation is erosion followed by expansion;

[0160] The erosion operation is used to remove a layer of pixels at the edge of the image, thereby reducing the image. The mathematical formula is expressed as:

[0161]

[0162] Among them, S1 represents the image after corrosion, B1 represents the structural element of corrosion, A1 represents the original image before corrosion, and z represents the position of the corroded pixel in the original image. Only when B1 is translated to position z and B1 completely belongs to A1 at this time, then z belongs to the image S1 after corrosion;

[0163] The dilation operation is used to increase the size of an image by adding a layer of pixels to the edge of the image. The mathematical formula is expressed as:

[0164]

[0165] Wherein, S2 represents the image after expansion, B2 represents the expanded structural element, A2 represents the original image before expansion, and z' represents the position of the expanded pixel in the original image. Only when B2 is translated to the position z' and B2 has a common point with A2 at this time, then z' belongs to the expanded image S2;

[0166] S5.2: Use morphological closing operation to connect adjacent blood vessel breaks to enhance the continuity of the blood vessels; the morphological closing operation is to first dilate and then corrode;

[0167] S5.3: repeating steps S5.1 to S5.2 several times, further smoothing the edge of the blood vessel by continuously applying morphological opening and morphological closing operations, and obtaining the final segmentation result;

[0168] S5.4: Map the final segmentation result to the medical image space, store the mapped data in the nii format, and complete the segmentation of the coronary artery;

[0169] In order to verify the effectiveness of this method, this embodiment also provides a comparative experiment. Specifically, the collected 480 original images are divided into 400 and 80 images; 400 of them use the five-fold training method as the training set and verification set input during model training, and the other 80 are used as the test set to test the best model saved separately; as shown in Table 1, the segmentation results of coronary artery images based on self-attention and multi-view feature attention are far better than other segmentation models, and each fold and the final average test are higher than the existing model;

[0170] Table 1 Dice test index table under the 5-fold training method

[0171] Name Fold_1(%) Fold_2(%) Fold_3(%) Fold_4(%) Fold_5(%) ACC(%) UNETR 76.32±2.87 77.87±2.89 78.07±2.63 76.28±2.72 77.71±3.11 77.29±2.84 UXNET 80.48±2.61 80.45±2.53 80.54±2.75 80.45±2.75 80.45±2.77 80.53±2.81 CANet 80.37±2.84 80.16±2.88 80.25±3.13 80.02±2.88 79.93±2.95 80.15±2.94 CSANet 80.45±2.62 80.31±3.09 80.54±2.84 80.75±2.54 80.58±3.24 80.53±2.87 MUNet 81.79±2.55 82.02±2.92 81.61±2.78 81.68±2.68 81.59±2.78 81.74±2.74

[0172] Therefore, compared with the traditional segmentation model, this method proposes a multi-view feature attention module based on the characteristics that three-dimensional medical images have different texture features when observed from different perspectives; secondly, a MUNet network model is constructed based on self-attention and multi-view feature attention, so that the coronary artery image segmentation has higher segmentation accuracy and efficiency; in addition, the post-processing module eliminates the small errors or noises that may be introduced in the coronary artery image segmentation, and is used to connect adjacent broken blood vessels to enhance the continuity of blood vessels, so that the segmentation effect is further improved, which provides help for subsequent clinical diagnosis and treatment;

[0173] This method constructs a MUNet network model based on self-attention and multi-view feature attention, and uses this model to perform global and local segmentation of coronary artery images, significantly improving its segmentation accuracy.

[0174] Example 3

[0175] like Figure 6 As shown, this embodiment provides a coronary artery image segmentation system based on MUNet, applying a coronary artery image segmentation method based on MUNet described in Embodiment 1 or 2, including:

[0176] Preprocessing unit 301: used to obtain a coronary artery image dataset and perform preprocessing;

[0177] The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels;

[0178] Model building unit 302: used to build a MUNet neural network model based on self-attention and multi-view feature attention;

[0179] The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence;

[0180] The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder;

[0181] Model training unit 303: used for inputting the preprocessed coronary artery vascular image data set into the MUNet neural network model for training, and obtaining a trained MUNet neural network model;

[0182] The preliminary segmentation unit 304 is used to obtain the coronary artery vascular image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain the preliminary segmentation result;

[0183] Post-processing unit 305 is used to smooth the preliminary segmentation result using an image morphology algorithm to obtain a final segmentation result and complete the segmentation of the coronary arteries.

[0184] In the specific implementation process, firstly, the preprocessing unit 301 acquires the coronary artery vascular image data set and performs preprocessing;

[0185] Then the model construction unit 302 constructs a MUNet neural network model based on self-attention and multi-view feature attention;

[0186] The MUNet neural network model includes: Transformer encoder, feature extraction network, feature fusion network and decoder; the feature extraction network includes: feature coarse extraction module and several multi-view feature attention modules connected in sequence;

[0187] The Transformer encoder and feature extraction network are set in parallel. The outputs of the Transformer encoder and feature extraction network are connected to the input of the feature fusion network respectively, and the output of the feature fusion network is connected to the decoder.

[0188] Then the model training unit 303 inputs the preprocessed coronary artery vascular image data set into the MUNet neural network model for training, and obtains a trained MUNet neural network model;

[0189] Then, the preliminary segmentation unit 304 obtains the coronary artery vascular image to be segmented and inputs it into the trained MUNet neural network model for segmentation to obtain a preliminary segmentation result;

[0190] Finally, the post-processing unit 305 uses an image morphological algorithm to smooth the preliminary segmentation result to obtain the final segmentation result and complete the segmentation of the coronary artery.

[0191] This system constructs a MUNet network model based on self-attention and multi-view feature attention, and uses this model to perform global and local segmentation of coronary artery images, significantly improving its segmentation accuracy.

[0192] The same or similar reference numerals correspond to the same or similar components;

[0193] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0194] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A coronary artery image segmentation method based on MUNet, characterized in that: The following steps are involved: S1: Obtain a coronary artery image dataset and perform preprocessing; The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels; S2: Construct MUNet neural network model based on self-attention and multi-view feature attention; The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence; The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder; S3: inputting the preprocessed coronary artery vascular image dataset into the MUNet neural network model for training, and obtaining a trained MUNet neural network model; S4: Obtain the coronary artery image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain a preliminary segmentation result; S5: Use image morphology algorithm to smooth the preliminary segmentation result to obtain the final segmentation result and complete the segmentation of the coronary artery.

2. The method for segmenting coronary artery images based on MUNet according to claim 1, characterized in that: The preprocessing in step S1 includes: any one or more of scale transformation, cropping, random flipping, random rotation, affine transformation and image value normalization.

3. The method for segmenting coronary artery images based on MUNet according to claim 1, characterized in that: In the step S2, the feature rough extraction module includes: a 3D convolution block, a Convnext feature extraction block 1 and a Convnext feature extraction block 2 connected in sequence; The structure of the 3D convolution block includes: a 4×4×4 convolution layer, a batch normalization BN layer, and a ReLU activation layer connected in sequence; The structure of each of the Convnext feature extraction blocks includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext feature extraction block also form a residual sum connection through a bypass.

4. The method for segmenting coronary artery images based on MUNet according to claim 1, characterized in that: In step S2, the structure of the multi-view feature attention module includes: branches 1, 2 and 3 arranged in parallel, wherein branches 1, 2 and 3 correspond to the coronal view, the sagittal view and the axial view respectively; The structure of the branch 1 includes: a convolution block 1 and a mixed attention block 1 connected in sequence; the structure of the branch 2 includes: a convolution block 2 and a mixed attention block 2 connected in sequence; the structure of the branch 3 includes: a convolution block 3 and a mixed attention block 3 connected in sequence; Each of the convolution blocks has the same structure, and includes: a 4×4×4 convolution layer, a batch normalization BN layer, a ReLU activation layer, and a 1×1×1 convolution layer connected in sequence; Each of the hybrid attention blocks has the same structure, and includes: a spatial domain attention branch and a channel domain attention branch arranged in parallel; the spatial domain attention branch includes: a 1×1×1 convolution layer, a 3×3×3 convolution layer, a 3×3×3 convolution layer, a 1×1×1 convolution layer, and a batch normalization BN layer connected in sequence; the channel domain attention branch includes: an adaptive pooling layer and a one-dimensional convolution layer connected in sequence; The outputs of the spatial domain attention branch and the channel domain attention branch are weightedly added as the output of the hybrid attention block; The outputs of branches 1, 2 and 3 are concatenated and fused as the output of the multi-view feature attention module.

5. The method for segmenting coronary artery images based on MUNet according to claim 4, characterized in that: In the step S2, a transition layer is also provided before each of the multi-view feature fusion modules, and the transition layer includes: a 1×1×1 convolution layer, a batch normalization BN layer, a ReLU activation layer, a maximum pooling downsampling layer and a Convnext block connected in sequence; The Convnext block includes: a 7×7×7 convolutional layer, a layer normalization LN layer, a 1×1×1 convolutional layer, a GeRU activation layer and a 1×1×1 convolutional layer connected in sequence; the input and output of the Convnext block also form a residual sum connection through a bypass.

6. The method for segmenting coronary artery images based on MUNet according to claim 1, characterized in that: In step S2, the Transformer encoder includes: an embedding layer and a plurality of Transformer blocks connected in sequence; Each of the Transformer blocks has the same structure, and includes: a layer-normalized LN layer 1, a multi-head self-attention layer, a layer-normalized LN layer 2, and a multi-layer perceptron layer connected in sequence; the input of the layer-normalized LN layer 1 also forms a residual sum connection with the output of the multi-head self-attention layer; the input of the layer-normalized LN layer 2 also forms a residual sum connection with the output of the multi-layer perceptron layer.

7. A coronary artery image segmentation method based on MUNet according to any one of claims 1 to 6, characterized in that: In step S2, the Transformer encoder includes four Transformer blocks, and the outputs of the four Transformer blocks are respectively recorded as T1, T2, T3 and T4; The number of the multi-view feature attention modules is 3; the output of the feature rough extraction module is recorded as S1, and the outputs of the three multi-view feature attention modules are recorded as M1, M2 and M3 respectively; In the feature fusion network, T1 and S1 are fused and output as F1; T2 and M1 are fused and output as F2; ​​T3 and M2 are fused and output as F3; Fuse the features of T4 and M3 and output them as F4; In the decoder, F4 is fused with F3 after upsampling, and the output is C3; C3 is fused with F2 after upsampling, and the output is C2; C2 is fused with F1 after upsampling, and the output is C1; C1 is upsampled and outputs the segmentation prediction result of the coronary artery vascular image as the output of the MUNet neural network model.

8. The method for segmenting coronary artery images based on MUNet according to claim 7, characterized in that: In step S3, the loss function used for MUNet neural network model training is a loss function L based on the similarity Dice coefficient. dice , specifically: Among them, P represents the prediction result of the MUNet neural network model, and R represents the true label of the preprocessed coronary artery image.

9. The method for segmenting coronary artery images based on MUNet according to claim 1, characterized in that: The step S5 comprises: S5.1: Use morphological opening operation to remove noise and eliminate errors or noise that may be introduced in the image segmentation process; the morphological opening operation is erosion followed by expansion; The erosion operation is used to remove a layer of pixels at the edge of the image, thereby reducing the image. The mathematical formula is expressed as: Among them, S1 represents the image after corrosion, B1 represents the structural element of corrosion, A1 represents the original image before corrosion, and z represents the position of the corroded pixel in the original image. Only when B1 is translated to position z and B1 completely belongs to A1 at this time, then z belongs to the image S1 after corrosion; The dilation operation is used to increase the size of an image by adding a layer of pixels to the edge of the image. The mathematical formula is expressed as: <h2 style=";text-align:left;direction:ltr">S2 = (A2⊕B2) = {z'|(B2)<h2 style=";text-align:left;direction:ltr"> z' <h2 style=";text-align:left;direction:ltr"> ∩A2≠0} Wherein, S2 represents the image after expansion, B2 represents the expanded structural element, A2 represents the original image before expansion, and z' represents the position of the expanded pixel in the original image. Only when B2 is translated to the position z' and B2 has a common point with A2 at this time, then z' belongs to the expanded image S2; S5.2: Use morphological closing operation to connect adjacent blood vessel breaks to enhance the continuity of the blood vessels; the morphological closing operation is to first dilate and then corrode; S5.3: repeating steps S5.1 to S5.2 several times, further smoothing the edge of the blood vessel by continuously applying morphological opening and morphological closing operations, and obtaining the final segmentation result; S5.4: Map the final segmentation result to the medical image space, store the mapped data in nii format, and complete the segmentation of the coronary artery.

10. A coronary artery image segmentation system based on MUNet, using a coronary artery image segmentation method based on MUNet as claimed in any one of claims 1 to 9, characterized in that: include: Preprocessing unit: used to obtain and preprocess the coronary artery image data set; The coronary artery image dataset includes a plurality of 3D CTA images of coronary artery vessels; Model building unit: used to build a MUNet neural network model based on self-attention and multi-view feature attention; The MUNet neural network model includes: a Transformer encoder, a feature extraction network, a feature fusion network and a decoder; the feature extraction network includes: a feature rough extraction module and a plurality of multi-view feature attention modules connected in sequence; The Transformer encoder and the feature extraction network are arranged in parallel, the outputs of the Transformer encoder and the feature extraction network are respectively connected to the input of the feature fusion network, and the output of the feature fusion network is connected to the decoder; Model training unit: used to input the preprocessed coronary artery vascular image data set into the MUNet neural network model for training, and obtain the trained MUNet neural network model; Preliminary segmentation unit: used to obtain the coronary artery vascular image to be segmented and input it into the trained MUNet neural network model for segmentation to obtain the preliminary segmentation result; Post-processing unit: used to use image morphology algorithm to smooth the preliminary segmentation results, obtain the final segmentation results, and complete the segmentation of coronary arteries.

Images