Myocardial segmentation method, system and device based on myocardial contrast echocardiography

Abstract

This invention relates to the field of ultrasound image processing technology, and more particularly to a method, system, and device for myocardial segmentation based on myocardial contrast echocardiography. The invention introduces a dilated convolution module in the encoder, each module comprising multiple sequentially connected dilated convolutional layers with different dilation rates. Each dilated convolutional layer includes a sequentially connected dilated convolution, a batch normalization layer, and a ReLU activation function. The output features of all dilated convolutional layers in each module are stacked along the channel dimension to obtain the output feature map of each module. A Transformer block is introduced in the bottleneck layer, comprising multiple sequentially connected Transformer layers. Each Transformer layer includes a sequentially connected layer normalization, a multi-head attention mechanism, and a multilayer perceptron. This invention can adapt to fluctuations in the shape and position of the myocardium due to relaxation and contraction, enabling complete and accurate segmentation of the myocardium.

Description

Technical Field

[0001] This invention relates to the field of ultrasound image processing technology, and in particular to a method, system and device for myocardial segmentation based on myocardial contrast echocardiography. Background Technology

[0002] With the widespread application of deep learning technology, the development of medical image analysis will inevitably advance further. In myocardial contrast echocardiography (MCE), accurate segmentation and identification of the myocardium are of great significance for the diagnosis and treatment of coronary heart disease. However, in practice, due to factors such as individual patient differences, image quality, and noise interference, artifacts or abnormal signals inevitably appear in the images. If not handled properly, this often leads to inaccurate diagnostic results and affects the patient's treatment plan.

[0003] Traditional image segmentation methods, such as thresholding, edge detection, and region growing, are commonly used for myocardial contrast echocardiography (MCE) analysis. Thresholding methods divide the image into different regions by setting a specific grayscale threshold, attempting to distinguish myocardium from other tissues. Edge detection uses abrupt changes in pixel grayscale to determine the boundaries of myocardial tissue. Region growing starts from a seed point in the image and gradually merges adjacent pixels according to certain similarity criteria until a complete myocardial region is formed. However, myocardial tissue itself has a complex structure, and its texture, grayscale, and other characteristics are not uniform. Moreover, the quality of myocardial contrast echocardiography (MCE) is often affected by various factors, such as noise and uneven distribution of contrast agents. These factors make it difficult for traditional methods to accurately determine myocardial boundaries, and the segmentation results are often not precise enough to meet the high-precision requirements of clinical myocardial segmentation. In view of this, the U-Net network has been introduced into myocardial segmentation tasks. The U-Net network adopts a unique encoder-decoder structure. The encoder downsamples the image and extracts features at different levels, which can capture global information of the image and overcome the problem of insufficient feature extraction of complex myocardial tissue by traditional methods. The decoder upsamples and fuses features from the corresponding levels of the encoder, enabling the network to better recover image details, thereby improving segmentation accuracy and compensating for the shortcomings of traditional methods in accurately segmenting myocardium.

[0004] However, while the U-Net network has achieved some success in image segmentation, it still has significant limitations in myocardial segmentation applications using contrast echocardiography. As an architecture primarily based on convolution operations, the U-Net network's receptive field is relatively limited by the characteristics of ordinary convolution. When dealing with complex and finely structured objects like myocardial tissue, the network struggles to comprehensively capture the local features of all parts of the myocardium and its relationships with surrounding tissues. Furthermore, the U-Net network has limited ability to model long-distance spatial dependencies, making it difficult to fully capture the correlation features of myocardial tissue in a large-scale image, resulting in poor myocardial segmentation accuracy. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of existing myocardial segmentation methods, which have limited receptive fields, make it difficult to capture local features of myocardium and surrounding tissues, and make it difficult to fully capture the correlation features of myocardial tissue in a large range of images, resulting in poor myocardial segmentation accuracy.

[0006] To address the aforementioned technical problems, this invention provides a method for myocardial segmentation based on myocardial contrast echocardiography, comprising the following steps:

[0007] A myocardial segmentation model is constructed, which includes an encoder, a bottleneck layer, and a decoder.

[0008] The encoder includes multiple layers connected sequentially along the forward propagation direction. Each layer includes multiple dilated convolution modules connected sequentially. Each dilated convolution module includes multiple dilated convolution layers with different dilation rates connected sequentially. Each dilated convolution layer includes a dilated convolution, a batch normalization layer, and a ReLU activation function connected sequentially. The output features of all dilated convolution layers in each dilated convolution module are stacked in the channel dimension to obtain the output feature map of each dilated convolution module.

[0009] The myocardial contrast echocardiogram is input into the encoder to extract features, resulting in the output feature maps of each layer of the encoder;

[0010] The output feature map of the last layer of the encoder is input into the bottleneck layer for feature fusion to obtain fused features;

[0011] The output feature maps of each layer of the encoder are input into the decoder along with the fused features to obtain the myocardial segmentation results of myocardial contrast echocardiography.

[0012] Preferably, when the encoder has 5 layers, the first and second layers of the encoder each include two dilated convolution modules connected in sequence, and the third, fourth and fifth layers of the encoder each include three dilated convolution modules connected in sequence.

[0013] When the number of layers in the encoder is greater than 5, the first and second layers of the encoder each include two dilated convolutional modules connected in sequence, the third, fourth and fifth layers of the encoder each include three dilated convolutional modules connected in sequence, and the remaining layers of the encoder each include multiple dilated convolutional modules connected in sequence.

[0014] Preferably, the input features of the dilated convolution module are input into the dilated convolution module, and the output feature map of the dilated convolution module is output, including:

[0015] The input features of the i-th dilated convolutional layer in the dilated convolution module are passed through the dilated convolution of the i-th dilated convolutional layer to obtain the output features of the i-th dilated convolutional layer, as shown in the formula:

[0016] Z i =ω i *F i +b,

[0017] The output features of the dilated convolution of the i-th dilated convolution layer are processed by a batch normalization layer to obtain the batch-normalized output features of the dilated convolution of the i-th dilated convolution layer, as shown in the formula:

[0018]

[0019] The output features of the i-th dilated convolutional layer after batch normalization are processed by the ReLU activation function to obtain the output features of the i-th dilated convolutional layer, as shown in the formula:

[0020]

[0021] The output features of all dilated convolutional layers are stacked along the channel dimension to obtain the output feature map of the dilated convolutional module, as shown in the formula:

[0022]

[0023] Among them, Z i Let ω be the output feature of the dilated convolution of the i-th dilated convolutional layer. i F is the convolution kernel of the dilated convolution of the i-th dilated convolution layer in the dilated convolution module. i Let be the input feature of the i-th dilated convolutional layer in the dilated convolutional module, b be the bias term, * be the convolution operation, i = 1, 2, ..., I, and I be the number of dilated convolutional layers in the dilated convolutional module. The output feature of the dilated convolution of the i-th dilated convolutional layer after batch normalization, where · represents the dot product, and μ B This represents the mean of the output features of the dilated convolution of the i-th dilated convolution layer corresponding to the myocardial contrast echocardiogram. Let ε be the variance of the output features of the dilated convolution of the i-th dilated convolution layer corresponding to the myocardial contrast echocardiogram, where ε is a constant, γ is a learnable first parameter, and β is a learnable second parameter. Let max(.) be the output feature of the i-th dilated convolutional layer, and max(.) be the maximum value. represents the output feature map of the dilated convolution module, and ⊕ represents stacking along the channel dimension.

[0024] Preferably, the input features of the dilated convolution module are input into the dilated convolution module, and the output feature map of the dilated convolution module is output, as follows:

[0025]

[0026] in, Here, DCM(.) represents the output feature map of the dilated convolutional module, ⊕ indicates stacking along the channel dimension, ReLU(.) is the ReLU activation function, BN(.) is the batch normalization layer, and DilatedConv... i (.) represents the dilated convolution of the i-th dilated convolutional layer in the dilated convolutional module, where I is the number of dilated convolutional layers in the dilated convolutional module, and F i F1 represents the input features of the i-th dilated convolutional layer in the dilated convolutional module.

[0027] Preferably, the step of inputting the output feature map of the last layer of the encoder into the bottleneck layer for feature fusion to obtain fused features includes:

[0028] The output feature map of the last layer of the encoder is reshaped to obtain a reshaped feature map. This reshaped feature map is then flattened into N patches. The set of all patches forms a shape... , where, The height of the output feature map of the last layer of the encoder. P is the width of the output feature map of the last layer of the encoder, and P is the size of the patch block;

[0029] The matrix is ​​linearly mapped to a D-dimensional space, and each patch block is converted into its corresponding D-dimensional vector;

[0030] After adding positional encoding to the D-dimensional vector corresponding to each patch block of the output feature map of the last layer of the encoder, it is input into the Transformer block for feature fusion to obtain the initial fused features;

[0031] Perform a reshape operation on the initial fused features to make the size of the initial fused features similar to that of the reshaped features. Figure 1 Thus, the fusion characteristics are obtained.

[0032] Preferably, the Transformer block includes multiple Transformer layers connected in sequence, and each Transformer layer includes: layer normalization, multi-head attention mechanism and multilayer perceptron connected in sequence.

[0033] Preferably, the decoder includes: multiple upsampling blocks, 3×3 convolutional layers and a segmentation head, each upsampling block consists of a 3×3 convolutional layer and a 2×2 upconvolution, and each 3×3 convolutional layer includes a 3×3 convolution and a ReLU activation function.

[0034] Preferably, the step of inputting the output feature maps of each layer of the encoder and the fused features into the decoder to obtain the myocardial segmentation result of the myocardial contrast echocardiography includes:

[0035] The fused features are processed through the first upsampling block to obtain the output feature map of the first upsampling block;

[0036] The output feature map of the j-th upsampled block is stacked with the output feature map of the (M-j+1)-th layer of the encoder in the channel dimension, and then used as the input of the (j+1)-th upsampled block; where j = 1, 2, ..., M, and M is the number of upsampled blocks;

[0037] The output feature map of the Mth upsampling block is stacked with the output feature map of the first layer of the encoder in the channel dimension, and then processed by a 3×3 convolutional layer and a segmentation head to obtain the myocardial segmentation result of myocardial contrast echocardiography.

[0038] The present invention also provides a myocardial segmentation system based on myocardial contrast echocardiography, comprising:

[0039] The model building module is used to build a myocardial segmentation model, which includes an encoder, a bottleneck layer, and a decoder.

[0040] The encoder includes multiple layers connected sequentially along the forward propagation direction. Each layer includes multiple dilated convolution modules connected sequentially. Each dilated convolution module includes multiple dilated convolution layers with different dilation rates connected sequentially. Each dilated convolution layer includes a dilated convolution, a batch normalization layer, and a ReLU activation function connected sequentially. The output features of all dilated convolution layers in each dilated convolution module are stacked in the channel dimension to obtain the output feature map of each dilated convolution module.

[0041] The feature extraction module is used to input the myocardial contrast echocardiogram into the encoder to extract features and obtain the output feature maps of each layer of the encoder;

[0042] The fusion module is used to input the output feature map of the last layer of the encoder into the bottleneck layer for feature fusion to obtain fused features.

[0043] The segmentation module is used to input the output feature maps of each layer of the encoder and the fused features into the decoder to obtain the myocardial segmentation results of myocardial contrast echocardiography.

[0044] The present invention also provides a myocardial segmentation device based on myocardial contrast echocardiography, comprising:

[0045] A memory for storing a computer program; a processor for executing the computer program to implement the steps of the above-described myocardial segmentation method based on myocardial contrast echocardiography.

[0046] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0047] The present invention discloses a myocardial segmentation method, system, and device based on myocardial contrast echocardiography. By setting different numbers of dilated convolution modules in different layers of the encoder, the dilated convolution modules expand the receptive field during feature extraction through the stacking of convolutions with different dilation rates, thereby effectively extracting multi-scale feature information and improving the perception ability of tissue structures at different scales in myocardial contrast images. Meanwhile, this structure is particularly helpful in capturing a wide range of low-frequency information, such as the overall structure and background of the myocardium, making up for the shortcomings of traditional U-Net networks, which are limited by the receptive field of ordinary convolution and thus cannot fully perceive local details. In addition, this invention flattens the output feature map of the last layer into patch blocks and constructs a matrix, which reintegrates the originally scattered and difficult-to-fully-mined local features. Each patch block becomes a unit containing local information, which can more meticulously reflect the local details of myocardial tissue and surrounding tissues. By introducing Transformer blocks in the bottleneck layer, each Transformer block includes multiple Transformer layers connected in sequence. Each Transformer layer includes: layer normalization, multi-head attention mechanism and multilayer perceptron connected in sequence. The multi-head attention mechanism can break through the locality limitation of traditional convolution and fully capture the long-distance spatial dependencies between various parts of the myocardium and its surrounding tissues. Through multiple Transformer layers, spatial dependencies of different scales or types can be focused on, which can adapt to the irregular shape and complex structure of the myocardium in contrast echocardiography. The extracted features can be interactively fused, making the segmentation of the myocardial contour more complete, thereby improving the prediction accuracy of the myocardial boundary region. Attached Figure Description

[0048] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0049] Figure 1This is a flowchart of a myocardial segmentation method based on myocardial contrast echocardiography according to the present invention. Figure 1 (a) in the diagram is a structural diagram of the myocardial segmentation model. Figure 1 (b) in the diagram is the structure diagram of the Transformer layer.

[0050] Figure 2 This is a structural diagram of the dilated convolution module.

[0051] Figure 3 This is a schematic diagram of the structure of a single dilated convolution in a dilated convolution layer.

[0052] Figure 4 This is the receptive field map of the dilated convolution module.

[0053] Figure 5 The simulation results are obtained by using contrast echocardiograms of the myocardium from three apical perspectives to test the myocardial segmentation model proposed in this invention and other comparative models. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0055] Embodiment 1 of the present invention provides a method for myocardial segmentation based on myocardial contrast echocardiography, comprising the following steps:

[0056] Figure 1 This is a flowchart of a myocardial segmentation method based on myocardial contrast echocardiography according to the present invention. Figure 1 (a) in the diagram is a structural diagram of the myocardial segmentation model. Figure 1 (b) in the diagram is the structure diagram of the Transformer layer.

[0057] Step S1: Construct a myocardial segmentation model (DillateUNet), which includes an encoder, a bottleneck layer, and a decoder;

[0058] The encoder includes multiple layers connected sequentially along the forward propagation direction. Each layer includes multiple dilated convolutional modules connected sequentially. Each dilated convolutional module includes multiple dilated convolutional layers with different dilation rates connected sequentially. Each dilated convolutional layer includes a dilated convolution, a batch normalization layer, and a ReLU activation function connected sequentially. The output features of all dilated convolutional layers in each dilated convolutional module are stacked in the channel dimension to obtain the output feature map of each dilated convolutional module.

[0059] In this embodiment, specifically, the output feature maps of all layers except the last layer are downsampled and used as the input of the next layer. The downsampling method is either max pooling or average pooling.

[0060] In this embodiment, preferably, when the encoder has 5 layers, the first and second layers of the encoder each include two dilated convolutional modules connected in sequence, and the third, fourth, and fifth layers of the encoder each include three dilated convolutional modules connected in sequence. The five-layer encoder structure can progressively extract features and gradually enrich feature representation. As the number of layers increases, from the two dilated convolutional modules in the first and second layers to the three dilated convolutional modules in the third, fourth, and fifth layers, the receptive field can be gradually expanded to capture a wider range of contextual information. At the same time, it avoids problems such as gradient vanishing or gradient explosion caused by excessive network depth, enabling the network to effectively learn features at different scales. This ensures both the ability to extract local detailed features and the ability to fully acquire global semantic information, thereby improving the model's overall ability to represent and process complex data.

[0061] When the number of layers in the encoder is greater than 5, the first and second layers of the encoder each include two dilated convolutional modules connected in sequence, the third, fourth and fifth layers of the encoder each include three dilated convolutional modules connected in sequence, and the remaining layers of the encoder each include multiple dilated convolutional modules connected in sequence.

[0062] like Figure 2 As shown, Figure 2 This is a structural diagram of the dilated convolution module.

[0063] In this embodiment, specifically, the input features of the dilated convolution module are input into the dilated convolution module, and the output feature map of the dilated convolution module is output, including:

[0064] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a single dilated convolution in a dilated convolutional layer. Each dilated convolution includes a convolution, batch normalization, and ReLU activation function connected in sequence.

[0065] The input features of the i-th dilated convolutional layer in the dilated convolution module are passed through the dilated convolution of the i-th dilated convolutional layer to obtain the output features of the i-th dilated convolutional layer, as shown in the formula:

[0066] Z i =ω i *F i +b,

[0067] The output features of the dilated convolution of the i-th dilated convolution layer are processed by a batch normalization layer to obtain the batch-normalized output features of the dilated convolution of the i-th dilated convolution layer, as shown in the formula:

[0068]

[0069] The output features of the i-th dilated convolutional layer after batch normalization are processed by the ReLU activation function to obtain the output features of the i-th dilated convolutional layer, as shown in the formula:

[0070]

[0071] The output features of all dilated convolutional layers are stacked along the channel dimension to obtain the output feature map of the dilated convolutional module, as shown in the formula:

[0072]

[0073] Among them, Z i Let ω be the output feature of the dilated convolution of the i-th dilated convolutional layer. i F is the convolution kernel of the dilated convolution of the i-th dilated convolution layer in the dilated convolution module. i Let be the input feature of the i-th dilated convolutional layer in the dilated convolutional module, b be the bias term, * be the convolution operation, i = 1, 2, ..., I, and I be the number of dilated convolutional layers in the dilated convolutional module. The output feature of the dilated convolution of the i-th dilated convolutional layer after batch normalization, where · represents the dot product, and μ B This represents the mean of the output features of the dilated convolution of the i-th dilated convolution layer corresponding to the myocardial contrast echocardiogram. Let ε be the variance of the output features of the dilated convolution of the i-th dilated convolution layer corresponding to the myocardial contrast echocardiogram, where ε is a constant, γ is a learnable first parameter, and β is a learnable second parameter. Let max(.) be the output feature of the i-th dilated convolutional layer, and max(.) be the maximum value. represents the output feature map of the dilated convolution module, and ⊕ represents stacking along the channel dimension.

[0074] The input features of the dilated convolution module are input into the dilated convolution module, and the output feature map of the dilated convolution module is output. The formula is:

[0075]

[0076] in, Here, DCM(.) represents the output feature map of the dilated convolutional module, ⊕ indicates stacking along the channel dimension, ReLU(.) is the ReLU activation function, BN(.) is the batch normalization layer, and DilatedConv... i (.) represents the dilated convolution of the i-th dilated convolutional layer in the dilated convolutional module, where I is the number of dilated convolutional layers in the dilated convolutional module, and F iF1 represents the input features of the i-th dilated convolutional layer in the dilated convolutional module.

[0077] like Figure 4 As shown, Figure 4 This is a receptive field map of the dilated convolution module. This invention introduces a dilated convolution module (DCM), which expands the receptive field during feature extraction by stacking convolutions with different dilation rates. This effectively extracts multi-scale feature information, thereby improving the perception of tissue structures at different scales in contrastive myocardial images. Simultaneously, this structure is particularly helpful in capturing a wide range of low-frequency information, such as the overall structure of the myocardium and the background.

[0078] Step S2: Input the myocardial contrast echocardiogram into the encoder to extract features and obtain the output feature maps of each layer of the encoder;

[0079] Step S3: Input the output feature map of the last layer of the encoder into the bottleneck layer to perform feature fusion and obtain fused features;

[0080] In this embodiment, specifically, the step of inputting the output feature map of the last layer of the encoder into the bottleneck layer for feature fusion to obtain fused features includes:

[0081] The output feature map of the last layer of the encoder is reshaped to obtain a reshaped feature map. This reshaped feature map is then flattened into N patches. The set of all patches forms a shape... , where, The height of the output feature map of the last layer of the encoder. P is the width of the output feature map of the last layer of the encoder, and P is the size of the patch block;

[0082] The matrix is ​​linearly mapped to a D-dimensional space, and each patch block is converted into its corresponding D-dimensional vector;

[0083] After adding positional encoding to the D-dimensional vector corresponding to each patch block of the output feature map of the last layer of the encoder, it is input into the Transformer block for feature fusion to obtain the initial fused features;

[0084] Perform a reshape operation on the initial fused features to make the size of the initial fused features similar to that of the reshaped features. Figure 1 Thus, the fusion characteristics are obtained.

[0085] In this embodiment, preferably, the Transformer block includes multiple Transformer layers connected in sequence, and each Transformer layer includes: layer normalization, multi-head attention mechanism and multilayer perceptron connected in sequence.

[0086] This invention introduces a Transformer block, which can efficiently capture long-distance dependencies between different regions in myocardial contrast echocardiography images. It can adapt to the irregular shape and complex structure of the myocardium in myocardial contrast echocardiography images. The extracted features can be interactively fused, resulting in more complete segmentation of the myocardial contour and thus improving the prediction accuracy of the myocardial boundary region.

[0087] Step S4: Input the output feature maps of each layer of the encoder and the fused features into the decoder to obtain the myocardial segmentation results of the myocardial contrast echocardiography.

[0088] In this embodiment, preferably, the decoder includes: multiple upsampling blocks, 3×3 convolutional layers, and a segmentation head. Each upsampling block consists of a 3×3 convolutional layer and a 2×2 upconvolution. Each 3×3 convolutional layer includes a 3×3 convolution and a ReLU activation function. The number of layers in the decoder is the same as the number of layers in the encoder. The 3×3 convolutional layers can further extract and optimize features from the upsampled feature maps. By sliding the convolutional kernels on the feature maps, features are further fused and filtered to adjust their representation, making them more suitable for subsequent category determination by the segmentation head.

[0089] In this embodiment, specifically, the step of inputting the output feature maps of each layer of the encoder with the fused features into the decoder to obtain the myocardial segmentation result of the myocardial contrast echocardiography includes:

[0090] The fused features are processed through the first upsampling block to obtain the output feature map of the first upsampling block;

[0091] The output feature map of the j-th upsampled block is stacked with the output feature map of the (M-j+1)-th layer of the encoder in the channel dimension, and then used as the input of the (j+1)-th upsampled block; where j = 1, 2, ..., M, and M is the number of upsampled blocks;

[0092] The output feature map of the Mth upsampling block is stacked with the output feature map of the first layer of the encoder in the channel dimension, and then processed by a 3×3 convolutional layer and a segmentation head to obtain the myocardial segmentation result of myocardial contrast echocardiography.

[0093] In this embodiment, specifically, the segmentation head is composed of 3×3 convolutions, and the number of output channels of the segmentation head is equal to the total number of categories. The segmentation head, composed of 3×3 convolutions and with the number of output channels equal to the total number of categories, can accurately classify different myocardial structures and possible background categories, assigning each pixel in the feature map to its corresponding category, and ultimately outputting a clear and concise myocardial segmentation result.

[0094] This embodiment uses a classic model for semantic segmentation and the myocardial segmentation model (DillateUNet) proposed in this invention to segment myocardial contrast echocardiograms (including three classes: two-chamber (A2C), three-chamber (A3C), and four-chamber (A4C)). The segmentation results are as follows: Figure 5 As shown, Figure 5 The simulation results are shown in the figure after comparing the myocardial segmentation model proposed in this invention with other comparative models using three apical perspective echocardiograms. Figure 5 The first row, from left to right, shows the original echocardiograms of two-chamber (A2C) heart muscle, three-chamber (A3C) heart muscle, and four-chamber (A4C) heart muscle. Figure 5 The second row, from left to right, shows the original images and corresponding true labels for two-chamber (A2C) myocardial contrast echocardiograms, three-chamber (A3C) myocardial contrast echocardiograms, and four-chamber (A4C) myocardial contrast echocardiograms. Figure 5 The third to last rows show the simulation results of myocardial segmentation of three types of myocardium using the U-Net model, U-Net++ model, U-Net+++ model, U2-Net model, AttentionUNet model, Deeplabv3+ model, Segformer model, PSPNet model, and the myocardial segmentation model (DillateUNet) proposed in this invention, respectively. Figure 5 It is evident that the myocardial segmentation model (DillateUNet) proposed in this invention provides the most accurate myocardial segmentation, and can well adapt to the fluctuations in shape and position of the myocardium due to relaxation and contraction, thus performing complete and accurate segmentation of the myocardium.

[0095] Based on this embodiment, the input to the myocardial segmentation model (DillateUNet) can be any one of the three categories of myocardial contrast echocardiograms and myocardial contrast echocardiogram videos. This embodiment uses a batch of different myocardial contrast echocardiograms. For example, B represents the batch size, C represents the number of channels, H represents the image height, and W represents the image width. The initial input image is reshaped to adjust its resolution to 512×512. Each input image originally has 3 channels (R, G, and B channels). The batch size B is set according to the computing power of the experimental equipment. In this second embodiment, 4 Nvidia Geforce RTX 4090 GPUs are used, and the batch size B is set to 16.

[0096] In this second embodiment, the encoder has 5 layers, the decoder has 4 upsampling blocks (M), each dilated convolution module has 4 dilated convolution layers, P is set to 1, the Transformer block has 12 Transformer layers, and the downsampling method is max pooling. The k-th myocardial image is compared with echocardiography. The input image undergoes processing by the myocardial segmentation model, including:

[0097] Compare the k-th myocardial image with echocardiography image x k The input encoder extracts features, resulting in the output feature maps of each layer of the encoder, including:

[0098] Will After passing through the first layer of the encoder, which consists of two dilated convolutional modules, the output feature map of the first layer of the encoder is obtained. Then, downsampling is performed to obtain the output feature map of the first layer after downsampling.

[0099] Output feature map of the first layer after downsampling After passing through the second layer of the encoder, which consists of two dilated convolutional modules, the output feature map of the second layer of the encoder is obtained. Then, downsampling is performed to obtain the output feature map of the second layer after downsampling.

[0100] Output feature map of the second layer after downsampling After passing through the third layer of the encoder, which consists of three dilated convolutional modules, the output feature map of the third layer of the encoder is obtained. Then, downsampling is performed to obtain the output feature map of the third layer after downsampling.

[0101] Output feature map of the third layer after downsampling After passing through the fourth layer of the encoder, which consists of three dilated convolutional modules, the output feature map of the fourth layer of the encoder is obtained. Then, downsampling is performed to obtain the output feature map of the fourth layer after downsampling.

[0102]

[0103] Output feature map of the fourth layer after downsampling After passing through the fifth layer of the encoder, which consists of three dilated convolutional modules, the output feature map of the fifth layer of the encoder is obtained.

[0104] The output feature map of the fifth layer of the encoder Input the bottleneck layer, perform feature fusion, and obtain fused features, including:

[0105] In this second embodiment, the output feature map of the fifth layer of the encoder is used. Reshape the output feature map of the fifth layer of the encoder. The channel dimension was transformed from 1024 to 768, resulting in the reshaped feature map. Reshape the feature map Flatten into N patch blocks The collection of all patch blocks forms a shape of The matrix y p ;in,

[0106] Let matrix y p A linear mapping to a D-dimensional space, each patch from The formula for mapping dimension to dimension D is:

[0107]

[0108] in, Let be the D-dimensional vector corresponding to the e-th patch block. Let y be the mapping matrix. p e For the e-th patch block, b p As the bias term for linear mapping, this embodiment performs linear mapping through ordinary convolution, setting both the kernel size and stride of the ordinary convolution to P.

[0109] After adding positional encoding to the D-dimensional vector corresponding to each patch block of the output feature map of the last layer of the encoder, a two-dimensional sequence is obtained. The formula is:

[0110]

[0111] Among them, X Feat It is a two-dimensional sequence. E is the D-dimensional vector corresponding to the Nth patch block. pos This is the position encoding matrix.

[0112] Two-dimensional sequence The input is fed into the Transformer block for feature fusion, resulting in the initial fused feature t0. The size and shape of the initial fused feature t0 do not change after passing through the Transformer block; the shape of the initial fused feature t0 remains (1024, 768).

[0113] The initial fused feature t0 is input into the first Transformer layer in the Transformer block to obtain the output features of the first Transformer layer, including:

[0114] Initial fusion features By normalizing the first Transformer layer (LN layer), we obtain the output features of the LN layer of the first Transformer layer, as shown in the formula:

[0115]

[0116] Among them, t1 ′ Let μ be the output feature of the LN layer of the first Transformer layer, and let μ be X. Feat The mean along the feature dimension, σ 2 For X Feat The variance in the feature dimension is ε, which is a constant, ⊙ is the Hadamard product, γ1 is a learnable scaling parameter used to adjust the range of normalized feature values, and β1 is a learnable offset parameter used to adjust the offset of normalized feature values.

[0117] The output feature t1 of the LN layer of the first Transformer layer ′ The output feature S1 of the multi-head attention mechanism of the first Transformer layer is obtained through the multi-head attention mechanism of the first Transformer layer. ′ ,include:

[0118] The output feature t1 of the LN layer of the first Transformer layer ′ Applying three different linear projections, we obtain t1. ′ The query vector Q = W Q t1 ′ Key vector K = W K t1 ′ Value vector V = W V t1 ′ Among them, W Q W is a learnable query projection matrix. K W is a learnable bond projection matrix. V W is the projected matrix of learnable values. Q W K , h represents the number of attention heads;

[0119] Based on t1 ′ The query vector Q of the attention mechanism of the τth head τ Key vector K τ Value vector Vτ Calculate t1 ′ The self-attention score S of the τth head is obtained. τ The formula is:

[0120]

[0121] Among them, S τ For t1 ′ The self-attention score of the τth head is obtained. Softmax(.) is the softmax function, which normalizes the attention score into a probability distribution. T For the transpose, τ = 1, 2…h.

[0122] The self-attention scores of h heads are concatenated after final linear projection to obtain the output feature S1 of the multi-head attention mechanism of the first Transformer layer. ′ The formula is:

[0123]

[0124] Among them, S1 ′ W represents the output feature of the multi-head attention mechanism in the first Transformer layer. o This is the final linear projection.

[0125] The output feature t1 of the LN layer of the first Transformer layer ′ Output feature S1 of the multi-head attention mechanism of the first Transformer layer ′ Add them together to obtain the fused attention feature t1″ of the first Transformer layer;

[0126] The fused attention feature t1″ of the first Transformer layer is passed through a multilayer perceptron to obtain the output feature t1″′ of the multilayer perceptron of the first Transformer layer. The multilayer perceptron consists of two linear layers and one non-linear activation function (GELU), as shown in the formula:

[0127] t1″′=ω fc1 (GELU(ω fc2 (LN(S1″))+b fc2 ))+b fc1 ,

[0128] Where t1″′ represents the output feature of the multilayer perceptron in the first Transformer layer. Let be the first projection matrix. Here, M is the second projection matrix, M is the dimension of the intermediate layer projection, GELU(.) is a non-linear activation function that scales the input according to a normal probability distribution, and b fc1 For the first bias term, b fc2 This is the second bias term.

[0129] The output feature t1″′ of the multilayer perceptron in the first Transformer layer is added to the fused attention feature t1″ to obtain the output feature of the first Transformer layer.

[0130] Similarly, the formulas for the output features of other Transformer layers are:

[0131] t ′ θ ′ =MAS(LN(t) θ；1 ))+t θ；1 θ = 1, 2…12

[0132]

[0133] Among them, t ′ θ ′ Let t be the fused attention feature of the θ-th Transformer layer, MAS(.) be the multi-head self-attention mechanism, LN(.) be the layer normalization, and MLP(.) be the multilayer perceptron. θ；1 For the input features of the θ-th Transformer layer, Let θ be the output feature of the θ-th Transformer layer.

[0134] The initial fused features are reshaped so that their dimensions are similar to those of the reshaped feature map. Consistency, resulting in fusion features

[0135] The output feature maps of each layer of the encoder are combined with the fused features Input the signal to the decoder to obtain the k-th myocardial contrast echocardiogram x. k The myocardial segmentation results include:

[0136] Fusion features The output feature map of the 3×3 convolutional layer of the first upsampled block is obtained through the first upsampled block. The output feature map of the 3×3 convolutional layer of the first upsampled block The output feature map of the first upsampled block is obtained by performing a 2×2 upconvolution on the first upsampled block.

[0137] The output feature map of the first upsampled block With the output feature map of the fourth layer of the encoder After stacking along the channel dimension, the output feature map of the first upsampled block after stacking is obtained.

[0138] The output feature map of the first upsampled block after stacking The output feature map of the second upsampling block's 3×3 convolutional layer is obtained through the second upsampling block. The output feature map of the 3×3 convolutional layer of the second upsampling block The output feature map of the second upsampled block is obtained by performing a 2×2 upconvolution on the second upsampled block.

[0139] The output feature map of the second upsampling block With the output feature map of the third layer of the encoder After stacking along the channel dimension, the output feature map of the second upsampled block is obtained.

[0140] The output feature map of the stacked second upsampled block The output feature map of the 3×3 convolutional layer of the third upsampling block is obtained through the third upsampling block. The output feature map of the 3×3 convolutional layer of the third upsampling block The output feature map of the third upsampled block is obtained by performing a 2×2 upconvolution on the third upsampled block.

[0141] The output feature map of the third upsampling block With the output feature map of the second layer of the encoder After stacking along the channel dimension, the output feature map of the third upsampled block is obtained.

[0142] The output feature map of the third upsampled block after stacking The output feature map of the fourth upsampling block's 3×3 convolutional layer is obtained through the fourth upsampling block. The output feature map of the 3×3 convolutional layer of the fourth upsampling block The output feature map of the fourth upsampling block is obtained by performing a 2×2 upconvolution on the fourth upsampling block.

[0143] The output feature map of the fourth upsampling block Compared with the output feature map of the first layer of the encoder After stacking along the channel dimension, the output feature map of the fourth upsampled block is obtained.

[0144] The output feature map of the fourth upsampled block after stacking After passing through a 3×3 convolutional layer, we obtain the output feature map of the 3×3 convolutional layer.

[0145] The output feature map of the 3×3 convolutional layer Input to the segmentation head, and output the k-th myocardial contrast echocardiogram x. k All pixels are divided into two categories, and the final output is the k-th myocardial contrast echocardiogram x. k Myocardial segmentation results

[0146] In this second embodiment, by setting the number of encoder layers to 5, rich and hierarchical feature information can be extracted stepwise at different depth levels from myocardial contrast echocardiography, making feature extraction more comprehensive and refined. Setting the number of dilated convolutional layers in each dilated convolutional module to 4 allows for the effective expansion of the receptive field without excessive computation, enabling better capture of contextual information and detailed features in the image through combinations of dilated convolutional layers with different dilation rates. Setting P to 1 transforms each pixel in the output feature map of the fifth encoder layer into a patch, allowing interaction between each pixel in the feature map and maximizing feature fusion. Since the Transformer has many parameters, setting the number of Transformer layers in the Transformer block to 12 layers reduces model parameters and computation while fully utilizing the powerful feature fusion and representation capabilities of the Transformer architecture. This allows for deep and comprehensive feature interaction and fusion on the position-encoded patch block vectors, resulting in more representative and accurate initial fused features. Ultimately, this leads to more accurate myocardial segmentation results from myocardial contrast echocardiography, improving the overall performance and effectiveness of the myocardial segmentation model.

[0147] This third embodiment provides a myocardial segmentation system based on myocardial contrast echocardiography, including:

[0148] The model building module is used to build a myocardial segmentation model, which includes an encoder, a bottleneck layer, and a decoder.

[0149] The encoder includes multiple layers connected sequentially along the forward propagation direction. Each layer includes multiple dilated convolution modules connected sequentially. Each dilated convolution module includes multiple dilated convolution layers with different dilation rates connected sequentially. Each dilated convolution layer includes a dilated convolution, a batch normalization layer, and a ReLU activation function connected sequentially. The output features of all dilated convolution layers in each dilated convolution module are stacked in the channel dimension to obtain the output feature map of each dilated convolution module.

[0150] The feature extraction module is used to input the myocardial contrast echocardiogram into the encoder to extract features and obtain the output feature maps of each layer of the encoder;

[0151] The fusion module is used to input the output feature map of the last layer of the encoder into the bottleneck layer for feature fusion to obtain fused features.

[0152] The segmentation module is used to input the output feature maps of each layer of the encoder and the fused features into the decoder to obtain the myocardial segmentation results of myocardial contrast echocardiography.

[0153] This fourth embodiment provides a myocardial segmentation device based on myocardial contrast echocardiography, including:

[0154] A memory for storing a computer program; a processor for executing the computer program to implement the steps of the above-described myocardial segmentation method based on myocardial contrast echocardiography.

[0155] In this embodiment, specifically, a myocardial segmentation device based on myocardial contrast echocardiography includes: a storage device for storing datasets and intermediate results during training; a central processing unit (CPU) for controlling the process and handling auxiliary tasks, including data preprocessing; an image processing unit (GPU) for accelerating deep learning training; RAM for temporarily storing data and model parameters; a power supply and cooling system for ensuring stable system operation and avoiding overheating and power instability; and a network device for inter-node communication in large-scale distributed training.

[0156] The GPU can be any one of the brands such as NVIDIA, RTX, or Tesla.

[0157] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0161] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for myocardial segmentation based on myocardial contrast echocardiography, characterized in that, Includes the following steps: A myocardial segmentation model is constructed, which includes an encoder, a bottleneck layer, and a decoder. The encoder includes multiple layers connected sequentially along the forward propagation direction. Each layer includes multiple dilated convolution modules connected sequentially. Each dilated convolution module includes multiple dilated convolution layers with different dilation rates connected sequentially. Each dilated convolution layer includes a dilated convolution, a batch normalization layer, and a ReLU activation function connected sequentially. The output features of all dilated convolution layers in each dilated convolution module are stacked in the channel dimension to obtain the output feature map of each dilated convolution module. The myocardial contrast echocardiogram is input into the encoder to extract features, resulting in the output feature maps of each layer of the encoder; The output feature map of the last layer of the encoder is input into the bottleneck layer for feature fusion to obtain fused features; The output feature maps of each layer of the encoder are input into the decoder along with the fused features to obtain the myocardial segmentation results of myocardial contrast echocardiography.

2. The method for myocardial segmentation based on myocardial contrast echocardiography according to claim 1, characterized in that, When the encoder has 5 layers, the first and second layers of the encoder each include two dilated convolution modules connected in sequence, and the third, fourth and fifth layers of the encoder each include three dilated convolution modules connected in sequence. When the number of layers in the encoder is greater than 5, the first and second layers of the encoder each include two dilated convolutional modules connected in sequence, the third, fourth and fifth layers of the encoder each include three dilated convolutional modules connected in sequence, and the remaining layers of the encoder each include multiple dilated convolutional modules connected in sequence.

3. The myocardial segmentation method based on myocardial contrast echocardiography according to claim 1, characterized in that, The input features of the dilated convolution module are input into the dilated convolution module, and the output feature map of the dilated convolution module is output, including: The input features of the i-th dilated convolutional layer in the dilated convolution module are passed through the dilated convolution of the i-th dilated convolutional layer to obtain the output features of the i-th dilated convolutional layer, as shown in the formula: Z i =ω i *F i +b, The output features of the dilated convolution of the i-th dilated convolution layer are processed by a batch normalization layer to obtain the batch-normalized output features of the dilated convolution of the i-th dilated convolution layer, as shown in the formula: The output features of the i-th dilated convolutional layer after batch normalization are processed by the ReLU activation function to obtain the output features of the i-th dilated convolutional layer, as shown in the formula: The output features of all dilated convolutional layers are stacked along the channel dimension to obtain the output feature map of the dilated convolutional module, as shown in the formula: Among them, Z i Let ω be the output feature of the dilated convolution of the i-th dilated convolutional layer. i F is the convolution kernel of the dilated convolution of the i-th dilated convolution layer in the dilated convolution module. i Let be the input feature of the i-th dilated convolutional layer in the dilated convolutional module, b be the bias term, * be the convolution operation, i = 1, 2, ..., I, and I be the number of dilated convolutional layers in the dilated convolutional module. The output feature of the dilated convolution of the i-th dilated convolutional layer after batch normalization, where · represents the dot product, and μ B This represents the mean of the output features of the dilated convolution of the i-th dilated convolution layer corresponding to the myocardial contrast echocardiogram. Let ε be the variance of the output features of the dilated convolution of the i-th dilated convolution layer corresponding to the myocardial contrast echocardiogram, where ε is a constant, γ is a learnable first parameter, and β is a learnable second parameter. Let max(.) be the output feature of the i-th dilated convolutional layer, and max(.) be the maximum value. This is the output feature map of the dilated convolution module. This is for stacking along the channel dimension.

4. The method for myocardial segmentation based on myocardial contrast echocardiography according to claim 3, characterized in that, The input features of the dilated convolution module are input into the dilated convolution module, and the output feature map of the dilated convolution module is output. The formula is: in, DCM(.) represents the output feature map of the dilated convolution module. To stack the layers along the channel dimension, ReLU(.) is the ReLU activation function, BN(.) is the batch normalization layer, and DilatedConv... i (.) represents the dilated convolution of the i-th dilated convolutional layer in the dilated convolutional module, where I is the number of dilated convolutional layers in the dilated convolutional module, and F i F1 represents the input features of the i-th dilated convolutional layer in the dilated convolutional module.

5. The method for myocardial segmentation based on myocardial contrast echocardiography according to claim 1, characterized in that, The step of inputting the output feature map of the last layer of the encoder into the bottleneck layer for feature fusion to obtain fused features includes: The output feature map of the last layer of the encoder is reshaped to obtain a reshaped feature map. This reshaped feature map is then flattened into N patches. The set of all patches forms a shape... , where, The height of the output feature map of the last layer of the encoder. P represents the width of the output feature map of the last layer of the encoder, and P is the size of the patch block. The matrix is ​​linearly mapped to a D-dimensional space, and each patch block is converted into its corresponding D-dimensional vector; After adding positional encoding to the D-dimensional vector corresponding to each patch block of the output feature map of the last layer of the encoder, it is input into the Transformer block for feature fusion to obtain the initial fused features; The initial fused features are reshaped to match the size of the reshaped feature map, thus obtaining the fused features.

6. The method for myocardial segmentation based on myocardial contrast echocardiography according to claim 5, characterized in that, The Transformer block consists of multiple Transformer layers connected in sequence. Each Transformer layer includes: layer normalization, multi-head attention mechanism and multilayer perceptron connected in sequence.

7. The method for myocardial segmentation based on myocardial contrast echocardiography according to claim 1, characterized in that, The decoder includes: Multiple upsampling blocks, 3×3 convolutional layers, and a segmentation head. Each upsampling block consists of a 3×3 convolutional layer and a 2×2 upconvolution. Each 3×3 convolutional layer includes a 3×3 convolution and a ReLU activation function.

8. A method for myocardial segmentation based on myocardial contrast echocardiography according to claim 7, characterized in that, The process of inputting the output feature maps of each layer of the encoder and the fused features into the decoder to obtain the myocardial segmentation result of myocardial contrast echocardiography includes: The fused features are processed through the first upsampling block to obtain the output feature map of the first upsampling block; The output feature map of the j-th upsampled block is stacked with the output feature map of the (M-j+1)-th layer of the encoder in the channel dimension, and then used as the input of the (j+1)-th upsampled block; where j = 1, 2, ..., M, and M is the number of upsampled blocks; The output feature map of the Mth upsampling block is stacked with the output feature map of the first layer of the encoder in the channel dimension, and then processed by a 3×3 convolutional layer and a segmentation head to obtain the myocardial segmentation result of myocardial contrast echocardiography.

9. A myocardial segmentation system based on myocardial contrast echocardiography, characterized in that, include: The model building module is used to build a myocardial segmentation model, which includes an encoder, a bottleneck layer, and a decoder. The encoder includes multiple layers connected sequentially along the forward propagation direction. Each layer includes multiple dilated convolution modules connected sequentially. Each dilated convolution module includes multiple dilated convolution layers with different dilation rates connected sequentially. Each dilated convolution layer includes a dilated convolution, a batch normalization layer, and a ReLU activation function connected sequentially. The output features of all dilated convolution layers in each dilated convolution module are stacked in the channel dimension to obtain the output feature map of each dilated convolution module. The feature extraction module is used to input the myocardial contrast echocardiogram into the encoder to extract features and obtain the output feature maps of each layer of the encoder; The fusion module is used to input the output feature map of the last layer of the encoder into the bottleneck layer for feature fusion to obtain fused features. The segmentation module is used to input the output feature maps of each layer of the encoder and the fused features into the decoder to obtain the myocardial segmentation results of myocardial contrast echocardiography.

10. A myocardial segmentation device based on myocardial contrast echocardiography, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the myocardial segmentation method based on myocardial contrast echocardiography as described in claims 1 to 8 when executing the computer program.

Images