Lightweight segmentation algorithm for cardiac ultrasound images

Abstract

The application discloses a lightweight segmentation algorithm for heart ultrasound images and relates to the technical field of image processing, which adopts a trained lightweight segmentation algorithm model to segment heart ultrasound images, and comprises the following steps: a MobileViT feature extraction network with ordered combined channel pruning and weight conversion shared feature extraction network is used to extract features of the preprocessed heart ultrasound images, and four effective feature layers are obtained; according to the fourth effective feature layer, an effective compressed feature layer is obtained based on a lightweight separable spatial pyramid feature extraction network; the first, second, third effective feature layers and the effective compressed feature layer are fused, then the channel number is adjusted by using 1x1 convolution, and finally, the final prediction result is obtained through 4 times of up sampling. The application achieves remarkable effects in the lightweight of the model, designs a multi-scale feature fusion method to find more rich feature information to enhance the robustness of the model, and improves the image segmentation efficiency under the condition of high segmentation precision.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a lightweight segmentation algorithm for cardiac ultrasound images. Background Technology

[0002] A cardiac ultrasound image shows four main structures: the left ventricle (LV), the right ventricle (RV), the left atrium (LA), and the right atrium (RA), as follows: Figure 1 As shown, the main structural boundaries of the left ventricle are typically segmented at end-systole and end-diastole for quantitative functional analysis. For example, the left ventricular endocardium is manually annotated, and the ejection fraction is calculated by the difference between the end-diastolic and end-systolic volumes. However, this process is lengthy, time-consuming, and subjective. Therefore, automated echocardiographic segmentation is urgently needed clinically. However, due to the low contrast, noise, and uncertainty inherent in ultrasound images, segmentation tasks are challenging. Current research focuses on improving segmentation accuracy. Compared to image classification techniques, semantic segmentation networks, which require pixel-level prediction, demand complex structural designs and a large number of parameters to achieve higher accuracy. This results in massive computational demands for training and inference, limiting the need for platforms with high computational and storage capabilities. The large storage space and high computational complexity also restrict the large-scale clinical application of current mainstream deep learning methods. In practical applications such as echocardiographic segmentation, the challenge lies in reducing computational load and increasing the speed of cardiac ultrasound image segmentation while maintaining accuracy. Summary of the Invention

[0003] The present invention provides a lightweight segmentation algorithm for cardiac ultrasound images that can solve the above-mentioned problems.

[0004] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0005] This invention provides a lightweight segmentation algorithm for cardiac ultrasound images, employing a pre-trained lightweight segmentation model to segment cardiac ultrasound images. The basic structure for echocardiogram segmentation is built upon the DeepLab v3+ network. To ensure a lightweight base model and facilitate subsequent expansion and improvement, the lightweight and universal mobile device Vision Transformer, MobileViT, is selected as the backbone network.

[0006] The encoder network of the lightweight segmentation algorithm model includes the MobileViT feature extraction network, which combines ordered channel pruning and weight transformation sharing, as well as a lightweight separable spatial pyramid feature extraction module. The decoder network of the lightweight segmentation algorithm model includes a multi-scale feature fusion module.

[0007] The lightweight separable spatial pyramid feature extraction module is obtained by optimizing the dilated spatial convolution pyramid pooling module using depthwise separable convolution and spatially separable convolution.

[0008] The method for segmenting cardiac ultrasound images includes the following steps:

[0009] Step 1: Preprocess the cardiac ultrasound images;

[0010] Step 2: Use the MobileViT feature extraction network to extract features from the preprocessed echocardiogram image. The four stages of the MobileViT feature extraction network output the first effective feature layer, the second effective feature layer, the third effective feature layer, and the fourth effective feature layer, respectively.

[0011] Step 3: Input the fourth effective feature layer into the lightweight separable spatial pyramid feature extraction network, use parallel dilated convolutions with different rates to extract features, then merge them, and finally compress the features through 1x1 convolution to obtain the effective compressed feature layer.

[0012] Step 4: Use the multi-scale feature fusion module to fuse the first effective feature layer, the second effective feature layer, the third effective feature layer and the effective compressed feature layer to obtain the stacked effective feature layer;

[0013] Step 5: For the stacked effective feature layers, adjust the number of channels using 1x1 convolution, and then perform 4x upsampling to obtain the final prediction result, thus completing the segmentation of the cardiac ultrasound image.

[0014] In a preferred embodiment of the present invention, step 1 includes a preprocessing operation on the cardiac ultrasound image, which is then centered and adjusted to a size of 512*512.

[0015] In a preferred embodiment of the present invention, in step 2, for the feature extraction network MobileViT, redundant or weakened parameters need to be found and pruned. Then, at a pruning rate of 70%, weight reuse and sharing are performed, that is, all blocks in each stage share parameters, and the parameters of the Transformation Kernels are not shared between layers.

[0016] In a preferred embodiment of the present invention, in step 2: the first effective feature layer is an image feature of size 24×128×128 obtained by compressing the height and width of the preprocessed cardiac ultrasound image 4 times; the second effective feature layer is an image feature of size 48×64×64 obtained by compressing the height and width of the preprocessed cardiac ultrasound image 8 times; the third effective feature layer is an image feature of size 64×32×32 obtained by compressing the height and width of the preprocessed cardiac ultrasound image 16 times; and the fourth effective feature layer is an image feature of size 16×16×16 obtained by compressing the height and width of the preprocessed cardiac ultrasound image 32 times.

[0017] In a preferred embodiment of the present invention, step 4 specifically includes the following steps:

[0018] Step 41: After upsampling the effective compressed feature layer by 2 times, merge it with the third effective feature layer to obtain the first feature merging result;

[0019] Step 42: After upsampling the first feature merging result by 2 times, merge it with the second effective feature layer to obtain the second feature merging result;

[0020] Step 43: After upsampling the second feature merged result by 2 times, merge it with the first effective feature layer to obtain the stacked effective feature layer.

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] 1) The Atrous Spatial Pyramid Pooling (ASPP) module is improved through convolution optimization. By combining structured model pruning techniques such as channel pruning and cross-layer weight sharing, redundant parameters of the vision transformer backbone are reduced, thus achieving a significant effect on model lightweighting. Finally, a multi-scale feature fusion method is designed to find richer feature information to enhance the robustness of the model.

[0023] 2) Compared with existing technologies, the method of the present invention significantly reduces the number of model parameters and computational load, and greatly improves image segmentation efficiency while maintaining high segmentation accuracy. It is more suitable for many medical scenarios that require real-time semantic segmentation or have device limitations.

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an example image of a cardiac ultrasound.

[0027] Figure 2 This is an architecture diagram of the lightweight segmentation algorithm model described in this invention;

[0028] Figure 3 This is a flowchart illustrating how the invention derives the final lightweight segmentation algorithm model from the original model.

[0029] Figure 4 This is a comparison chart showing the effects of segmenting cardiac ultrasound images using existing methods and the method of this invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0031] This invention proposes a lightweight segmentation algorithm for cardiac ultrasound images. The basic architecture of the lightweight segmentation algorithm model uses a DeepLab v3+ network with encoder and decoder networks, such as... Figure 2 As shown, the lightweight segmentation algorithm model includes the MobileViT feature extraction network, which combines ordered channel pruning and weight transformation sharing; a lightweight separable spatial pyramid feature extraction network; and a multi-scale feature fusion module.

[0032] We optimized the ASPP module using depthwise separable convolution and spatially separable convolution to construct a lightweight separable spatial pyramid feature extraction network, which improved the bottleneck structure of ASPP and efficiently extracted local and contextual information.

[0033] We designed a structured channel pruning technique for the vision transformer in the MobileViT feature extraction network. By adding masks to the Multihead Self-Attention (MHSA) and Multilayer Perceptron (MLP) mechanisms in MobileViT and applying L1 regularization to achieve sparsity, we can identify and prune redundant or weakened parameters in the model. Pruning these parameters reduces the model's complexity, improves its generalization performance, and avoids overfitting. Subsequently, at a 70% pruning rate, we implemented weight reuse and sharing, meaning that all blocks in each stage share parameters, while the Transformation Kernels parameters are not shared between layers. We also introduced linear transformations before and after the softmax layer to introduce non-linearity, thereby increasing the model's diversity and flexibility, and thus better fitting the data.

[0034] Based on the stage characteristics of feature extraction networks, a multi-scale feature fusion module is designed. By using multi-scale feature fusion, the missing details in the predicted features are enriched, the model's learning ability is enhanced, and the receptive field is increased. This improves segmentation performance with limited computation.

[0035] The method for segmenting cardiac ultrasound images according to the present invention is as follows:

[0036] Step 1: Since the original echocardiogram image is large in size, data preprocessing is required to speed up the training process. This includes: centering the echocardiogram image to reduce the influence of the background region on the segmentation results, adjusting the 2D echocardiogram image to 512*512 size, and then inputting it into the lightweight segmentation algorithm model.

[0037] Step 2: The lightweight segmentation algorithm model uses the MobileViT feature extraction network, which consists of four stages (stage 1-stage 4). Each stage generates output feature maps of different spatial sizes. After the preprocessed echocardiogram image passes through the MobileViT feature extraction network, four effective feature layers are obtained: the first effective feature layer (stage 1, which compresses the height and width of the input image four times, resulting in a size of 24×128×128), the second effective feature layer (stage 2, which compresses the height and width of the input image eight times, resulting in a size of 48×64×64), the third effective feature layer (stage 3, which compresses the height and width of the input image 16 times, resulting in a size of 64×32×32), and the fourth effective feature layer (stage 4, which compresses the height and width of the input image 32 times, resulting in a size of 16×16×16).

[0038] Step 3: In the encoder network, the fourth effective feature layer is input into the lightweight separable spatial pyramid feature extraction network. Parallel dilated convolutions are used to extract features with dilated convolutions of different rates, which are then merged. Finally, the features are compressed through 1x1 convolutions to obtain the effective compressed feature layer.

[0039] Step 4: Since shallow features have higher resolution and contain more spatial information and details, while deep features have stronger semantic information, this step effectively fuses the output features at different scales in the MobileViT feature extraction network with the upsampled predicted features to enrich the detailed information missing from the predicted features. Specifically, this includes:

[0040] Step 41: After upsampling the effective compressed feature layer obtained in step 3 by 2, merge it with the third effective feature layer to obtain the first feature merging result;

[0041] Step 42: Then, after upsampling the first feature merging result by 2 times, merge it with the second effective feature layer to obtain the second feature merging result;

[0042] Step 43: After upsampling the second feature merging result by 2 times, merge it with the first effective feature layer to obtain the third feature merging result.

[0043] Step 5: In the decoder network, the number of channels in the stacked effective feature layers, i.e., the third feature merging result, is adjusted using a 1x1 convolution. Finally, the final prediction result is obtained after a 4x upsampling. The main process of obtaining the final compressed model from the original model is as follows: Figure 3 As shown.

[0044] Based on the lightweight segmentation algorithm for cardiac ultrasound images proposed in this invention, we used the fully annotated CAMUS (Cardiac Acquisition with Multi-Structure Ultrasound Segmentation) dataset to segment three important substructures of the left ventricle: the left ventricular endocardium (LVEndo), the left ventricular epicardium (LVEpi), and the left atrium (LA). Figure 4 The segmentation performance of the original model and the compressed model after lightweighting is compared and presented. (a) represents the input image; (b) represents the ground truth; (c) represents the segmentation result of the original model; and (d) represents the segmentation result of the compressed model. Observation of the visualized prediction results reveals that, under the same conditions, our proposed lightweight model predicts segmentation contours that are closer to the manually annotated contours by doctors. Our method also demonstrates good segmentation performance in terms of the segmentation of different tissue structures in the left ventricle and the detail of edge points.

[0045] We compared the segmentation efficiency and performance of the improved lightweight model with the original model (based on the DeepLab v3+ network and using MobileViT as the backbone network) without any compression measures. We selected IoU (Intersection over Union) and Dice similarity coefficient as segmentation performance evaluation metrics, and model parameter count (parameters, parameters) and floating-point operations (FLOPs) as segmentation efficiency evaluation metrics. Table 1 shows the overall segmentation effect of the proposed model compression method.

[0046] Table 1

[0047]

[0048] The lightweight segmentation algorithm for cardiac ultrasound images described in this invention significantly reduces the number of model parameters and computational cost in terms of segmentation efficiency. The number of parameters is reduced from 3.716M to 800.070K, saving 79%, and the computational cost is reduced from 4.016G to 2.356G, saving 41.3%. This makes it more suitable for many medical scenarios that require real-time semantic segmentation or have device limitations. In terms of segmentation performance, the segmentation accuracy of the lightweight model is almost the same as that of the original model (original mIou and mDice are 0.8459 and 0.9081, respectively; after optimization, they are 0.8456 and 0.9081, respectively). We use the lightweight model to segment cardiac ultrasound images, and cardiac boundary segmentation in echocardiography is of great significance for cardiac function assessment and disease diagnosis.

[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A lightweight segmentation algorithm for cardiac ultrasound images, characterized in that, A trained lightweight segmentation algorithm model was used to segment cardiac ultrasound images. The basic structure of the lightweight segmentation algorithm model was built on the DeepLab v3+ network. Its encoder network includes the MobileViT feature extraction network, which combines ordered channel pruning and weight transformation sharing, and a lightweight separable spatial pyramid feature extraction module. Its decoder network includes a multi-scale feature fusion module. Sparsity was achieved by adding masks and L1 regularization to the multi-head self-attention mechanism and multilayer perceptron in MobileViT, and redundant or weakened parameters in the model were identified and pruned. Subsequently, at a pruning rate of 70%, weight reuse and sharing were performed, that is, all blocks in each stage share parameters, but the parameters of the Transformation Kernels are not shared between layers, and linear transformations are introduced before and after the softmax layer to introduce nonlinearity. The lightweight separable spatial pyramid feature extraction module was obtained by optimizing the dilated spatial convolution pyramid pooling module using depthwise separable convolution and spatially separable convolution. The method for segmenting cardiac ultrasound images includes the following steps: Step 1: Preprocess the cardiac ultrasound images; Step 2: Use the MobileViT feature extraction network to extract features from the preprocessed echocardiogram image. The four stages of the MobileViT feature extraction network output the first effective feature layer, the second effective feature layer, the third effective feature layer, and the fourth effective feature layer, respectively. Step 3: Input the fourth effective feature layer into the lightweight separable spatial pyramid feature extraction module, use parallel dilated convolutions with different rates to extract features, then merge them, and finally compress the features through 1x1 convolution to obtain the effective compressed feature layer. Step 4: Use the multi-scale feature fusion module to fuse the first effective feature layer, the second effective feature layer, the third effective feature layer and the effective compressed feature layer to obtain the stacked effective feature layer; Step 5: For the stacked effective feature layers, adjust the number of channels using 1x1 convolution, and then perform 4x upsampling to obtain the final prediction result, thus completing the segmentation of the cardiac ultrasound image.

2. The lightweight segmentation algorithm for cardiac ultrasound images according to claim 1, characterized in that, In step 1, the preprocessing process includes center cropping the cardiac ultrasound image and then resizing it to 512*512 pixels.

3. The lightweight segmentation algorithm for cardiac ultrasound images according to claim 1, characterized in that, In step 2: the first effective feature layer is the image feature with a size of 24×128×128 obtained by compressing the height and width of the preprocessed cardiac ultrasound image four times; The second effective feature layer is the image feature with a size of 48×64×64 obtained by compressing the height and width of the preprocessed cardiac ultrasound image eight times. The third effective feature layer is the image feature with a size of 64×32×32 obtained by compressing the height and width of the preprocessed cardiac ultrasound image 16 times; the fourth effective feature layer is the image feature with a size of 16×16×16 obtained by compressing the height and width of the preprocessed cardiac ultrasound image 32 times.

4. The lightweight segmentation algorithm for cardiac ultrasound images according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 41: After upsampling the effective compressed feature layer by 2 times, merge it with the third effective feature layer to obtain the first feature merging result; Step 42: After upsampling the first feature merging result by 2 times, merge it with the second effective feature layer to obtain the second feature merging result; Step 43: After upsampling the second feature merged result by 2 times, merge it with the first effective feature layer to obtain the stacked effective feature layer.

Images