Multi-scale cross ultrasonic medical image segmentation method based on parallel coding structure

Abstract

The invention discloses a multi-scale cross ultrasonic medical image segmentation method based on a parallel coding structure, which is called MSC-LSAM. Belongs to the technical field of medical image processing, and comprises the following steps of: adding an efficient adapter block into a parallel UNet encoder and a frozen segmentation all-model visual encoder, and adding an efficient adapter block into the frozen segmentation all-model visual encoder, so that the segmentation all-model visual encoder keeps a relatively low parameter quantity and keeps the learning capability; in addition, multi-scale cross attention is introduced into the UNet global network, cross fusion of multi-scale features is realized, the edge segmentation capability is effectively improved, and overfitting is inhibited; after the parallel encoder features are spliced, an efficient channel attention feature fusion block is added, and efficient fusion of the parallel encoder multi-scale features is achieved. The MSC-LSAM network can ensure the training precision and maintain a low calculation amount at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a multi-scale cross-ultrasound medical image segmentation method based on a parallel coding structure. Background Art

[0002] Cardiac cavity segmentation and carotid lumen segmentation can assist doctors in quantitatively assessing the condition of the patient's heart and carotid artery, and assist in the diagnosis of cardiovascular and cerebrovascular diseases. The manual segmentation process of ultrasound images is cumbersome and largely relies on the experience of medical experts. However, ultrasound images have problems such as low contrast, high noise, image blur, and many artifacts, which make it difficult to identify the edges of the cardiac cavity and carotid lumen, confuse the boundaries between normal tissue and diseased tissue, and significantly affect the segmentation accuracy. In recent years, the development of advanced models such as convolutional neural networks (CNN) and Transformer has brought new solutions to ultrasound image segmentation, but due to the resolution of ultrasound data, noise interference, diverse lesion morphology, and the lack of large-scale high-quality annotated data, how to effectively train a model with high robustness and strong generalization ability is still a major difficulty. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a multi-scale cross-ultrasound medical image segmentation method based on a parallel coding structure to achieve accurate segmentation of the heart cavity and carotid artery lumen from complex heart and carotid artery ultrasound images.

[0004] Technical solution: A multi-scale cross-ultrasound medical image segmentation method based on a parallel coding structure of the present invention comprises the following steps:

[0005] Step 1, obtaining ultrasound medical images from the public two-dimensional cardiac ultrasound dataset CAMUS and the self-built three-dimensional carotid artery ultrasound dataset CAUS;

[0006] Step 2: Encode the ultrasound medical image using the parallel segmentation model visual encoder and UNet encoder;

[0007] Step 3: Introduce an adapter into the frozen segmentation model visual encoder to form a learnable segmentation model while keeping the computational complexity low.

[0008] Step 4: Introduce a multi-scale cross-axis attention module into the UNet global network to achieve cross-fusion of multi-scale features;

[0009] Step 5: The fusion of dual encoder multi-scale features is realized through the channel attention module;

[0010] Step 6: Use the Dice similarity coefficient and 95% Hausdorff distance to evaluate the segmentation accuracy of the model, and set indicators to evaluate the model's ability to handle abnormal situations;

[0011] Step 7: Input the image data into the segmentation network for ultrasound image segmentation, complete the image segmentation, and output the model segmentation prediction result.

[0012] Furthermore, the visual encoder of the segmentation model remains frozen during the encoding process, reducing computational complexity.

[0013] Furthermore, the maximum number of channels of the UNet encoder is reduced from 1024 to 256 to reduce the amount of computation and the risk of overfitting.

[0014] Furthermore, in step 3, the adapter is a bottleneck model, including a downward projected MLP layer, a GELU activation function, and an upward projected MLP layer, and the formula is as follows

[0015]

[0016] Among them, F i Represents the output of each Transformer layer, MLP down Responsible for projecting these outputs downward to compress the input data to a lower dimension. The GELU function is introduced as an activation function, which adds nonlinearity and helps the model learn more complex feature representations. Subsequently, MLP up The layer restores the data from low dimensions to its original shape, realizing the up-projection of the adapter. This bottleneck structure design introduces more flexibility to the adapter, allowing the model to learn adaptability for specific tasks while maintaining computational efficiency.

[0017] Furthermore, in step 4, the multi-scale cross-axis attention module is divided into two parallel branches to calculate the horizontal and vertical axis attention respectively; each branch first undergoes 1D convolution with kernel sizes of 1×7, 1×11 and 1×21 to encode multi-scale context information along one spatial dimension. and represents the one-dimensional convolution along the x-axis and y-axis, Norm(·) represents the layer normalization, and F x and F y They are x-axis and y-axis outputs respectively; the formula can be written as:

[0018]

[0019] In cross attention, the input data is converted into three different vector sets: K for Key, Q for Query, and V for Value; but unlike axial attention, the Q matrix of one branch is passed to another branch, thus establishing interaction between the two parallel axes and utilizing multi-scale features and global information; F xK and F xV is the key and value matrix of the x-axis, F yQ is the query matrix of the y-axis. Similarly, F yV and F yK is the key and value matrix of the y-axis, F xQ is the query matrix on the x-axis, MHCA y (·,·,·) represents multi-head cross attention along the x-axis, MHCA x (·,·,·) represents multi-head cross attention along the y-axis, with:

[0020] F T =MHCA y (F yQ , F xV , F xK ), (4)

[0021] F B =MHCA x (F xQ , F yV , F yK ), (5)

[0022] The output of multi-scale cross-axis attention is expressed as:

[0023] F out =Conv 1×1 (F T )+Conv 1×1 (F B )+F.#(6).

[0024] Furthermore, in step 5, the channel attention module obtains the global spatial information of each channel of the input feature map through a global average pooling operation, and generates a vector representing the global information; then, a one-dimensional convolution kernel is used to realize the dynamic association between channels, and by adjusting the size of the convolution kernel, efficient fusion of multi-scale features of the dual encoders is achieved to reduce model mis-segmentation.

[0025] Furthermore, in step 6, the segmentation accuracy of the model is evaluated by the Dice similarity coefficient and the 95% Hausdorff distance. Specifically, the Dice similarity coefficient DSC measures the performance of the model by calculating the overlap between the segmentation result and the segmentation label, with T and P representing the true value and predicted value of all voxels respectively:

[0026]

[0027] The 95% Hausdorff distance HD95 measures the performance of the model by calculating the maximum distance between the segmentation result and 95% of the pixels of the segmentation label, with d T′P′ represents the maximum Hausdorff distance between the real voxel set T′ and the predicted voxel set P′, d P′T′ It represents the maximum Hausdorff distance between the predicted voxel set P′ and the real voxel set T′.

[0028] d H (T′, P′)=max(d T′P′ , d P′T′ ),#(8).

[0029] Further, in step 6, the setting index is used to evaluate the abnormal situation handling ability of the model, specifically: the accuracy ACC, precision PRE, sensitivity SE and F1 score F1-Score are used to evaluate the abnormal situation handling ability of the model, wherein TP represents the number of pixels correctly segmented as targets, FP represents the number of background pixels incorrectly segmented as targets, TN represents the number of pixels correctly segmented as backgrounds, and FN represents the number of target pixels incorrectly segmented as backgrounds, and there are:

[0030]

[0031]

[0032]

[0033] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0034] The present invention also discloses a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the present invention are implemented.

[0035] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0036] (1) By introducing multi-scale cross-attention and efficient channel attention modules, the accuracy and robustness of ultrasound medical image segmentation are significantly improved. Multi-scale cross-attention can effectively capture feature information of different scales and enhance the ability to handle complex boundaries, while efficient channel attention optimizes the feature fusion process and reduces the occurrence of mis-segmentation by dynamically adjusting channel weights.

[0037] (2) It improves the shortcomings of traditional ultrasound image segmentation methods in dealing with noise and blurred boundaries, especially in low-contrast and high-noise environments that are common in ultrasound images. MSC-LSAM can effectively suppress noise interference and improve segmentation accuracy, thereby better assisting doctors in disease diagnosis.

[0038] (3) A dual encoder structure is adopted, in which a learnable segmentation model and a UNet encoder are used in parallel. This allows the model to fully utilize the advantages of different models in the feature extraction process, improves the segmentation ability of complex structures, and effectively reduces the computational complexity of the model.

[0039] (4) The model architecture of the present invention achieves efficient feature fusion and forms an implicit consistency regularization mechanism through a cross-teaching strategy. This mechanism can produce more stable and accurate pseudo-labels, reduce the situation where different networks predict the same but wrong results, and thus improve the overall performance of the model.

[0040] (5) The present invention can effectively utilize a small number of annotated images and a large number of unannotated images to segment ultrasonic medical images, significantly reducing the dependence on high-quality pixel-level annotation and alleviating the pressure of annotation of ultrasonic medical image datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the overall architecture of MSC-LSAM;

[0042] Figure 2 The overall architecture of the LSAM image encoder and the architecture of the adapter;

[0043] Figure 3 The overall architecture of multi-scale cross-axis attention;

[0044] Figure 4 The overall architecture for efficient channel attention;

[0045] Figure 5 Visual comparison of MSC-LSAM and other algorithms on the two-chamber view of the CAMUS dataset;

[0046] Figure 6 Visual comparison of MSC-LSAM and other algorithms on the four-chamber view of the CAMUS dataset;

[0047] Figure 7 Visual comparison of MSC-LSAM and other algorithms on CAUS dataset;

[0048] Figure 8 Visualization results of ablation experiments on the two-chamber view of the CAMUS dataset;

[0049] Fig. 9 Visualization results of ablation experiments on the four-chamber view of the CAMUS dataset;

[0050] Fig.10 Visualization of the ablation experiment results on the CAUS dataset. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0052] Step 1: Obtain ultrasound medical image dataset.

[0053] The medical image datasets include but are not limited to:

[0054] The CAMUS dataset contains 500 2D ultrasound sequences in the two-chamber view and the four-chamber view, as well as manually annotated labels for each ultrasound sequence at the end-diastole and end-systole frames, of which 900 are used for training and 100 for testing. The manually annotated cardiac structures include the left ventricular endocardium, myocardium, and left atrial endocardium.

[0055] CAUS is a private dataset from the Affiliated Hospital of Nantong University. The dataset contains 2439 2D carotid ultrasound images, of which 1952 are used for training and 487 are used for testing. The dataset is annotated by 5 ultrasound doctors with more than 5 years of clinical experience. The manually annotated carotid structures include the internal carotid artery and the external carotid artery.

[0056] Step 2: Reduce the maximum number of channels of the UNet network from 1024 to 256, which reduces the amount of computation and the risk of overfitting.

[0057] Step 3: See Figure 2 , a simple and efficient adapter block is introduced between each Transformer layer, and the adapter is designed as a bottleneck model, including a downward-projected MLP layer, a GELU activation function, and an upward-projected MLP layer. i Represents the output of each Transformer layer, MLP down Responsible for projecting these outputs down to compress the input data into a lower dimension.

[0058] P i =MLP up (GELU(MLP down (F i ))),#(1)

[0059] In step 3, the GELU function introduced as the activation function adds nonlinearity to help the model learn more complex feature representations. Subsequently, the layer restores the data from the low dimension to the original shape to achieve the upper projection of the adapter.

[0060] Step 4: Embed multi-scale features into axial attention, see Figure 3 , which aims to better segment regions with different individual sizes and shapes. Among them, the multi-scale cross attention is divided into two parallel branches, which calculate the horizontal and vertical axis attention respectively. Each branch first passes through three 1D convolutions with different kernel sizes (1×7, 1×11 and 1×21) to encode multi-scale context information along one spatial dimension.

[0061] by and represents the one-dimensional convolution along the x-axis and y-axis, Norm(·) represents the layer normalization, and F x and F y For x-axis and y-axis output respectively, there are:

[0062]

[0063] For criss-cross attention, the input data is converted into three different vector sets: Key (K), Query (Q), and Value (V). However, unlike axial attention, the q matrix of one branch is passed to the other branch, thereby establishing interaction between the two parallel axes and more effectively utilizing multi-scale features and global information. xK and F xV is the key and value matrix of the x-axis, F yQ is the query matrix of the y-axis. Similarly, F yV and F yK is the key and value matrix of the y-axis, F xQ is the query matrix on the x-axis. y (·,·,·) represents multi-head cross attention along the x-axis, MHCA x (·,·,·) represents multi-head cross attention along the y-axis, with:

[0064] F T =MHCA y (F yQ , F xV , F xK ),#(4)

[0065] F B =MHCA x (F xQ , F yV , F yK ),#(5)

[0066] The output of multi-scale cross-axis attention can be expressed as:

[0067] F out =Conv 1×1(F T )+Conv 1×1 (F B )+F.#(6)

[0068] Step 5: Obtain the global spatial information of each channel of the input feature map through the global average pooling operation and generate a vector representing the global information. Then, use a one-dimensional convolution kernel to achieve dynamic association between channels by adjusting the size of the convolution kernel. Figure 4 ,The ECA module is introduced to adapt to the feature expression of different scales, effectively integrates the information from different modalities and different scales, and reduces the channel weights of useless features.

[0069] In order to evaluate the segmentation performance of the multi-scale cross-ultrasound medical image segmentation method proposed in the present invention on ultrasonic medical images, the Dice similarity coefficient (DSC) and 95% Hausdorf fdistance (HD95) are used to evaluate the segmentation accuracy of the model. In addition, the accuracy (ACC), precision (PRE), sensitivity (SE) and F1 score (F1-Score) are used to evaluate the abnormal situation handling ability of the model.

[0070] After comparative experiments, the comparative results show that MSC-LSAM has achieved a high level in the ultrasound medical image segmentation task in terms of DSC, HD95, ACC, PRE, SE, F1-Score and other indicators, see Tables 1, 2, and 3, showing excellent segmentation performance and abnormal situation handling capabilities.

[0071] The present invention conducts a comprehensive ablation experiment on the algorithm network, uses LSAM, ECA and MCA as variables, and analyzes the improvement of each module on the performance and segmentation effect of the neural network by gradually increasing the modules, see Tables 4, 5, and 6. In MSC-LSAM, the introduction of the learnable segmentation model significantly enhances the feature extraction capability of the encoder part, enables the model to focus on key features, and improves the model's understanding and capture capabilities of complex data patterns. However, due to the scale mismatch between the features generated by the learnable segmentation model and the features output by the UNet encoder, difficulties arise in the feature fusion process. This mismatch causes inefficient fusion of features, which leads to an increase in the mis-segmentation rate.

[0072] Table 1: Comparative experimental results on the two-chamber view of the CAMUS dataset

[0073]

[0074] Among them, R50 is Resnet50 and R101 is Resnet101.

[0075] Table 2: Comparative experimental results on the four-chamber view of the CAMUS dataset

[0076]

[0077] Among them, R50 is Resnet50 and R101 is Resnet101.

[0078] Table 3: Comparative experimental results on the CAUS dataset

[0079]

[0080]

[0081] Among them, R50 is Resnet50 and R101 is Resnet101.

[0082] Table 4: Ablation experimental results on the two-chamber section of the CAMUS dataset

[0083]

[0084] Label1 represents the left ventricular endocardium, Label2 represents the myocardium, and Label3 represents the left atrial endocardium.

[0085] Table 5: Ablation experimental results on the four-chamber section of the CAMUS dataset

[0086]

[0087]

[0088] Label1 represents the left ventricular endocardium, Label2 represents the myocardium, and Label3 represents the left atrial endocardium.

[0089] Table 6: Ablation experiment results on the CAUS dataset

[0090]

[0091] Label1 represents the internal carotid artery, and Label2 represents the external carotid artery.

Claims

1. A multi-scale cross-ultrasound medical image segmentation method based on a parallel coding structure, characterized in that: The steps include: Step 1, obtaining ultrasound medical images from the public two-dimensional cardiac ultrasound dataset CAMUS and the self-built three-dimensional carotid artery ultrasound dataset CAUS; Step 2: Encode the ultrasound medical image using the parallel segmentation model visual encoder and UNet encoder; Step 3: Introduce an adapter into the frozen segmentation model visual encoder to form a learnable segmentation model while keeping the computational complexity low. Step 4: Introduce a multi-scale cross-axis attention module into the UNet global network to achieve cross-fusion of multi-scale features; Step 5: The fusion of dual encoder multi-scale features is realized through the channel attention module; Step 6: Use the Dice similarity coefficient and 95% Hausdorff distance to evaluate the segmentation accuracy of the model, and set indicators to evaluate the model's ability to handle abnormal situations; Step 7: Input the image data into the segmentation network for ultrasound image segmentation, complete the image segmentation, and output the model segmentation prediction result.

2. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: The visual encoder of the segmentation model remains frozen during the encoding process to reduce computational complexity.

3. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: Reduce the maximum number of channels of the UNet encoder from 1024 to 256 to reduce the amount of computation and the risk of overfitting.

4. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: In step 3, the adapter is a bottleneck model, including a downward-projected MLP layer, a GELU activation function, and an upward-projected MLP layer, and the formula is as follows: P i =MLP up (GELU(MLP down (F i ))).#(1) where F i Represents the output of each Transformer layer, MLP down Responsible for projecting these outputs down to compress the input data into a lower dimension.

5. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: In step 4, the multi-scale cross-axis attention module is divided into two parallel branches to calculate the horizontal and vertical axis attention respectively; each branch first undergoes 1D convolution with kernel sizes of 1×7, 1×11, and 1×21 to encode multi-scale contextual information along one spatial dimension. and represents the one-dimensional convolution along the x-axis and y-axis, Norm(·) represents the layer normalization, and F x and F y They are x-axis and y-axis outputs respectively; the formula can be written as: In cross attention, the input data is converted into three different vector sets: K for Key, Q for Query, and V for Value; but unlike axial attention, the Q matrix of one branch is passed to another branch, thus establishing interaction between the two parallel axes and utilizing multi-scale features and global information; F xK and F xV is the key and value matrix of the x-axis, F yQ is the query matrix of the y-axis. Similarly, F yV and F yK is the key and value matrix of the y-axis, F xQ is the query matrix on the x-axis, MHCA y (·,·,·) represents multi-head cross attention along the x-axis, MHCA x (·,·,·) represents the multi-head cross attention along the y-axis, with: F T =MHCA y (F yQ ,F xV ,F xK ), (4) F B =MHCA x (F xQ ,F yV ,F yK ), (5) The output of multi-scale cross-axis attention is expressed as: F out =Conv 1×1 (F T )+Conv 1×1 (F B )+F.#(6)。 6. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: In step 5, the channel attention module obtains the global spatial information of each channel of the input feature map through a global average pooling operation and generates a vector representing the global information; then, a one-dimensional convolution kernel is used to realize the dynamic association between channels, and by adjusting the size of the convolution kernel, efficient fusion of multi-scale features of the dual encoders is achieved to reduce model mis-segmentation.

7. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: In step 6, the segmentation accuracy of the model is evaluated by the Dice similarity coefficient and the 95% Hausdorff distance. Specifically, the Dice similarity coefficient DSC measures the performance of the model by calculating the overlap between the segmentation result and the segmentation label. T and P represent the true value and predicted value of all voxels, respectively. The 95% Hausdorff distance HD95 measures the performance of the model by calculating the maximum distance between the segmentation result and 95% of the pixels of the segmentation label, with d T'P' represents the maximum Hausdorff distance between the real voxel set T' and the predicted voxel set P', d P'T' Represents the maximum Hausdorff distance from the predicted voxel set P' to the true voxel set T', which is: d H (T’,P’)=max(d T'P0' ,d P'T' ),#(8)。 8. The multi-scale cross-ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that: In step 6, the setting index is used to evaluate the abnormal situation handling ability of the model, specifically: the accuracy ACC, precision PRE, sensitivity SE and F1 score F1-Score are used to evaluate the abnormal situation handling ability of the model, where TP represents the number of pixels correctly segmented as targets, FP represents the number of background pixels incorrectly segmented as targets, TN represents the number of pixels correctly segmented as backgrounds, and FN represents the number of target pixels incorrectly segmented as backgrounds.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images