Multi-scale cross-modality ultrasound medical image segmentation method based on parallel coding structure

Abstract

The application discloses a kind of multi-scale cross ultrasonic medical image segmentation methods based on parallel coding structure, called MSC-LSAM.Belong to medical image processing technical field, the application includes in parallel UNet encoder and frozen segmentation everything model visual encoder, and frozen segmentation everything model visual encoder is joined into efficient adapter block, so that segmentation everything model visual encoder is kept while retaining learning ability with lower parameter amount;In addition, the application introduces multi-scale cross attention in UNet global network, realizes the cross fusion of multi-scale feature, effectively improves edge segmentation ability, inhibits overfitting;After parallel encoder feature splicing, join efficient channel attention feature fusion block, realize the efficient fusion of parallel encoder multi-scale feature.MSC-LSAM network can guarantee training accuracy while keeping lower operation amount.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a multi-scale cross-ultrasound medical image segmentation method based on a parallel coding structure. Background Technology

[0002] Heart chamber segmentation and carotid artery segmentation can assist doctors in quantitatively assessing the condition of a patient's heart and carotid arteries, aiding in the diagnosis of cardiovascular and cerebrovascular diseases. Manual segmentation of ultrasound images is cumbersome and largely relies on the experience of medical experts. However, ultrasound images suffer from low contrast, high noise levels, image blurring, and numerous artifacts, making it difficult to identify the edges of heart chambers and carotid artery lumens, blurring the boundaries between normal and diseased tissues, and significantly affecting segmentation accuracy. In recent years, the development of advanced models such as Convolutional Neural Networks (CNNs) and Transformers has brought new solutions to ultrasound image segmentation. However, due to the resolution of ultrasound data, noise interference, diverse lesion morphologies, and the lack of large-scale, high-quality labeled data, effectively training robust and generalizable models remains a major challenge. Summary of the Invention

[0003] Objective of the Invention: The objective of this invention is to provide a multi-scale cross-segmentation method for ultrasound medical images based on a parallel coding structure. This method enables precise segmentation of cardiac cavities and carotid artery lumens from complex cardiac and carotid ultrasound images.

[0004] Technical solution: The present invention provides a multi-scale cross-linked ultrasound medical image segmentation method based on a parallel coding structure, comprising the following steps:

[0005] Step 1: Obtain ultrasound medical images from the publicly available 2D cardiac ultrasound dataset CAMUS and the self-built 3D carotid ultrasound dataset CAUS;

[0006] Step 2: Encode the ultrasound medical images using a parallel visual encoder for segmenting all models and a UNet encoder;

[0007] Step 3: Introduce an adapter into the frozen segmentation model visual encoder to form a learnable segmentation model while maintaining low computational cost.

[0008] Step 4: Introduce a multi-scale cross-axis attention module into the UNet global network to achieve cross-fusion of multi-scale features;

[0009] Step 5: Achieve multi-scale feature fusion between dual encoders through the channel attention module;

[0010] Step 6: Evaluate the segmentation accuracy of the model using the Dice similarity coefficient and the 95% Hausdorff distance, and set indicators to evaluate the model's ability to handle outliers.

[0011] Step 7: Input the image data into the segmentation network to perform ultrasound image segmentation, complete the image segmentation, and output the model segmentation prediction results.

[0012] Furthermore, the visual encoder that segments all models remains frozen during the encoding process, reducing computational complexity.

[0013] Furthermore, the maximum number of channels in the UNet encoder was reduced from 1024 to 256, reducing computational load and mitigating the risk of overfitting.

[0014] Furthermore, in step 3, the adapter is a bottleneck model, including a downward-projected MLP layer, a GELU activation function, and an upward-projected MLP layer, as shown in the following formula.

[0015]

[0016] Where F i This represents the output of each Transformer layer, MLP down This is responsible for projecting these outputs downwards, compressing the input data to a lower dimension. The GELU function is introduced as an activation function, adding non-linearity and helping the model learn more complex feature representations. Subsequently, MLP... up The layer restores the data from low dimension to its original shape, enabling the adapter's up projection. This bottleneck structure design introduces more flexibility to the adapter, allowing the model to learn task-specific adaptability while maintaining computational efficiency.

[0017] Furthermore, in step 4, the multi-scale cross-axis attention module is divided into two parallel branches, which respectively calculate the horizontal and vertical axial attention; each branch first undergoes 1D convolutions with kernel sizes of 1×7, 1×11, and 1×21 to encode multi-scale contextual information along one spatial dimension, so as to... and This represents a one-dimensional convolution along the x and y axes, Norm(·) represents layer normalization, and F... x and F y Outputs are provided for the x-axis and y-axis respectively; the formula can be written as:

[0018]

[0019] Cross-attention transforms the input data into three distinct vector sets: K for Key, Q for Query, and V for Value. However, unlike axial attention, the Q-matrix of one branch is passed to the other, establishing an interaction between the two parallel axes and leveraging multi-scale features and global information. xK and F xV F is the key-value matrix along the x-axis. yQ It is the query matrix along the y-axis, and similarly F yV and F yK F is the key and value matrix along the y-axis. xQ It is the query matrix along the x-axis, MHCA y (·,·,·) represents multi-head cross attention along the x-axis, MHCA x (·,·,·) represents a multi-headed intersection along the y-axis. Note that:

[0020] F T =MHCA y (F yQ F xV F xK (4)

[0021] F B =MHCA x (F xQ F yV F yK (5)

[0022] The output of multi-scale cross-axis attention is represented as:

[0023] F out =Conv 1×1 (F T )+Conv 1×1 (F B )+F.#(6).

[0024] Furthermore, in step 5, the channel attention module obtains the global spatial information of each channel of the input feature map through global average pooling operation and generates a vector representing the global information; then, a one-dimensional convolutional kernel is used to realize the dynamic correlation between channels. By adjusting the size of the convolutional kernel, the efficient fusion of multi-scale features of the dual encoder is achieved, reducing model missegmentation.

[0025] Furthermore, in step 6, the segmentation accuracy of the model is evaluated using the Dice similarity coefficient and the 95% Hausdorff distance. Specifically, the Dice similarity coefficient (DSC) measures the model's performance by calculating the overlap between the segmentation results and the segmentation labels, with T and P representing the true and predicted values ​​of all voxels, respectively.

[0026]

[0027] The 95% Hausdorff distance HD95 measures the model's performance by calculating the maximum distance between the segmentation result and 95% of the pixels in the segmentation label, in d T′P′ d represents the maximum Hausdorff distance between the true voxel set T′ and the predicted voxel set P′. P′T′ Let represent the maximum Hausdorff distance between the predicted voxel set P′ and the true voxel set T′. We have:

[0028] d H (T′,P′)=max(d T′P′ d P′T′ ), #(8).

[0029] Furthermore, in step 6, the setting of metrics to evaluate the model's ability to handle outliers specifically involves using accuracy (ACC), precision (PRE), sensitivity (SE), and F1 score (F1-Score) to evaluate the model's outlier handling capability. Here, TP represents the number of pixels correctly segmented as targets, FP represents the number of background pixels incorrectly segmented as targets, TN represents the number of pixels correctly segmented as background, and FN represents the number of target pixels incorrectly segmented as background.

[0030]

[0031]

[0032]

[0033] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0034] The present invention also discloses a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method of the present invention.

[0035] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0036] (1) By introducing multi-scale cross-attention and efficient channel attention modules, the accuracy and robustness of ultrasound medical image segmentation are significantly improved. Multi-scale cross-attention can effectively capture feature information at different scales and enhance the ability to handle complex boundaries, while efficient channel attention optimizes the feature fusion process by dynamically adjusting channel weights and reduces the occurrence of missegmentation.

[0037] (2) It improves the shortcomings of traditional ultrasound image segmentation methods in dealing with noise and blurred boundaries. Especially in the low contrast and high noise environment common in ultrasound images, MSC-LSAM can effectively suppress noise interference and improve the accuracy of segmentation, thereby better assisting doctors in disease diagnosis.

[0038] (3) A dual encoder structure was adopted, in which a learnable segmentation model and a UNet encoder were used in parallel, which enabled the model to make full use of the advantages of different models during feature extraction, improve the segmentation ability of complex structures, and effectively reduce the computational complexity of the model.

[0039] (4) The model architecture of this invention achieves efficient feature fusion and forms an implicit consistency regularization mechanism through a cross-teaching strategy. This mechanism can generate more stable and accurate pseudo-labels, reduce the situation where different networks predict the same but incorrect labels, thereby improving the overall performance of the model.

[0040] (5) The present invention can effectively utilize a small number of labeled images and a large number of unlabeled images for the segmentation of ultrasound medical images, significantly reducing the dependence on high-quality pixel-level annotations and alleviating the pressure of labeling ultrasound medical image datasets. Attached Figure Description

[0041] Figure 1 The overall architecture of MSC-LSAM;

[0042] Figure 2 The overall architecture of the LSAM image encoder and the architecture of the adapter;

[0043] Figure 3 The overall architecture for multi-scale cross-axis attention;

[0044] Figure 4 An overall architecture for efficient channel attention;

[0045] Figure 5 Visual comparison of MSC-LSAM with other algorithms on the two-cavity cross section of the CAMUS dataset;

[0046] Figure 6 Visual comparison of MSC-LSAM with other algorithms on a four-cavity section of the CAMUS dataset;

[0047] Figure 7 A visual comparison of MSC-LSAM with other algorithms on the CAUS dataset;

[0048] Figure 8 Visualization of ablation experiments on a two-cavity section of the CAMUS dataset;

[0049] Figure 9 Visualization of ablation experiments on a four-cavity section of the CAMUS dataset;

[0050] Figure 10 Visualization of ablation experiments on the CAUS dataset. Detailed Implementation

[0051] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0052] Step 1: Obtain the ultrasound medical image dataset.

[0053] The medical image dataset mentioned includes, but is not limited to:

[0054] The CAMUS dataset contains 500 2D ultrasound sequences each in two-chamber and four-chamber views, along with manually annotated labels for each ultrasound sequence at the end-diastolic and end-systolic frames. 900 sequences were used for training, and 100 were used for testing. The manually annotated cardiac structures include the left ventricular endocardium, myocardium, and left atrial endocardium.

[0055] CAUS is a private dataset from the Affiliated Hospital of Nantong University. The dataset contains 2439 2D carotid ultrasound images, of which 1952 were used for training and 487 for testing. The dataset was manually annotated by five ultrasound physicians with over five years of clinical experience, and the annotated carotid structures include the internal and external carotid arteries.

[0056] Step 2: The maximum number of channels in the UNet network was reduced from 1024 to 256, which reduced the computational load and the risk of overfitting.

[0057] Step 3: See Figure 2 A simple and efficient adapter block is introduced between each Transformer layer. The adapter is designed as a bottleneck model, consisting of a downward-projected MLP layer, a GELU activation function, and an upward-projected MLP layer. Using F... i This represents the output of each Transformer layer, MLP down It is responsible for projecting these outputs downwards, compressing the input data into a lower dimension.

[0058] P i =MLP up (GELU(MLP down (F i ))), #(1)

[0059] In step 3, the introduced GELU function serves as the activation function, adding non-linearity and helping the model learn more complex feature representations. Subsequently, the layer restores the data from low dimension to its original shape, achieving the upprojection of the adapter.

[0060] Step 4: Embed multi-scale features into axial attention, see [link to relevant documentation] Figure 3 The aim is to better segment regions with different individual sizes and shapes. The multi-scale cross-attention mechanism consists of two parallel branches that compute horizontal and vertical axial attention respectively. Each branch first undergoes three 1D convolutions with different kernel sizes (1×7, 1×11, and 1×21) to encode multi-scale contextual information along one spatial dimension.

[0061] by and This represents a one-dimensional convolution along the x and y axes, Norm(·) represents layer normalization, and F... x and F y The x-axis and y-axis outputs are respectively:

[0062]

[0063] For cross-attention, the input data is transformed into three distinct vector sets: Key(K), Query(Q), and Value(V). However, unlike axial attention, the q matrix of one branch is passed to the other, thus establishing an interaction between the two parallel axes and more effectively utilizing multi-scale features and global information. xK and F xV F is the key-value matrix along the x-axis. yQ It is the query matrix along the y-axis, and similarly F yV and F yK F is the key-value matrix along the y-axis. xQ This is the query matrix along the x-axis. MHCA y (·,·,·) represents multi-head cross attention along the x-axis, MHCA x (·,·,·) represents a multi-headed intersection along the y-axis. Note that:

[0064] F T =MHCA y (F yQ F xV F xK ), #(4)

[0065] F B =MHCA x (F xQ F yV F yK ), #(5)

[0066] The output of multi-scale cross-axis attention can be expressed as:

[0067] F out =Conv 1×1(F T )+Conv 1×1 (F B )+F.#(6)

[0068] Step 5: Obtain the global spatial information of each channel of the input feature map through global average pooling, generating a vector representing the global information. Then, use a one-dimensional convolutional kernel to realize the dynamic correlation between channels by adjusting the size of the convolutional kernel. See also Figure 4 The ECA module is introduced to adapt to feature representations at different scales, effectively integrating information from different modalities and scales, and reducing the channel weights of useless features.

[0069] To evaluate the segmentation performance of the proposed multi-scale cross-segmentation ultrasound medical image method on ultrasound medical images, the Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) were used to assess the model's segmentation accuracy. Furthermore, accuracy (ACC), precision (PRE), sensitivity (SE), and F1 score were used to evaluate the model's ability to handle outliers.

[0070] Comparative experiments showed that MSC-LSAM achieved high levels of performance in ultrasound medical image segmentation tasks, including DSC, HD95, ACC, PRE, SE, and F1-Score (see Tables 1, 2, and 3), demonstrating excellent segmentation performance and ability to handle anomalies.

[0071] This invention conducted comprehensive ablation experiments on the algorithm network, using LSAM, ECA, and MCA as variables. The improvement of each module on neural network performance and segmentation accuracy was analyzed by gradually adding modules (see Tables 4, 5, and 6). In MSC-LSAM, the introduction of a learnable segmentation model significantly enhanced the feature extraction capability of the encoder, allowing the model to focus on key features and improving its understanding and capture of complex data patterns. However, due to a scale mismatch between the features generated by the learnable segmentation model and the features output by the UNet encoder, difficulties arose during feature fusion. This mismatch led to inefficient feature fusion, resulting in an increased missegmentation rate.

[0072] Table 1: Comparative experimental results on the two-cavity section of the CAMUS dataset.

[0073]

[0074] R50 is Resnet50, and R101 is Resnet101.

[0075] Table 2: Comparative experimental results on the four-cavity section of the CAMUS dataset.

[0076]

[0077] R50 is Resnet50, and R101 is Resnet101.

[0078] Table 3: Comparative Experiment Results on the CAUS Dataset

[0079]

[0080]

[0081] R50 is Resnet50, and R101 is Resnet101.

[0082] Table 4: Ablation experimental results on the two-cavity section of the CAMUS dataset.

[0083]

[0084] Label1 represents the endocardium of the left ventricle, Label2 represents the myocardium, and Label3 represents the endocardium of the left atrium.

[0085] Table 5: Ablation experimental results on the four-cavity section of the CAMUS dataset.

[0086]

[0087]

[0088] Label1 represents the endocardium of the left ventricle, Label2 represents the myocardium, and Label3 represents the endocardium of the left atrium.

[0089] Table 6: Ablation experimental results on the CAUS dataset

[0090]

[0091] Label1 represents the internal carotid artery, and Label2 represents the external carotid artery.

Claims

1. A multi-scale cross-sonography medical image segmentation method based on a parallel coding structure, characterized in that, Includes the following steps: Step 1: Obtain ultrasound medical images from the publicly available 2D cardiac ultrasound dataset CAMUS and the self-built 3D carotid ultrasound dataset CAUS; Step 2: Encode the ultrasound medical images using a parallel visual encoder for segmenting all models and a UNet encoder; Step 3: Introduce an adapter into the frozen segmentation model visual encoder to form a learnable segmentation model while maintaining low computational cost. Step 4: Introduce a multi-scale cross-axis attention module into the UNet global network to achieve cross-fusion of multi-scale features; Step 5: Achieve multi-scale feature fusion between dual encoders through the channel attention module; Step 6: Evaluate the segmentation accuracy of the model using the Dice similarity coefficient and the 95% Hausdorff distance, and set indicators to evaluate the model's ability to handle outliers. Step 7: Input the image data into the segmentation network to perform ultrasound image segmentation, complete the image segmentation, and output the model segmentation prediction results.

2. A multi-scale cross-modality ultrasound medical image segmentation method based on parallel coding structure according to claim 1, characterized in that, The visual encoder that segments all models remains frozen during the encoding process, reducing computational complexity.

3. The multi-scale cross-mode ultrasound medical image segmentation method based on a parallel coding structure according to claim 1, characterized in that, The maximum number of channels in the UNet encoder was reduced from 1024 to 256, which reduced the computational load and the risk of overfitting.

4. The multi-scale cross-mode ultrasound medical image segmentation method based on a parallel coding structure according to claim 1, characterized in that, In step 3, the adapter is a bottleneck model, including a downward-projected MLP layer, a GELU activation function, and an upward-projected MLP layer, as shown in the following formula: P i = MLP up (GELU(MLP down (F i ))).#(1) where F i represents the output of each Transformer layer, MLP down is responsible for projecting these outputs downward, compressing the input data to a lower dimension.

5. The multi-scale cross-mode ultrasound medical image segmentation method based on a parallel coding structure according to claim 1, characterized in that, In step 4, the multi-scale cross-axis attention module is divided into two parallel branches, which compute horizontal and vertical axial attention respectively. Each branch first undergoes 1D convolutions with kernel sizes of 1×7, 1×11, and 1×21 to encode multi-scale contextual information along one spatial dimension. and This represents a one-dimensional convolution along the x and y axes, Norm(·) represents layer normalization, and F... x and F y Outputs are provided for the x-axis and y-axis respectively; the formula can be written as: Cross-attention, input data is converted into three different vector sets: K represents Key, Q represents Query, and V represents Value; but unlike the axial attention, the Q matrix of one branch is passed to the other branch, thereby establishing interaction between the two parallel axes, utilizing multi-scale features and global information; F xK and F xV is the key and value matrix of the x-axis, F yQ is the query matrix of the y-axis, and F yV and F yK is the key and value matrix of the y-axis, F xQ is the query matrix of the x-axis, MHCA y (·,·,·) represents multi-head cross-attention along the x-axis, MHCA x (·,·,·) represents multi-head cross-attention along the y-axis, and F T = MHCA y (F yQ ,F xV ,F xK ), (4) F B = MHCA x (F xQ ,F yV ,F yK ), (5) The output of multi-scale cross-axis attention is represented as: F out = Conv 1×1 (F T )+ Conv 1×1 (F B )+ F. (6) 6. The multi-scale cross-mode ultrasound medical image segmentation method based on a parallel coding structure according to claim 1, characterized in that, In step 5, the channel attention module obtains the global spatial information of each channel of the input feature map through global average pooling operation and generates a vector representing the global information. Then, a one-dimensional convolution kernel is used to realize the dynamic correlation between channels. By adjusting the size of the convolution kernel, the efficient fusion of multi-scale features of the dual encoder is achieved, reducing model missegmentation.

7. The multi-scale cross-mode ultrasound medical image segmentation method based on a parallel coding structure according to claim 1, characterized in that, In step 6, the segmentation accuracy of the model is evaluated using the Dice similarity coefficient and the 95% Hausdorff distance. Specifically, the Dice similarity coefficient (DSC) measures the model's performance by calculating the overlap between the segmentation results and the segmentation labels. Let T and P represent the true and predicted values ​​of all voxels, respectively. The 95% Hausdorff distance HD95 measures the performance of the model by computing the maximum distance between the segmentation result and the 95% of the pixels of the segmentation label, with d T'P' denotes the maximum Hausdorff distance between the set of real voxels T' and the set of predicted voxels P', with d P'T' denotes the maximum Hausdorff distance from the set of predicted voxels P' to the set of real voxels T', with d d H (T',P') = max(d T'P0' ,d P'T' ),#(8).

8. The multi-scale cross-mode ultrasound medical image segmentation method based on a parallel coding structure according to claim 1, characterized in that, In step 6, the set metrics for evaluating the model's outlier handling capability specifically include: using accuracy (ACC), precision (PRE), sensitivity (SE), and F1 score (F1-Score) to evaluate the model's outlier handling capability. Here, TP represents the number of pixels correctly segmented as targets, FP represents the number of background pixels incorrectly segmented as targets, TN represents the number of pixels correctly segmented as background, and FN represents the number of target pixels incorrectly segmented as background.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

Images