Carotid artery plaque segmentation method based on dual-branch multi-scale cross-fusion network

Abstract

The present invention discloses a carotid plaque segmentation method based on a dual-branch multi-scale cross-fusion network, which mainly solves the problem that the existing technology has poor segmentation effect on carotid plaques in ultrasound images. Its implementation scheme is: the data set of the acquired carotid ultrasound image is divided into a training set, a validation set and a test set; an integrated segmentation network composed of an encoder, a dual-branch decoder, a cross-attention feature fusion module, and a multi-scale feature fusion module is constructed, and its loss function is defined; the integrated network is trained by the segmentation data of the training set; the segmentation data of the test set is input into the trained integrated network to obtain the predicted segmentation result corresponding to the test set data. The present invention improves the segmentation effect of carotid plaques, can accurately locate and extract effective features for plaques of different positions, shapes and sizes, and obtains a good segmentation effect, which can be used to help doctors complete the automatic delineation of carotid plaques before the treatment of cardiovascular and cerebrovascular diseases.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a carotid artery plaque segmentation method, which can be used to help doctors automatically delineate carotid artery plaques before treating cardiovascular and cerebrovascular diseases. Background Art

[0002] In recent years, cardiovascular and cerebrovascular diseases have extremely high morbidity and mortality, and it is very necessary to propose an effective evaluation method to detect potential cardiovascular and cerebrovascular diseases. Carotid atherosclerosis is the main cause of cardiovascular disease, which is generally formed in the blood vessel wall. Plaques composed of cholesterol, lipoproteins and other cellular molecules. Therefore, accurate monitoring and evaluation of carotid plaques are of great clinical significance for the prevention of cardiovascular events. The size and shape of different plaques also vary greatly. Even for the same plaque, the size and shape may vary greatly due to different scanning positions. Therefore, it is crucial to achieve automatic segmentation of carotid plaques in clinical applications.

[0003] At present, it is a very time-consuming task for clinicians to quantify lesions. Not only is the workload large, but the doctor's diagnosis results are also very subjective and difficult to reproduce. It also requires certain clinical knowledge and experience of doctors. The segmentation results between different doctors are also very different. Although the doctor's diagnosis is very accurate, it may also be affected by environmental factors such as time, energy, and emotional changes, which is likely to affect the accuracy of the diagnosis. In recent years, the emergence of deep learning methods has not only effectively solved the complex problems faced in the process of medical image processing and analysis, but also greatly reduced the workload of clinicians. It can be seen that establishing a reliable computer-aided diagnosis method for automatic segmentation of carotid plaques can be of great help to clinical diagnosis, surgical implementation, and treatment plan design.

[0004] The U-Net network proposed by Ronneberger et al. in the article "U-Net: Convolutional Networks for Biomedical Image Segmentation" is widely used in the segmentation of carotid plaques in ultrasound images, and they have been adopted in the segmentation of carotid plaques in different types of ultrasound, MRI and CT images. It adopts an encoder-decoder structure and splices shallow features with deep features at the horizontal skip connection structure before feature fusion. However, when the shape and size of the segmentation target vary greatly, the segmentation performance of the network will decrease.

[0005] Zhou et al. proposed a UNet++ ensemble algorithm in the article "Deep learning-based measurement of total plaque area in B-mode ultrasound images" to segment plaques from 2D carotid ultrasound images. UNet++ replaces the skip connection structure in U-Net with a more compact and dense nested connection structure, making feature splicing more comprehensive. This method reduces the size of ultrasound images by manually cropping the region of interest around each plaque to achieve a more accurate segmentation effect. However, this method relies heavily on manually labeled regions of interest, so the segmentation effect of the entire image is poor.

[0006] With the widespread application of Transformer in medical image segmentation, Chen et al. proposed the TransUNet network with mixed encoding of CNN and Transformer in the article "TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation". This network adds a multi-layer Transformer structure after the CNN of the encoder to process global features. It can effectively process images of different sizes and shapes and can perform segmentation at different resolutions. However, due to the strong artifacts in ultrasound images and the blurred boundaries between carotid plaques and blood vessels, the model may suffer from under-segmentation problems.

[0007] Although all of the above existing methods can achieve automatic segmentation of carotid plaques, they treat the segmentation of carotid plaques and blood vessels as two independent tasks, lack the exploration of the implicit relationship and complementary information between plaques and blood vessels, and do not use blood vessel features to constrain and locate plaque features. In addition, since the shape and size of plaques vary greatly among different patients, these existing segmentation methods are not well suited for the segmentation of plaques of different scanning positions, shapes, and sizes. Summary of the invention

[0008] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a carotid plaque segmentation method based on a dual-branch multi-scale cross-fusion network, so as to make full use of the information complementarity between plaques and blood vessels, better segment plaques of different positions, sizes and shapes, and improve the versatility of the segmentation model.

[0009] The technical idea of ​​the present invention is: by constructing and training a segmentation network for carotid artery plaques, blood vessels and plaques are segmented at the same time; by using a multi-scale feature fusion method at the end stage of the encoder, plaques of different positions, sizes and shapes can be segmented; by cross-feature fusion of blood vessel and plaque features between dual decoder branches, the plaques can be constrained and positioned in the blood vessel area.

[0010] According to the above ideas, the implementation steps of the present invention include the following:

[0011] (1) obtaining an ultrasound image dataset of carotid artery plaques and blood vessels, annotating the dataset, and performing standardization and downsampling preprocessing on the annotated segmented dataset to obtain a segmented dataset;

[0012] (2) Divide the segmentation data set;

[0013] (2a) The segmented dataset is divided into patient sets, with 70% of the patient images used as training sets, 15% of the patient images used as validation sets, and 15% of the patient images used as test sets, and it is ensured that there are no repeated subsets between the datasets;

[0014] (2b) The images of the segmentation training set after the division are sequentially flipped up and down, flipped left and right, and randomly rotated by -30° to 30° to obtain the segmentation training set after data augmentation;

[0015] (3) Constructing a dual-branch integrated network PDCSNet:

[0016] (3a) establishing a connected basic segmentation network consisting of an encoder and a decoder, wherein the encoder includes five cascaded encoding layers (E1, E2, E3, E4, E5), and the decoder is composed of a plaque branch and a vascular branch connected in parallel, wherein the plaque branch includes five cascaded decoding layers (Dp1, Dp2, Dp3, Dp4, Dp5), and the vascular branch includes five cascaded decoding layers (Dv1, Dv2, Dv3, Dv4, Dv5);

[0017] (3b) Establish a cross-attention feature fusion module CAFF including channel attention CA and dual-branch convolutional layer, add this module between the two branches of the decoder, so that it can cross-transfer features between the vascular branch and the plaque branch, forming the first segmentation sub-network CAFFNet;

[0018] (3c) Establish a multi-scale feature fusion module MSFF consisting of three parallel convolution branches, and cascade the module to the last stage of the encoder to form the second segmentation sub-network MSFFNet;

[0019] (3d) The first segmentation sub-network CAFFNet is connected in parallel with the second segmentation sub-network MSFFNet to form an integrated segmentation network PDCSNet;

[0020] (4) According to the segmentation loss of the patch Plaque CAFF and blood vessel segmentation loss Loss Vessel CAFF Construct the loss of the first segmentation subnetwork CAFFNet CAFF , according to the segmentation loss of the patch Plaque MSFF and blood vessel segmentation loss Loss Vessel MSFF The loss of the second segmentation sub-network MSFFNet MSFF :

[0021] Loss CAFF =αLoss Plaque CAFF +βLoss Vessel CAFF

[0022] Loss MSFF =αLoss Plaque MSFF +βLoss Vessel MSFF

[0023] Where α is the loss ratio coefficient of segmenting plaques, and β is the loss ratio coefficient of segmenting blood vessels;

[0024] (5) Using the mini-batch gradient descent algorithm, the constructed dual-branch integrated network PDCSNet is trained by splitting the training set to obtain the trained optimal network model;

[0025] (6) Input the segmentation data of the test set into the trained two-branch integrated network PDCSNet to obtain the predicted segmentation results corresponding to the test set data.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] 1. The present invention adopts a shared encoder and a dual-branch decoder to simultaneously segment plaques and blood vessels based on the traditional UNet segmentation network, and introduces a cross-attention feature fusion module between the dual decoders. The segmentation of plaques is constrained and located by blood vessel features, which enables the network to better learn the complementary information between plaques and blood vessels, effectively improving the segmentation accuracy of the model.

[0028] 2. The present invention introduces a multi-scale feature fusion module, so it can better extract the features of plaques of different positions, shapes and sizes. When the differences in plaques between different patients vary greatly, effective features can be extracted to obtain accurate results, making the model more universal;

[0029] 3. Since the present invention adopts the model integration method, the final result can make the best use of the strengths of different models and complement each other, integrating the learning capabilities of each model and improving the generalization ability of the final model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the implementation of the present invention;

[0031] Figure 2 It is a structural diagram of the dual-branch integrated network PDCANet constructed in the present invention;

[0032] Figure 3 yes Figure 2 Schematic diagram of the cross-attention feature fusion module CAFF;

[0033] Figure 4 yes Figure 2 Schematic diagram of the multi-scale feature fusion module MSFF;

[0034] Figure 5 yes Figure 3 Schematic diagram of the channel attention module CA in the cross-attention feature fusion module CAFF;

[0035] Figure 6 This is a comparison diagram of the plaque segmentation results of the present invention and six existing segmentation algorithms;

[0036] Figure 7 This is a comparison chart of the results of simultaneous segmentation of plaques and blood vessels by the present invention and other six segmentation algorithms. DETAILED DESCRIPTION

[0037] The examples and effects of the present invention are further described in detail below with reference to the accompanying drawings.

[0038] See also Figure 1 The specific implementation steps of this example include the following:

[0039] Step 1: Obtain ultrasound image datasets of plaques and blood vessels, and preprocess the ultrasound image datasets.

[0040] The ultrasound image dataset in this example comes from the ultrasound department of a hospital, and contains images of carotid artery plaques and blood vessels collected by six ultrasound physicians.

[0041] Preprocess the above ultrasound image dataset:

[0042] (1.1) For each ultrasound image, the plaque area and the vascular area are labeled respectively. Each image corresponds to a pixel-level plaque segmentation annotation and a pixel-level vascular segmentation annotation, forming the original segmentation dataset;

[0043] (1.2) Z-Score standardization is used to standardize the data based on the mean and standard deviation of the original data, so that all image values ​​in the ultrasound image data set obey the standard normal distribution after standardization;

[0044] (1.3) The standardized data in (1.2) are uniformly downsampled to 192×256, the segmentation label map is downsampled using nearest neighbor interpolation, and the ultrasound image is downsampled using linear interpolation to obtain the segmentation data set after preprocessing.

[0045] Step 2: Divide the segmentation dataset.

[0046] (2.1) 70% of the images in the segmented data set are used as the training set, 15% of the images are used as the validation set, and 15% of the images are used as the test set, and it is ensured that there are no repeated subsets between the data sets. The segmented data set in this example includes 223 patients and a total of 884 2D ultrasound images. After the training set is divided into 156 patients and a total of 614 images, the validation set has 33 patients and a total of 124 images, and the test set has 34 patients and a total of 146 images;

[0047] (2.2) The images of the segmentation training set after division are flipped up and down, flipped left and right, and randomly rotated by -30° to 30° to obtain the segmentation training set after data augmentation.

[0048] Step 3. Construct the dual-branch integrated network PDCSNet.

[0049] (3.1) Establish a basic segmentation network:

[0050] Set up five encoding layers E1, E2, E3, E4, and E5, and cascade them to form an encoder;

[0051] Set five patch branch decoding layers Dp1, Dp2, Dp3, Dp4, and Dp5, and cascade them to form a patch branch decoder;

[0052] Set five blood vessel branch decoding layers Dv1, Dv2, Dv3, Dv4, and Dv5, and cascade them to form a blood vessel branch decoder;

[0053] The plaque branch decoder and the vascular branch decoder are connected in parallel to form a dual-branch decoder, and then the encoder and the dual-branch decoder are cascaded to form a basic segmentation network, where:

[0054] Each encoding layer in the encoder includes two cascaded convolution blocks, each convolution block is composed of a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a BN layer, and a ReLU activation function cascade. Except for the last encoding layer, each other encoding layer is cascaded with a maximum pooling operation with a filter size of 2×2 and a step size of 2 to achieve downsampling;

[0055] The plaque branch and the blood vessel branch in the decoder have the same structure. Each decoding layer in each branch includes two consecutive convolution blocks. Each convolution block is composed of a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a BN layer, and a ReLU activation function cascade. Except for the last decoding layer, each other decoding layer is cascaded with an upsampling layer, and bilinear interpolation is used to make the output image size twice the input image size.

[0056] (3.2) Construct the cross-attention feature fusion module CAFF:

[0057] Set the channel attention CA, which consists of a composite pooling layer, a convolution layer with a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and a sigmoid activation layer cascade. The composite pooling layer consists of a maximum pooling and an average pooling in parallel, such as Figure 5 As shown;

[0058] Set up a two-branch convolutional layer, where each branch consists of a cascade of convolutional layers with a convolution kernel size of 1×1, a stride of 1, and padding of 0;

[0059] The channel attention CA is cascaded on the branch where the vascular branch features are passed to the plaque branch decoder, the first convolutional layer branch in the dual-branch convolutional layer is cascaded on the feature map after the plaque branch features and the vascular branch features are spliced ​​and fused, and the second convolutional layer branch is cascaded on the feature map after the vascular branch features and the plaque branch features are spliced ​​and fused, forming a cross-attention feature fusion module CAFF, as shown in Figure 3 As shown;

[0060] (3.3) Construct multi-scale feature fusion module MSFF:

[0061] (3.3.1) Establish a normal convolution kernel branch consisting of three convolution blocks in cascade, where:

[0062] The first convolution block is a convolution block with a kernel size of 1×1, a step size of 1, and a padding of 0×0.

[0063] The second convolution block is a convolution block with a kernel size of 3×3, a stride of 1, and a padding of 1×1.

[0064] The third convolution block is a convolution block with a convolution kernel size of 5×5, a stride of 1, and a padding of 2×2;

[0065] (3.3.2) Establish a vertical convolution kernel branch consisting of three convolution blocks in cascade, where:

[0066] The first convolution block is a convolution block with a kernel size of 1×1, a step size of 1, and a padding of 0×0.

[0067] The second convolution block is a convolution block with a kernel size of 3×1, a stride of 1, and a padding of 1×0.

[0068] The third convolution block is a convolution block with a convolution kernel size of 5×1, a stride of 1, and a padding of 2×0;

[0069] (3.3.3) Establish a horizontal convolution kernel branch consisting of three convolution blocks in cascade, where:

[0070] The first convolution block is a convolution block with a kernel size of 1×1, a step size of 1, and a padding of 0×0.

[0071] The second convolution block is a convolution block with a kernel size of 1×3, a step size of 1, and a padding of 0×1.

[0072] The third convolution block is a convolution block with a kernel size of 1×5, a stride of 1, and a padding of 0×2;

[0073] (3.3.4) The normal convolution kernel branch, the vertical convolution kernel branch and the horizontal convolution kernel branch are connected in parallel, and the feature maps of these three branches are spliced ​​in the channel dimension. After a 1×1 convolution to change the number of channels, a multi-scale feature fusion module MSFF is formed, as shown in Figure 4 As shown;

[0074] (3.4) The cross-attention feature fusion module CAFF is added between the two branches of the decoder to form the first segmentation sub-network CAFFNet. The multi-scale feature fusion module MSFF is cascaded at the last stage of the encoder to form the second segmentation sub-network MSFFNet. The first segmentation sub-network is connected to the second segmentation sub-network in parallel to form the integrated segmentation network PDCSNet, as shown in Figure 2 shown.

[0075] Step 4. Set the loss function of the integrated segmentation network PDCSNet.

[0076] (4.1) The target area to be segmented is set as the foreground category, and the remaining areas are set as the background category. That is, in the plaque label, the plaque area is set as the foreground category, and the other areas are set as the background area; in the blood vessel label, the blood vessel area is set as the foreground category, and the other areas are set as the background category;

[0077] (4.2) Construct the loss of the first segmentation sub-network CAFFNet CAFF :

[0078] (4.2.1) Select the existing binary cross entropy loss function Loss BCE p1 And Dice loss function Loss Dice p1 As the loss of the patch in the first segmentation subnetwork; select the existing binary cross entropy loss function Loss BCE v1 And Dice loss function Loss Dice v1 As the loss for the first segmentation subnetwork for blood vessels:

[0079]

[0080]

[0081]

[0082]

[0083] where y p1 is the actual segmentation result of the first segmentation subnetwork patch, is the predicted segmentation result of the first segmentation subnetwork patch, y v1 is the actual segmentation result of the blood vessels of the first segmentation sub-network, is the predicted segmentation result of the blood vessel of the first segmentation sub-network, ε is the smoothing coefficient to ensure that the denominator of the Dice loss function calculation formula is not 0;

[0084] (4.2.2) According to Loss BCE p1 and Loss Dice p1 Construct the patch segmentation loss Loss of the first segmentation subnetwork Plaque CAFF According to Loss BCE v1 and Loss Dice v1 Construct the blood vessel segmentation loss Loss of the first segmentation subnetwork Vessel CAFF :

[0085] Loss Plaque CAFF =λLoss BCE p1 +γLoss Dice p1

[0086] Loss Vessel CAFF =λLoss BCE v1 +γLoss Dice v1

[0087] Where λ is the proportional coefficient of the binary cross entropy loss function, and γ is the proportional coefficient of the Dice loss function;

[0088] (4.2.3) According to the segmentation loss of the patch Plaque CAFF and blood vessel segmentation loss Loss Vessel CAFF Construct the loss of the first segmentation subnetwork CAFF :

[0089] Loss CAFF =αLoss Plaque CAFF +βLoss Vessel CAFF

[0090] Where α is the loss ratio coefficient of segmenting plaques, and β is the loss ratio coefficient of segmenting blood vessels;

[0091] (4.3) Construct the loss of the second segmentation sub-network MSFFNet MSFF :

[0092] (4.3.1) Select the existing binary cross entropy loss function Loss BCE p2 And Dice loss function Loss Dice p2 As the loss of the patch in the second segmentation sub-network; select the existing binary cross entropy loss function Loss BCE v2 And Dice loss function Loss Dice v2 As the loss for the blood vessels in the second segmentation subnetwork:

[0093]

[0094]

[0095]

[0096]

[0097] where y p2 is the actual segmentation result of the second segmentation subnetwork patch, is the predicted segmentation result of the second segmentation subnetwork patch, y v2 is the actual segmentation result of the blood vessels of the second segmentation sub-network. is the predicted segmentation result of the blood vessel of the second segmentation sub-network, ε is the smoothing coefficient to ensure that the denominator of the Dice loss function calculation formula is not 0;

[0098] (4.3.2) According to Loss BCE p2 and Loss Dice p2 Construct the patch segmentation loss Loss of the second segmentation subnetwork Plaque MSFF According to Loss BCE v2 and Loss Dice v2 Construct the blood vessel segmentation loss Loss of the second segmentation sub-network Vessel MSFF :

[0099] Loss Plaque MSFF =λLoss BCE p2 +γLoss Dice p2

[0100] Loss Vessel MSFF =λLoss BCE v2 +γLoss Dice v2

[0101] Where λ is the proportional coefficient of the binary cross entropy loss function, and γ is the proportional coefficient of the Dice loss function;

[0102] (4.3.3) According to the segmentation loss of the patch Plaque MSFF and blood vessel segmentation loss Loss Vessel MSFF Construct the loss of the second segmentation sub-network MSFF :

[0103] Loss MSFF =αLoss PlaqueMSFF +βLoss Vessel MSFF

[0104] Where α is the loss ratio coefficient of segmenting plaques, and β is the loss ratio coefficient of segmenting blood vessels;

[0105] (4.4) The loss of the first segmentation sub-network is Loss CAFF and the loss of the second segmentation sub-network MSFF Add together and get the total loss of the dual-branch integrated network PDCSNet:

[0106] Loss All =aLoss CAFF +bLoss MSFF

[0107] Where a is the loss coefficient of the first segmentation sub-network, and b is the loss coefficient of the second segmentation sub-network.

[0108] Step 5. Use the back-propagation method to iteratively train the constructed dual-branch integrated network PDCSNet.

[0109] (5.1) Set the total number of iterations to 150, the batch size to 8, the initial learning rate to 0.0001, and use the Adam optimizer to update the parameters in the two-branch integrated network PDCSNet;

[0110] (5.2) Input the ultrasound image of the training set into the dual-branch integrated network PDCSNet to obtain 4 predicted segmentation results, among which the first sub-segmentation network CAFFNet in the integrated network returns a predicted segmentation result of a plaque And the predicted segmentation result of a blood vessel The second sub-segmentation network MSFFNet returns a predicted segmentation result of a patch And the predicted segmentation result of a blood vessel

[0111] (5.3) Calculate the loss of the first sub-segmentation network Loss CAFF And the loss of the second sub-segmentation network Loss MSFF , the total loss of the dual-branch integrated network PDCSNet is calculated by the total loss function;

[0112] (5.4) Back-propagate the loss value obtained in (5.3), update the network parameters, and obtain the model after preliminary training;

[0113] (5.5) The process (5.2)-(5.4) is repeated for the model after preliminary training until the loss value of the validation set does not decrease after 10 rounds or the total number of iterations reaches 150. The training is stopped to obtain the optimal model of the trained dual-branch integrated network PDCSNet.

[0114] Step 6. Use the segmentation data of the test set to test the trained two-branch integrated network PDCSNet.

[0115] (6.1) Take out ultrasound images from the segmentation data of the test set in sequence and input them into the trained two-branch integrated network PDCSNet. Each image gets the plaque prediction result. and vascular prediction results These two prediction results;

[0116] (6.2) Repeat step (6.1) until all images in the test set are input, and the segmentation results of the carotid plaque and blood vessels are obtained, and the test is terminated.

[0117] The numbers of the above steps are for more clearly describing the embodiments of the present invention, and the order of the numbers is not limited.

[0118] The effect of the present invention can be further illustrated by the following simulation.

[0119] 1. Simulation conditions

[0120] The simulation experiment platform of the present invention is a Linux operating system, configured as follows Core TM i7-12700KF CPU and NVIDIARTX3090 GPU, 64GB memory, using PyTorch deep learning framework, CUDA version is 12.0, and the development language is Python.

[0121] The segmentation data set of this example includes 223 patients and a total of 884 2D ultrasound images. 15% of the images in the segmentation data set are used as an ultrasound image test set, that is, the test set includes 34 patients and a total of 146 images.

[0122] 2. Simulation content and result analysis

[0123] Simulation 1: Under the above simulation experimental conditions, the present invention and the existing six segmentation methods UNet, UNet++, ResUNet, Attention UNet, TransUNet, and SwinUNet are used to segment the plaques of the above ultrasound image test set. The results are as follows: Figure 6 As shown, where:

[0124] Figure 6 (a) is the original ultrasound image.

[0125] 6(b) is the true label of patch segmentation.

[0126] 6(c) is the patch segmentation result diagram of the existing UNet segmentation method.

[0127] 6(d) is the patch segmentation result diagram of the existing UNet++ segmentation method.

[0128] 6(e) is the patch segmentation result diagram of the existing ResUNet segmentation method.

[0129] 6(f) is the patch segmentation result diagram of the existing Attention UNet segmentation method.

[0130] 6(g) is the patch segmentation result diagram of the existing TransUNet segmentation method.

[0131] 6(h) is the patch segmentation result diagram of the existing SwinUNet segmentation method.

[0132] 6(i) is a diagram showing the plaque segmentation result of the present invention.

[0133] Figure 6 The solid-line frame in the figure represents the patch area of ​​the original image. For patches where the results are difficult to observe, the patch prediction area is enlarged in the lower right corner of the result image to make it easier to observe the prediction results of different models.

[0134] like Figure 6 As shown, compared with the existing six methods, the patch segmentation result of the present invention is closest to the real annotation, and avoids the problems of under-segmentation and over-segmentation of other methods, and achieves better segmentation effect for patches of different shapes, sizes and positions.

[0135] Simulation 2: Under the above simulation experimental conditions, the present invention and the existing six segmentation methods UNet, UNet++, ResUNet, Attention UNet, TransUNet, and SwinUNet are used to perform simultaneous segmentation of plaques and blood vessels on the above ultrasound image test set. The results are shown in the figure. Figure 7 As shown, where:

[0136] Figure 7 (a) is the original ultrasound image.

[0137] 7(b) is the true label of plaque and blood vessel segmentation.

[0138] 7(c) shows the segmentation results of plaques and blood vessels using the existing UNet segmentation method.

[0139] 7(d) shows the segmentation results of plaques and blood vessels using the existing UNet++ segmentation method.

[0140] 7(e) is the segmentation result of plaque and blood vessels by the existing ResUNet segmentation method.

[0141] 7(f) shows the segmentation results of plaques and blood vessels using the existing Attention UNet segmentation method.

[0142] 7(g) is the segmentation result of plaque and blood vessels by the existing TransUNet segmentation method.

[0143] 7(h) is the segmentation result of plaque and blood vessels by the existing SwinUNet segmentation method.

[0144] 7(i) is a diagram showing the segmentation result of plaque and blood vessels according to the present invention.

[0145] like Figure 7 As shown, compared with the existing six methods, the distribution of plaques segmented by the present invention in blood vessels is closest to the distribution of true labels.

[0146] Evaluation indicators of segmentation results

[0147] Using Dice coefficient, intersection over union (IoU), Hausdorff distance (HD), sensitivity (SEN), and precision (PRE) as evaluation indicators, the indicators of the segmentation results of the present invention and the existing six methods are statistically analyzed. The indicators of the plaque segmentation results are compared in Table 1, and the indicators of the blood vessel segmentation results are compared in Table 2:

[0148] Table 1 Segmentation indexes of carotid artery plaques in ultrasound images by the present invention and six existing methods

[0149]

[0150] As can be seen from Table 1, the Dice index of the plaque in the present invention can reach 80.26%, the intersection over union ratio IoU can reach 70.28%, the Hausdorff distance HD is 20.11, the sensitivity SEN is 83.28%, and the precision PRE is 83.99%. Compared with the best method among the existing 6 methods, the Dice index of the plaque in the present invention is improved by 2.21%, the IoU index is improved by 2.99%, the Hausdorff distance HD is reduced by 1.04, and the sensitivity SEN is improved by 2.76%.

[0151] Table 2 Segmentation indexes of carotid artery in ultrasound images by the present invention and six existing methods

[0152]

[0153] As can be seen from Table 2, the Dice index of the blood vessels in the present invention can reach 93.77%, the IoU can reach 88.96%, the Hausdorff distance HD is 7.23, the sensitivity SEN is 96.15%, and the accuracy PRE is 92.00%. Compared with the best method among the existing 6 methods, the Dice index of the blood vessels in the present invention is improved by 1.02%, the IoU index is improved by 1.09%, and the sensitivity SEN is improved by 1.58%.

[0154] In summary, the carotid plaque segmentation method based on a dual-branch multi-scale cross-fusion network in the present invention can obtain better segmentation results than the existing UNet, UNet++, ResUNet, Attention UNet, TransUNet and SwinUNet segmentation methods. The present invention introduces two feature fusion modules to guide and constrain the segmentation of plaques through vascular features, so that the network can better utilize the complementary information between learning plaques and blood vessels, extract the features of plaques in different positions, shapes and sizes, and effectively improve the segmentation performance. Even for plaques with large differences between different patients, effective features can be extracted to obtain accurate results, so that the PDCSNet network has better versatility and generalization ability.

Claims

1. A carotid plaque segmentation method based on a dual-branch multi-scale cross-fusion network, characterized in that: include: (1) obtaining an ultrasound image dataset of carotid artery plaques and blood vessels, annotating the dataset, and performing standardization and downsampling preprocessing on the annotated segmented dataset to obtain a segmented dataset; (2) Divide the segmentation data set; (2a) The segmented dataset is divided into patient sets, with 70% of the patient images used as training sets, 15% of the patient images used as validation sets, and 15% of the patient images used as test sets, and it is ensured that there are no repeated subsets between the datasets; (2b) The images of the segmentation training set after the division are sequentially flipped up and down, flipped left and right, and randomly rotated by -30° to 30° to obtain the segmentation training set after data augmentation; (3) Constructing a dual-branch integrated network PDCSNet: (3a) establishing a basic segmentation network connected by an encoder and a decoder, wherein the encoder includes five cascaded encoding layers (E1, E2, E3, E4, E5), and the decoder is composed of a plaque branch and a vascular branch connected in parallel, wherein the plaque branch includes five cascaded decoding layers (Dp1, Dp2, Dp3, Dp4, Dp5), and the vascular branch includes five cascaded decoding layers (Dv1, Dv2, Dv3, Dv4, Dv5); (3b) Establish a cross-attention feature fusion module CAFF including channel attention CA and dual-branch convolutional layer, add this module between the two branches of the decoder, so that it can cross-transfer features between the vascular branch and the plaque branch, forming the first segmentation sub-network CAFFNet; (3c) Establish a multi-scale feature fusion module MSFF consisting of three parallel convolution branches, and cascade the module to the last stage of the encoder to form the second segmentation sub-network MSFFNet; (3d) The first segmentation sub-network CAFFNet is connected in parallel with the second segmentation sub-network MSFFNet to form a dual-branch integrated segmentation network PDCSNet; (4) According to the segmentation loss of the patch Plaque CAFF and blood vessel segmentation loss Loss Vessel CAFF Construct the loss of the first segmentation subnetwork CAFFNet CAFF , according to the segmentation loss of the patch Plaque MSFF and blood vessel segmentation loss Loss Vessel MSFF The loss of the second segmentation sub-network MSFFNet MSFF : Loss CAFF =αLoss Plaque CAFF +βLoss Vessel CAFF Loss MSFF =αLoss Plaque MSFF +βLoss Vessel MSFF Where α is the loss ratio coefficient of segmenting plaques, and β is the loss ratio coefficient of segmenting blood vessels; (5) Using a small batch gradient descent algorithm, the constructed dual-branch integrated network PDCSNet is trained by splitting the training set to obtain the trained optimal network model; (6) Input the segmentation data of the test set into the trained two-branch integrated network PDCSNet to obtain the predicted segmentation results corresponding to the test set data.

2. The method according to claim 1, characterized in that: The segmented data set in step (1) is preprocessed by labeling, standardization and downsampling as follows: (2a) For an ultrasound image, the plaque area and the blood vessel area are marked respectively. Each image corresponds to two labels, plaque and blood vessel, forming the original segmentation data set; (2b) Z-Score standardization is used to standardize the data based on the mean and standard deviation of the original data, so that all image values ​​in the ultrasound image data set obey the standard normal distribution after standardization; (2c) The standardized data in (2a) are uniformly downsampled to 192×256, the segmentation label map is interpolated by the nearest neighbor, and the ultrasound image is interpolated by linear interpolation to obtain the segmentation data set after preprocessing.

3. The method according to claim 1, characterized in that The encoder and decoder in the basic segmentation network in step (3a) have the following structures and parameters: Each encoding layer in the encoder includes two consecutive convolution blocks, each convolution block is composed of a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a BN layer, and a ReLU activation function cascade. Except for the last encoding layer, each encoding layer is cascaded with a maximum pooling operation with a filter size of 2×2 and a step size of 2 to achieve downsampling; The plaque branch and the blood vessel branch in the decoder have the same structure. Each decoding layer in each branch contains two consecutive convolution blocks. Each convolution block is composed of a cascade of a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1, a BN layer, and a ReLU activation function. Except for the last decoding layer, each other decoding layer is cascaded with an upsampling layer, and bilinear interpolation is used to make the output image size twice the size of the input image.

4. The method according to claim 1, characterized in that: In the step (3b), a cross attention feature fusion module CAFF including a channel attention CA and a dual-branch convolutional layer is constructed, and its structure and parameters are as follows: The channel attention CA cascade is on the branch where the vascular branch feature is transmitted to the plaque branch, and is composed of a composite pooling layer, a convolution layer with a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and a sigmoid activation layer cascade, wherein the composite pooling layer is composed of a maximum pooling and an average pooling in parallel; The dual-branch convolutional layer, each branch of which is composed of a cascade of convolutional layers with a convolution kernel size of 1×1, a step size of 1, and a padding of 0, and the first convolutional layer branch is cascaded on the feature map after the plaque branch feature and the blood vessel branch feature are spliced ​​and fused, and the second convolutional layer branch is cascaded on the feature map after the blood vessel branch feature and the plaque branch feature are spliced ​​and fused.

5. The method according to claim 1, characterized in that The three branches of the multi-scale feature fusion module MSFF in step (3c) have the following structures and parameters: The three branches include a normal convolution kernel branch, a vertical convolution kernel branch and a horizontal convolution kernel branch; The normal convolution kernel branch includes three cascaded convolution blocks, namely, a convolution block with a convolution kernel size of 1×1, a step size of 1, and a padding of 0×0, a convolution block with a convolution kernel size of 3×3, a step size of 1, and a padding of 1×1, and a convolution block with a convolution kernel size of 5×5, a step size of 1, and a padding of 2×2. The vertical convolution kernel branch includes three cascaded convolution blocks, namely, a convolution block with a convolution kernel size of 1×1, a stride of 1, and a padding of 0×0, a convolution block with a convolution kernel size of 3×1, a stride of 1, and a padding of 1×0, and a convolution block with a convolution kernel size of 5×1, a stride of 1, and a padding of 2×0. The horizontal convolution kernel branch includes three cascaded convolution blocks, namely, a convolution block with a convolution kernel size of 1×1, a stride of 1, and a padding of 0×0, a convolution block with a convolution kernel size of 1×3, a stride of 1, and a padding of 0×1, and a convolution block with a convolution kernel size of 1×5, a stride of 1, and a padding of 0×2. The normal convolution kernel branch, the vertical convolution kernel branch and the horizontal convolution kernel branch are connected in parallel to form a multi-scale feature fusion module MSFF. The feature maps of these three branches are spliced ​​in the channel dimension, and the number of channels is changed through a 1×1 convolution to obtain the final output result map.

6. The method according to claim 1, characterized in that In step (4), the segmentation loss Loss of the patch is Plaque CAFF and blood vessel segmentation loss Loss Vessel CAFF Construct the loss of the first segmentation subnetwork CAFFNet CAFF , according to the segmentation loss of the patch Plaque MSFF and blood vessel segmentation loss Loss Vessel MSFF The loss of the second segmentation sub-network MSFFNet MSFF , the implementation steps include the following: (4a) The target area to be segmented is set as the foreground category, and the remaining areas are set as the background category. That is, in the plaque label, the plaque area is set as the foreground category, and the other areas are set as the background area; in the blood vessel label, the blood vessel area is set as the foreground category, and the other areas are set as the background category; (4b) Construct the loss of the first segmentation sub-network CAFFNet CAFF : (4b1) Setting the binary cross entropy loss function Loss BCE p1 And Dice loss function Loss Dice p1 As the loss of the patch in the first segmentation subnetwork, set the binary cross entropy loss function Loss BCE v1 And Dice loss function Loss Dice v1 As the loss for the first segmentation subnetwork for blood vessels: where y p1 is the actual segmentation result of the first segmentation subnetwork patch, is the predicted segmentation result of the first segmentation subnetwork patch, y v1 is the actual segmentation result of the blood vessels of the first segmentation sub-network, is the predicted segmentation result of the blood vessel of the first segmentation sub-network, ε is the smoothing coefficient to ensure that the denominator of the Dice loss function calculation formula is not 0; (4b2) According to Loss BCE p1 and Loss Dice p1 Construct the patch segmentation loss Loss of the first segmentation subnetwork Plaque CAFF , according to Loss BCE v1 and Loss Dice v1 Construct the blood vessel segmentation loss Loss of the first segmentation subnetwork Vessel CAFF : Loss Plaque CAFF =λLoss BCE p1 +γLoss Dice p1 Loss Vessel CAFF =λLoss BCE v1 +γLoss Dice v1 Where λ is the proportional coefficient of the binary cross entropy loss function, and γ is the proportional coefficient of the Dice loss function; (4b3) According to the segmentation loss of the patch Plaque CAFF and blood vessel segmentation loss Loss Vessel CAFF Construct the loss of the first segmentation subnetwork CAFF : Loss CAFF =αLoss Plaque CAFF +βLoss Vessel CAFF Where α is the loss ratio coefficient of segmenting plaques, and β is the loss ratio coefficient of segmenting blood vessels; (4c) Construct the loss of the second segmentation sub-network MSFFNet MSFF : (4c1) Setting the binary cross entropy loss function Loss BCE p2 And Dice loss function Loss Dice p2 As the loss of the patch in the second segmentation subnetwork, set the binary cross entropy loss function Loss BCE v2 And Dice loss function Loss Dice v2 As the loss for the blood vessels in the second segmentation subnetwork: where y p2 is the actual segmentation result of the second segmentation subnetwork patch, is the predicted segmentation result of the second segmentation subnetwork patch, y v2 is the actual segmentation result of the blood vessels of the second segmentation sub-network. is the predicted segmentation result of the blood vessel of the second segmentation sub-network, ε is the smoothing coefficient to ensure that the denominator of the Dice loss function calculation formula is not 0; (4c2) According to Loss BCE p2 and Loss Dice p2 Construct the patch segmentation loss Loss of the second segmentation subnetwork Plaque MSFF , according to Loss BCE v2 and Loss Dice v2 Construct the blood vessel segmentation loss Loss of the second segmentation sub-network Vessel MSFF : Loss Plaque MSFF =λLoss BCE p2 +γLoss Dice p2 Loss Vessel MSFF =λLoss BCE v2 +γLoss Dice v2 Where λ is the proportional coefficient of the binary cross entropy loss function, and γ is the proportional coefficient of the Dice loss function; (4c3) According to the segmentation loss of the patch Plaque MSFF and blood vessel segmentation loss Loss Vessel MSFF Construct the loss of the second segmentation sub-network MSFF : Loss MSFF =αLoss Plaque MSFF +βLoss Vessel MSFF Where α is the loss ratio coefficient of segmenting plaques, and β is the loss ratio coefficient of segmenting blood vessels; (4d) The loss of the first segmentation sub-network is Loss CAFF and the loss of the second segmentation sub-network MSFF Add together and get the total loss of the dual-branch integrated network PDCSNet: Loss All =aLoss CAFF +bLoss MSFF Where a is the loss coefficient of the first segmentation sub-network, and b is the loss coefficient of the second segmentation sub-network.

7. The method according to claim 1, characterized in that In step (5), based on the segmented data of the training set, the constructed dual-branch integrated network PDCSNet is trained using the back propagation method, and the implementation steps include the following: (5a) Set the total number of iterations to 150, the batch size to 8, the initial learning rate to 0.0001, and use the Adam optimizer to update the parameters in the two-branch integrated network PDCSNet; (5b) Input the ultrasound image of the training set into the dual-branch integrated network PDCSNet to obtain four predicted segmentation results, among which the first sub-segmentation network CAFFNet in the integrated network returns a predicted segmentation result of a plaque And the predicted segmentation result of a blood vessel The second sub-segmentation network MSFFNet returns a predicted segmentation result of a patch And the predicted segmentation result of a blood vessel (5c) Calculate the loss of the first sub-segmentation network CAFFNet CAFF And the loss of the second sub-segmentation network MSFFNet MSFF , the total loss of the dual-branch integrated network PDCSNet is calculated by the total loss function; (5d) Back-propagate the loss value obtained in (5c), update the network parameters, and obtain the model after preliminary training; (5e) The process (5b)-(5d) is repeated for the model after preliminary training until the loss value of the validation set does not decrease after 10 rounds or the total number of iterations reaches 150. Then the training is stopped to obtain the optimal model of the trained dual-branch integrated network PDCSNet.

8. The method according to claim 1, characterized in that: Step (6) uses the segmentation data of the test set to test the trained two-branch integrated network PDCSNet. The implementation steps include the following: (6a) Take the ultrasound images from the segmentation data of the test set in sequence and input them into the trained two-branch integrated network PDCSNet. The plaque prediction results are obtained for each image. and vascular prediction results These two prediction results; (6b) Repeat step (6a) until all images in the test set are input, and the segmentation results of carotid artery plaques and blood vessels are obtained.

Images