Intravascular lipid plaque automatic segmentation method based on deep learning

Abstract

The invention discloses an intravascular lipid plaque automatic segmentation method based on deep learning, and the method employs an improved U-Net + + network architecture to achieve the high-precision segmentation of the internal and external membranes of a blood vessel, employs a K-means clustering algorithm based on gray values, edge features and texture features, and achieves the segmentation of the internal and external membranes of the blood vessel through adding a memory mechanism. Stability and consistency in the clustering process are ensured, and accurate classification of plaque types is realized. According to the method, the improved U-Net + + network is used for segmenting the inner and outer membranes of the ultrasonic and photoacoustic images, so that different structures of the blood vessel wall can be accurately distinguished, accurate identification of the lipid plaque and high-precision segmentation and classification of the lipid plaque are realized, and the method is suitable for large-scale popularization and application. By means of the K-means clustering algorithm based on feature selection and memory mechanism optimization, an accurate and stable classification effect can be achieved in virtual histology, and a reliable basis is provided for further lesion analysis and clinical application.

Description

Technical Field

[0001] The present invention belongs to deep learning and image segmentation, and relates to a plaque segmentation method, specifically an automatic segmentation method for intravascular lipid plaques based on deep learning. Background Art

[0002] With the aging of the population and the changes in lifestyle, the incidence of cardiovascular diseases related to atherosclerosis and the like has been increasing year by year. Early diagnosis and treatment are crucial for improving the prognosis of patients. Plaque segmentation can help doctors more accurately evaluate the size, location, and composition of plaques, and has important value for formulating personalized treatment plans. Plaque segmentation is a key task in medical image analysis, which involves identifying and separating specific structural or lesion areas from complex backgrounds. This technology is particularly important in the medical fields of cardiovascular diseases, brain diseases, and other fields involving soft tissue imaging. Currently, common methods for plaque segmentation include threshold-based methods, methods using edge detection with gradient operators (such as Sobel, Canny), region growing methods, active contour models (Active Contour Model), deep learning methods, graph cut algorithms (GraphCut), etc., but each of these methods has some problems.

[0003] For example, the threshold-based method distinguishes the target area and the background by setting thresholds of gray values or other features. Therefore, this method has high requirements for image quality, is easily affected by noise, and the thresholds for different patients may need to be adjusted; edge detection is difficult to handle discontinuous or blurred boundaries and is prone to generating artifacts; the region growing method is sensitive to the selection of initial seed points and may over-segment or under-segment; the active contour model has a high computational cost, is difficult to initialize, and is easily trapped in local optimal solutions, etc. Summary of the Invention

[0004] To solve the problems of accurate recognition and classification of intravascular lipid plaques, the present invention provides an automatic segmentation method for intravascular lipid plaques based on deep learning. After using the improved U-Net++ to segment the inner and outer membranes of blood vessels, the present invention further uses K-means clustering with an increased memory factor to segment the components of the blood vessel wall (calcification, plaque, inflammatory response, and normal tissue), and improves the assessment of the degree of atherosclerosis, predicts the risk of cardiovascular events, and provides a basis for personalized treatment plans through advanced image processing techniques.

[0005] The object of the present invention is achieved through the following technical solutions:

[0006] An automatic segmentation method for intravascular lipid plaques based on deep learning, comprising the following steps:

[0007] Step 1: Obtain the ultrasound (US) and photoacoustic (PA) vascular image datasets for training and testing. These images should contain different types of lipid plaques within the blood vessels;

[0008] Step 2: Perform preprocessing operations of standardization and normalization on the original vascular images;

[0009] Step 3: Improve the U-Net++ network to construct an improved multi-layer U-Net++ network architecture:

[0010] Adopt the classic U-Net++ network structure as the initial network structure, which includes nested U-shaped structures and dense connections; on this basis, introduce the MLP module and KAN module to optimize the U-Net++ network structure. The specific steps are as follows:

[0011] Step 3-1: On the basis of the classic U-Net++ network structure, introduce the MLP module at the X 0,0 node. Input the feature X 0,0 Perform a linear transformation through the linear layer W 1 The obtained output is processed through a non-linear activation function. After multiple such linear transformations and non-linear activation function processes, the final feature representation is output;

[0012] Step 3-2: Introduce the KAN module at the X 0,0 node. The input feature is X 0,0 which is the result after being processed by the MLP module. Convert the feature map into tokens through the Tokenization layer. The tokens undergo non-linear transformation through the KAN Layer, and the DwConv uses depth convolution to further process the feature map. The obtained result is then normalized through the LayerNorm application layer to stabilize the training process, and finally the feature map processed by the KAN module is output;

[0013] Step 4: Train the multi-layer U-Net++ network constructed in Step 3 on the training set to update the parameters;

[0014] Step 5: Run the network trained in Step 4 on the validation set. Based on the deep supervision operation, select the network with the highest evaluation index as the segmentation network;

[0015] Step 6: Run the optimal network selected in Step 5 on the test set for image segmentation, and perform pruning according to the performance of each layer of the network on the test set;

[0016] Step 7: Apply the final improved U-Net++ model obtained in Step 6 to segment the newly acquired ultrasound and photoacoustic vascular images, and clearly mark the positions of the inner and outer membranes of the blood vessels;

[0017] Step 8: Use the K-means clustering method with an increased memory factor to segment the vascular wall components.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] By using the improved U-Net++ network to segment the intima and adventitia of ultrasonic and photoacoustic images, the present invention can accurately distinguish different structures of the vascular wall, thereby realizing the accurate identification, high-precision segmentation and classification of lipid plaques. Relying on the K-means clustering algorithm optimized based on feature selection and memory mechanism, it can achieve a relatively accurate and stable classification effect in virtual histology, providing a reliable basis for further lesion analysis and clinical applications. Description of the Drawings

[0020] Figure 1 is the automatic segmentation flowchart of intravascular lipid plaques in multimodal medical images based on improved U-Net++ and optimized K-means clustering;

[0021] Figure 2 is the overall framework of the classic U-Net++ network structure;

[0022] Figure 3 is the skip path diagram of the U-Net++ network;

[0023] Figure 4 is the decomposition diagram of U-Net++;

[0024] Figure 5 is the TOK-KAN module in the improved U-Net++ network;

[0025] Figure 6 is the structure diagram of the KAN Layer;

[0026] Figure 7 is the schematic diagram of processing vascular images using the improved U-Net++ network structure;

[0027] Figure 8 is the schematic diagram of the method of the improved U-Net++ network;

[0028] Figure 9 is the segmentation result of the U-Net++ network, (a) original ultrasonic IVUS image, (b) intima and adventitia parts of the blood vessel after U-Net++ segmentation;

[0029] Figure 10 is the change trend of relevant indicators during the training process, (a) change trend of the loss function, (b) change trend of mIoU;

[0030] Figure 11It is K-means plaque clustering. (a) Original ultrasound IVUS image, (b) Virtual histological image after clustering. Detailed implementation manner

[0031] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings, but it is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention shall be covered by the protection scope of the present invention.

[0032] The present invention provides an automatic segmentation method for intravascular lipid plaques based on deep learning. The method uses an improved U-Net++ network architecture to achieve high-precision segmentation of the inner and outer membranes of blood vessels. Then, on this basis, a K-means clustering algorithm based on gray value, edge feature and texture feature is adopted. By adding a memory mechanism, the stability and consistency in the clustering process are ensured, and accurate classification of plaque types is achieved. As Figure 1 shown, the specific steps are as follows:

[0033] Step 1: Obtain ultrasound (US) and photoacoustic (PA) vascular image datasets for training and testing. These images should contain different types of lipid plaques in blood vessels. The specific steps are as follows:

[0034] Step 1-1: Collect ultrasound and photoacoustic vascular images with lipid plaques, ensuring that different types of lipid plaques are covered, including fibrous plaques, fibro-fatty plaques, necrotic cores and calcifications.

[0035] Step 1-2: Remove irrelevant or low-quality data;

[0036] Step 1-3: Rotate, flip and scale the images to increase the diversity of the dataset, and obtain an ultrasound (US) and photoacoustic (PA) vascular image dataset containing different types of lipid plaques in blood vessels for training and testing.

[0037] Step 2: Perform preprocessing operations such as standardization and normalization on the original vascular images in the dataset obtained in Step 1 to ensure the consistency and comparability of the data input into the neural network. The preprocessing operations include steps such as size adjustment and noise removal.

[0038] Step 3: Improve the classic U-Net++ network to construct a multi-layer U-Net++ network architecture. First, construct the encoder part to extract image features layer by layer. Each layer will reduce the spatial resolution but increase the number of channels to capture deeper information. Then, design the decoder to gradually restore the spatial resolution and restore the feature map to the original image size. At the same time, map the low-resolution features back to high-resolution through upsampling operations. Establish dense skip connections between the encoder and the decoder to directly transfer early features to later layers, thereby retaining more detailed information and enhancing feature reusability.

[0039] U-Net is a convolutional neural network (CNN) designed specifically for biomedical image segmentation, initially proposed by Ronneberger et al. in 2015. It adopts an encoder-decoder structure, where the encoder is responsible for extracting features, and the decoder is used to reconstruct the spatial dimensions of the input image. The uniqueness of U-Net lies in its skip connections, which connect the feature maps at different levels in the encoder to the corresponding-level feature maps in the decoder, thereby retaining more positional information, which is particularly important for high-precision pixel-level segmentation tasks.

[0040] U-Net++ is an improvement on the original U-Net architecture, aiming to solve some specific problems and enhance model performance. Based on traditional skip connections, U-Net++ introduces more direct connections between layers, forming dense skip connections. This design allows lower-level features to be directly transferred to deeper levels, enhancing feature reusability and expressive power. U-Net++ introduces multiple auxiliary loss functions at different stages of the decoder, enabling the model to receive feedback from multiple levels during training. This not only speeds up the training process but also improves the quality of the final output. By adding additional paths to integrate features at different scales, U-Net++ can better capture global and local context information, which is particularly beneficial for dealing with complex structures. U-Net++ performs well in various medical image segmentation tasks, such as tumor detection, organ segmentation, and the intima-media segmentation of blood vessels mentioned in this article. Its efficiency and accuracy make it an important tool in the biomedical field.

[0041] As Figure 2 shown, the U-Net++ network architecture mainly consists of an encoder, a decoder, and skip connections, where:

[0042] The encoder gradually extracts high-level features of the image through a series of convolutional layers and pooling layers, while reducing the spatial size of the feature map, where: each convolutional layer contains two 3×3 convolutional operations, followed by a 2×2 max pooling operation. The specific process is shown in the following formula:

[0043] Y encoder = MaxPool(σ encoder (W encoder * X + b encoder ))

[0044] Wherein, X is the input feature map, W encoder and b encoder are the convolution kernel weights and biases respectively, and σ encoder is the activation function, usually ReLu, and Y encoder is the output of the convolutional layer, the feature map after convolution and activation function processing.

[0045] The decoder gradually restores the spatial dimension of the feature map through upsampling operations and fuses it with the feature map of the corresponding encoder layer, where: each upsampling layer contains a 2×2 transposed convolution operation, followed by two 3×3 convolution operations. The specific process is shown in the following formula:

[0046] Y decoder = σ decoder (W decoder * UpSample(X) + b decoder )

[0047] Wherein, X is the input feature map, W decoder and b decoder are the convolution kernel weights and biases respectively, and σ decoder is the activation function, and Y decoder is the output of the decoding layer, the feature map after upsampling and convolution processing.

[0048] The skip connection directly passes the feature map of each layer of the encoder to the corresponding layer of the decoder, helping the network to better capture detailed information and context features. The specific process is shown in the following formula:

[0049] Y decoder = Concat(Y encoder , Y decoder )

[0050] Wherein, Y encoder is the output feature map of the encoder layer, and Y decoder is the output feature map of the decoder layer.

[0051] The meaning of the above formula is to concatenate the output feature map Y encoder of the encoder layer with the output feature map Y decoder of the decoder layer to generate a new feature map Y decoder, this new feature map contains context information from the encoder and detail information from the decoder. This operation helps to better utilize global and local features during the decoding process, thereby improving the performance of the model.

[0052] U-NET++ also introduces dense skip connections, which are connected between the encoder and decoder at each level, as well as inside each sub-U-Net structure, to enhance the transfer and utilization efficiency of features. The specific process is shown as follows:

[0053]

[0054] Among them, Y i,j represents the output of the j-th sub-network at the i-th layer, W i,j is the convolutional kernel, b i,j is the bias term, σ is the activation function, Y i-1,j represents the output of the j-th sub-network at the (i - 1)-th layer, and Y i,j-1 is the output of the (j - 1)-th sub-network at the i-th layer.

[0055] Through the nested U-shaped structure, U-Net++ achieves multi-scale feature fusion, effectively enhancing the network's ability to capture details at different scales. The specific process is shown as follows:

[0056]

[0057] Among them, Y i,j represents the output of the j-th sub-network at the i-th layer, W i,j is the convolutional kernel, b i,j is the bias term, σ is the activation function, represents the concatenation of the output feature maps from the first n sub-networks at the i-th layer.

[0058] From Figure 2 it can be found that, compared with the network structure of U-Net, U-Net++ adds a series of dense convolutional blocks on the skip path, that is, the green part in the figure. And feature map concatenation is performed between every two convolutional blocks. The specific concatenation process can be carried out according to the following formula:

[0059]

[0060] Among them, x i,j represents the output of the j-th sub-network at the i-th layer, H(x i-1,j ) represents performing two convolutional operations on the input feature map x i-1,j , followed by an activation function, and U(x i+1,j-1 ) represents performing upsampling on the input feature map x i+1,j-1 , and the method used is still the bilinear interpolation method in U-Net. Denotes the concatenation of all output feature maps from the 0th sub-network of the i-th layer to the (j-1)-th sub-network.

[0061] Figure 3 Is the convolutional block of the topmost layer in the U-Net++ structure. Where U(X 1，0 ) is to perform an upsampling on the result of X 1，0 , and X 0，1 is the result of convolution after concatenating U(X 1，0 ) and X 0，0 . Similarly, X 0，4 is the result of convolution after concatenating U(X 1，3 ) and X 0，0 , X 0 ，2 , X 0，3 . By adjusting the number of convolutional kernels, the sizes of the images obtained by the five convolutional blocks in the first layer are made exactly the same.

[0062] This innovative skip path fuses image features at different depths in a more diverse way through a series of dense convolutional blocks, improving the segmentation accuracy. Essentially, the dense convolutional blocks make the semantic level of the feature maps of the encoder closer to the semantic level of the feature maps waiting in the decoder. Assuming that when the received encoder feature maps and the corresponding decoder feature maps are semantically similar, the optimizer will solve an easier optimization problem.

[0063] The U-Net++ network uses a combination of binary cross-entropy and Dice coefficient as the loss function. The deep supervision operation adds the following loss function to the output layer of each sub-network, as shown in Figure 4 at layer L 1 , layer L 2 , layer L 3 and layer L 4 . The calculation location of the loss function is marked at the output of each sub-network in these layers. The calculation formula of the loss function is:

[0064]

[0065] where, denotes the loss function, which is used to measure the difference between the true label Y and the predicted label , N represents the number of samples, b represents the sample index, Y b represents the true label of the b-th sample, and represents the predicted label of the b-th sample.

[0066] The present invention improves the above-mentioned classical U-Net++ network. By combining U-Net++ and KAN, in intravascular plaque segmentation, U-Net++ improves the segmentation accuracy through multi-scale skip connections, while KAN accelerates model training and inference by reducing the number of parameters, thus enhancing the overall efficiency. The improved U-Net++ network structure is as shown in Figure 8 as follows. The specific improvement methods are as follows:

[0067] Step 3-1: Based on the classical U-Net++ network structure, an MLP module is introduced at the X 0,0 node. The input feature X 0,0 is linearly transformed through the linear layer W 1 . The obtained output is processed by a non-linear activation function. After multiple such linear transformations and non-linear activation function processes, the final feature representation is output.

[0068] Step 3-2: A KAN module is introduced at the X 0,0 node, as shown in Figure 5 . The input feature is the result after being processed by the MLP module for X 0,0 . The feature map is converted into tokens through the Tokenization layer. The tokens undergo non-linear transformation through the KAN Layer. DwConv further processes the feature map using depth convolution. The obtained result is then normalized by LayerNorm to stabilize the training process. Finally, the feature map processed by the KAN module is output.

[0069] Step 4: Train the multi-layer U-Net++ network constructed in Step 3 on the training set and update the parameters.

[0070] Step 5: Run the network trained in Step 4 on the validation set. Based on the deep supervision operation, select the network with the highest evaluation metric as the segmentation network.

[0071] Step 6: Run the optimal network selected in Step 5 on the test set for image segmentation, and perform pruning based on the performance of each layer of the network on the test set.

[0072] Step 7: Apply the finally improved U-Net++ model obtained in Step 6 to segment the newly acquired ultrasound and photoacoustic vascular images, and clearly mark the positions of the intima and adventitia of the blood vessels.

[0073] Step 8: Use the K-means clustering method with an increased memory factor to segment the components of the blood vessel wall (calcification, plaque, inflammatory response, and normal tissue).

[0074] The goal of K-means is to partition a dataset into K clusters such that each data point belongs to the cluster with the nearest cluster center. By repeatedly adjusting the positions of the cluster centers, K-means continuously optimizes the compactness within the clusters, thereby obtaining clusters that are as compact and separated from each other as possible, that is, minimizing the Within-Cluster Sum of Squares (WCSS), which is the sum of the squares of the distances from each point to the center of its belonging cluster. The formula is as follows:

[0075]

[0076] where K is the number of clusters, C i is the set of points in the i-th cluster, x is the data point belonging to C i , μ i is the centroid of the i-th cluster, and ||x - μ i || 2 is the square of the Euclidean distance between x and μ i .

[0077] The process of classical K-means clustering is divided into two main steps: Assignment and Update. The following are the detailed steps:

[0078] Step (1) Select the value of K: Set the number of clusters K.

[0079] Step (2) Initialize the cluster centers: Randomly select K data points as the initial cluster centers (centroids).

[0080] Step (3) Assignment Step: For each point in the dataset, assign it to the cluster corresponding to the nearest cluster center. Here, the "distance" usually uses the Euclidean distance. For each data point, find the nearest cluster center:

[0081]

[0082] where x i is a data point in the dataset, c i is the index of the cluster to which the data point x i is assigned, and μ j is the center of the j-th cluster.

[0083] Step (4) Update Step: According to the current cluster assignment, recalculate the center of each cluster and update the center μ j of each cluster to the mean of all points within the cluster:

[0084]

[0085] Among them, a data point in the x dataset is μ j is the center of the j-th cluster, C j the set of all data points in the j-th cluster, |C j | the number of data points in the j-th cluster.

[0086] Step (5) continuously repeats the assignment and update steps until the cluster centers no longer change (converge) or the specified maximum number of iterations is reached.

[0087] Based on the classical K-means clustering algorithm, the present invention introduces a memory mechanism and provides a K-means clustering method with an increased memory factor to ensure high stability for different input images during the clustering process. The specific implementation steps are as follows:

[0088] Step 8-1: First, ensure that the U-Net++ model has been trained and can accurately segment the vascular region. Initialize previous_centroids to None and use the default initialization method for the first clustering.

[0089] Step 8-2: Define the memory factor and introduce a variable memory_factor, which is a value between 0 and 1. This value determines the influence of the previous cluster centers on the current cluster centers. If memory_factor = 0, it means that historical information is not considered at all, while memory_factor = 1 indicates complete dependence on historical information.

[0090] Step 8-3: Modify the apply_kmeans function to incorporate the influence of the memory factor when calculating the new cluster centers. In the main loop, when processing each image, call the apply_kmeans function with the memory factor and pass the previous_centroids and memory_factor parameters. Update previous_centroids to the new cluster centers of the current frame to influence the clustering process of the next frame.

[0091] Step 8-4: Perform operations such as segmentation, extraction of the region of interest (vascular wall), application of K-means clustering with the memory factor, mapping the clustering results to colors, and generating a color image for each frame of the image, and add each processed image to the video stream. After processing all frames, release the video writing object and save the final output video.

[0092] Example:

[0093] Step 1: Collect intravascular ultrasound (IVUS) and photoacoustic (PA) images of blood vessels with lipid plaques, ensuring that a sufficient variety of plaque types are covered (including fibrous plaques, fibro-fatty plaques, necrotic cores, and calcifications). Then, remove irrelevant or low-quality data, such as blurred, overly noisy, or low-resolution images. Image rotation, flipping, scaling, etc. can be performed to increase the diversity of the dataset. After the above preprocessing steps, an IVUS and PA blood vessel image dataset containing different types of lipid plaques within the blood vessels for training and testing is obtained.

[0094] Step 2: Improve the classic U-Net++ network.

[0095] Compare the training losses and related parameters of the improved U-Net++ network in the present invention with those of the classic U-Net and U-Net++ networks. The results are shown in Table 1.

[0096] Table 1

[0097]

[0098] From the comparison results, it can be found that the improved U-Net++ network reduces the number of parameters while improving various indicators of the segmentation results. Moreover, some tiny targets in complex backgrounds can also be accurately segmented by the network of the present invention, demonstrating its generalization ability at different scales.

[0099] Step 3: Use the improved U-Net++ network to perform intima-media segmentation on blood vessel images.

[0100] Figure 9 Shows the segmentation effect of the improved U-Net++ network in intravascular ultrasound (IVUS) images. Figure 9 (a) is the original IVUS image, Figure 9 (b) is the result after segmentation by U-Net++, marking the positions of the intima and media of the blood vessel. It can be seen that the U-Net++ network can clearly identify the boundary regions of the intima and media, achieving accurate segmentation of the blood vessel structure. This shows the superior performance of U-Net++ in multi-modal image segmentation tasks, being able to capture important structural features in IVUS images well.

[0101] Figure 10 Shows the changing trends of the loss function and mean intersection over union (mIoU) during the training process. Figure 10 (a) shows the convergence curves of the training loss and validation loss. It can be seen that both the training and validation losses decrease rapidly and gradually become stable during the training process, indicating good convergence of the model on this task. In addition, the validation loss tends to be stable in the later stage of training and is relatively close to the training loss, indicating that the model has good generalization ability. Figure 10(b) shows the variation of mIoU with the number of training iterations. The mIoU increases rapidly in the initial stage and reaches stability at nearly 20 iterations, being 97.88%. This result indicates that U-Net++ has achieved a high segmentation accuracy in this segmentation task and can effectively separate different tissue regions in IVUS images.

[0102] Step 4: Use the K-means clustering algorithm with an increased memory factor to segment the vascular wall components. This algorithm successfully classifies the plaques in the IVUS image into four types: fibrous plaque, fibro-fatty plaque, necrotic core, and calcification. This precise segmentation and classification provide a solid foundation for the study of virtual histology, enabling the position and size of each type of plaque to be clearly marked, thus providing effective support for the quantitative analysis of plaques.

[0103] Compared with the classical K-means, in the present invention, for the first frame image, K-means uses k-means++ for initialization (when previous_centroids is None). For subsequent frames, the centroid positions obtained from the clustering of the previous frame are used as the initial centroids (i.e., the case when previous_centroids is not None), thus enabling the K-means clustering algorithm to have a "memory" characteristic. This method is particularly useful when processing consecutive frames (such as video sequences) because it can maintain the consistency and coherence of the clustering results, thereby improving the stability of time-series data. And in the present invention, the K-means clustering is restricted within a specific region (i.e., region2_mask), which is different from the classical K-means that usually performs unconstrained clustering on the entire dataset. Such a constraint helps to focus on the key parts of the image and avoid the influence of irrelevant regions.

[0104] From Figure 11 It can be seen that the K-means clustering algorithm with a memory mechanism has successfully classified the plaques in the IVUS image into four types: fibrous plaque (green), fibro-fatty plaque (yellow), necrotic core (red), and calcification (white). This preliminary classification provides strong support for the study of virtual histology. The quantitative evaluation index shows that the Silhouette Score of this method is 0.8654, indicating a high degree of separation between clustering categories and good internal compactness. The Davies-Bouldin index is 0.2376, indicating a low similarity between categories, that is, a good clustering effect. These results show that the K-means clustering algorithm optimized based on feature selection and memory mechanism can achieve a relatively accurate and stable classification effect in virtual histology, providing a reliable basis for further lesion analysis and clinical applications.

Claims

1. A deep learning-based automatic segmentation method for intravascular lipid plaques, characterized in that The method comprises the following steps: Step 1: Obtain ultrasound and photoacoustic vascular image datasets for training and testing; Step 2: Perform standardization and normalization preprocessing operations on the original vascular image; Step 3: Improve the U-Net++ network and build an improved multi-layer U-Net++ network architecture: The classic U-Net++ network structure is used as the initialization network structure, which contains a nested U-shaped structure and dense connections. On this basis, the MLP module and KAN module are introduced to optimize the U-Net++ network structure. Step 4: Train the multi-layer U-Net++ network constructed in step 3 on the training set and update the parameters; Step 5: Run the network trained in step 4 on the validation set, and select the network with the highest evaluation index as the segmentation network based on the deep supervision operation; Step 6: Run the optimal network selected in step 5 on the test set to perform image segmentation, and prune each layer of the network based on its performance on the test set; Step 7: Apply the final improved U-Net++ model obtained in step 6 to segment the newly acquired ultrasound and photoacoustic vascular images, and clearly mark the locations of the inner and outer membranes of the blood vessels; Step 8: Use the K-means clustering method with an increased memory factor to segment the vascular wall components.

2. The deep learning-based automatic segmentation method for intravascular lipid plaques according to claim 1 is characterized in that The specific steps of step 1 are as follows: Step 1-1: Collect ultrasound and photoacoustic images of lipid plaque vessels, ensuring that different types of lipid plaques are covered, including fibrous plaques, fibrofatty plaques, necrotic cores, and calcifications; Step 1-2: Remove irrelevant or low-quality data; Step 1-3: Rotate, flip, and scale the image to increase the diversity of the data set, and obtain an ultrasound and photoacoustic vascular image data set containing different types of lipid plaques in blood vessels for training and testing.

3. The deep learning-based automatic segmentation method for intravascular lipid plaques according to claim 1 is characterized in that In step 2, the preprocessing operation includes resizing and noise removal steps.

4. The deep learning-based automatic segmentation method for intravascular lipid plaques according to claim 1 is characterized in that The specific steps of step 3 are as follows: Step 3-1: Based on the classic U-Net++ network structure, 0,0 The MLP module is introduced at the node and the feature X is input 0,0 The linear transformation is performed through the linear layer W1, and the obtained output is processed by a nonlinear activation function. After multiple linear transformations and nonlinear activation function processing, the final feature representation is output; Step 3-2: On X 0,0 The KAN module is introduced at the node, and the input feature is X 0,0 After the result is processed by the MLP module, the feature map is converted into tokens through the Tokenization layer. The token is transformed nonlinearly through the KAN Layer. DwConv uses deep convolution to further process the feature map. The result is then normalized by the LayerNorm application layer to stabilize the training process, and finally the feature map processed by the KAN module is output.

5. The method for automatic segmentation of intravascular lipid plaques based on deep learning according to claim 1, characterized in that The specific steps of step 8 are as follows: Step 8-1: Initialize previous_centroids to None and use the default initialization method for the first clustering; Step 8-2: Define the memory factor and introduce a variable memory_factor. If memory_factor = 0, it means that historical information is not considered at all, while memory_factor = 1 means that historical information is completely relied on; Step 8-3: Modify the apply_kmeans function, add the effect of the memory factor when calculating the new cluster center. In the main loop, when processing each picture, call the apply_kmeans function with the memory factor, pass the previous_centroids and memory_factor parameters, and update previous_centroids to the new cluster center of the current frame to affect the clustering process of the next frame. Step 8-4: Perform segmentation on each frame image, extract the region of interest, apply K-means clustering with a memory factor, map the clustering results to colors and generate a color image, and add each processed image to the video stream. After completing the processing of all frames, release the video writing object and save the final output video.

Images