Automatic segmentation method of intravascular lipid plaque based on deep learning

Abstract

The present invention discloses a deep learning-based method for automatic segmentation of intravascular lipid plaques. The method utilizes an improved U-Net++ network architecture to achieve high-precision segmentation of the inner and outer membranes of blood vessels. Based on this, a K-means clustering algorithm based on grayscale values, edge features, and texture features is then employed. By incorporating a memory mechanism, stability and consistency are ensured during the clustering process, enabling accurate classification of plaque types. By using an improved U-Net++ network for inner and outer membrane segmentation in ultrasound and photoacoustic images, the present invention can accurately distinguish between different structures of the vascular wall, thereby achieving accurate identification, high-precision segmentation, and classification of lipid plaques. The K-means clustering algorithm, optimized based on feature selection and a memory mechanism, achieves relatively accurate and stable classification results in virtual histology, providing a reliable foundation for further lesion analysis and clinical application.

Description

Technical Field

[0001] The present invention belongs to deep learning and image segmentation, and relates to a plaque segmentation method, and specifically to an automatic segmentation method for intravascular lipid plaques based on deep learning. Background Art

[0002] With the aging of the population and changes in lifestyle, the incidence of cardiovascular diseases such as those related to atherosclerosis is increasing year by year. Early diagnosis and treatment are crucial to improving patient prognosis. Plaque segmentation can help doctors more accurately assess the size, location, and composition of plaques, which is of great value in formulating personalized treatment plans. Plaque segmentation is a key task in medical image analysis, which involves identifying and separating specific structures or lesion areas from complex backgrounds. This technology is particularly important in cardiovascular disease, brain disease, and other medical fields involving soft tissue imaging. Currently, commonly used methods for plaque segmentation include threshold-based methods, edge detection methods using gradient operators (such as Sobel and Canny), region growing methods, active contour models (Active Contour Model), deep learning methods, graph cut algorithms (GraphCut), etc., but each of these methods has some problems.

[0003] For example, the threshold-based method distinguishes the target area from the background by setting the threshold of grayscale value or other features. Therefore, this method has high requirements on image quality and is easily affected by noise. The threshold of different patients may need to be adjusted; edge detection has difficulty in processing discontinuous or blurred boundaries and is prone to artifacts; the region growing method is sensitive to the selection of initial seed points and may be over-segmented or under-segmented; the active contour model has a high computational cost, is difficult to initialize, and is prone to falling into local optimal solutions, etc. Summary of the Invention

[0004] To accurately identify and classify intravascular lipid plaques, this paper provides a deep learning-based automated segmentation method for intravascular lipid plaques. After using a modified U-Net++ to segment the inner and outer tunica of the vessel, this method further employs K-means clustering with an increased memory factor to segment vessel wall components (calcification, plaques, inflammatory responses, and normal tissue). This advanced image processing technology improves the assessment of atherosclerosis severity, predicts the risk of cardiovascular events, and provides a basis for personalized treatment plans.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A deep learning-based automatic segmentation method for intravascular lipid plaques includes the following steps:

[0007] Step 1: Obtain ultrasound (US) and photoacoustic (PA) vascular image datasets for training and testing. These images should contain different types of lipid plaques in blood vessels.

[0008] Step 2: Perform standardization and normalization preprocessing operations on the original vascular image;

[0009] Step 3: Improve the U-Net++ network and build an improved multi-layer U-Net++ network architecture:

[0010] The classic U-Net++ network structure is used as the initial network structure, which contains a nested U-shaped structure and dense connections. On this basis, the MLP module and KAN module are introduced to optimize the U-Net++ network structure. The specific steps are as follows:

[0011] Step 3-1: Based on the classic U-Net++ network structure in X 0,0 The MLP module is introduced at the node and the feature X is input 0,0 The linear transformation is performed through the linear layer W1, and the output is processed by a nonlinear activation function. After multiple such linear transformations and nonlinear activation function processing, the final feature representation is output;

[0012] Step 3-2: In X 0,0 The KAN module is introduced at the node, and the input feature is X 0,0 After the results of the MLP module processing, the feature map is converted into tokens through the Tokenization layer. The tokens are nonlinearly transformed through the KAN Layer. DwConv uses deep convolution to further process the feature map. The obtained results are then normalized by the LayerNorm application layer to stabilize the training process, and finally the feature map processed by the KAN module is output;

[0013] Step 4: Train the multi-layer U-Net++ network constructed in step 3 on the training set and update the parameters;

[0014] Step 5: Run the network trained in step 4 on the validation set. Based on the deep supervision operation, select the network with the highest evaluation index as the segmentation network.

[0015] Step 6: Run the optimal network selected in step 5 on the test set to perform image segmentation, and prune each layer of the network based on its performance on the test set;

[0016] Step 7: Apply the final improved U-Net++ model obtained in step 6 to segment the newly acquired ultrasound and photoacoustic vascular images, clearly marking the locations of the inner and outer membranes of the blood vessels;

[0017] Step 8: Use the K-means clustering method with an increased memory factor to segment the vascular wall components.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] The present invention uses an improved U-Net++ network to perform inner and outer membrane segmentation of ultrasound and photoacoustic images, which can accurately distinguish different structures of the blood vessel wall, thereby achieving accurate identification, high-precision lipid plaque segmentation and classification. Relying on the K-means clustering algorithm based on feature selection and memory mechanism optimization, it can achieve relatively accurate and stable classification effects in virtual histology, providing a reliable foundation for further lesion analysis and clinical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart for automatic segmentation of intravascular lipid plaques in multimodal medical images based on improved U-Net++ and optimized K-means clustering;

[0021] Figure 2 It is the overall framework of the classic U-Net++ network structure;

[0022] Figure 3 It is the U-Net++ network jump path graph;

[0023] Figure 4 This is the decomposition diagram of U-Net++;

[0024] Figure 5 It improves the TOK-KAN module in the U-Net++ network;

[0025] Figure 6 This is the KAN Layer structure diagram;

[0026] Figure 7 This is the principle diagram of using the improved U-Net++ network structure to process blood vessel images;

[0027] Figure 8 This is a schematic diagram of the method for improving the U-Net++ network;

[0028] Figure 9 The U-Net++ network segmentation results are: (a) original IVUS image, (b) vascular intima and tunica intima after U-Net++ segmentation;

[0029] Figure 10 The changing trends of related indicators during training are: (a) loss function changing trend, (b) mIoU changing trend;

[0030] Figure 11K-means plaque clustering, (a) original IVUS image, (b) virtual histology image after clustering. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0032] The present invention provides a deep learning-based automatic segmentation method for intravascular lipid plaques. The method utilizes an improved U-Net++ network architecture to achieve high-precision segmentation of the inner and outer membranes of blood vessels. On this basis, a K-means clustering algorithm based on grayscale values, edge features, and texture features is adopted. By adding a memory mechanism, the stability and consistency of the clustering process are ensured, thereby achieving accurate classification of plaque types. Figure 1 As shown, the specific steps include:

[0033] Step 1: Obtain ultrasound (US) and photoacoustic (PA) vascular image datasets for training and testing. These images should contain different types of lipid plaques in blood vessels. The specific steps are as follows:

[0034] Step 1-1: Collect ultrasound and photoacoustic images of lipid plaque vessels, ensuring coverage of different types of lipid plaques, including fibrous plaques, fibrofatty plaques, necrotic cores, and calcifications.

[0035] Step 1-2: Remove irrelevant or low-quality data;

[0036] Steps 1-3: Rotate, flip, and scale the images to increase the diversity of the dataset, and obtain an ultrasound (US) and photoacoustic (PA) vascular image dataset containing different types of lipid plaques in blood vessels for training and testing.

[0037] Step 2: Perform preprocessing operations such as standardization and normalization on the original vascular images in the dataset obtained in step 1 to ensure the consistency and comparability of the data input to the neural network. The preprocessing operations include resizing, noise removal, and other steps.

[0038] Step 3: Improve the classic U-Net++ network and construct a multi-layer U-Net++ network architecture. First, construct the encoder to extract image features layer by layer. Each layer reduces the spatial resolution but increases the number of channels to capture deeper information. Then, design a decoder to gradually restore the spatial resolution and restore the feature maps to the original image size. Simultaneously, upsampling is used to map low-resolution features back to high resolution. Dense skip connections are established between the encoder and decoder to pass early features directly to later layers, preserving more detailed information and enhancing feature reuse.

[0039] U-Net is a convolutional neural network (CNN) designed specifically for biomedical image segmentation, originally proposed by Ronneberger et al. in 2015. It employs an encoder-decoder architecture, where the encoder extracts features and the decoder reconstructs the spatial dimensions of the input image. A unique feature of U-Net is its skip connections, which connect feature maps from different levels of the encoder to feature maps from the corresponding levels of the decoder. This preserves more positional information, which is particularly important for high-precision pixel-level segmentation tasks.

[0040] U-Net++ is an improvement to the original U-Net architecture, designed to address specific issues and enhance model performance. Building on traditional skip connections, U-Net++ introduces more direct connections between layers, forming dense skip connections. This design allows features from lower layers to be directly passed to deeper layers, enhancing feature reuse and expressiveness. U-Net++ introduces multiple auxiliary loss functions at different stages of the decoder, allowing the model to receive feedback from multiple levels during training. This not only speeds up training but also improves the quality of the final output. By adding additional paths to integrate features at different scales, U-Net++ is able to better capture global and local contextual information, which is particularly beneficial for processing complex structures. U-Net++ has performed well in various medical image segmentation tasks, such as tumor detection, organ segmentation, and the vascular intima and endothelium segmentation mentioned in this article. Its efficiency and accuracy make it an important tool in the biomedical field.

[0041] like Figure 2 As shown in Figure 1, the U-Net++ network architecture mainly consists of an encoder, a decoder, and skip connections, where:

[0042] The encoder gradually extracts high-level features of the image through a series of convolutional and pooling layers, while reducing the spatial size of the feature map. Each convolutional layer consists of two 3×3 convolution operations followed by a 2×2 maximum pooling operation. The specific process is shown in the following formula:

[0043] Y encoder =MaxPool(σ encoder (W encoder *X+b encoder ))

[0044] Among them, X is the input feature map, W encoder and b encoder are the convolution kernel weight and bias, σ encoder is the activation function, usually ReLu, Y encoder It is the output of the convolutional layer, the feature map after convolution and activation function processing.

[0045] The decoder gradually restores the spatial size of the feature map through upsampling operations and fuses it with the feature map of the corresponding encoder layer, where each upsampling layer contains a 2×2 deconvolution operation followed by two 3×3 convolution operations. The specific process is shown in the following formula:

[0046] Y decoder =σ decoder (W decoder *UpSample(X)+b decoder )

[0047] Among them, X is the input feature map, W decoder and b decoder are the convolution kernel weight and bias, σ decoder is the activation function, Y decoder It is the output of the decoding layer, the feature map after upsampling and convolution.

[0048] The skip connection directly transmits the feature map of each layer of the encoder to the corresponding layer of the decoder, helping the network to better capture detailed information and contextual features. The specific process is shown in the following formula:

[0049] Y decoder =Concat(Y encoder ,Y decoder )

[0050] Among them, Y encoder is the output feature map of the encoder layer, Y decoder is the output feature map of the decoder layer.

[0051] The meaning of the above formula is to convert the output feature map Y of the encoder layer encoder and the output feature map Y of the decoder layer decoder Splice and generate a new feature map Y decoder, this new feature map contains contextual information from the encoder and detailed information from the decoder. This operation helps to better utilize global and local features during the decoding process, thereby improving the performance of the model.

[0052] U-NET++ also introduces dense skip connections between the encoder and decoder at each level, as well as within each sub-U-Net structure, to enhance the efficiency of feature transfer and utilization. The specific process is shown in the following formula:

[0053]

[0054] Among them, Y i,j represents the output of the jth sub-network in the i-th layer, W i,j is the convolution kernel, b i,j is the bias term, σ is the activation function, Y i-1,j represents the output of the jth sub-network in the i-1th layer, Y i,j-1 The output of the j-1th sub-network in the i-th layer.

[0055] Through the nested U-shaped structure, U-Net++ achieves multi-scale feature fusion, effectively improving the network's ability to capture details at different scales. The specific process is shown in the following formula:

[0056]

[0057] Among them, Y i,j represents the output of the jth sub-network in the i-th layer, W i,j is the convolution kernel, b i,j is the bias term, σ is the activation function, Represents the concatenation of the output feature maps of the first n sub-networks from the i-th layer.

[0058] from Figure 2 Compared to the U-Net architecture, U-Net++ adds a series of dense convolutional blocks (shown in green) along the skip path. Feature maps are concatenated between each convolutional block. The concatenation process can be described as follows:

[0059]

[0060] Among them, x i,j represents the output of the jth sub-network of the i-th layer, H(x i-1,j ) represents the input feature map x i-1,j Perform two convolution operations, followed by an activation function, U(x i+1,j-1 ) represents the input feature map x i+1,j-1 For upsampling, the method used is still the bilinear interpolation method in U-Net. Represents the concatenation of all output feature maps from the 0th sub-network to the j-1th sub-network of the i-th layer.

[0061] Figure 3 It is the top convolution block in the U-Net++ structure. 1，0 ) is X 1，0 The result is upsampled once, X 0，1 Is U(X 1，0 ) and X 0，0 The result of convolution after concatenation. Similarly, X 0，4 Is U(X 1，3 ) and X 0，0 、X 0 ，2 、X 0，3 The convolution result after concatenation. By adjusting the number of convolution kernels, the size of the images obtained by the five convolution blocks in the first layer are exactly the same.

[0062] This innovative skipping path combines image features at different depths in a more diverse way through a series of dense convolutional blocks, improving segmentation accuracy. Essentially, the dense convolutional blocks bring the semantic level of the encoder's feature maps closer to the semantic level of the feature maps waiting in the decoder. Assuming that the received encoder feature maps and the corresponding decoder feature maps are semantically similar, the optimizer will solve an easier optimization problem.

[0063] The U-Net++ network uses a combination of binary cross entropy and Dice coefficient as the loss function. The deep supervision operation adds the following loss function to the output layer of each sub-network, such as Figure 4 Shown in L 1 Layer, L 2 Layer L 3 Layer and L 4 The calculation location of the loss function is marked at the output of each sub-network of the layer. The calculation formula of the loss function is:

[0064]

[0065] in, Represents the loss function, which is used to measure the difference between the true label Y and the predicted label The difference between them, N represents the number of samples, b represents the sample index, and Y b represents the true label of the b-th sample, represents the predicted label of the b-th sample.

[0066] This paper improves the classic U-Net++ network mentioned above, combining U-Net++ and KAN. In the segmentation of intravascular plaque, U-Net++ improves segmentation accuracy through multi-scale skip connections, while KAN accelerates model training and reasoning by reducing the number of parameters, thereby improving overall efficiency. The improved U-Net++ network structure is as follows: Figure 8 The specific improvement methods are as follows:

[0067] Step 3-1: Based on the classic U-Net++ network structure in X 0,0 The MLP module is introduced at the node and the feature X is input 0,0 A linear transformation is performed through the linear layer W1, and the output is processed by a nonlinear activation function. After multiple such linear transformations and nonlinear activation function processing, the final feature representation is output.

[0068] Step 3-2: In X 0,0 The KAN module is introduced at the node, such as Figure 5 As shown, the input feature is X 0,0 After the results of the MLP module processing, the feature map is converted into tokens through the Tokenization layer. The tokens are nonlinearly transformed through the KAN Layer. DwConv uses deep convolution to further process the feature map. The results are then normalized by the LayerNorm application layer to stabilize the training process, and finally the feature map processed by the KAN module is output.

[0069] Step 4: Train the multi-layer U-Net++ network constructed in step 3 on the training set and update the parameters.

[0070] Step 5: Run the network trained in step 4 on the validation set. Based on the deep supervision operation, select the network with the highest evaluation index as the segmentation network.

[0071] Step 6: Run the optimal network selected in step 5 on the test set to perform image segmentation, and prune each layer of the network based on its performance on the test set.

[0072] Step 7: Apply the final improved U-Net++ model obtained in step 6 to segment the newly acquired ultrasound and photoacoustic vascular images and clearly mark the locations of the inner and outer membranes of the blood vessels.

[0073] Step 8: Use the K-means clustering method with an increased memory factor to segment the vascular wall components (calcification, plaque, inflammatory response, and normal tissue).

[0074] The goal of K-means is to partition a dataset into K clusters, such that each data point belongs to the nearest cluster center. By repeatedly adjusting the positions of cluster centers, K-means continuously optimizes the density within the clusters, thereby obtaining clusters that are as compact and separated as possible. This is done by minimizing the Within-Cluster Sum of Squares (WCSS), which is the sum of the squares of the distances from each point to the cluster center. The formula is as follows:

[0075]

[0076] Where K is the number of clusters, C i is the point set of cluster i, x is the point of cluster C i Data points, μ i is the centroid of the ith cluster, ||x-μ i || 2 is x and μ i The square of the Euclidean distance.

[0077] The classic K-means clustering process is divided into two main steps: assignment and update. The following are the detailed steps:

[0078] Step (1) Select K value: set the number of clusters K.

[0079] Step (2) Initialize cluster centers: randomly select K data points as initial cluster centers (centroids).

[0080] Step (3) Assignment Step: For each point in the dataset, assign it to the cluster corresponding to the nearest cluster center. The "distance" here is usually Euclidean distance. For each data point, find the nearest cluster center:

[0081]

[0082] Among them, x i is a data point in the dataset, c i is the data point x i The index of the cluster to be assigned, μ j is the center of the jth cluster.

[0083] Step (4) Update Step: Recalculate the center of each cluster based on the current cluster assignment and update the center μ of each cluster. j is the mean of all points in the cluster:

[0084]

[0085] Among them, a data point in the x data set is μ j is the center of the jth cluster, C j The set of all data points in the jth cluster, |C j |The number of data points in the jth cluster.

[0086] Step (5) repeats the assignment and update steps until the cluster centers no longer change (converge) or the specified maximum number of iterations is reached.

[0087] Based on the classic K-means clustering algorithm, this paper introduces a memory mechanism and provides a K-means clustering method with an increased memory factor to ensure high stability for different input images during the clustering process. The specific implementation steps are as follows:

[0088] Step 8-1: First, ensure that the U-Net++ model has been trained and can accurately segment the blood vessels. Initialize previous_centroids to None and use the default initialization method for the first clustering.

[0089] Step 8-2: Define the memory factor. Introduce a variable called memory_factor, which is a value between 0 and 1. This value determines the degree of influence of the previous cluster center on the current cluster center. If memory_factor = 0, it means that historical information is completely ignored, while memory_factor = 1 means that historical information is completely relied upon.

[0090] Step 8-3: Modify the apply_kmeans function to incorporate the memory factor when calculating new cluster centers. In the main loop, when processing each image, call the apply_kmeans function with the memory factor, passing it the previous_centroids and memory_factor parameters. Update previous_centroids to the new cluster centers for the current frame to influence the clustering process for the next frame.

[0091] Step 8-4: Perform segmentation on each frame image, extract the region of interest (vascular wall), apply K-means clustering with a memory factor, map the clustering results to colors and generate a color image, and add each processed image to the video stream. After completing the processing of all frames, release the video writing object and save the final output video.

[0092] Example:

[0093] Step 1: Collect ultrasound and photoacoustic images of blood vessels with lipid plaques, ensuring that a sufficient range of plaque types (including fibrous plaques, fibrofatty plaques, necrotic cores, and calcifications) are included. Irrelevant or low-quality data, such as blurry, noisy, or low-resolution images, are then removed. Images can be rotated, flipped, and scaled to increase the diversity of the dataset. After these preprocessing steps, a dataset of ultrasound (US) and photoacoustic (PA) blood vessel images containing different types of intravascular lipid plaques is obtained for training and testing.

[0094] Step 2: Improve the classic U-Net++ network.

[0095] The training losses and related parameters of the improved U-Net++ network in the present invention are compared with those of the classic U-Net and U-Net++ networks, and the results are shown in Table 1.

[0096] Table 1

[0097]

[0098] By comparing the results, it can be found that the improved U-Net++ network reduces the number of parameters while improving various indicators of the segmentation results. In addition, some tiny targets in complex backgrounds can also be accurately segmented by the network of the present invention, showing generalization at different scales.

[0099] Step 3: Use the improved U-Net++ network to segment the vascular image into the inner and outer membranes.

[0100] Figure 9 The segmentation effect of the improved U-Net++ network in intravascular ultrasound (IVUS) images is shown. Figure 9 (a) is the original IVUS ultrasound image, Figure 9 (b) shows the U-Net++ segmentation result, marking the locations of the inner and outer tunica of the vessel. As can be seen, the U-Net++ network clearly identifies the boundary between the inner and outer tunica, achieving accurate segmentation of the vascular structure. This demonstrates the superior performance of U-Net++ in multimodal image segmentation tasks, effectively capturing important structural features in IVUS images.

[0101] Figure 10 The changing trends of the loss function and mean intersection-over-union (mIoU) during training are shown. Figure 10 (a) shows the convergence curves of the training and validation losses. As can be seen, both the training and validation losses decrease rapidly and gradually stabilize during training, indicating that the model converges well on this task. Furthermore, the validation loss stabilizes in the later stages of training and is relatively close to the training loss, indicating that the model has good generalization capabilities. Figure 10(b) shows how mIoU changes with the number of training iterations. mIoU increases rapidly initially and stabilizes at 97.88% around the 20th iteration. This result demonstrates that U-Net++ achieves high segmentation accuracy in this segmentation task and is able to effectively separate different tissue regions in IVUS images.

[0102] Step 4: Segment the vessel wall components using a K-means clustering algorithm with an increased memory factor. This algorithm successfully classifies plaques in IVUS images into four types: fibrous plaque, fibrofatty plaque, necrotic core, and calcification. This precise segmentation and classification provides a solid foundation for virtual histology studies, enabling the location and size of each plaque type to be clearly labeled, effectively supporting quantitative plaque analysis.

[0103] Compared to the classic K-means, the present invention uses k-means++ to initialize K-means for the first frame of image (when previous_centroids is None). For subsequent frames, the centroid position obtained by clustering the previous frame is used as the initialization centroid (i.e., when previous_centroids is not None), so that the K-means clustering algorithm produces a "memory" feature. This method is particularly useful when processing continuous frames (such as video sequences) because it can maintain the consistency and coherence of the clustering results, thereby improving the stability of time series data. In addition, the K-means clustering in the present invention is limited to a specific area (i.e., region2_mask), which is different from the classic K-means, which usually performs unconstrained clustering on the entire data set. Such constraints help to focus on the key parts of the image and avoid the influence of irrelevant areas.

[0104] from Figure 11 As can be seen, the K-means clustering algorithm with the addition of a memory mechanism successfully classified the plaques in the IVUS images into four types: fibrous plaques (green), fibrofatty plaques (yellow), necrotic cores (red), and calcifications (white). This preliminary classification provides strong support for the study of virtual histology. Quantitative evaluation indicators show that the silhouette coefficient (Silhouette Score) of this method is 0.8654, indicating that the separation between the cluster categories is high and the internal compactness is good. The Davies-Bouldin index is 0.2376, indicating that the similarity between the categories is low, that is, the clustering effect is good. These results show that the K-means clustering algorithm based on feature selection and memory mechanism optimization can achieve relatively accurate and stable classification results in virtual histology, providing a reliable foundation for further lesion analysis and clinical application.

Claims

1. A deep learning-based automatic segmentation method for intravascular lipid plaques, characterized by The method comprises the following steps: Step 1: Obtain ultrasound and photoacoustic vascular image datasets for training and testing; Step 2: Perform standardization and normalization preprocessing operations on the original vascular image; Step 3: Improve the U-Net++ network and build an improved multi-layer U-Net++ network architecture: The classic U-Net++ network structure is used as the initial network structure, which contains a nested U-shaped structure and dense connections. On this basis, the MLP module and KAN module are introduced to optimize the U-Net++ network structure. Step 4: Train the multi-layer U-Net++ network constructed in step 3 on the training set and update the parameters; Step 5: Run the network trained in step 4 on the validation set. Based on the deep supervision operation, select the network with the highest evaluation index as the segmentation network. Step 6: Run the optimal network selected in step 5 on the test set to perform image segmentation, and prune each layer of the network based on its performance on the test set; Step 7: Apply the final improved U-Net++ model obtained in step 6 to segment the newly acquired ultrasound and photoacoustic vascular images, clearly marking the locations of the inner and outer membranes of the blood vessels; Step 8: Use the K-means clustering method with an increased memory factor to segment the vascular wall components. The specific steps are as follows: Step 8-1: Initialize previous_centroids to None and use the default initialization method for the first clustering; Step 8-2: Define the memory factor and introduce a variable memory_factor. If memory_factor = 0, it means that historical information is not considered at all, while memory_factor = 1 means that historical information is completely relied upon. Step 8-3: Modify the apply_kmeans function to add the effect of the memory factor when calculating the new cluster center. In the main loop, when processing each picture, call the apply_kmeans function with the memory factor and pass the previous_centroids and memory_factor parameters. Update previous_centroids to the new cluster center of the current frame to affect the clustering process of the next frame. Step 8-4: Perform segmentation on each frame image, extract the region of interest, apply K-means clustering with a memory factor, map the clustering results to colors and generate a color image, and add each processed image to the video stream. After completing the processing of all frames, release the video writing object and save the final output video.

2. The deep learning-based automatic segmentation method for intravascular lipid plaques according to claim 1 is characterized in that The specific steps of step 1 are as follows: Step 1-1: Collect ultrasound and photoacoustic images of lipid plaque vessels, ensuring coverage of different types of lipid plaques, including fibrous plaques, fibrofatty plaques, necrotic cores, and calcifications; Step 1-2: Remove irrelevant or low-quality data; Steps 1-3: Rotate, flip, and scale the images to increase the diversity of the dataset, and obtain an ultrasound and photoacoustic vascular image dataset containing different types of lipid plaques in blood vessels for training and testing.

3. The method for automatic segmentation of intravascular lipid plaques based on deep learning according to claim 1, characterized in that In step 2, the preprocessing operation includes resizing and noise removal steps.

4. The method for automatic segmentation of intravascular lipid plaques based on deep learning according to claim 1, characterized in that The specific steps of step 3 are as follows: Step 3-1: Based on the classic U-Net++ network structure The MLP module is introduced at the node and the input features Through the linear layer Perform linear transformation, and the obtained output is processed by nonlinear activation function. After multiple linear transformations and nonlinear activation function processing, the final feature representation is output; Step 3-2: The KAN module is introduced at the node, and the input feature is After the results of the MLP module processing, the feature map is converted into tokens through the Tokenization layer. The tokens are nonlinearly transformed through the KAN Layer. DwConv uses deep convolution to further process the feature map. The results are then normalized by the Layer Norm application layer to stabilize the training process, and finally the feature map processed by the KAN module is output.

Images