Intravascular ultrasound image segmentation reconstruction method based on multi-scale prior coordination mechanism

Abstract

The application discloses a kind of based on multiscale prior coordination mechanism intravascular ultrasound image segmentation reconstruction method.The application is as follows: collecting a variety of cardiovascular diseases IVUS dataset;IVUS dataset is divided into training set and test set;Multiscale prior coordination framework model is constructed, and the framework model specifically includes: prior correlation paradigm, hierarchical information enhancement module, adaptive fusion and upper and lower layer space coding module;Training set is preprocessed and data is enhanced;Input labeled training set to multiscale prior coordination framework model carries out IVUS image segmentation training;Adopt cross-entropy loss function, solve segmentation precision and class imbalance problem;The segmentation result is converted into three-dimensional model and is reconstructed to blood vessel, and generates blood vessel profile drawing.High-precision blood vessel structure segmentation result obtained using the application can generate accurate three-dimensional blood vessel model, and provide a intuitive three-dimensional view for cardiologist.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing technology and provides a method for segmenting and reconstructing intravascular ultrasound images based on a multi-scale prior coordination mechanism. The present invention uses a deep learning method to perform accurate segmentation and three-dimensional reconstruction of IVUS images. Background Art

[0002] With the development of medical imaging technology, intracardiac ultrasound (IVUS) is a medical imaging technique commonly used to assess lesions such as atherosclerotic plaques or vascular stenosis and is widely used in the assessment and treatment of coronary artery disease. However, distinguishing the vessel lumen and plaque areas from IVUS images remains a significant challenge. IVUS images are often affected by electronic noise, speckle noise, motion artifacts, and acoustic scattering, resulting in blurred images and difficulty in discerning details. Tiny bubbles or stray reflections in the blood can create bright spots in the image, interfering with the identification of plaques and vessel walls.

[0003] Especially at bends in blood vessels, uneven lighting and shadowing in IVUS images can obscure or exaggerate details in certain areas. The evolution from soft plaque to calcified plaque reflects the different stages of atherosclerosis. The varying stages of lesions create complex feature structures in IVUS images. Due to the diverse plaque types and complex vascular morphology, traditional network models struggle to capture certain advanced or complex features in IVUS images, resulting in numerous misclassifications and edge vanishing in segmentation results.

[0004] In addition, high-quality annotated IVUS data is very scarce and difficult to obtain, and manually annotating IVUS data is not only time-consuming but also labor-intensive. Relying solely on small data samples, the model may not be able to effectively learn sufficiently rich and robust feature representations, resulting in poor generalization of traditional models and poor segmentation accuracy. Therefore, factors such as low-quality images, insufficient capture of complex features, the influence of side branches, and the lack of high-quality annotated data have restricted the development of automated IVUS segmentation technology. The present invention was created in this context and aims to provide a novel multi-scale prior coordination framework to overcome the shortcomings of the existing technology and improve the accuracy and robustness of IVUS image segmentation. Summary of the Invention

[0005] In light of the shortcomings of existing technologies, this paper provides an intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism. This method aims to significantly improve the accuracy and efficiency of segmenting cardiovascular structures, such as vascular lumens and plaques, in IVUS images. Furthermore, based on these precise segmentation results, high-precision 3D reconstruction of cardiovascular structures can be performed.

[0006] The application innovatively integrates multi-scale feature extraction, deep learning algorithm and multi-dimensional information fusion technology, and overcomes the limitations of traditional segmentation technology in noise processing, anatomical difference adaptation, fine structure recognition and complex lesion state processing.

[0007] The technical scheme adopted by the application to solve the technical problems comprises the following steps:

[0008] S1: collecting IVUS data sets of various cardiovascular diseases;

[0009] S2: dividing the IVUS data set into a training set and a test set;

[0010] S3: constructing a multi-scale prior coordination framework model, which specifically comprises a prior correlation paradigm, a hierarchical information enhancement module, an adaptive fusion and an upper and lower layer space coding module;

[0011] 1.S4: preprocessing and data enhancement of the training set;

[0012] S5: inputting the labeled training set into the multi-scale prior coordination framework model for IVUS image segmentation training;

[0013] S6: using a cross-entropy loss function to solve the segmentation accuracy and class imbalance problem;

[0014] S7: on an independent test set, evaluating the model segmentation accuracy and robustness through Dice, IoU, MCC, G-Mean, Kappa and other indicators;

[0015] S8: converting the segmentation result into a three-dimensional model for blood vessel reconstruction, and generating a blood vessel profile graph from the original graph.

[0016] Further, in step S2, all IVUS images are uniformly scaled to 480*480 pixels to standardize the input data specifications.

[0017] Further, in step S3, the prior correlation paradigm comprises a prior encoder and a calibration module. The prior encoder relies on the trained weights for accurate feature extraction, and the calibration module performs feature matching in space-time dimensions according to the characteristics of the current task and data set.

[0018] A masked autoencoder (MAE) is used as the prior encoder. The MAE is based on the Transformer architecture and applies the principles of self-supervised learning. The MAE employs an asymmetric design, consisting of an encoder that extracts latent features from an image and a decoder that reconstructs the original image from these latent features. Both the encoder and decoder are visual transformers (ViTs), which divide the image into individual patches, randomly mask and remove some patches, and feed the remaining patches into a Transformer module for feature extraction. The decoder then fills in blank patches in the place of the removed patches, and then feeds the remaining patches into the Transformer module for image reconstruction. The MAE significantly reduces the computational requirements during training. This allows for the use of larger models or more efficient processing of larger datasets. Through the reconstruction task, the MAE forces the model to learn the intrinsic structure and characteristics of the input data, thereby improving data utilization efficiency and understanding depth.

[0019] The Pyramid Visual Transformer (PVT) is used as the calibration module. PVT designs a series of Transformer layers that increase the network width while generating multi-scale feature maps, which can be widely used in downstream tasks. This approach effectively enhances the model's ability to capture features at different scales and effectively addresses ViT's shortcomings in processing small objects or tasks that require fine feature analysis. PVT expands the application scope of ViT in the field of computer vision, especially in dense prediction tasks that traditionally rely on convolutional neural networks (CNNs). The spatial reduction attention mechanism introduced by PVT reduces computational complexity and memory usage by reducing the spatial dimensions of keys and values ​​in the attention mechanism. This enables PVT to process large-size images more efficiently. PVT uses dynamic position encoding, which can be dynamically adjusted according to the size of the input image, thereby improving the model's adaptability and generalization ability for different image sizes.

[0020] The proposed prior correlation paradigm addresses the challenges of the extreme scarcity of annotated datasets and the inadequate learning of fine-grained structures. Image mask reconstruction, contrastive learning, and other self-supervised methods are employed to train the prior encoder, integrating pre-trained high-level features and temporally coherent data into the framework. Furthermore, since the segmentation task relies on semantic information at different levels, the calibration module generates multi-scale feature maps and coordinates prior features obtained from different tasks based on the current task and dataset.

[0021] The layered information enhancement module is as follows:

[0022] The channel attention module and the spatial attention module are used to focus on the relevant features in the input image, giving the framework dynamic attention capabilities and attenuating less relevant features across spatial and channel dimensions.

[0023] The channel attention mechanism works by extracting channel-level statistical features using global average pooling and global maximum pooling. These two pooling operations each emphasize different feature responses. The feature maps generated by the two pooling operations are processed separately through a multi-layer perceptron (MLP) with shared weights. These are then added together and activated with a sigmoid function to generate a channel attention map. This channel attention map is multiplied channel by channel with the feature map output by the last layer of the prior correlation paradigm to enhance the features of important channels and suppress those of unimportant channels.

[0024] The spatial attention mechanism applies global average pooling and global max pooling to the feature map, but this time along the channel axis, generating two 2D feature maps (one obtained by average pooling and one by max pooling). These two feature maps are stacked across the channel and then passed through a convolutional layer to generate a spatial attention map. The output is processed through a sigmoid activation function to obtain the final spatial attention map.

[0025] The method further utilizes the correlation direction vectors between pixels to reconstruct a latent subspace from high-level features (feature maps output by the last layer of the prior correlation paradigm) with lower resolution and high-level semantic information. The correlation direction vectors between pixels are then extracted from the latent subspace. These correlation direction vectors are used to enhance the fine-grained structural features of vascular structures in IVUS images.

[0026] Through feature scaling and channel projection, the hierarchical information enhancement module re-encodes the fine-tuned features and extracts direction vectors from the latent subspace. These direction vectors encapsulate the contextual information of the input features, helping the framework to comprehensively interpret the global environment and high-level semantic features of the vascular structure in IVUS images.

[0027] The adaptive fusion and upper and lower layer spatial coding modules are as follows:

[0028] The adaptive fusion and upper-lower spatial encoding module consists of a lower spatial encoding layer and an adaptive fusion layer. These layers are designed to extract and re-encode spatial features from context-rich information, leveraging correlated scene and content features to promote a tighter integration of local and global information. They also implement cross-channel interaction of correlated scene features, improving the framework's generalizability and feature learning. A sophisticated spatial encoding strategy projects features into various directions and computes correlations between these projections, enhancing feature representation.

[0029] The adaptive fusion layer focuses on morphologically fusing the fused features and decoded features using deformable convolution and advanced pooling strategies. The deformable convolution layer in the adaptive fusion layer is good at adapting to the complex radial pattern characteristics of IVUS images.

[0030] Deformable convolution dynamically adjusts the position of each sampling point in the convolution kernel by introducing an additional offset, allowing the convolution kernel to deform freely according to the needs of the image content. Deformable convolution increases the network's adaptability to spatial transformations, especially in complex environments, such as image recognition and analysis of irregular objects and dynamic scenes.

[0031] In CNNs, a feature map consists of multiple channels, each representing a different type of feature response. Cross-channel interaction technology uses grouped convolution and shuffling operations to exchange and integrate information between channels, improving the model's overall representation and utilization efficiency of features.

[0032] Furthermore, in step S8, the three-dimensional reconstruction system generates depth estimation and constructs point clouds from the segmented IVUS images using deep learning technology, converts them into three-dimensional mesh models using Marching Cubes reconstruction technology, and enhances visual realism through texture mapping and advanced rendering technology.

[0033] Compared with the prior art, the present invention has the following significant advantages:

[0034] This multi-scale prior coordination framework integrates a prior correlation paradigm, a hierarchical information enhancement module, adaptive fusion, and upper and lower spatial encoding modules. The multi-scale prior coordination framework employs a prior coherence paradigm and trains the prior encoder via image mask reconstruction, contrastive learning, and other self-supervised methods. It integrates pre-trained high-level features and temporal coherence data into the framework, and uses a calibration module to generate multi-scale feature maps. The high-level feature representation is fine-tuned based on the task and dataset, enhancing the representation of fine-grained vascular structure in the feature maps. This module addresses the challenges of the extreme scarcity of annotated datasets and the inability of traditional networks to learn fine-grained structure. Upper-level information is lost during decoding, and key features are gradually attenuated in deep structures. Therefore, we introduce a hierarchical information enhancement module to extract global spatial context and enhance high-level semantic features of vascular structure in IVUS images, while attenuating less relevant elements across spatial and channel dimensions. To address the problems of insufficient utilization of coding features and mismatch of reconstruction information in existing decoders, we use adaptive fusion and upper and lower spatial coding modules in the decoding process. From the perspective of fusing global and local information and multi-scale features, we enhance local details in global features and enrich the reconstruction information. This improves the segmentation ability of fuzzy boundaries while taking into account the regional segmentation effect.

[0035] Furthermore, the present invention's 3D reconstruction technology further enhances its application value. Leveraging highly accurate vascular structure segmentation results, the present invention generates precise 3D vascular models, providing cardiologists with an intuitive 3D view. This not only assists physicians in plaque characterization and determining treatment strategies such as angioplasty or stent implantation, but also facilitates the study of vascular biomechanical properties, monitoring plaque changes over time, and assessing disease progression or plaque regression after treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a multi-scale prior coordination framework according to an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the segmentation of five types of labeled frames in the IVUS dataset according to an embodiment of the present invention;

[0038] Figure 3 This is an overall framework diagram of the multi-scale prior coordination framework in an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of a priori correlation paradigm in a multi-scale priori coordination framework in an embodiment of the present invention;

[0040] Figure 5 is a block diagram of a hierarchical information enhancement module in a multi-scale prior coordination framework according to an embodiment of the present invention;

[0041] Figure 6 4 is a framework diagram of the adaptive fusion and upper and lower layer spatial coding modules in the multi-scale prior coordination framework in an embodiment of the present invention;

[0042] Figure 7 This is a performance comparison of the multi-scale prior coordination framework of the embodiment of the present invention on the IVUS dataset with other existing methods;

[0043] Figure 8 Shows the segmentation diagram of each model for selected samples in the IVUS dataset;

[0044] Figure 9 The image shows the effect of converting the segmented image into a three-dimensional model. DETAILED DESCRIPTION

[0045] The present invention is further explained below with reference to the embodiments.

[0046] In particular, the present invention relates to a multi-scale prior coordination framework (PricoMS), which is designed to accurately identify and segment complex vascular structures such as vascular lumens and plaques, and to understand information such as intima-media thickness (IMT), blood flow velocity, degree of stenosis, pressure waveforms, and anatomical characteristics of bifurcations within the cardiac arteries to support the diagnosis and treatment of cardiovascular disease. Furthermore, the framework is capable of performing high-precision three-dimensional reconstruction of vascular structures based on these precise segmentation results. Three-dimensional reconstruction technology provides cardiologists with an intuitive three-dimensional view, helping them to gain a deeper understanding of the spatial distribution and pathological changes of cardiovascular structures, thereby improving diagnostic accuracy and treatment effectiveness.

[0047] like Figure 1 As shown, the present invention proposes a multi-scale prior coordination framework for segmenting IVUS images, which specifically includes the following steps:

[0048] S1: Collection of IVUS datasets for various cardiovascular diseases;

[0049] S2: Divide the IVUS dataset into training and testing sets;

[0050] S3: Construct a multi-scale prior coordination framework, which specifically includes: a prior correlation paradigm, a hierarchical information enhancement module, an adaptive fusion and upper and lower layer spatial coding module;

[0051] S4: Preprocessing and data enhancement of training set;

[0052] S5: Input the labeled training data to the multi-scale prior coordination framework for IVUS image segmentation training;

[0053] S6: Use cross entropy loss function to solve the problems of segmentation accuracy and category imbalance;

[0054] S7: On an independent test set, the segmentation accuracy and robustness of the model are evaluated using metrics such as Dice, IoU, MCC, G-Mean, and Kappa.

[0055] S8: Convert the segmentation results into a three-dimensional model for vascular reconstruction, and use the original image to generate a vascular cross-section.

[0056] For steps S1, S2, and S4:

[0057] This study used an IVUS dataset of 70 patients. Imaging was performed using an Infraredx 2.4F 50MHz Dualpro system, which captured IVUS video at 30 frames per second with a resolution of 480×480 pixels. The combined dataset included 197 vessels, yielding a total of 26,678 end-diastolic frames. These frames were meticulously annotated by four experts over a two-year period.

[0058] For our research, we compiled a dataset consisting of 3,500 labeled frames and an additional 10,000 unlabeled frames (a total of 13,500 frames) to aid in boundary segmentation, which requires analyzing adjacent frames. Our dataset ensures diversity by including training and test data from diverse patient populations. Evaluation of the training and test sets was performed by two experienced cardiologists, who annotated and assessed the patients' disease. Furthermore, a senior expert with over 2,000 days of experience selected the best annotations and manually refined them to ensure the highest quality and accuracy for our dataset analysis.

[0059] Specifically, the training set used 2,653 labeled frames, which were classified as follows: the first type of labeled frames consisted of 113 IVUS images with larger vessel radius; the second type of labeled frames consisted of 633 IVUS images with large shadows due to calcification; the third type of labeled frames consisted of 328 IVUS images with smaller shadows behind calcification; the fourth type of labeled frames consisted of 91 IVUS images with other side branch vascular structures; and the fifth type of labeled frames consisted of 1,488 IVUS images without the other four obvious features. Figure 2 As shown, the training set uses 1000 labeled frames.

[0060] A series of data augmentation operations, including random horizontal flipping, resizing, random cropping, normalization, and large-scale jittering, are performed on the training and validation sets to significantly enhance the diversity and complexity of the datasets. Dynamic combination transformations help the model gain better generalization capabilities during training, ensuring the consistency of input data and the effectiveness of model training.

[0061] Figure 3 The overall framework of the network model in the present invention is presented, and the following section will explain the framework in detail.

[0062] For the prior correlation paradigm, the schematic diagram is as follows Figure 4As shown, the unlabeled IVUS image is first randomly masked by the encoder so that the network can learn the features of the unmasked part. Subsequently, the decoder predicts the content of the masked area to complete the image reconstruction task. In view of the fact that the focus of the present invention is on image segmentation rather than reconstruction, the encoder and its training weights are retained. The retained encoder becomes the prior encoder. The prior encoder uses MAE to perform self-supervised tasks of image masking and reconstruction. The main component of the calibration module is the feature extractor, which consists of a certain number of convolutional layers and Transformer layers. A hierarchical structure is introduced to generate multi-scale feature maps, which is convenient for extracting fine-grained details and broader contextual information, while coordinating the prior features obtained from different tasks according to the current task and dataset. The present invention uses PVT as a calibration module.

[0063] The IVUS image with labels is input into the multi-scale prior coordination framework model and encoded through the prior correlation paradigm. The specific process is as follows: Input IVUS image X in ∈R H×W×3 After the pre-trained prior encoder, H, W, 3 are the length, width and number of channels of the IVUS image respectively. The IVUS image is decomposed into image patches, each of which is 16 × 16 × 3. The flattened image patches are then mapped into a linear space to generate a feature sequence of size Where C1 is the dimension of linear projection. Then, the linearly embedded feature sequence is added to the positional encoding and put into the Transformer layer with a layer number of d1. The feature is extracted through the self-attention mechanism to obtain the global feature as the final output of the prior encoder. The size of the global feature is still Reorganize the global features into a priori feature map e0∈R H×W×3 ,Then the prior feature map e0 is input into the calibration module.

[0064] Denote the encoder output at each stage in the calibration module as e n ,The calibration module has four encoding stages, and the encoding output of each stage is e2, e3, e4, and e5.

[0065] Perform convolution, batch normalization and activation operations on the prior feature map e0 to obtain the alignment map e1, which is reduced to

[0066] e1=δ(BN(W1·e0))

[0067] In this equation, δ represents the “ReLU” activation function, is a convolution kernel with a dimension of 7×7, a stride of 2, and a padding of 3. kIndicates the number of channels of the alignment graph e1, the size is 32; Represents the number of channels of the prior feature map e0, and “BN” is the batch normalization function.

[0068] The prior feature map e0 is then placed into a pyramid structure, passed through a Transformer module with layer l1, and the output shape is adjusted to Get the feature map e2.

[0069] By repeating the encoding strategy of the first stage in the calibration module, a series of feature maps are further generated using the feature map from the adjacent previous stage as input. The strides of the feature maps e3, e4 and e5 relative to the prior feature map e0 are 8, 16 and 32 pixels respectively, among which the feature map e5 is a high-level feature map; the multi-scale feature maps {e1, e2, e3, e4, e5} are applied to downstream segmentation tasks.

[0070] After the prior correlation paradigm operation is completed, the hierarchical information enhancement module is used to enhance the high-level feature map e5 obtained by the final encoder in the calibration module. This is because the high-level feature map e5 has a lower resolution and contains rich semantic information, which can promote the framework's comprehensive understanding of the global context. The specific operation of the hierarchical information enhancement module is as follows Figure 5 shown.

[0071] The high-level feature map e5 will be convolved, normalized and activated, and upsampled to the same size as the input IVUS image X in Size matching to obtain the latent space feature map This image contains a lot of overall spatial information of the image structure. Use 1×1 convolution kernel W2 for channel mapping and global average pooling to transform E pre Compress and finally extract the channel direction information v pre , v pre Essentially reflects the underlying vascular structure.

[0072] v pre =GAP(W2E pre )

[0073] Among them, GAP is global average pooling, convolution kernel Among them C N The size of is three times the number of output categories.

[0074] Then, without changing the size, the high-level feature map e5 is obtained by channel attention and spatial attention. This operation aims to highlight key features and reduce the correlation of edge elements in the spatial and channel dimensions, thereby improving the feature topology. Use 1×1 convolution Channel direction information v pre Re-encode the re-encoded channel direction information v pre Mapped to v enhance The same dimension. Then, the re-encoded channel direction information v pre Activate and normalize the projection vector through the sigmoid gate function σ:

[0075] α pre =σ(δ(W3v pre ))

[0076] α pre With v enhance Perform channel-level multiplication according to α pre The directional attributes contained in the convolution operation selectively enhance or weaken the features. The final stage involves using a 1×1 convolution kernel. Re-encode the output and combine it with the high-level feature map e5 residual:

[0077] e HCS =W4(α pre ·e enhance )+e5

[0078] Finally, the feature map e HCS Input to the decoder of the present invention, perform the final decoding operation to restore the input IVUS image X in The decoder uses adaptive fusion and upper and lower spatial coding modules. The specific process is as follows Figure 6 and Figure 2 As shown:

[0079] In the first stage, the feature map e HCS Upsampling is performed to make its size 4 times the original and the number of channels doubled, resulting in Subsequently, global average pooling is applied to d'4 to compress its embedding to produce After this, d'4 and are sent to the upper and lower spatial coding layers, and in the upper and lower spatial coding layers d'4 and After re-encoding to optimize the spatial characteristics of feature representation, d'4 is spatially encoded twice to obtain the associated scene features and content features Generated by compression After convolution and activation operations, a feature map focusing on spatial attributes is finally formed. Calculate the optimal k value to adjust The result after dimension adjustment is applied with one-dimensional convolution to reduce the number of channels, thereby promoting local cross-channel interaction and obtaining the final output calculate and The similarity between relation. Perform element-by-element multiplication to refine the features and finally derive the refined feature map d″4. The specific mathematical expressions of some parameters are as follows:

[0080]

[0081] Where w1∈R 1×1 , indicating that the size is k and the padding is set to The convolution kernel,

[0082] in Represents the number of channels of the prior feature map e5, squeeze and transpose operations to adjust the feature dimension to adapt to one-dimensional convolution, ⊙ represents element-by-element multiplication, and sum() represents summation over a specific dimension.

[0083] The refined feature map d″4 is fused with the feature map e4 of the same size in the encoder through a skip connection. The result of the skip connection is channel-compressed to obtain the final decoded feature d4.

[0084] Through the same decoding operation, the refined feature maps d″3, d″2, and d″1 are obtained, which are fused with the feature maps e3, e2, and e1 of the same size in the encoder through skip connections. The results of the skip connection are channel-compressed to obtain d3, d2, and d1, where d1 is the final decoded feature. At the same time, the decoded feature maps {d1, d2, d3, d4} are obtained.

[0085] The multi-scale feature map {e1, e2, e3, e4} obtained by the encoder is subjected to multi-scale feature fusion. The specific operation is: upsample {e2, e3, e4} to the same size and number of channels as e1, and merge all upsampled results with e1 to obtain the fused feature x1. Figure 3 As shown, the fused features x1 and d1 are passed through the adaptive fusion layer, the fused feature x1 is extracted through deformable convolution to obtain x'1 and averaged and max pooled. The two results after average and max pooling are dot-producted, and softmax is used to calculate the first attention map attention1:

[0086] attention1=Softmax(AvgPool(x′1)⊙MaxPool(x′1))

[0087] Multiply attention1 by d1 and reshape the merged feature map to generate out1.

[0088] At the same time, d1 will be re-encoded as d'1, and the vertical direction feature Pool(H,1) and the horizontal direction feature Pool(1,W) will be separated through the pooling operation in a specific direction. The features in the two orthogonal directions are multiplied to obtain the position-related attention map attention2:

[0089]

[0090] attention2=Softmax(Pool(1,W)(d′1)⊙Pool(H,1)(d′1))

[0091] Multiply attention2 by x1 to obtain out2. Finally, combine d1, x1, out1, and out2 to obtain the result feat, which is a fusion of multi-scale feature maps. Simply upsample the feat to restore it to the original input size and apply it to the subsequent segmentation task.

[0092] Furthermore, the framework of step S6 adopts a cross-loss entropy function, which can ensure the efficiency and performance of model learning while effectively addressing challenges in IVUS image analysis, such as category imbalance and detail segmentation accuracy.

[0093]

[0094] Where N is the number of samples, y i is the true label of sample i, is the model's predicted value for sample i.

[0095] Furthermore, in step S7, the model is used to perform segmentation on an independent test set, and the segmentation results are compared with the true labels. The evaluation indicators include Dice Coefficient (Dice), Intersection over Union (IoU), Matthews Correlation Coefficient (MCC), Geometric Mean (G-Mean), and Cohen's Kappa Coefficient (Kappa). The specific contents are as follows:

[0096] Dice is a statistical metric used to measure the similarity between two samples, especially for measuring the overlap between samples. In image segmentation, Dice is used to quantitatively evaluate the degree of overlap between predicted and true segmented regions.

[0097]

[0098] In this formula, S is the total number of categories, V pred.j is the region of the predicted class, Vtruth.j is the region of the real class.

[0099] IoU is a commonly used evaluation metric in image segmentation, which is used to quantify the ratio of the intersection and union between the predicted bounding box or segmentation area and the true bounding box or segmentation area.

[0100]

[0101] In this formula, |V pred.j ∪V truth.j ∣ represents j th The union of the predicted and true classes, i.e. the total number of unique elements in both sets.

[0102] MCC is a metric used to measure the quality of binary classification in image segmentation. It combines the true positive rate, false positive rate, true negative rate, and false negative rate. MCC is a value between -1 and +1 that measures the quality of the segmentation. +1 indicates a perfect segmentation, 0 indicates an average random segmentation, and -1 indicates a completely inconsistent segmentation.

[0103]

[0104] In this formula, for the jth th samples, TP j This is a true example, TN j It is a true negative example, FP j It is a false positive, FN j It is a false negative example.

[0105] G-Mean is a common statistical metric in image segmentation. It is used to measure the central tendency of images in a data set. It is the geometric mean of sensitivity and specificity and is used to measure the balanced performance of the model in processing positive and negative samples of the image.

[0106]

[0107] In this formula, for the jth th samples, TPR j This is a real example, TNR j j is a true negative example.

[0108] Kappa is a metric used to evaluate segmentation accuracy in image segmentation. It not only considers segmentation accuracy, but also factors such as random consistency and class imbalance, providing a more comprehensive performance evaluation method than traditional accuracy.

[0109]

[0110] In this formula, P o(j) is the relative observed agreement between raters, and P e (j) is the jth th The probability of the hypothesis that the samples are consistent by chance.

[0111] To verify the effectiveness of our proposed method, we conducted a comparative analysis of 13 different model methods in a uniformly controlled experimental setting, including UNet, DeepLab, ICNet, etc. We give the specific data of the model performance in Table 1. For data visualization and statistical analysis, Figure 7 In

[15] , we integrate the actual segmentation contours of the lumen and vessel regions with the predicted segmentation contours of each model onto the actual IVUS images.

[0112]

[0113] Table 1 Segmentation performance of each model for plaque and vascular lumen areas

[0114] Table 1 shows that our method outperforms the best model, DUNet, by 5.1% on Dice for plaque segmentation and DeepLab by 1.4% on Dice for vascular lumen segmentation. Some traditional models with symmetrical encoder-decoder structures suffer from a lack of multi-scale feature fusion and context loss. Our network, through upper and lower spatial encoding modules, implements spatially aware projection and re-encoding of spatial features, enhancing information reconstruction during decoding. Furthermore, through an adaptive fusion module, multi-scale features are incorporated throughout the entire process, enabling the interaction of global and deep structural information. Some networks perform poorly on small datasets such as medical images. Our network utilizes a prior encoder to achieve feature transfer and adaptation, thereby enhancing feature extraction. Some networks, while using rich attention mechanisms to capture broader context, experience performance degradation when processing scenes with complex textures or small objects. Our proposed hierarchical information enhancement module, through spatial resampling and re-encoding strategies, focuses on the hierarchical structure of the image, improving understanding of complex textures. The superior performance in Kappa and other data shows that the multi-scale prior coordination framework has high accuracy and consistency, and can effectively identify segmentation boundaries and correctly segment the lumen and plaque areas when processing unbalanced data sets.

[0115] Comprehensive data analysis revealed that the accuracy of the segmentation model for the plaque region was significantly lower than that for the vascular lumen. Plaques are composed of heterogeneous components such as lipids, cholesterol, calcification, and fibrous tissue, and exhibit rich reflective characteristics after onset. The acoustic shadowing and scattering effects of plaques in the blood can complicate the imaging of these regions. Therefore, achieving accurate segmentation of the plaque region is a major challenge and difficulty in IVUS image segmentation technology. The lumen region primarily contains blood, which is composed of red blood cells, white blood cells, platelets, and plasma. Its structural complexity is relatively low, and its contours are clearer, necessitating a deeper understanding of global and local fine-grained features. Improving the segmentation accuracy of the lumen region can further improve the accuracy of plaque region segmentation. Therefore, lumen segmentation is a key focus of IVUS image segmentation technology.

[0116] To demonstrate the key contributions of various components of our approach to enhanced segmentation performance, we conducted a series of ablation studies, as shown in Table 2. The experiments included variable control experiments on the prior encoder (a), calibration module (b), hierarchical information enhancement module (c), adaptive fusion, and upper and lower spatial encoding modules (d).

[0117]

[0118] Table 2 Segmentation performance of the framework after removing each module

[0119] The data presented in Table 2 demonstrate that the overall performance of the model is the result of the synergistic collaboration of these modules. Ignoring any module significantly reduces the model's segmentation ability. After removing the prior encoder, we observed a significant drop in segmentation performance due to the lack of prior features. Relying solely on a limited dataset of labeled IVUS images, the framework is unable to discern the complex, fine-grained structure of vascular structures in IVUS images. Removing the prior encoder resulted in a 6% reduction in the Dice score for plaque regions and a 1.7% reduction in the Dice score for lumen regions. This demonstrates that self-supervised learning can learn preliminary features of IVUS images through reconstruction tasks using unlabeled images. These preliminary features can be effectively applied to our segmentation task, improving the model's segmentation performance while reducing its reliance on labeled data. Replacing the calibration module with a traditional U-Net encoder architecture resulted in a 4.2% reduction in the Dice score for plaque regions and a 0.9% reduction in the Dice score for lumen regions. This demonstrates that without the calibration module's ability to match the features derived from the prior encoder, the features acquired from the large-scale image reconstruction task are not well suited for subsequent image segmentation tasks.

[0120] After removing the hierarchical information enhancement module, the segmentation Dice for plaque regions decreased by 2.6%, and for lumen regions decreased by 1.0%. Without the enhancement of potential subspace features, the network loses shallow semantic features and contextual information during the encoding process, which reduces the understanding of local information. At the same time, high-level semantic information is no longer adaptively adjusted by the channel attention and spatial attention modules, and the feature maps contain a large amount of features irrelevant to the fine-grained structure of blood vessels. The framework's ability to analyze complex and subtle structures, such as the fine structure of blood vessels, is reduced.

[0121] Without adaptive fusion and upper-lower spatial encoding modules, the framework can only rely on traditional upsampling and skip connections for decoding. During the decoding process, it is difficult to accurately recover detailed information through interpolation of surrounding pixels, often resulting in blurred edges and insufficient details. After removing the adaptive fusion and upper-lower spatial encoding modules, the segmentation Dice for plaque and lumen areas decreased by 4.3% and 1.4%, respectively. The deeper integration of local and global contextual cues can significantly improve the sensitivity of the segmentation model to subtle changes in IVUS images. The adaptive fusion module is a fusion process for multi-scale feature maps. Unlike traditional encoding-decoding structures, which only use decoded features, it ignores the rich features from the encoding process. The adaptive encoding module combines features from different resolutions during encoding and decoding, thereby improving segmentation accuracy.

[0122] Finally, to enhance the practical value of the research in clinical applications, this study used a multi-scale prior coordination framework to perform three-dimensional reconstruction of IVUS images. This technology generates three-dimensional models based on clinical needs, providing important support for doctors throughout the treatment process, including preoperative diagnosis, intraoperative decision-making, and postoperative effect monitoring, and has significant clinical significance.

[0123] In the specific practice of three-dimensional reconstruction, we selected three groups of IVUS image sequences for analysis, containing 110, 215, and 134 images, respectively. These three sequences are named sequence a, sequence b, and sequence c. Figure 9 The three-dimensional structure cross-sectional diagrams composed of the original images of these sequences are shown, as well as the corresponding three-dimensional vascular structure reconstruction results. Through these cross-sectional diagrams, we can clearly observe the outer contour, shape, and size of the corresponding part of the blood vessel, and detailedly display the specific situation of the vascular cross-section. This process not only helps doctors and researchers to deeply understand the pathological condition of the blood vessel from multiple angles and levels, but also enables doctors to obtain a panoramic view of the internal structure of the blood vessel through accurate segmentation of the sequence images and subsequent three-dimensional reconstruction.

Claims

1. An intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism, characterized in that: Collect IVUS datasets of various cardiovascular diseases; The IVUS dataset is divided into training set and test set; Constructing a multi-scale prior coordination framework model, the framework model specifically includes: a prior correlation paradigm, a hierarchical information enhancement module, an adaptive fusion and upper and lower layer spatial coding module; The a priori correlation paradigm includes a priori encoder and a calibration module. The priori encoder relies on trained weights for accurate feature extraction, and the calibration module is used for multi-scale feature map generation. Preprocess and enhance the training set; Input the labeled training set to the multi-scale prior coordination framework model for IVUS image segmentation training; input the labeled IVUS image to the multi-scale prior coordination framework model and perform encoding operation through the prior correlation paradigm. The specific process is as follows: Input IVUS image X in The pre-trained prior encoder is decomposed into multiple image blocks; the flattened image blocks are mapped to a linear space to generate a feature sequence; the linearly embedded feature sequence is added to the position code and put into the Transformer layer with a layer number of d1. The feature is extracted through the self-attention mechanism to obtain the global feature as the final output of the prior encoder. The global feature is reorganized into the prior feature map e0 and then input into the calibration module; the calibration module has a total of 4 encoding stages, and the encoding output of each encoding stage is d2, e3, e4 and e5 respectively; the prior feature map d0 is convolved The alignment map e1 is obtained by performing product, batch normalization, and activation operations, and the alignment map e1 is reduced. The prior feature map e0 is then placed in a pyramid structure, passed through a Transformer module with an l1 layer, and the output shape is adjusted to obtain the feature map e2. By repeating the encoding strategy of the first stage in the calibration module, the feature map from the adjacent previous stage is used as input to generate a series of feature maps: e3, e4, and e5, where the feature map e5 is a high-level feature map. The multi-scale feature maps {e1, e2, e3, e4, e5} are applied to the downstream segmentation task. Use cross entropy loss function to solve the problems of segmentation accuracy and category imbalance; The segmentation results are converted into a three-dimensional model for vascular reconstruction and a vascular cross-section image is generated.

2. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 1, characterized in that: The IVUS dataset is a collection of 3,500 labeled frames and an additional 10,000 unlabeled frames. The dataset ensures diversity by including training and test data from different patient groups. The training and test sets are professionally annotated and evaluated.

3. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 1, characterized in that: The training set was classified as follows: the first type of labeled frames were IVUS images with a vessel radius greater than the set threshold I; the second type of labeled frames were IVUS images with a large shadow area greater than the set threshold II due to the calcification area; the third type of labeled frames were IVUS images with a shadow area behind the calcification area less than the set threshold III; the fourth type of labeled frames were IVUS images with other side branch vascular structures; and the fifth type of labeled frames were IVUS images without the other four obvious features.

4. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 3, characterized in that: A mask autoencoder is used as the prior encoder; a pyramid visual transformer is used as the calibration module.

5. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 4, characterized in that: The hierarchical information enhancement module uses a channel attention module and a spatial attention module to focus on the relevant features of the input image; The channel attention mechanism works by using global average pooling and global maximum pooling to extract channel-level statistical features. The feature maps obtained by the two pooling are processed by a multi-layer perceptron with shared weights, and then added and activated by a sigmoid function to generate a channel attention map. This channel attention map is multiplied channel by channel with the feature map output by the last layer of the prior correlation paradigm to enhance the features of important channels. The spatial attention mechanism applies global average pooling and global maximum pooling to the feature map along the channel axis to generate two two-dimensional feature maps, stacks these two-dimensional feature maps on the channel axis, and then generates a spatial attention map through a convolutional layer. The output is processed by the sigmoid activation function to obtain the final spatial attention map.

6. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 5, characterized in that: The hierarchical information enhancement module is implemented as follows: the high-level feature map e5 is convolved, normalized, and activated, and upsampled to the same size as the input IVUS image X in Size matching to obtain the latent space feature map E pre , use convolution kernel W2 for channel mapping, and use global average pooling to pre Compress and finally extract the channel direction information v pre Without changing the size, the high-level feature map e5 is obtained by channel attention and spatial attention. enhance ; Use convolution kernel W3 to transform channel direction information v pre Re-encode the re-encoded channel direction information v pre Mapped to v enhance The same dimension; then the re-encoded channel direction information v pre Activate and normalize the projection vector α through the sigmoid gate function σ pre ; The projection vector α pre With v enhance Perform channel-level multiplication according to the projection vector α pre The directional attributes contained in it selectively strengthen or weaken the features; the final stage uses the convolution kernel W4 to pre With v enhance The multiplication result of re-encodes the output and combines the re-encoded result with the high-level feature map e5 residual to obtain the feature map e HCS .

7. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 6, characterized in that: The adaptive fusion and upper and lower spatial coding module consists of an upper and lower spatial coding layer and an adaptive fusion layer. The upper and lower spatial coding layers are designed to extract and re-encode spatial features from context-rich information, using associated scene features and content features to promote a tighter integration of local and global information. They also perform cross-channel interaction on associated scene features, projecting features into various directions through a complex spatial coding strategy and calculating the correlation between these projections to enhance feature representation. The adaptive fusion layer focuses on morphologically fusing fusion features and decoding features using deformable convolution and advanced pooling strategies. The deformable convolution layer in the adaptive fusion layer is good at adapting to the complex radial pattern characteristics of IVUS images. The deformable convolution dynamically adjusts the position of each sampling point in the convolution kernel by introducing an additional offset, so that the convolution kernel can be freely deformed according to the needs of the image content.

8. The intravascular ultrasound image segmentation and reconstruction method based on a multi-scale prior coordination mechanism according to claim 6, characterized in that: The specific implementation of the adaptive fusion and upper and lower layer spatial coding modules is as follows: The feature map e HCS Upsample to get d'4; apply global average pooling to d'4 to compress it and embed it to produce D'4 and are sent to the upper and lower spatial coding layers, where d'4 and After re-encoding to optimize the spatial characteristics of feature representation; d'4 is spatially encoded twice to obtain the associated scene features and content features Generated by compression After convolution and activation operations, a feature map focusing on spatial attributes is finally formed. Calculate the optimal k value to adjust The feature dimension of the resized result is applied with one-dimensional convolution to reduce the number of channels, thereby promoting local cross-channel interaction and obtaining the final output. calculate and The similarity between relation, relation and Perform element-by-element multiplication to refine the features and finally derive the refined feature map d″4; The refined feature map d″4 is fused with the feature map e4 of the same size in the encoder through a skip connection. The result of the skip connection is channel-compressed to obtain the final decoded feature d4. Through the same decoding operation, the refined feature maps d″3, d″2 and d″1 are obtained, which are fused with the feature maps e3, e2 and e1 of the same size in the encoder through jump connections respectively; the results of the jump connection are channel compressed to obtain d3, d2 and d1, where d1 is the final decoded feature; at the same time, the decoded feature maps {d1, d2, d3, d4} are obtained; The multi-scale feature map {e1, d2, e3, e4} obtained by the encoder is subjected to multi-scale feature fusion. The specific operation is as follows: {e2, e3, e4} is upsampled to the same size and number of channels as e1, and all upsampling results are merged with e1 to obtain the fusion feature x1; the fusion feature x1 and d1 are passed through the adaptive fusion layer, and the fusion feature x1 is extracted by deformable convolution to obtain x'1 and averaged and maximum pooled. The two results after average and maximum pooling are dot-producted, and softmax is used to calculate the first attention map attention1. The first attention map attention1 is combined with d1. Multiply and reshape the merged feature map to generate out1; at the same time, d1 will be re-encoded as d'1, and the vertical feature Pool(H,1) and the horizontal feature Pool(1,W) will be separated through the pooling operation in a specific direction. The features in the two orthogonal directions are multiplied to obtain the position-related attention map attention2, and the attention2 is multiplied with the fusion feature x1 to obtain out2; finally, d1, x1, out1 and out2 are combined to obtain the result feat that fuses the multi-scale feature map; feat is upsampled to restore it to the original input size and applied to subsequent segmentation tasks.

Images