A cross-modal distillation coupled method for coronary plaque classification

Abstract

This invention presents a cross-modal distillation-coupled coronary plaque classification method, aiming to address the limitations of CTA (Coronary Artery Therapy) due to its low spatial resolution and susceptibility to artifacts, while high-precision IVUS (Intravascular Ultrasound) requires inserting the probe into the vessel, potentially causing discomfort and complications for patients. To combine the advantages of both CTA and IVUS, this invention utilizes artificial intelligence to model the mapping relationship between CTA and IVUS. Through automatic delineation and identification of coronary vessels and plaques, the high-resolution characteristics of IVUS overcome the limitations of CTA, endowing CTA images with the discriminative capabilities of IVUS.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, deep learning, and computer vision content technology, and in particular to a coronary plaque classification method based on cross-modal distillation coupling. Background Technology

[0002] Accurate classification of coronary artery plaques is a crucial step in the diagnosis and treatment of cardiovascular diseases. Currently, clinical practice mainly relies on two imaging modalities: CT angiography (CTA) and intravascular ultrasound (IVUS). Traditional methods are mostly based on single-modal feature analysis. While CTA has the advantage of being non-invasive, it has inherent limitations such as insufficient spatial resolution and susceptibility to calcification artifacts, resulting in an accuracy rate of only 68-75% in identifying key features such as fibrous plaques and lipid cores. Although IVUS can provide high-resolution cross-sectional images down to 0.1 mm, it requires invasive catheter manipulation, carrying a 3-5% risk of vascular injury and raising concerns about patient tolerance.

[0003] Zreik et al. proposed a deep learning-based method for coronary plaque classification, utilizing CTA images for automatic plaque feature extraction and classification. This method extracts features from CTA images using a convolutional neural network (CNN) and combines this with clinical data for plaque classification, achieving good results. However, it does not fully utilize the high-resolution characteristics of IVUS. Due to insufficient spatial resolution and calcification artifacts in CTA images, the accuracy in identifying key features such as fibrous plaques and lipid cores is limited. Especially in complex lesion areas, traditional methods struggle to effectively separate anatomical structural features from artifact noise.

[0004] This invention presents a cross-modal distillation-coupled coronary plaque classification method, aiming to address the limitations of CTA (Coronary Artery Therapy) due to its low spatial resolution and susceptibility to artifacts, while high-precision IVUS (Intravascular Ultrasound) requires inserting the probe into the vessel, potentially causing discomfort and complications for patients. To combine the advantages of both CTA and IVUS, this invention utilizes artificial intelligence to model the mapping relationship between CTA and IVUS. Based on the automatic delineation of coronary vessels and plaques, the high-resolution characteristics of IVUS are leveraged to overcome the limitations of CTA, endowing CTA images with the discriminative capabilities of IVUS.

[0005] This invention innovatively constructs a cross-modal distillation coupling framework: First, an adaptive feature decoupling module separates the anatomical structural features and artifact noise of CTA images; then, a hierarchical knowledge distillation mechanism is designed to establish multi-scale feature mapping relationships using the high-resolution characteristics of IVUS; finally, a dynamic contrastive learning strategy is introduced to construct cross-modal positive and negative sample pairs in the latent space, and modality-invariant feature extraction is achieved through a learnable similarity metric function. This method overcomes the limitations of traditional single-modal analysis, significantly improves the plaque classification accuracy of CTA images, and maintains the non-invasive clinical advantage. Summary of the Invention

[0006] The purpose of this invention is to address the issue that coronary CTA has low spatial resolution and is susceptible to artifacts, while high-precision IVUS requires inserting a probe into the blood vessel, which may cause discomfort and potential complications for patients. This invention employs a cross-modal distillation coupling method to directly learn the discrimination capability of IVUS from CTA images and combines it with CTA clues to comprehensively assess and judge coronary plaques, thereby improving the accuracy of medical diagnosis.

[0007] The technical solution of the present invention is as follows:

[0008] A cross-modal distillation-coupled coronary plaque classification method, comprising the following steps:

[0009] Step 1: Extract image features from CT angiography (CTA) and intravascular ultrasound (IVUS) using a backbone network;

[0010] Given CTA modal input and IVUS modal input Backbone network extraction I CTA and I IVUS Feature map and Where N represents the number of CTA or IVUS images of the input sample, H and W represent the height and width of the image, respectively, and C represents the channel dimension of the image.

[0011] Step 2: Use the feature enhancement module to obtain the feature-enhanced representation vector, and use the cross-modal distillation module to realize the transfer of IVUS modal knowledge to CTA modality through feature alignment;

[0012] V CTA V IVUS The feature enhancement module Transformer1 then obtains the feature-enhanced representation vector V′. CTA and V′ IVUS V CTA V′ is learned through Transformer2 and the cross-modal distillation module. IVUS The discriminative clues in the data yielded V″. CTA .

[0013] During Transformer2 processing, a feature-based cross-modal loss function is used to achieve knowledge distillation from the IVUS modality to the CTA modality, minimizing the Euclidean distance between the enhanced feature vectors of these two modalities. The Euclidean distance expression is as follows:

[0014]

[0015] Step 3: Use a cross-modal coupling module to fuse multimodal features;

[0016] The cross-modal coupling module includes a cross-modal self-attention module and a cross-modal cross-attention module. The cross-modal self-attention module includes a normalized layer Norm, a multi-head self-attention network MHSA, a convolutional feedforward network Conv-FFN, and a connector. The cross-modal cross-attention module includes a normalized layer Norm, a multi-head cross-attention network MHCA, a coupling connection unit, a convolutional feedforward network Conv-FFN, and a connector.

[0017] The aforementioned cross-modal coupling module first performs reconstruction and self-reinforcement through two cross-modal self-attention modules, then enters a cross-modal cross-attention module for feature fusion, and finally outputs features V for classification. cls and V′ cls .

[0018] Furthermore, the specific process of the cross-modal self-attention module is expressed by the following formula: For input X, output Y can be obtained through the following formula:

[0019] X′=MHSA(Norm(X))+X

[0020] Y = Conv - FFN(Norm(X′)) + X′

[0021] Where X is the input V′ of the cross-modal self-attention module. CTA V′ IVUS and V″ CTA Let Y be the output of the cross-modal self-attention module, which acts on the cross-modal cross-attention module. Let the outputs of the cross-modal self-attention module be respectively. The output of the cross-modal self-attention module is set as follows:

[0022] Furthermore, the specific process of the cross-modal attention module is as follows: the cross-modal attention module includes two branches, each with its own data input, wherein... For the data input of a branch, or The data input for another branch is then processed through a normalization layer. The results are fed to the multi-head cross-attention layer of the current branch and the multi-head cross-attention layer of the other branch. The results from the multi-head cross-attention layers are concatenated with the input of the current branch. The result of the two concatenated branches is then fed into a coupling connection unit for coupling. The result of the coupling connection then passes through a normalization layer and a convolutional feedforward network before being concatenated with the result of the coupling connection. When the data input for the other branch is... At that time, the output of the cross-modal attention module is V′ cls When the data input of another branch is At that time, the output of the cross-modal attention module is V. cls .

[0023] Furthermore, the specific process of the coupling connection unit is as follows: first, splice V′ along the channel. CTA and V″ CTA Then, the channel attention mechanism is used to fuse the two branches to obtain the fused feature vector V. cls The channel attention mechanism is implemented using a multilayer perceptron containing two 1×1 convolutional layers. Its computation process is as follows:

[0024] V cls =MLP(Cat(V′) CTA ,V″ CTA ))·Cat(V′ CTA ,V″ CTA )

[0025] Cat(·) represents splicing along the channel, and MLP(·) represents a multilayer perceptron.

[0026] Similarly, V′ CTA and V′ IVUS The feature vector V′ is obtained by fusing in the same way. cls .

[0027] Step 4: Establish spatial consistency constraints to model the spatial context relationships between slices.

[0028] First, sample positive and negative sample pairs, then constrain V by the comparison loss function. cls and V′ cls The spatial semantic continuity ensures that the model's predictions satisfy the local continuity assumption. The expression for this loss function is as follows.

[0029]

[0030] in The feature vector at the current position. For positive samples of the same class as x, randomly sampled from the neighborhood of x in space. Negative samples that are randomly sampled from the neighborhood of x and are of a different class than x.

[0031] After semantic continuity calculation, the L2 loss function is used to assist in regularizing the feature vectors. The auxiliary loss is as follows.

[0032]

[0033] When y is a positive sample, b is 1; when y is a negative sample, b is 0. The auxiliary loss considers additional spatial information to prevent the model from overfitting. The total spatial consistency loss... for:

[0034]

[0035] Step 5: Classify using a constrained classification network;

[0036] The classification features V processed in step four cls V′ cls After a fully connected layer, cosine similarity is used to constrain V. cls and V′ cls Consistency, and apply cross-entropy loss respectively. and To regularize V cls and V′ cls This makes the predicted category approximate the true category. The cosine similarity formula is:

[0037]

[0038] The beneficial effects of this invention are as follows: This invention utilizes the high-resolution characteristics of IVUS to enhance the discriminative ability of CTA images through a cross-modal distillation coupling framework, overcoming the inherent defects of insufficient spatial resolution and artifact interference in CTA images, while avoiding the risks of vascular damage and patient discomfort associated with invasive IVUS procedures. Furthermore, this invention introduces a dynamic contrastive learning strategy, achieving modality-invariant feature extraction by constructing cross-modal coupled sample pairs. The introduction of spatial semantic continuity constraints and a contrastive learning loss function further enhances the model's ability to model local continuity and spatial context relationships, improving the robustness of classification results. This invention not only significantly improves the plaque classification accuracy of CTA images but also maintains the non-invasive clinical advantage, providing strong technical support for the precision diagnosis and treatment of cardiovascular diseases. Attached Figure Description

[0039] Figure 1 This is the overall flowchart.

[0040] Figure 2 This is a schematic diagram of a cross-modal coupling unit.

[0041] Figure 3 This is a schematic diagram of a coupled connection unit. Detailed Implementation

[0042] A cross-modal distillation-coupled coronary plaque classification method, comprising the following steps:

[0043] Network Framework Construction: The network framework of this invention consists of a dual-branch structure, processing CTA and IVUS modal inputs respectively. The backbone network uses a convolutional neural network (CNN) to extract initial features, followed by a feature enhancement module (Transformer1), a cross-modal distillation module, a cross-modal coupling module, and a constraint classification network. Specifically, the cross-modal distillation module achieves the transfer of IVUS modal knowledge to the CTA modality through feature alignment, the coupling module fuses multimodal features through an attention mechanism, and finally, patch classification is completed by combining spatial semantic continuity constraints and a classifier.

[0044] Step 1: Extract image features from CT angiography (CTA) and intravascular ultrasound (IVUS) using a backbone network;

[0045] Given CTA modal input and IVUS modal input Backbone network extraction I CTA and I IVUS Feature map and Where N represents the number of CTA or IVUS images of the input sample, H and W represent the height and width of the image, respectively, and C represents the channel dimension of the image.

[0046] Step 2: Obtain the enhanced representation vector using the feature enhancement module, and use the cross-modal distillation module to transfer IVUS modal knowledge to CTA modality through feature alignment.

[0047] V CTA V IVUS The feature enhancement module Transformer1 then obtains the feature-enhanced representation vector V′. CTA and V′ IVUS (Using the Transformer1 module to generate V′) CTA (Preserves the original information implied by the CTA mode); V CTA V′ is learned through Transformer2 and the cross-modal distillation module. IVUS The discriminative clues in the data yielded V″. CTA .

[0048] During Transformer2 processing, a feature-based cross-modal loss function is used to achieve knowledge distillation from the IVUS modality to the CTA modality, minimizing the Euclidean distance between the enhanced feature vectors of these two modalities. The Euclidean distance expression is as follows:

[0049]

[0050] Step 3: Use a cross-modal coupling module to fuse multimodal features

[0051] Because CTA and IVUS are modally heterogeneous, in order to make V′ CTA and V″ CTA To ensure compatibility of feature information between the two modes, this invention designs a cross-modal coupling mechanism to achieve the alignment and fusion of information from both modes, thereby obtaining classification features. Cross-modal coupling enables the organic combination of CTA modal information and information distilled from IVUS modal information, avoiding modal information conflicts and contamination caused by simple operations such as splicing.

[0052] The cross-modal coupling modules include a cross-modal self-attention module and a cross-modal cross-attention module. Both modules employ a multi-head design and use Conv-FFN instead of the fully connected layers in the attention modules. A coupling connection module replaces the cascading operations in the cross-attention module, resulting in better feature coupling. The cross-modal self-attention module includes a normalization layer (Norm), a multi-head self-attention network (MHSA), a convolutional feedforward network (Conv-FFN), and a connector. The cross-modal cross-attention module includes a normalization layer (Norm), a multi-head cross-attention network (MHCA), a coupling connection unit, a convolutional feedforward network (Conv-FFN), and a connector.

[0053] The aforementioned cross-modal coupling module first performs reconstruction and self-reinforcement through two cross-modal self-attention modules, then enters a cross-modal cross-attention module for feature fusion, and finally outputs features V for classification. cls and V′ cls .

[0054] Furthermore, the specific process of the cross-modal self-attention module is expressed by the following formula: For input X, output Y can be obtained through the following formula:

[0055] X′=MHSA(Norm(X))+X

[0056] Y = Conv - FFN(Norm(X′)) + X′

[0057] Where X is the input V′ of the cross-modal self-attention module. CTA V′ IVUS and V″CTA Let Y be the output of the cross-modal self-attention module, which acts on the cross-modal cross-attention module. Let the outputs of the cross-modal self-attention module be respectively. The output of the cross-modal self-attention module is set as follows:

[0058] While Transformers can model long-range dependencies well, local dependencies reveal more valuable information in this task. The fully connected portion of the Transformer only performs channel transformations on each element individually, lacking attention to the local environment. Therefore, this invention adjusts the FFN through convolution, replacing the original two fully connected layers with a 3×3 convolutional layer. A "3×3 convolutional layer - batch normalization - activation" approach is used to generate a FFN with the same dimensions as the original.

[0059] Furthermore, the specific process of the cross-modal attention module is as follows: the cross-modal attention module includes two branches, each with its own data input, wherein... For the data input of a branch, or The data input for another branch is then processed through a normalization layer. The results are fed to the multi-head cross-attention layer of the current branch and the multi-head cross-attention layer of the other branch. The results from the multi-head cross-attention layers are concatenated with the input of the current branch. The result of the two concatenated branches is then fed into a coupling connection unit for coupling. The result of the coupling connection then passes through a normalization layer and a convolutional feedforward network before being concatenated with the result of the coupling connection. When the data input for the other branch is... At that time, the output of the cross-modal attention module is V′ cls When the data input of another branch is At that time, the output of the cross-modal attention module is V. cls .

[0060] Furthermore, the specific process of the coupling connection unit is as follows: first, splice V′ along the channel. CTA and V″ CTA Then, the channel attention mechanism is used to fuse the two branches to obtain the fused feature vector V. cls The channel attention mechanism is implemented using a multilayer perceptron containing two 1×1 convolutional layers. Its computation process is as follows:

[0061] V cls =MLP(Cat(V′) CTA ,V″ CTA ))·Cat(V′ CTA ,V″ CTA )

[0062] Cat(·) represents splicing along the channel, and MLP(·) represents a multilayer perceptron.

[0063] Similarly, V′ CTA and V′ IVUS The feature vector V′ is obtained by fusing in the same way. cls .

[0064] Step 4: Establish spatial consistency constraints to model the spatial context relationships between slices.

[0065] Considering that plaque lesions along the blood vessel course occupy a certain spatial area, the prediction results of adjacent slices have continuity. Utilizing this prior knowledge helps improve the robustness of classification. This invention establishes spatial consistency constraints to model the spatial context relationship between slices. First, semantic continuity is calculated on the generated feature vectors. Specifically, positive and negative sample pairs are sampled first, and then V is constrained by a contrastive loss function. cls and V′ cls The spatial semantic continuity ensures that the model's predictions satisfy the local continuity assumption. The expression for this loss function is as follows.

[0066]

[0067] in The feature vector at the current position. For positive samples of the same class as x, randomly sampled from the neighborhood of x in space. Negative samples, randomly sampled from the neighborhood of x and belonging to a different class than x, are considered. This loss function brings positive samples closer to each other in the feature space while pushing away negative samples. After semantic continuity calculation, an L2 loss function is used to further regularize the feature vector. The auxiliary loss is as follows:

[0068]

[0069] When y is a positive sample, b is 1; when y is a negative sample, b is 0. The auxiliary loss considers additional spatial information to prevent the model from overfitting. The total spatial consistency loss... for:

[0070]

[0071] Step 5: Classify using a constrained classification network.

[0072] The classification features V processed in step four cls V′ cls After a fully connected layer, cosine similarity is used to constrain V. cls and V′ cls Consistency, and apply cross-entropy loss respectively. and To regularize V cls and V′ cls This makes the predicted category approximate the true category. The cosine similarity formula is:

[0073]

[0074] Experiment content:

[0075] This invention utilizes publicly available CTA and IVUS datasets, as well as data provided by collaborating hospitals. The CTA dataset includes 200 patients, with 25 CTA slices per patient, totaling approximately 5000 images; the IVUS dataset includes 150 patients, with 20 cross-sectional slices per patient, totaling 3000 images. All images have been annotated by professional physicians.

[0076]

[0077] In the experiment, the initial learning rate was set to 1e-4, the batch size was 16, the Adam optimizer was used, and the weight decay was 1e-5. Evaluation metrics included classification accuracy, precision, recall, F1 score, and AUC (area under the curve). Experimental results showed that the model achieved an overall accuracy of 89.5%, significantly higher than the 68%-75% of traditional unimodal methods; the AUC value improved to 0.93-0.94, better than the traditional 0.8-0.9. Simultaneously, the model maintained its advantage of non-invasive diagnosis.

Claims

1. A coronary plaque classification method based on cross-modal distillation coupling, characterized in that, The steps are as follows: Step 1: Extract image features from CT angiography (CTA) and intravascular ultrasound (IVUS) using a backbone network; Given CTA modal input and IVUS modal input Backbone network extraction I CTA and I IVUS Feature map and Where N represents the number of CTA or IVUS images of the input sample, H and W represent the height and width of the image, respectively, and C represents the channel dimension of the image; Step 2: Use the feature enhancement module to obtain the feature-enhanced representation vector, and use the cross-modal distillation module to realize the transfer of IVUS modal knowledge to CTA modality through feature alignment; V CTA V IVUS The feature enhancement module Transformer1 then obtains the feature-enhanced representation vector V′. CTA and V′ IVUS V CTA V′ is learned through Transformer2 and the cross-modal distillation module. IVUS The discriminative clues in the data yielded V″. CTA ; During Transformer2 processing, a feature-based cross-modal loss function is used to achieve knowledge distillation from the IVUS modality to the CTA modality, minimizing the Euclidean distance between the enhanced feature vectors of these two modalities. The Euclidean distance expression is as follows: Step 3: Use a cross-modal coupling module to fuse multimodal features; The cross-modal coupling module includes a cross-modal self-attention module and a cross-modal cross-attention module. The cross-modal self-attention module includes a normalized layer Norm, a multi-head self-attention network MHSA, a convolutional feedforward network Conv-FFN, and a connector. The cross-modal cross-attention module includes a normalized layer Norm, a multi-head cross-attention network MHCA, a coupling connection unit, a convolutional feedforward network Conv-FFN, and a connector. The aforementioned cross-modal coupling module first performs reconstruction and self-reinforcement through two cross-modal self-attention modules, then enters a cross-modal cross-attention module for feature fusion, and finally outputs features V for classification. cls and V′ cls ; Step 4: Establish spatial consistency constraints to model the spatial context relationships between slices; First, sample positive and negative sample pairs, then constrain V by the comparison loss function. cls and V′ cls The spatial semantic continuity ensures that the model's prediction results satisfy the local continuity assumption; the expression for this loss function is as follows. in The feature vector at the current position. For positive samples of the same class as x, randomly sampled from the neighborhood of x in space. Negative samples of different class than x, randomly sampled from the spatial neighborhood of x; After semantic continuity calculation, the L2 loss function is used to assist in regularizing the feature vectors. The auxiliary loss is as follows. b is 1 when y is a positive sample and b is 0 when y is a negative sample; the auxiliary loss considers additional spatial information to avoid overfitting of the model; the total spatial consistency loss... for: Step 5: Classify using a constrained classification network; The classification features V processed in step four cls V′ cls After a fully connected layer, cosine similarity is used to constrain V. cls and V′ cls Consistency, and apply cross-entropy loss respectively. and To regularize V cls and V′ cls This makes the predicted category approximate the true category; the cosine similarity formula is:

2. The coronary plaque classification method based on cross-modal distillation coupling according to claim 1, characterized in that, The specific process of the cross-modal self-attention module is expressed by the following formula: For input X, output Y can be obtained through the following formula: X′=MHSA(Norm(X))+X Y = Conv - FFN(Norm(X′)) + X′ Where X is the input V′ of the cross-modal self-attention module. CTA V′ IVUS and V″ CTA Y is the output of the cross-modal self-attention module, which acts on the cross-modal cross-attention module; let the outputs of the cross-modal self-attention module be respectively; the output of the cross-modal self-attention module is set as follows.

3. The coronary plaque classification method based on cross-modal distillation coupling according to claim 1, characterized in that, The specific process of the cross-modal attention module is as follows: The cross-modal attention module includes two branches, each with its own data input, wherein... For the data input of a branch, or The data input for another branch is processed through a normalization layer, and the results are then fed into the multi-head cross-attention layer of the current branch and the multi-head cross-attention layer of the other branch. The results from the multi-head cross-attention layers are concatenated with the input of the current branch. The result of the two concatenated branches is then fed into a coupling connection unit for coupling. The result of the coupling connection then passes through a normalization layer and a convolutional feedforward network before being concatenated with the result of the coupling connection. When the data input for the other branch is... At that time, the output of the cross-modal attention module is V′ cls When the data input of another branch is At that time, the output of the cross-modal attention module is V. cls .

4. The coronary plaque classification method based on cross-modal distillation coupling according to claim 3, characterized in that, The specific process of the coupling connection unit is as follows: first, splice V′ along the channel. CTA and V″ CTA Then, the channel attention mechanism is used to fuse the two branches to obtain the fused feature vector V. cls The channel attention mechanism is implemented using a multilayer perceptron containing two 1×1 convolutional layers, and its computation process is as follows: In cls =MLP(Cat(V′ CTA "In" CTA ))·Cat(V′ CTA "In" CTA ) Cat(·) represents splicing along the channel, and MLP(·) represents a multilayer perceptron. Similarly, V′ CTA and V′ IVUS The feature vector V′ is obtained by fusing in the same way. cls .

Images