Retinal vessel marker concept perception and multi-mode fusion coronary artery lesion detection method

Abstract

The invention discloses a coronary artery lesion detection method based on retinal vessel marker concept perception and multi-modal fusion, and relates to the problem that in the coronary artery lesion detection technology, the existing method is difficult to effectively utilize retinal vessel markers and realize multi-modal feature fusion detection. Coronary artery lesion is a main cardiovascular disease threatening human health, and early diagnosis is of great significance. Traditional detection depends on contrast examination and has the limitations of trauma, high cost and the like. The retinal blood vessel reflects the microcirculation state of the whole body, and a new way is provided for non-invasive detection by combining clinical indexes. The invention provides a coronary artery lesion detection method based on fusion of retinal vascular marker concept perception and multi-mode. The method has the advantages that (1) markers such as blood vessel density and curvature are calculated, and retinal blood vessel morphology is quantified; and (2) image features and clinical features are fused through a cross-modal attention aggregation mechanism, so that coronary artery disease diagnosis performance is improved. The method can be used for auxiliary detection of coronary artery lesions.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, specifically to a method for detecting coronary artery lesions based on concept perception and multimodal fusion of retinal vascular markers. Background Technology

[0002] Coronary artery disease is a significant manifestation of cardiovascular disease and a leading cause of morbidity and mortality worldwide. With an aging population and changing lifestyles, the prevalence of coronary artery disease is increasing annually, placing a heavy burden on patients' quality of life and societal healthcare resources. Clinically, the diagnosis of coronary artery disease typically relies on coronary angiography. While accurate, this method is highly invasive, costly, and carries significant risks, making it unsuitable for large-scale screening.

[0003] In recent years, studies have shown that retinal vessels can reflect the state of the systemic microcirculation, and their morphological characteristics are closely related to coronary artery lesions. Therefore, non-invasive detection methods for coronary artery lesions based on retinal images have gradually become a research hotspot. Common retinal vascular markers include vessel density, vessel tortuosity, and the number of vessel branches and intersections, which can provide a reference for risk assessment of coronary artery lesions. However, existing methods still have limitations in utilizing vascular markers, failing to fully model the semantic information of vascular morphological changes, resulting in insufficient accuracy and interpretability of detection results. To address these issues, this paper proposes a coronary artery lesion detection method based on retinal vascular marker concept perception and multimodal fusion. This method first constructs a cross-sectional dataset of coronary artery lesions, collecting retinal fundus images and clinical information from selected patients, and establishing positive and negative groups based on coronary angiography results. Then, a trained retinal arteriovenous segmentation model is used to extract arteries and veins, and retinal vascular markers are calculated. Subsequently, a coronary artery lesion detection model with retinal vascular marker concept awareness is constructed. A marker concept awareness module generates learnable marker concept tokens to quantify vascular morphological changes, and a clinical feature representation module and a cross-modal attention aggregation module are combined to fuse vascular morphological features with clinical features. Finally, using the calculated vascular markers and the positive / negative results of coronary angiography as labels, the model is trained to convergence and applied to the non-invasive diagnosis of coronary artery lesions. This method can improve the accuracy and interpretability of detection, providing a new approach for early screening and clinical auxiliary diagnosis of coronary artery lesions. Summary of the Invention

[0004] The purpose of this invention is to address the problems of existing coronary artery disease diagnosis, which mainly relies on angiography, is invasive, and does not make sufficient use of retinal vascular morphological changes. The invention proposes a coronary artery disease detection method that integrates retinal vascular marker concept perception and multimodal fusion.

[0005] The aforementioned objectives are primarily achieved through the following technical solutions:

[0006] S1. Construct a cross-sectional dataset of coronary artery lesions. After screening patients, include those who meet the criteria in the dataset. Collect retinal fundus images of the patients and record their age, gender, blood pressure, smoking status, and body mass index (BMI). Based on the coronary angiography results, divide the patients into a positive coronary artery lesion group and a negative coronary artery lesion group. The steps are as follows:

[0007] (1) Screen patients and include those who meet the inclusion criteria into the coronary artery lesion cross-sectional dataset. Exclusion criteria include: patients with a previous confirmed diagnosis of type 1 diabetes, patients with liver or kidney disease or severe organic lesions, patients with impaired consciousness or mental disorders, patients with metabolic disorders and malignant tumors, pregnant women and women within 6 months postpartum, and patients with ophthalmic diseases that affect retinal images.

[0008] (2) Collect retinal fundus images of patients included in the dataset and record the patients' age, gender, blood pressure, smoking status, and body mass index (BMI);

[0009] (3) Perform coronary angiography on the patients, determine whether there are lesions in the coronary arteries based on the angiography results, and divide the patients into a positive coronary artery lesion group and a negative coronary artery lesion group.

[0010] S2. Construct and train a retinal artery and vein segmentation model. Use the trained retinal artery and vein segmentation model to segment the retinal arteries and veins in the coronary artery cross-section dataset, and calculate retinal vascular markers to quantify retinal vascular morphology. The steps are as follows:

[0011] (1) Construct a retinal arteriovenous vessel segmentation model, train the retinal arteriovenous vessel segmentation model using the retinal arteriovenous segmentation dataset, and segment the retinal arteriovenous vessels in the coronary artery cross-section dataset using the converged retinal arteriovenous vessel segmentation model.

[0012] (2) Calculate the arterial density and vein density using the segmented retinal arteriovenous vessels to quantify the sparsity of the retinal arteriovenous vessels, calculate the tortuosity of the vessels to quantify the tortuosity, and calculate the number of vessel intersections to quantify the complexity of vessel branches and intersections.

[0013] S3. Construct a coronary artery lesion detection model based on retinal vascular biomarker concept awareness, simultaneously achieving coronary artery disease diagnosis and retinal vascular biomarker prediction. This model uses a biomarker concept awareness module to construct learnable biomarker concept tokens to quantify retinal vascular morphological changes, a clinical feature representation module to learn clinical features, and a cross-modal attention aggregation module to fuse retinal vascular morphological features and clinical features, thereby achieving the diagnosis of coronary artery lesions. The steps are as follows:

[0014] (1) A coronary artery lesion detection model based on the concept perception of retinal vascular markers quantifies retinal vascular morphology through a biomarker concept perception module:

[0015] The biomarker concept perception module crops the input retinal fundus image into Each patch is mapped to generate a patch token. ,in D Indicates the embedding dimension. ;

[0016] Biomarker concept perception module definition A learnable biomarker token ,in D Representing the embedding dimension, the biomarker concept perception module will embed retinal vascular marker tokens. and patch token Input the retinal vascular marker feature extractor to learn the global representation of retinal vascular markers;

[0017] The biomarker concept perception module further mines fine-grained information in the patch token. Reshape as And the channel dimension is obtained by 1×1 convolution. intermediate features , respectively and biomarker tokens Max pooling and average pooling operations were performed and fused to obtain the regression results of the biomarkers, as shown in formula (1):

[0018] (1)

[0019] in Indicates the first c Regression results of biomarkers D Represents biomarker tokens Embedding dimension Indicates intermediate features. N Indicates the number of patch tokens. This indicates the operation of finding the extreme value. n Indicates an index. ;

[0020] (2) The clinical feature representation module of the retinal vascular marker concept-aware coronary artery lesion detection model adopts a dual-path coding structure, which represents the categorical and continuous variables of clinical indicators through class embedding and learnable affine transformation, respectively, and generates clinical feature tokens:

[0021] The clinical characteristic representation module categorizes patients' clinical indicators into categorical variables. and continuous variables ,in B Indicates batch size. and These are the dimensions of categorical and continuous variables, respectively.

[0022] For categorical variables, the clinical feature representation module converts them into sparse high-dimensional representations using one-hot encoding, and then transforms them into continuous high-dimensional representations using a learnable class embedding matrix, as shown in Equation (2):

[0023] (2)

[0024] in Represents the high-dimensional embedding of categorical variables. This indicates a one-hot encoding operation. For categorical variables of clinical indicators, This represents a learnable offset vector. Represents the category embedding matrix, This represents the total number of possible values ​​for a categorical variable.

[0025] For continuous variables, the clinical feature representation module projects them into an embedding dimension space consistent with that of the categorical variables through a learnable affine transformation, as shown in Equation (3):

[0026] (3)

[0027] in Representing high-dimensional embeddings of continuous variables, Representing continuous variables of clinical indicators, Represents a learnable affine transformation matrix. This represents the tensor product operation. Represents a learnable bias term;

[0028] The clinical feature representation module defines learnable clinical feature tokens and concatenates them with high-dimensional embeddings of clinical indicators into the clinical feature encoder to learn the features of clinical indicators. This process is shown in formula (4):

[0029] (4)

[0030] in Represents a learnable clinical feature token. and These represent the high-dimensional embeddings of categorical and continuous variables, respectively. This represents the clinical feature token output after learning. This indicates a splicing operation. Indicates a clinical feature encoder. Used to extract learned clinical feature tokens through matrix operations. ;

[0031] (3) The cross-modal attention aggregation module defines two learnable disease tokens, learns the feature representations of positive and negative coronary artery lesions respectively, and fuses the disease tokens with retinal vascular marker tokens and clinical feature tokens to output the diagnosis results of coronary artery lesions.

[0032] S4. Using the calculated positive and negative results of retinal vascular markers and coronary artery lesions as labels, train the retinal vascular marker-based coronary artery lesion detection model until convergence, and apply it to the diagnosis of coronary artery lesions after convergence. The steps are as follows:

[0033] (1) The concept-aware coronary artery lesion detection model based on retinal vascular biomarkers uses calculated retinal vascular biomarkers as labels to optimize the model's ability to predict retinal vascular biomarkers. The biomarker prediction uses a smoothed L1 loss function, as shown in formula (5):

[0034] (5)

[0035] in This indicates that retinal vascular markers predict loss. I Represents the total number of samples. Indicates the predicted number of biomarkers. Indicates sample i The l The loss is predicted by a biomarker, as shown in formula (6):

[0036] (6)

[0037] in Indicates sample i The l Biomarkers predict loss. Indicates sample i No.l True values ​​of biomarkers Represents the sample i No. l Predicted values ​​of biomarkers;

[0038] (2) The concept-aware coronary artery lesion detection model based on retinal vascular markers uses binary cross-entropy as the loss function for coronary artery diagnosis, as shown in formula (7):

[0039] (7)

[0040] in Indicates coronary artery diagnostic loss. I Represents the total number of samples. Represents the true value of coronary artery lesions. This indicates the prediction results for coronary artery lesions;

[0041] (3) The loss function of the coronary artery lesion detection model based on the concept perception of retinal vascular markers is the sum of the coronary artery diagnosis loss and the biomarker prediction loss, as shown in formula (8):

[0042] (8)

[0043] in This represents the total loss of the model. This indicates a loss of diagnostic capability in coronary artery disease. This indicates that retinal vascular markers predict loss.

[0044] (4) After the coronary artery lesion detection model based on the concept perception of retinal vascular markers is trained and converged, the patient's retinal fundus images and clinical indicators are input for the diagnosis of coronary artery lesions.

[0045] Invention Effects

[0046] This invention provides a method for detecting coronary artery lesions based on retinal vascular marker concept perception and multimodal fusion. This method first segments arteries and veins in fundus images based on a retinal arteriovenous segmentation model and calculates vascular markers such as arterial density, vein density, vascular tortuosity, and the number of vascular intersections. Then, a learnable marker concept token is constructed through a biomarker concept perception module to quantify changes in retinal vascular morphology. Next, a clinical feature representation module and a cross-modal attention aggregation module are combined to achieve deep fusion of retinal vascular features and clinical features, simultaneously predicting vascular markers and diagnosing coronary artery lesions. Finally, the positive and negative results of retinal vascular markers and coronary angiography are used as labels to train the detection model until convergence, and it is applied to the auxiliary diagnosis of coronary artery lesions. Experiments show that this method has the following advantages: (1) It effectively captures changes in retinal vascular morphology through the marker concept perception module, improving the accuracy of coronary artery lesion diagnosis; (2) It effectively fuses retinal vascular features and clinical features through a cross-modal attention aggregation mechanism, enhancing the interpretability of the model; (3) This method provides a non-invasive diagnostic approach for coronary artery lesions, reducing the burden on patients. Attached Figure Description

[0047] Figure 1 This is a flowchart of the coronary artery lesion detection method based on retinal vascular marker concept perception and multimodal fusion in an example of the present invention.

[0048] Figure 2 This is a structural diagram of a coronary artery lesion detection model based on the concept of retinal vascular markers in an example of the present invention.

[0049] Figure 3 This is a flowchart of the high-dimensional embedding of categorical variables in clinical indicators in an example of the present invention;

[0050] Figure 4 This is a flowchart of high-dimensional embedding of continuous variables in clinical indicators in an example of the present invention. Specific implementation methods Specific implementation method one:

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1As shown, the method for detecting coronary artery lesions based on retinal vascular marker concept perception and multimodal fusion includes the following steps:

[0054] S1. Construct a cross-sectional dataset of coronary artery lesions, include patients who meet the inclusion criteria into the dataset, collect images and clinical information from the included patients, acquire retinal fundus images using standardized fundus photography equipment, and record patients' clinical indicators, including age, gender, blood pressure, body mass index (BMI), and whether they smoke. Perform coronary angiography on all patients, and have experienced cardiologists interpret the angiography results. Based on the angiography results, determine whether patients have coronary artery stenosis or occlusion, and divide patients into a positive coronary artery lesion group and a negative coronary artery lesion group.

[0055] S2. Construct and train a retinal arteriovenous vessel segmentation model. The model uses Attention U-Net as the backbone network and the publicly available arteriovenous segmentation dataset DRIVE is used to train the retinal arteriovenous vessel segmentation model to convergence. The trained retinal arteriovenous vessel segmentation model is used to segment the retinal arteriovenous vessels in the coronary artery cross-section dataset, and retinal vessel markers are calculated to quantify the retinal vessel morphology.

[0056] S3. Construct a coronary artery lesion detection model based on retinal vascular biomarker concept perception. This model can simultaneously diagnose coronary artery lesions and predict retinal vascular biomarkers. The model includes a biomarker concept perception module, a clinical feature representation module, and a cross-modal attention aggregation module. The biomarker concept perception module constructs learnable biomarker concept tokens to quantitatively represent changes in vascular morphology. The clinical feature representation module performs vectorized modeling of patients' clinical indicators. The cross-modal attention aggregation module integrates vascular features and clinical features to output the diagnostic results of coronary artery lesions.

[0057] S4. The positive and negative results of the calculated retinal vascular biomarkers and coronary artery lesions are used as training labels to train the coronary artery lesion detection model based on the concept perception of retinal vascular biomarkers. During the training process, the model is optimized using the corresponding loss function and the parameters are continuously updated through backpropagation until the convergence criterion is reached on the validation set. After the model training converges, the patient's retinal fundus image and clinical feature information are input. The model can simultaneously output the predicted values ​​of retinal vascular biomarkers and provide the diagnostic results of coronary artery lesions, realizing non-invasive detection of coronary artery lesions.

[0058] The embodiments of the present invention will be described in detail below:

[0059] The specific implementation of this invention is as follows.

[0060] S1, such as Figure 1 As shown, a cross-sectional dataset of coronary artery lesions was constructed. Patients meeting the inclusion criteria were included in the dataset. Imaging and clinical information were collected from the included patients. Retinal fundus images were acquired using standardized fundus photography equipment. Simultaneously, patients' clinical indicators, including age, sex, blood pressure, body mass index (BMI), and smoking status, were recorded. All patients underwent coronary angiography, and the results were interpreted by experienced cardiologists. Based on the angiography results, the presence of coronary artery stenosis or occlusion was determined, and patients were divided into a positive coronary artery lesion group and a negative coronary artery lesion group. The steps are as follows:

[0061] (1) Initial screening of subjects was conducted, and patients who met the inclusion criteria were included in the coronary artery lesion cross-sectional dataset. Patients who did not meet the requirements were excluded during the inclusion process. The exclusion criteria included: patients with a previous confirmed diagnosis of type 1 diabetes, patients with liver or kidney disease or serious organic lesions, patients with impaired consciousness or mental disorders, patients with metabolic disorders or malignant tumors, pregnant women and women within 6 months postpartum, and patients with ophthalmic diseases such as glaucoma, macular degeneration, and severe cataracts that affect the quality of retinal images.

[0062] (2) For patients who are selected and included in the dataset, retinal fundus images are collected using a standardized fundus camera with a resolution of not less than 2000×2000 pixels to ensure clear visibility of vascular details. At the same time, the patient's basic clinical information is recorded, including age, gender, blood pressure, smoking status and body mass index (BMI).

[0063] (3) For patients whose images and clinical information have been collected, coronary angiography is performed. Experienced cardiovascular specialists interpret the angiography results and divide the patients into a positive coronary artery disease group and a negative coronary artery disease group based on whether there are stenosis, occlusion or other lesions in the coronary arteries.

[0064] S2, such as Figure 1 As shown, a retinal arteriovenous segmentation model was constructed and trained. This model uses AttentionU-Net as the backbone network and is trained to convergence using the publicly available arteriovenous segmentation dataset DRIVE. The trained retinal arteriovenous segmentation model is then used to segment the retinal arteries and veins from the coronary artery cross-section dataset, and retinal vascular markers are calculated to quantify retinal vessel morphology. The steps are as follows:

[0065] (1) Construct a retinal arteriovenous segmentation model. The model uses Attention U-Net as the backbone network. The training data is selected from the publicly available DRIVE arteriovenous segmentation dataset. The Dice loss function is used to optimize the model parameters until the model achieves the required segmentation accuracy on the validation set and converges. After the model converges, the trained model is applied to the retinal images of the coronary artery cross-section dataset to obtain the segmentation results of the arteriovenous vessels.

[0066] (2) After obtaining the arterial and venous vessel segmentation results, retinal vascular markers are further calculated to quantify the morphological characteristics of the vessels. The calculation methods include: arterial density is the ratio of the number of arterial vessel pixels in the segmentation results to the total number of pixels in the whole image; venous density is defined as the ratio of the number of venous vessel pixels in the segmentation results to the total number of pixels in the whole image; vascular curvature is extracted through the vascular skeleton and the ratio of the length of the vascular curve to the straight distance between its two endpoints is calculated; the number of vascular intersections is based on the vascular skeletonization results and the number of bifurcation points and intersections is detected.

[0067] S3, such as Figure 2 As shown, a coronary artery disease detection model based on retinal vascular biomarker concept awareness is constructed. This model can simultaneously diagnose coronary artery disease and predict retinal vascular biomarkers. The model includes a biomarker concept awareness module, a clinical feature representation module, and a cross-modal attention aggregation module. The biomarker concept awareness module constructs learnable biomarker concept tokens to quantitatively represent changes in vascular morphology. The clinical feature representation module performs vectorized modeling of patients' clinical indicators. The cross-modal attention aggregation module fuses vascular features and clinical features to output the diagnostic results of coronary artery disease. The steps are as follows:

[0068] (1) The biomarker concept perception module crops the input retinal fundus image into Each patch is mapped to generate a patch token. The embedding dimension is 1024. ;

[0069] The biomarker concept awareness module defines four learnable biomarker tokens. Representing arterial density, venous density, vascular tortuosity, and number of intersections, respectively, with an embedding dimension of 1024, the biomarker concept perception module will use retinal vascular marker tokens. and patch token In the input retinal vascular marker feature extractor, the retinal vascular marker feature extractor uses Vision Transformer Large as the backbone network. Vision Transformer is a visual model based on the Transformer architecture. Its core idea is to divide the image into fixed-size patches and map each patch into a vector sequence input into the Transformer. The global relationship between patches is modeled through a self-attention mechanism.

[0070] The biomarker concept awareness module further mines fine-grained information from the patch token:

[0071] Patch Token Reshape as And intermediate features with a channel dimension of 4 are obtained through 1×1 convolution. , respectively and biomarker tokens Max pooling and average pooling operations were performed and fused to obtain the regression results of the biomarkers, as shown in formula (1):

[0072] (1)

[0073] in Indicates the first c Regression results of biomarkers Indicates intermediate features. N Indicates the number of patch tokens. This indicates the operation of finding the extreme value. n Indicates an index. ;

[0074] (2) The clinical feature representation module of the retinal vascular marker concept-aware coronary artery lesion detection model adopts a dual-path coding structure, which represents the categorical and continuous variables of clinical indicators through class embedding and learnable affine transformation, respectively, and generates clinical feature tokens:

[0075] The clinical characteristic representation module categorizes patients' clinical indicators into categorical variables. and continuous variables ,in B This indicates the batch size. Categorical variables include gender and smoking status, while continuous variables include age, blood pressure, and body mass index (BMI).

[0076] like Figure 3 As shown, for categorical variables, the clinical feature representation module converts them into sparse high-dimensional representations using one-hot encoding, and then transforms them into continuous high-dimensional representations using a learnable categorical embedding matrix, as shown in Equation (2):

[0077] (2)

[0078] in Represents the high-dimensional embedding of categorical variables. This indicates a one-hot encoding operation. For categorical variables of clinical indicators, This represents a learnable offset vector. This represents the category embedding matrix, where 4 represents the total number of values ​​for the category variables;

[0079] like Figure 4 As shown, for continuous variables, the clinical feature representation module projects them into an embedding dimension space consistent with that of categorical variables through a learnable affine transformation, as follows:

[0080] (3)

[0081] in Representing high-dimensional embeddings of continuous variables, Representing continuous variables of clinical indicators, Represents a learnable affine transformation matrix. This represents the tensor product operation. Represents a learnable bias term;

[0082] The clinical feature representation module defines learnable clinical feature tokens and concatenates them with high-dimensional embeddings of clinical indicators into the clinical feature encoder to learn the features of clinical indicators. This process is shown in formula (4):

[0083] (4)

[0084] in Represents a learnable clinical feature token. and These represent the high-dimensional embeddings of categorical and continuous variables, respectively. This represents the clinical feature token output after learning. This indicates a splicing operation. Indicates a clinical feature encoder. Used to extract learned clinical feature tokens through matrix operations. ;

[0085] (3) such as Figure 2 As shown, the cross-modal attention aggregation module constructs two learnable disease tokens. The module learns feature representations for positive and negative coronary artery lesions, respectively. The cross-modal attention aggregation module then learns clinical feature tokens. splicing with biomarker tokens It interacts with the disease token through cross-attention, and the Query, Key, and Value matrix is ​​constructed as follows:

[0086] (5)

[0087] in , and These represent the learnable Query, Key, and Value mapping matrices, respectively. , and These represent biomarkers, clinical features, and disease tokens, respectively. , and These represent the Query, Key, and Value matrices, respectively.

[0088] An 8-head attention mechanism is used to model cross-modal information interactions, as shown below:

[0089] (6)

[0090] in Represents the attention weight matrix. , and These represent the Query, Key, and Value matrices, respectively.

[0091] Using attention weights For the Value matrix Weighted aggregation is performed, as shown in formula (7):

[0092] (7)

[0093] in This represents the feature matrix of the weighted fusion. Represents the attention weight matrix. Represents the Value matrix;

[0094] The weighted fused feature matrix is ​​mapped to the output dimension by a linear projection matrix, and a global average pooling operation is performed to obtain the coronary artery lesion prediction result. This process is shown in Equation (8):

[0095] (8)

[0096] in This indicates the prediction results of coronary artery lesions. For the mapping matrix, This represents the feature matrix of the weighted fusion.

[0097] S4. Using the calculated positive and negative results of retinal vascular biomarkers and coronary artery lesions as training labels, train the coronary artery lesion detection model based on the concept of retinal vascular biomarkers. During training, the model is optimized using an appropriate loss function and its parameters are continuously updated through backpropagation until convergence is achieved on the validation set. After model training convergence, the patient's retinal fundus image and clinical feature information are input. The model can simultaneously output the predicted values ​​of retinal vascular biomarkers and provide the diagnostic results of coronary artery lesions, achieving non-invasive detection of coronary artery lesions. The steps are as follows:

[0098] (1) The concept-aware coronary artery lesion detection model based on retinal vascular biomarkers uses calculated retinal vascular biomarkers as labels to optimize the model's retinal vascular biomarker prediction ability. The biomarker prediction uses a smoothed L1 loss function, as shown in formula (9):

[0099] (9)

[0100] in This represents the prediction loss for retinal vascular biomarkers, where I represents the total number of samples. Let the predicted loss of the l-th biomarker in sample i be represented by formula (10):

[0101] (10)

[0102] in Indicates sample i The l Biomarkers predict loss. Indicates sample i No. l True values ​​of biomarkers Represents the sample i No. l Predicted values ​​of biomarkers;

[0103] (2) The concept-aware coronary artery lesion detection model based on retinal vascular markers uses binary cross-entropy as the loss function for coronary artery diagnosis, as shown in formula (11):

[0104] (11)

[0105] in Indicates coronary artery diagnostic loss. I Represents the total number of samples. Represents the true value of coronary artery lesions. This indicates the prediction results for coronary artery lesions;

[0106] (3) The loss function of the coronary artery lesion detection model based on the concept perception of retinal vascular markers is the sum of the coronary artery diagnosis loss and the biomarker prediction loss, as shown in formula (12):

[0107] (12)

[0108] in This represents the total loss of the model. This indicates a loss of diagnostic capability in coronary artery disease. This indicates the predicted loss of retinal vascular biomarkers;

[0109] (4) After the coronary artery lesion detection model based on the concept perception of retinal vascular markers is trained and converged, the patient's retinal fundus images and clinical indicators are input for the diagnosis of coronary artery lesions.

[0110] Table 1 compares the detection performance of the method of the present invention with other advanced methods on the coronary artery lesion cross-sectional dataset, and Table 2 compares the regression performance of the method of the present invention on the coronary artery lesion cross-sectional dataset. The results fully demonstrate the superior performance of the method of the present invention in the detection of coronary artery lesions.

[0111] Table 1 compares the performance of the proposed method with other advanced methods in detecting coronary artery lesions.

[0112]

[0113] Table 2 compares the regression performance of the proposed method for retinal vascular markers.

[0114]

Claims

1. A method for detecting coronary artery lesions based on the concept of retinal vascular markers perception and multi-modal fusion, characterized in that, Comprise the following steps: S1, build a coronary artery lesion cross-sectional data set, after screening the patients, the patients who meet the conditions are included in the data set, the retinal fundus images of the patients are collected, and the age, gender, blood pressure, smoking condition, body mass index BMI of the patients are recorded, the patients are divided into a coronary artery lesion positive group and a coronary artery lesion negative group through coronary angiography results; S2, build and train a retinal artery and vein vessel segmentation model, use the trained retinal artery and vein vessel segmentation model to segment the retinal artery and vein vessels of the coronary artery cross-sectional data set, and calculate the retinal vascular marker to quantify the retinal vascular morphology; S3, build a coronary artery lesion detection model with retinal vascular marker concept perception, realize the diagnosis of coronary artery and the prediction of retinal vascular marker at the same time, the model learns the biomarker concept Token to quantify the retinal vascular morphology change through the biomarker concept perception module, learns the clinical characteristics through the clinical characteristic representation module, realizes the fusion of retinal vascular morphology characteristics and clinical characteristics through the cross-modal attention aggregation module, and realizes the diagnosis of coronary artery lesion; S4, take the calculated retinal vascular marker and the positive and negative results of coronary artery lesion as labels, train the coronary artery lesion detection model with retinal vascular marker concept perception to convergence, and apply it to the diagnosis of coronary artery lesion after convergence. 2.The method of claim 1, wherein the method comprises: In step S1, the coronary artery lesion cross-sectional data set is built, the patients who meet the conditions are included in the data set after screening, the retinal fundus images of the patients are collected, and the age, gender, blood pressure, smoking condition, body mass index BMI of the patients are recorded, the patients are divided into a coronary artery lesion positive group and a coronary artery lesion negative group, which comprises the following steps: S11, screen the patients, and include the patients who meet the inclusion criteria in the coronary artery lesion cross-sectional data set, the exclusion conditions include: patients with a clear diagnosis of type 1 diabetes mellitus, patients with liver and kidney diseases or serious organic lesions, patients with unconsciousness or mental disorders, patients with metabolic disorders and malignant tumor diseases, pregnant women and women within 6 months after delivery, patients with eye diseases affecting retinal images; S12, collect the retinal fundus images of the patients included in the data set, and record the age, gender, blood pressure, smoking condition, body mass index BMI of the patients; S13, perform coronary angiography on the patients, judge whether the coronary artery has lesions according to the angiography results, and divide the patients into a coronary artery lesion positive group and a coronary artery lesion negative group. 3.The method of claim 1, wherein the method comprises: In step S2, the retinal artery and vein vessel segmentation model is built and trained, the trained retinal artery and vein vessel segmentation model is used to segment the retinal artery and vein vessels of the coronary artery cross-sectional data set, and the retinal vascular marker is calculated to quantify the retinal vascular morphology, which comprises the following steps: S21, build a retinal artery and vein vessel segmentation model, train the retinal artery and vein vessel segmentation model using the retinal artery and vein segmentation data set, and segment the retinal artery and vein vessels in the coronary artery cross-sectional data set using the converged retinal artery and vein vessel segmentation model; S22, calculate the arterial density and venous density of the segmented retinal arteriovenous vessels to quantify the degree of retinal arteriovenous sparsity, calculate the vessel tortuosity to quantify the tortuosity, and calculate the number of vessel intersection points to quantify the complexity of vessel branching and intersection. 4.The method of claim 1, wherein the method comprises: In step S3, the coronary artery lesion detection model with retinal vascular marker concept perception is constructed, and the diagnosis of the coronary artery and the prediction of the retinal vascular marker are realized. The model quantifies the changes in retinal vascular morphology through the biomarker concept perception module, learns the clinical features through the clinical feature representation module, and realizes the fusion of retinal vascular morphology features and clinical features through the cross-modal attention aggregation module. The diagnosis of coronary artery lesions includes the following steps: S31, the coronary artery lesion detection model with retinal vascular marker concept perception quantifies the retinal vascular morphology through the biomarker concept perception module: The biomarker concept perception module crops the input retinal fundus image into a patch, and generates patch Token by mapping the patch wherein D represents the embedding dimension, ; Biomarker concept perception module definition Individual learnable biomarker Token wherein D denotes the embedding dimension, the biomarker concept perception module will learn a global representation of the retinal vessel marker Token and the patch Token Input retinal vessel marker feature extractor, learn global representation of retinal vessel marker; The biomarker concept perception module further excavates fine-grained information in the patch Token, the patch Token is reshaped to , and an intermediate feature with a channel dimension of is obtained through a 1x1 convolution . Max-pooling and average-pooling operations are performed on the and biomarker Token respectively, and are fused to obtain the biomarker regression result, as shown in equation (1): (1) wherein represents the regression result of the c kind of biomarker, D represents the embedding dimension of the biomarker Token , represents the intermediate feature, N represents the number of patch Tokens, represents the operation of taking the extreme value, n represents the index, ; S32, the clinical feature representation module of the coronary artery lesion detection model with retinal vascular marker concept perception adopts a double-path encoding structure to represent the category variables and continuous variables of the clinical indicators through category embedding and learnable affine transformation, respectively, to generate clinical feature tokens: The clinical feature characterization module divides the clinical indicators of the patient into categorical variables and continuous variables wherein B denotes the batch size, and are the dimensions of the categorical and continuous variables, respectively; For category variables, the clinical feature representation module converts them into sparse high-dimensional representations through One-hot encoding, and uses a learnable category embedding matrix to convert them into continuous high-dimensional representations, as shown in equation (2): (2) wherein represents a high-dimensional embedding of a categorical variable, represents a one-hot encoding operation, is a categorical variable for a clinical indicator, represents a learnable offset vector, represents a categorical embedding matrix, represents a total number of values of a categorical variable; For continuous variables, the clinical feature representation module projects them into the embedding dimension space consistent with the category variables through a learnable affine transformation, as shown in equation (3): (3) wherein represents a high-dimensional embedding of continuous variables, represents a continuous variable of a clinical indicator, represents a learnable affine transformation matrix, represents a tensor product operation, represents a learnable bias term; The clinical feature representation module defines learnable clinical feature tokens and concatenates them with the high-dimensional embedding of the clinical indicators to input into the clinical feature encoder to learn the features of the clinical indicators, as shown in equation (4): (4) wherein denotes learnable clinical feature Tokens, with denotes high-dimensional embeddings of categorical and continuous variables, respectively, denotes output clinical feature Tokens after learning, denotes a concatenation operation, denotes a clinical feature encoder, for extracting learned clinical feature Tokens by matrix operations, ; S33, the cross-modal attention aggregation module defines two learnable disease tokens to learn the feature representations of coronary artery lesion positive and coronary artery negative, respectively, and fuses the disease tokens, retinal vascular marker tokens, and clinical feature tokens to output the coronary artery lesion diagnosis result. 5.The method of claim 1, wherein the method comprises: In step S4, the retinal vascular marker and the positive and negative results of coronary artery lesions calculated are used as labels to train the coronary artery lesion detection model with retinal vascular marker concept perception to convergence, and the model is applied to the diagnosis of coronary artery lesions after convergence, including the following steps: S41, the coronary artery lesion detection model with retinal vascular marker concept perception uses the calculated retinal vascular marker as a label to optimize the retinal vascular marker prediction ability of the model. The biomarker prediction uses a smooth L1 loss function, as shown in equation (5): (5) wherein denotes the retinal vessel marker prediction loss, I denotes the total number of samples, denotes the number of predicted biomarkers, denotes the sample i the l th biomarker prediction loss of the sample as shown in equation (6): (6) wherein denotes a sample i of the first l biomarker of the sample, denotes a sample i of the first l biomarker true value, denotes a sample i of the first l biomarker predicted value; S42, the coronary artery lesion detection model with retinal vascular marker concept perception uses binary cross-entropy as the loss function for coronary artery diagnosis, as shown in equation (7): (7) wherein denotes the coronary diagnostic loss, I denotes the total number of samples, denotes the coronary lesion ground truth, denotes the coronary lesion prediction result; S43, the loss function of the coronary artery lesion detection model with retinal vascular marker concept perception is the sum of the coronary artery diagnosis loss and the biomarker prediction loss, as shown in equation (8): (8) wherein denotes the total loss of the model, denotes the coronary lesion diagnosis loss, denotes the retinal vessel marker prediction loss; S44, after the training of the coronary artery lesion detection model based on the concept of retinal blood vessel markers is converged, the retinal fundus image and the clinical indicators of the patient are input to diagnose the coronary artery lesion.

Images