Coronary heart disease risk prediction method and system based on multi-modal eye fundus image

Abstract

The invention provides a coronary heart disease risk prediction method and system based on a multi-modal eye fundus image, and relates to the technical field of intelligent medical systems. The method comprises the following steps: acquiring a multi-modal eye fundus image group corresponding to a target user; based on a multi-scale attention model, extracting a fundus plane feature group corresponding to the first fundus image; extracting a fundus stereoscopic feature group corresponding to the second fundus image based on a region guidance attention model; and inputting the fundus plane feature group and the fundus stereo feature group into a risk prediction model to determine a corresponding coronary heart disease risk prediction result, the risk prediction model adopting a multi-modal fusion deep learning model. Therefore, big data analysis is carried out on the multi-modal and multi-dimensional fundus images, a coronary heart disease prevention and treatment gateway is moved forward, and non-invasive, rapid and accurate risk prediction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of smart medical systems, and in particular to a method and system for predicting the risk of coronary heart disease based on multimodal fundus images. Background Art

[0002] Coronary Heart Disease (CHD) is a heart disease caused by vascular stenosis or blockage due to coronary atherosclerosis, which in turn leads to myocardial ischemia, hypoxia or necrosis. CHD is not only one of the leading causes of death and disability worldwide, but also the disease with the highest morbidity and mortality among cardiovascular diseases. Therefore, early detection and risk prediction of CHD are of vital importance to reducing morbidity and mortality.

[0003] At present, the diagnosis and prognosis of coronary heart disease mainly rely on coronary CTA (Coronary Computed Tomography Angiography), cardiac magnetic resonance imaging, coronary angiography and other imaging examinations, which are expensive, time-consuming, partially invasive and require high operation, making it difficult to achieve early non-invasive screening, diagnosis, prognosis assessment and long-term follow-up. Most patients have entered the middle and late stages when they are diagnosed, resulting in complex coronary artery lesions, increased difficulty in interventional and surgical treatments, and poor prognosis.

[0004] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention

[0005] The embodiments of the present invention provide a method and system for predicting the risk of coronary heart disease based on multimodal fundus images, which are used to at least solve the problems of high cost, long time consumption and strong invasiveness existing in current imaging examination methods.

[0006] In a first aspect, an embodiment of the present invention provides a method for predicting the risk of coronary heart disease based on multimodal fundus images, the method comprising: obtaining a multimodal fundus image group corresponding to a target user; the multimodal fundus image group comprises a first fundus image and a second fundus image; the first fundus image is obtained by color fundus photography, and the second fundus image is obtained by optical coherence tomography blood flow imaging; based on a multiscale attention model, extracting a fundus plane feature group corresponding to the first fundus image; based on a region-guided attention model, extracting a fundus stereo feature group corresponding to the second fundus image; inputting the fundus plane feature group and the fundus stereo feature group into a risk prediction model to determine a corresponding coronary heart disease risk prediction result, the risk prediction model adopting a multimodal fusion deep learning model.

[0007] In a second aspect, an embodiment of the present invention provides a coronary heart disease risk prediction system based on multimodal fundus images, the system comprising: a multimodal data acquisition unit, used to acquire a multimodal fundus image group corresponding to a target user; the multimodal fundus image group comprises a first fundus image and a second fundus image; the first fundus image is obtained by color fundus photography, and the second fundus image is obtained by optical coherence tomography blood flow imaging; a plane feature extraction unit, used to extract the fundus plane feature group corresponding to the first fundus image based on a multi-scale attention model; a stereo feature extraction unit, used to extract the fundus stereo feature group corresponding to the second fundus image based on a region-guided attention model; a risk prediction unit, used to input the fundus plane feature group and the fundus stereo feature group into a risk prediction model to determine the corresponding coronary heart disease risk prediction result, the risk prediction model adopts a multimodal fusion deep learning model.

[0008] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the above method.

[0009] In a fourth aspect, an embodiment of the present invention provides a storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of the above-mentioned method of the present invention.

[0010] In a fifth aspect, an embodiment of the present invention further provides a computer program product, wherein the computer program product comprises a computer program stored on a storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the above method.

[0011] The method, system, electronic device and non-transitory computer-readable storage medium for predicting the risk of coronary heart disease based on multimodal fundus images provided by the present invention can produce at least the following technical effects:

[0012] (1) The risk prediction of coronary heart disease is completely non-invasive by using multimodal fundus images, which reduces the pain of patients and the consumption of medical resources. It also avoids the high cost, time-consuming, and invasiveness of traditional imaging examination methods. Through this technical solution, color fundus photography (CFP) and optical coherence tomography angiography (OCTA) are used to comprehensively capture the planar and three-dimensional features of the fundus. Through the multimodal fusion deep learning model, the sensitivity and specificity of early detection of coronary heart disease are improved, and screening can be performed in the early stage when the patient has no obvious symptoms, thereby achieving early detection and intervention of coronary heart disease.

[0013] (2) In this technical solution, a multi-scale attention model is used to extract the planar features of color fundus images. The multi-scale attention model can capture the detailed features of fundus images at different scales, and a regional guided attention model is used to extract the stereoscopic features of optical coherence tomography blood flow imaging images. The regional guided attention model can focus on regional features with important clinical significance, thereby improving the accuracy and efficiency of feature extraction.

[0014] (3) By inputting the feature groups of fundus images of different modalities into the multimodal fusion deep learning model, a comprehensive analysis of the features of different modalities was achieved. Compared with the analysis and prediction method of single modality images, the multimodal fusion model can more comprehensively reflect the complex pathological characteristics of coronary heart disease and improve the accuracy and reliability of risk prediction.

[0015] Through this technical solution, fundus imaging technology is used to predict coronary heart disease before its onset, and important features in the two types of fundus imaging are extracted respectively, and feature fusion is performed. By performing big data analysis on multimodal and multi-dimensional fundus images, the threshold for coronary heart disease prevention and treatment is moved forward, and non-invasive, rapid and accurate predictions are achieved, making this platform more valuable for disease prevention and greatly facilitating clinical applications. It is especially suitable for community hospitals and primary medical institutions, and improves the coverage and popularity of coronary heart disease screening. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1A flowchart of an example of a method for predicting the risk of coronary heart disease based on multimodal fundus images according to an embodiment of the present invention is shown;

[0018] Figure 2 A schematic diagram showing a structural connection of an example of a risk prediction model according to an embodiment of the present invention is shown;

[0019] Figure 3 A schematic diagram showing a structural connection of an example of a multi-scale attention model according to an embodiment of the present invention is shown;

[0020] Figure 4 A schematic diagram showing a structural connection of an example of a region-guided attention model according to an embodiment of the present invention is shown;

[0021] Figure 5 It shows an operation flow chart of an example of determining a film layer region image by an adaptive threshold segmentation method according to an embodiment of the present invention;

[0022] Figure 6 A structural block diagram of an example of a coronary heart disease risk prediction system based on multimodal fundus images according to an embodiment of the present invention is shown;

[0023] Figure 7 It is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "one" or "the" do not indicate a quantitative limitation, but indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0026] It should be noted that the terms "up", "down", "left", "right", "front", "back", etc. used in the present invention are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0027] Fundus microcirculation has similar pathophysiological characteristics to cardiovascular circulation. Many studies have discovered fundus imaging markers related to cardiovascular disease, such as retinal vessel diameter, retinal and choroidal blood flow density, thickness, etc., which are related to the incidence of cardiovascular events, and pupil area is related to the prognosis of heart failure. Fundus images can generally be obtained from opticians or ophthalmology clinics. The collection method is safe, non-invasive, convenient and inexpensive, with high health economic benefits and various imaging methods. Therefore, artificial intelligence is used to integrate multimodal information of fundus imaging, and then non-invasively and accurately predict the risk of coronary heart disease, and achieve early warning and intervention. The development of an efficient, rapid, economical, non-invasive, popular and scalable coronary heart disease prediction and evaluation model is crucial for early warning and prevention for people with coronary heart disease.

[0028] Figure 1 A flowchart of an example of a method for predicting the risk of coronary heart disease based on multimodal fundus images according to an embodiment of the present invention is shown.

[0029] Regarding the executor of the method of the embodiment of the present invention, it can be any controller or processor with computing or processing capabilities, which can be integrated or partially integrated in a server or a coronary heart disease risk prediction platform, so as to achieve non-invasive, early, high-precision, rapid and economical coronary heart disease risk prediction by combining multimodal fundus images and deep learning models, thereby greatly optimizing the existing detection and evaluation methods of coronary heart disease.

[0030] In some examples, it can be integrated into the configuration server by software, hardware, or a combination of software and hardware, which should not be limited here. It should be understood that one or more steps involved in the following process can be implemented by one or more controllers or software installed and deployed in the client or server.

[0031] like Figure 1 As shown, in step S110, a multimodal fundus image group corresponding to the target user is acquired, and the multimodal fundus image group includes a first fundus image and a second fundus image.

[0032] Here, the first fundus image is obtained by color fundus photography, and the second fundus image is obtained by optical coherence tomography blood flow imaging.

[0033] In some embodiments, a high-resolution color fundus camera is used to photograph the fundus of the target user to obtain a color fundus image. Since the patient only needs to look in a certain direction to complete the shooting, the whole process is non-invasive and fast. In addition, an optical coherence tomography (OCT) device is used to scan the user's retina to obtain a high-resolution stereo fundus image, especially a blood flow imaging image. The sampling process is also non-invasive, and the user only needs to keep his eyes fixed and look in a specific direction for a few seconds. Thus, a non-invasive and fast acquisition method is achieved, which improves the patient's comfort and compliance and facilitates large-scale screening.

[0034] In step S120, a fundus plane feature group corresponding to the first fundus image is extracted based on the multi-scale attention model.

[0035] In some embodiments, the first fundus image is subjected to preprocessing operations such as denoising, contrast enhancement, and standardization, and multi-level feature extraction is performed on the color fundus image through a multi-scale attention mechanism of deep learning. Through this embodiment, the planar features of the fundus are accurately extracted through a multi-scale attention mechanism, which is highly correlated with the risk of coronary heart disease, and can effectively capture important feature details at different scales, thereby improving the accuracy and completeness of feature extraction.

[0036] In some examples of embodiments of the present invention, the fundus plane feature group includes at least one of the following: cup-to-disc ratio of the left and right eyes, vertical cup-to-disc ratio, optic cup area, optic cup volume, optic disc area, average thickness of the retinal nerve fiber layer, foveal avascular zone area, foveal avascular zone perimeter and near-circularity index.

[0037] It should be noted that the cup-to-disc ratio (CDR) of the left and right eyes is the ratio of the diameter of the optic cup to the diameter of the optic disc, which is used to evaluate the morphology of the optic disc. A higher CDR may be associated with optic nerve damage, and optic nerve health is related to overall cardiovascular health. The vertical cup-to-disc ratio is the ratio of the vertical diameter of the optic cup to the vertical diameter of the optic disc. Similar to the horizontal CDR, the vertical cup-to-disc ratio provides another dimension of information on optic nerve health, which helps to comprehensively evaluate optic disc changes and increase the accuracy of coronary heart disease risk prediction. The optic cup area is the area of ​​the concave part of the center of the optic disc. Changes in the optic cup area may reflect the damage to the optic nerve fibers, which is related to the health of the blood vessels and can indirectly reflect the risk of coronary heart disease. The optic cup volume is the three-dimensional volume of the optic cup. The optic cup volume can more three-dimensionally reflect the changes in the size and shape of the optic cup, which helps to more comprehensively evaluate the health of the optic nerve and blood vessels. The optic disc area is the area of ​​the area where the optic nerve enters the retina. Changes in the optic disc area may be related to fundus blood flow and vascular conditions. Monitoring changes in the optic disc area can provide an important reference for coronary heart disease risk assessment. The average thickness of the retinal nerve fiber layer is the average thickness value of the retinal nerve fiber layer. The thickness of the retinal nerve fiber layer is closely related to the health of the optic nerve. Abnormal thickness may indicate vascular lesions and help predict the risk of coronary heart disease at an early stage. The foveal avascular zone is an area in the center of the retina without vascular supply. Its area reflects the size of the avascular zone. Changes in the area of ​​the foveal avascular zone may be related to abnormal retinal blood flow. Monitoring changes in this area can provide an indirect basis for the risk of coronary heart disease. The perimeter of the foveal avascular zone can help quantify changes in the avascular zone and provide more information on the status of retinal blood flow. The near-circularity index measures the degree to which the shape of the foveal avascular zone is close to a circle. Changes in the near-circularity index may reflect abnormalities in the retinal microcirculation status.

[0038] In step S130 , a fundus stereo feature group corresponding to the second fundus image is extracted based on the region guided attention model.

[0039] In some embodiments, the second fundus image is subjected to preprocessing operations such as denoising, contrast enhancement, and standardization, and the OCT blood flow imaging image is deeply analyzed through the deep learning regional guided attention mechanism to extract the characteristics of the three-dimensional structure. Thus, the three-dimensional characteristics of the fundus blood vessels can be accurately extracted through the regional guided attention model, and the key areas of the blood vessel structure can be focused on, thereby improving the accuracy of three-dimensional feature extraction, being able to effectively distinguish and analyze the characteristics of arteries and veins, and providing data support for more detailed risk assessment.

[0040] In some examples of embodiments of the present invention, the fundus stereoscopic feature group includes at least one of the following: total vessel diameter, arterial diameter, venous diameter, total vessel fractal dimension, arterial fractal dimension, venous fractal dimension, total vessel tortuosity, arterial tortuosity, venous tortuosity, total vessel tortuosity density, arterial tortuosity density, venous tortuosity density and total vessel branching angle.

[0041] It should be noted that the total vessel diameter is the total diameter of all major blood vessels on the retina, reflecting the overall health of the retinal blood vessels. Abnormal vessel diameter may indicate systemic vascular health problems, including coronary heart disease. Arterial diameter, that is, the diameter of the retinal artery, changes in arterial diameter can directly reflect arteriosclerosis and stenosis, which is an important risk factor for coronary heart disease. Venous diameter, that is, the diameter of the retinal vein, changes in venous diameter may reflect venous return disorders and abnormal hemodynamics, which helps to assess the overall vascular health. The total fractal dimension of the blood vessels reflects the complexity and fractal characteristics of the retinal blood vessels. The fractal dimension of the blood vessels is related to their health status and function. A higher fractal dimension may indicate a healthier vascular network structure, which helps to assess the risk of coronary heart disease. The arterial fractal dimension is used to measure the fractal characteristics of the retinal arteries, which can provide detailed information on the health of the arteries and help identify early risk factors for coronary heart disease. The venous fractal dimension is used to measure the fractal characteristics of the retinal veins, which reflects the complexity of the venous network. Abnormalities may indicate blood flow disorders and indirectly reflect the risk of coronary heart disease. Total vascular curvature indicates the degree of curvature of all blood vessels on the retina. Changes in vascular curvature may reflect vascular elasticity and functional status, and have important reference value for the prediction of coronary heart disease risk. Arterial curvature is the degree of curvature of the retinal artery. Abnormal arterial curvature may indicate arteriosclerosis and blood flow disorders, which are directly related to the risk of coronary heart disease. Venous curvature indicates the degree of curvature of the retinal veins, and its changes may reflect abnormal venous return and hemodynamics. Total vascular curvature density indicates the curvature density of all blood vessels on the retina, reflecting the density and health of the vascular network. Arterial curvature density, that is, the curvature density of retinal arteries, its changes can indicate the health status of arteries, and have important reference value for the prediction of coronary heart disease risk. Changes in venous curvature density can reflect venous blood flow and vascular health, which helps to comprehensively assess the risk of coronary heart disease. Total vascular branch angles indicate the angles of all vascular branches on the retina, and their changes may reflect the growth and functional status of blood vessels, providing quantitative analysis for the prediction of coronary heart disease risk.

[0042] Through this embodiment, by fully extracting and analyzing these planar and three-dimensional features, it is possible to fully utilize fundus image information to provide accurate and comprehensive coronary heart disease risk prediction. Through the complementary and comprehensive analysis of these features, it is helpful to improve the accuracy and reliability of the prediction model and provide a scientific basis for early intervention and treatment.

[0043] In step S140, the fundus plane feature group and the fundus stereo feature group are input into the risk prediction model to determine the corresponding coronary heart disease risk prediction result. The risk prediction model adopts a multimodal fusion deep learning model.

[0044] In some embodiments, the extracted fundus planar feature group and stereo feature group are fused to construct a unified feature vector. A multimodal fusion deep learning model is used to effectively fuse and comprehensively analyze the features of different modalities to predict the risk of coronary heart disease and improve the accuracy and robustness of the prediction results. Exemplarily, supervised learning is performed using a large amount of annotated data, and training is performed using a large-scale annotated data set to optimize model parameters and improve prediction accuracy.

[0045] Through this embodiment, the multimodal fusion deep learning model is used to comprehensively utilize the advantages of planar and three-dimensional features to provide more accurate and comprehensive risk prediction. Through the risk prediction results of the model, early screening and intervention can be achieved to improve the prognosis of patients. Therefore, through the acquisition and analysis of multimodal fundus images, combined with the multi-scale attention model and the regional guided attention model, efficient feature extraction and analysis are achieved, which significantly improves the accuracy and reliability of the prediction, has the advantages of early non-invasive screening, strong real-time performance, and low cost, and can effectively reduce the morbidity and mortality of coronary heart disease.

[0046] It should be noted that color fundus photography (CFP) and optical coherence tomography (OCT) are two fundus imaging methods. Optical coherence tomography blood flow imaging (OCTA) is a non-invasive blood flow detection technology recently developed based on OCT (optical coherence tomography). It uses moving blood as an intrinsic contrast agent to detect the difference in the forward and backward movement of red blood cells in the retinal blood vessels, and obtains clear fundus images through computer algorithms. OCTA is a three-dimensional stereoscopic imaging that can observe vascular images at different levels and perform high-resolution imaging of the retinal membrane layers, but the observation range is limited. CFP can obtain a large range of color two-dimensional fundus images, but cannot observe the deep structure and membrane structure of the retina. The image features of CFP and OCTA have their own characteristics, and traditional deep learning cannot effectively extract complementary features.

[0047] Through the embodiments of the present application, a modality-specific attention deep learning network is proposed. Through the modality-specific attention module, important features in CFP and OCTA images are extracted respectively, providing information of different dimensions for the diagnosis of coronary heart disease, and incorporating clinical and biological features to establish a multimodal information fusion model to achieve early diagnosis and prognosis evaluation of coronary heart disease.

[0048] Figure 2 A schematic diagram of structural connections of an example of a risk prediction model according to an embodiment of the present invention is shown.

[0049] like Figure 2 As shown, the risk prediction model 200 includes a feature fusion layer 210, a deep residual network layer (ResNet) 220, a bidirectional long short-term memory network layer (Bidirectional Long Short-Term Memory, BiLSTM) 230, a multi-head self-attention layer 240 and a fully connected output layer 250.

[0050] The feature fusion layer 210 is used to fuse the fundus plane feature group and the fundus stereo feature group. Specifically, the fundus plane feature group and the fundus stereo feature group are connected to obtain a fused feature representation:

[0051] F fusion =[F p ,F s ] , Formula (1)

[0052] In the formula, F p represents the fundus plane feature group, F s represents the fundus stereo feature group, and F fusion Represents the fundus fusion feature group.

[0053] Through the feature fusion layer, the planar features and stereoscopic features in the color fundus photography (CFP) and planar optical coherence tomography (OCT-A) images are integrated to ensure the diversity and richness of the input features. By fusing the planar and stereoscopic features of the fundus, it is possible to fully capture the different dimensional information of the fundus blood vessels, such as the cup-to-disc ratio, cup area, blood vessel diameter, blood vessel tortuosity, etc., and improve the ability to comprehensively assess the risk of coronary heart disease.

[0054] The deep residual network layer 220 includes multiple residual blocks, each of which includes multiple convolutional layers and short connections between different convolutional layers, for example, a short connection is constructed between every two convolutional layers. The deep residual network layer 220 is used to extract the deep convolutional features corresponding to the fundus fusion feature group:

[0055] H=F fusion +F(F fusion ,{W i}), Formula (2)

[0056] Where F represents the convolution operation in the residual block, W i represents the weight of the i-th convolutional layer in the residual block, and H represents the deep convolutional features.

[0057] Through the deep residual network (ResNet) layer, multi-level deep features can be effectively extracted through residual blocks. The depth of the model is increased based on the ResNet layer, the ability of feature representation is improved, and the complex fundus vascular characteristics can be better captured, thereby improving the model's ability to distinguish the risk of coronary heart disease.

[0058] The bidirectional long short-term memory network layer 230 is used to determine the bidirectional temporal dependency features corresponding to the deep convolutional features. Specifically, in order to capture the time series information and global dependency in the features, the forward and backward LSTM networks in the bidirectional long short-term memory network are used to simultaneously consider the forward and backward dependencies of the sequence.

[0059]

[0060] In the formula, H BiLSTM Represents the bidirectional timing dependency feature, represents the output features of the forward LSTM of H, Represents the output features of the backward LSTM of H.

[0061] The BiLSTM layer can simultaneously consider the forward and backward dependencies of the feature sequence, enhancing the time series information and global dependencies of the feature representation. The BiLSTM layer improves the model's ability to process time series data by capturing the temporal dynamic changes in fundus image features, and can more accurately predict the risk of coronary heart disease.

[0062] The multi-head self-attention layer 240 is used to process the bidirectional temporal dependency features to obtain the corresponding global attention features. By using the multi-head self-attention mechanism, the global dependency and important information in the features can be further captured.

[0063] H attention =MultiHead(Q,K,V)=[head1,head2,…,head h ]W O , formula (4)

[0064]

[0065] Q=H BiLSTM W Q , Formula (6)

[0066] K=H BiLSTM W K , Formula (7)

[0067] V=H BiLSTM W V , Formula (8)

[0068] Where WQ , W K and W V denote the query weight matrix, key weight matrix, and value weight matrix, respectively; Q, K, and V denote the query matrix, key matrix, and value matrix, respectively; softmax denotes the softmax function, Indicates the dimension of the key vector, head i represents the output of the i-th attention head; W O represents the multi-head output weight matrix, h represents the total number of attention heads, H attention Represents the global attention feature.

[0069] Through the multi-head self-attention layer, the correlation between features is calculated based on multiple attention heads to capture the global dependency between features. The multi-head self-attention mechanism can focus on the key information in the features, enhance the global consistency and discrimination of feature representation, and thus improve the model's ability to identify the risk of coronary heart disease.

[0070] The fully connected output layer 250 is used to process the global attention features to obtain the corresponding coronary heart disease risk probability. Here, the global attention features are further processed by the fully connected layer, and the coronary heart disease risk probability is output through the Sigmoid activation function.

[0071] H FC =ReLU(W FC H attention +b FC ), Formula (9)

[0072] y=Sigmoid(W y H FC +b y ), Formula (10)

[0073] Where y represents the final output of the risk probability of coronary heart disease, W FC and b FC Respectively represent the weight matrix and bias term of the fully connected layer; ReLU represents the nonlinear activation function, W y and b y They represent the weight matrix and bias term of the output layer respectively; Sigmoid represents the Sigmoid activation function.

[0074] The fused deep features are further processed through the fully connected layer and the output layer, and the risk probability of coronary heart disease is finally output through the Sigmoid activation function. The fully connected layer optimizes the nonlinear transformation capability of feature representation through the calculation of multiple layers of neurons, and the output layer normalizes the prediction results into probability values ​​through the Sigmoid function, which is convenient for risk assessment and interpretation.

[0075] Through this embodiment, through the comprehensive application of multimodal feature fusion, deep residual network, bidirectional long short-term memory network and multi-head self-attention mechanism, it is possible to comprehensively and deeply extract and fuse multi-dimensional information in fundus images. Furthermore, the combination of multimodal feature fusion and advanced attention mechanism makes the model more adaptable to data of different types and sources, and improves the robustness and generalization ability of the model. Therefore, when facing different individuals and diverse fundus image data, the model can maintain stable prediction performance, providing an effective technical means for the early prediction of coronary heart disease.

[0076] In some examples of the embodiments of the present invention, the loss function of the risk prediction model can adopt a comprehensive loss function that combines focal loss and Bayesian optimization loss. By introducing focal loss, the model can better handle the problem of class imbalance, give higher weights to difficult-to-classify samples, and improve the performance of the model on small sample categories. By introducing Bayesian optimization loss, combining the uncertainty information of the model, and guiding parameter optimization, the prediction accuracy and robustness of the model can be effectively improved, and the prediction error can be reduced.

[0077] More specifically, the loss function L of the risk prediction model is total for:

[0078] L total =λ1·L focal +λ2·L bayesian , Formula (11)

[0079]

[0080]

[0081] Where, L focal represents the focal loss term, L bayesian represents the Bayesian optimization loss term, λ1 and λ2 represent the importance coefficients of the corresponding loss terms respectively; N represents the total number of samples in the data sample set; β t represents the category weight of the tth sample, p t is the model's predicted probability that the tth sample belongs to the true category, γ represents the adjustment factor; σ t is the prediction uncertainty of the t-th sample, y t is the true label of the tth sample, is the predicted value of the tth sample.

[0082] In some embodiments, by adjusting the weight coefficients λ1 and λ2, the proportion of focal loss and Bayesian optimization loss in the total loss can be flexibly adjusted, which can achieve a better balance in dealing with class imbalance and optimizing model parameters, thereby achieving optimal performance in different application scenarios.

[0083] It should be noted that for the Bayesian optimization loss term in this embodiment, in the coronary heart disease risk prediction model, the prediction uncertainty of the sample (σ i ) is a key step. Here, it can be achieved through Bayesian Neural Network (BNN). Based on Bayesian Neural Network, uncertainty is taken into account in the prediction process by introducing probability distribution on model parameters.

[0084] Specifically, Bayesian neural networks quantify the uncertainty of the model by introducing probability distribution on the weights and biases of the neural network. Prediction uncertainty can be divided into model uncertainty and data uncertainty.

[0085] Model uncertainty comes from the variation of model parameters, which is achieved by sampling the weights of the neural network, and the Monte Carlo (MC) sampling method can be used to estimate model uncertainty. In addition, data uncertainty comes from the noise and unpredictability of the input data, which can be achieved by adding additional predictive distribution parameters (e.g., variance) to the output layer.

[0086] For example, a probability distribution is introduced on the weights and biases of the neural network, for example, using a Gaussian distribution Then, multiple forward propagations are performed using the Monte Carlo sampling method to calculate the predicted mean and variance. Assuming Q samplings are performed:

[0087]

[0088] Where W q is the weight of the qth sampling.

[0089] Furthermore, based on the sampling results, the predicted mean is calculated by the following formula: and prediction uncertainty σ i :

[0090]

[0091] In this way, based on the Bayesian neural network, the prediction uncertainty of the sample can be effectively estimated. In the coronary heart disease risk prediction model, multiple forward propagations are performed through the Monte Carlo sampling method to calculate the mean and variance of the prediction, thereby quantifying the uncertainty of the prediction. In this way, not only the uncertainty of the model parameters but also the uncertainty of the data itself is considered, providing the model with stronger robustness and higher prediction accuracy.

[0092] The comprehensive loss function provided in this embodiment can produce at least the following technical effects:

[0093] Based on the focal loss term, in medical image analysis, positive samples (those with coronary heart disease) are usually small. The focal loss can effectively enhance the model's attention to these small sample categories and improve the recognition rate of positive samples. Therefore, by reducing the weight of easy-to-classify samples, the model can better focus on difficult-to-classify samples, thereby improving the overall classification effect.

[0094] Based on the Bayesian optimization loss term, the Bayesian neural network can quantify the uncertainty of the prediction results by introducing the uncertainty of the weights, providing more reliable information for subsequent decision-making. Through multiple sampling and uncertainty assessment, the model can adjust parameters more accurately, reduce prediction errors, and improve prediction accuracy.

[0095] By adjusting the weight coefficients in the loss function, the model can adjust the weight coefficients of focal loss and Bayesian optimization loss according to actual application needs, so that the model can be widely used in different medical image analysis tasks, improving the practicality and versatility of the model.

[0096] Figure 3 A schematic diagram of the structural connection of an example of a multi-scale attention model according to an embodiment of the present invention is shown.

[0097] like Figure 3 As shown, the multi-scale attention model 300 includes multiple (i.e., K) cascaded attention modules 310 and a combination module 320. Each attention module 310 includes a convolution layer 311 and an attention layer 312, respectively, for extracting attention features at the corresponding scale. Therefore, a corresponding attention module is assigned to each feature scale. The combination module 320 is used to determine the corresponding multi-scale attention feature map by combining the attention features of each scale, thereby obtaining a fundus plane feature group.

[0098] Through the convolutional layers 311 in each attention module 310, the multi-scale image features corresponding to the first fundus image are extracted:

[0099] F c =Conv c (I1), formula (17)

[0100] Where c = 1, 2, ..., K, K represents the total number of attention modules, and the convolutional layers in each attention module have corresponding convolution kernel sizes; I1 represents the first fundus image; Conv c represents the convolution operation of the convolutional layer in the cth attention module, F c Represents the feature map extracted by the c-th convolutional layer.

[0101] Using multiple convolutional layers to extract feature maps of different sizes can capture vascular and retinal structural information at different resolutions. Multi-scale representation helps to identify subtle features that are difficult to detect at a single scale.

[0102] Based on the first-scale attention module ranked first among the attention modules 310, since there is no feature of the previous scale for comparison, a self-attention feature operation is performed to obtain the corresponding self-attention feature:

[0103]

[0104] G1=[g 1,1 ,g 1,2 ,...,g 1,C ] , Formula (19)

[0105] α1=softmax(W a1 G1+b a1 ), Formula (20)

[0106] A1=α1·F1 , Formula (21)

[0107] Where G1 represents the global feature vector extracted by the first scale attention module, W a1 and b a1 They represent the weight matrix and bias term of the first-scale attention module respectively; α1 represents the attention weight corresponding to the first-scale attention module, and A1 represents the self-attention feature output by the first-scale attention module; f u,v,q represents the value of the qth channel of the feature map F1 at position (u, v), g 1,q It represents the value of the qth channel after global average pooling, and H, W, and C represent the height, width, and number of channels of F1, respectively.

[0108] Here, firstly, a global feature vector is generated by averaging the spatial dimensions of the entire feature map through the global average pooling (GAP) operation, which can significantly reduce the number of parameters and effectively avoid overfitting. In addition, the self-attention mechanism of the first scale can assign different weights to each position of the feature map, highlight the features of the key areas, and suppress noise and irrelevant information.

[0109] Based on the non-first-ranked attention modules, the calculated attention features are fused with the attention features output by the previous attention module to establish the connection between features of different scales:

[0110] Q e =F e W Qe , Ke-1 =A e-1 W Ke , V e-1 =A e-1 W Ve , Formula (22)

[0111] In the formula, Q e represents the query matrix output by the e-th attention module, K e-1 and V e-1 They represent the key matrix and value matrix output by the e-1th attention module respectively; W Qe , W Ke and W Ve Respectively represent the query weight matrix, key weight matrix and value weight matrix of the e-th attention module; α e,g represents the attention weight of the e-th attention module to the g-th position, e>1.

[0112] The attention weight is determined by calculating the cosine similarity:

[0113]

[0114] Calculate the attention features based on the attention weights:

[0115]

[0116] In the formula, A e represents the attention feature of the e-th scale.

[0117] Through this embodiment, the subsequent scale calculates the similarity with the attention feature of the previous scale, thereby establishing a connection between features of different scales, so that the model can better understand and integrate multi-scale information, further enhancing the feature representation capability.

[0118] After pooling all scales of attention features, we can get a complete multi-scale attention feature map:

[0119] P e =Pooling(A e ), Formula (25)

[0120] F MSA =Concat(P1,P2,...,P K ), Formula (26)

[0121] Where P e represents the pooling result of the attention feature of the e-th scale, F MSA Represents a multi-scale attention feature map.

[0122] Through this embodiment, features of different scales are fused through the attention mechanism, so that the model can consider local and global information at the same time, improving the performance of the model when processing complex and diverse fundus images. In addition, the attention mechanism weights the features according to their importance, so that the model can effectively suppress unimportant or interfering features while focusing on key features, achieving the ability of adaptive weighting, and enhancing the stability of the model under different image quality and shooting conditions.

[0123] Figure 4 A structural connection diagram of an example of a region-guided attention model according to an embodiment of the present invention is shown.

[0124] The Region-Guided Attention (RGA) model aims to effectively extract important features in optical coherence tomography (OCT-A) images, especially those concentrated in the central membrane region, while suppressing the interference of background noise on the model.

[0125] like Figure 4 As shown, the regional guided attention model 400 includes multiple (i.e., U) regional attention modules 410 and a fusion module 420. Each regional attention module 410 includes a cascaded regional convolution layer 411 and a regional guided attention layer 412 to extract the regional guided attention features of the corresponding region. The fusion module 420 is used to fuse the regional guided attention features of each region to obtain a corresponding fundus stereo feature group.

[0126] The regional convolutional layer 411 in each regional attention module 410 is used to extract the regional convolutional feature map of the second fundus image in each specific area:

[0127] R l =Conv l (I2), formula (27)

[0128] Where, l = 1, 2, ..., U, U is the total number of regional attention modules, each of which is used to process the corresponding demarcated pixel area; I2 represents the second fundus image; R l represents the regional convolution feature map extracted by the regional convolution layer in the lth regional attention module, Conv l Represents the convolution operation of the regional convolutional layer in the l-th regional attention module.

[0129] By extracting the feature differences of different regions in the second fundus image through multiple regional convolutional layers, the membrane details at different resolutions can be captured, and local feature calculations can be performed on the details of different regions, so that the model can pay more attention to the features of each region, further improving the feature representation capability.

[0130] The regional guided attention layer 412 in each regional attention module 410 uses the membrane layer regional image to weight the regional convolution feature map to obtain the corresponding regional guided feature map:

[0131] R lM =R l ·M , Formula (28)

[0132] Where M represents the film area image, R lM Represents the region-guided feature map determined by the region-guided attention layer in the l-th region attention module.

[0133] It should be noted that the membrane region image is extracted from the optical coherence tomography blood flow imaging (OCT-A) image by image segmentation technology, and includes a binary image of the membrane region, which represents the important membrane region in the OCT-A image, wherein a value of 1 represents the membrane region, and a value of 0 represents the background region. The membrane region image can be implemented by fixed threshold segmentation, global threshold segmentation, or k-means clustering method. More preferably, it can be implemented by adaptive threshold segmentation, and more details will be described below in conjunction with other examples.

[0134] Through the region-guided attention mechanism, the feature map is weighted using the membrane region image, so that the model can suppress the influence of background noise while paying attention to the characteristics of the membrane region, thereby increasing the robustness of the model.

[0135] The region guides the feature map R lM Divide into multiple sub-regions to guide feature maps Where s represents the sth sub-region.

[0136] Here, further dividing the sub-regions according to the region-guided feature map can achieve image segmentation based on feature differentiation within the region and effectively improve the feature representation capabilities of different sub-regions.

[0137] For each sub-region guided feature map The corresponding local feature vector is obtained by local average pooling

[0138] Calculate the local feature vector through the fully connected layer The corresponding local attention weight

[0139]

[0140] In the formula, They represent the weight matrix and bias term calculated by the fully connected layer for the attention weight of the s-th sub-region of the l-th region, respectively. represents the attention weight of the s-th sub-region of the l-th region.

[0141] Through this embodiment, the attention weight is calculated based on the local feature vector, so that the model can adaptively adjust the degree of attention to different areas, thereby improving the generalization ability of the model in different image scenes.

[0142] Guide feature maps for each sub-region Weighted, we get the sub-region attention-guiding features:

[0143]

[0144] In the formula, represents the sub-region guided attention feature of the s-th sub-region of the l-th region.

[0145] Through this embodiment, by utilizing multi-scale feature extraction and local feature calculation, the model can capture subtle features in OCT-A images, thereby improving the ability to identify lesions related to coronary heart disease.

[0146] The attention features of all sub-regions are concatenated to obtain a regional guided attention feature map, and then fused through the fusion module 420:

[0147]

[0148] F RGA =Concat(F RGA,1 ,F RGA,2 ,...,F RGA,U ), Formula (32)

[0149] In the formula, F RGA,l represents the regional attention feature map of the lth region, S represents the total number of sub-regions contained in the lth region, and F RGA Represents the fused features based on the region-guided attention features of each region.

[0150] Through this embodiment, the attention weight calculation method based on the local feature vector enables the model to dynamically adjust the attention to different regions, thereby improving the accuracy of feature representation and prediction. In addition, the local feature calculation and weighting operation can effectively reduce the consumption of computing resources by processing the features of each sub-region in parallel.

[0151] Figure 5 The flowchart shows an example of determining a membrane region image by an adaptive threshold segmentation method according to an embodiment of the present invention.

[0152] like Figure 5As shown, in step S510, for each pixel in the second fundus image, a local mean within a window centered on the pixel is calculated.

[0153] More specifically, it is calculated by the following formula:

[0154]

[0155] Wherein, (i, j) represents the position of the pixel in the image, μ(i, j) represents the local mean pixel value at (i, j), I2(i+m, j+n) represents the pixel value of the second fundus image I2 at the position (i+m, j+n), and w represents the window size.

[0156] In step S520, the local standard deviation is calculated.

[0157] More specifically, it is calculated by the following formula:

[0158]

[0159] Where σ(i,j) represents the local standard deviation at (i,j).

[0160] In step S530, an adaptive threshold is calculated using the local mean and standard deviation.

[0161] More specifically, it is calculated as follows:

[0162]

[0163] Where r represents the adjustment parameter and T(i,j) represents the adaptive threshold at (i,j).

[0164] In step S540, the second fundus image is binarized according to the adaptive threshold to obtain a membrane layer area image.

[0165] More specifically, it is processed in the following ways:

[0166]

[0167] Where M(i, j) represents the pixel value of the film region image M at (i, j), and I2(i, j) represents the pixel value of the image I2 at (i, j).

[0168] In the region-guided attention model, the key steps of the adaptive threshold segmentation algorithm involve local standard deviation calculation, adaptive threshold calculation, and generation of membrane region images. The local standard deviation σ(i,j) is used to measure the discreteness of pixel values ​​within a specific window (centered on pixel (i,j)), which reflects the variation of pixel values ​​within the region, that is, the degree to which the pixel values ​​deviate from their local mean μ(i,j). The adaptive threshold T(i,j) is used to dynamically determine the threshold of the segmented image based on local statistics (mean and standard deviation) to distinguish between the membrane region and the background region, so that the threshold can be adaptively adjusted to adapt to the image characteristics of different regions.

[0169] Through the embodiment of the present invention, the adaptive threshold segmentation algorithm is used to dynamically adjust the threshold to adapt to the image characteristics of different regions, effectively separate the membrane region, reduce the interference of background noise, improve the accuracy of feature extraction, ensure the accurate separation of the membrane region and the background region, and thus improve the accuracy of coronary heart disease risk prediction. In addition, by measuring the discrete degree of local pixel values, the segmentation threshold is dynamically adjusted to adapt to the characteristics of different regions of the image, and has a stronger generalization ability.

[0170] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0171] Figure 6 A structural block diagram of an example of a coronary heart disease risk prediction system based on multimodal fundus images according to an embodiment of the present invention is shown.

[0172] like Figure 6 As shown, the coronary heart disease risk prediction system 600 based on multimodal fundus images includes a multimodal data acquisition unit 610 , a plane feature extraction unit 620 , a stereo feature extraction unit 630 and a risk prediction unit 640 .

[0173] The multimodal data acquisition unit 610 is used to acquire a multimodal fundus image group corresponding to the target user; the multimodal fundus image group includes a first fundus image and a second fundus image; the first fundus image is obtained by color fundus photography, and the second fundus image is obtained by optical coherence tomography blood flow imaging.

[0174] The plane feature extraction unit 620 is used to extract a fundus plane feature group corresponding to the first fundus image based on a multi-scale attention model.

[0175] The stereo feature extraction unit 630 is used to extract the fundus stereo feature group corresponding to the second fundus image based on the region guided attention model.

[0176] The risk prediction unit 640 is used to input the fundus plane feature group and the fundus stereoscopic feature group into a risk prediction model to determine a corresponding coronary heart disease risk prediction result, and the risk prediction model adopts a multimodal fusion deep learning model.

[0177] In some embodiments, an embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the above-mentioned coronary heart disease risk prediction method based on multimodal fundus images of the present invention.

[0178] In some embodiments, an embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned coronary heart disease risk prediction method based on multimodal fundus images.

[0179] In some embodiments, an embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a coronary heart disease risk prediction method based on multimodal fundus images.

[0180] Figure 7 is a schematic diagram of the hardware structure of an electronic device for executing a method for predicting coronary heart disease risk based on multimodal fundus images provided by another embodiment of the present invention, such as Figure 7 As shown, the device includes:

[0181] One or more processors 710 and memory 720, Figure 7 A processor 710 is taken as an example.

[0182] The device for executing the method for predicting the risk of coronary heart disease based on multimodal fundus images may further include: an input device 730 and an output device 740 .

[0183] The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0184] The memory 720, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for predicting the risk of coronary heart disease based on multimodal fundus images in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 720, that is, the method for predicting the risk of coronary heart disease based on multimodal fundus images in the above method embodiment is realized.

[0185] The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 720 may optionally include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The input device 730 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0187] The one or more modules are stored in the memory 720, and when executed by the one or more processors 710, the coronary heart disease risk prediction method based on multimodal fundus images in any of the above method embodiments is executed.

[0188] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0189] The electronic devices of the embodiments of the present invention exist in various forms, including but not limited to:

[0190] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0191] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0192] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0193] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0194] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for predicting the risk of coronary heart disease based on multimodal fundus images, characterized in that: The method comprises: Acquire a multimodal fundus image group corresponding to a target user; the multimodal fundus image group includes a first fundus image and a second fundus image; the first fundus image is obtained by color fundus photography, and the second fundus image is obtained by optical coherence tomography blood flow imaging; Extracting a fundus plane feature group corresponding to the first fundus image based on a multi-scale attention model; Extracting a fundus stereo feature group corresponding to the second fundus image based on a region-guided attention model; The fundus plane feature group and the fundus stereoscopic feature group are input into a risk prediction model to determine a corresponding coronary heart disease risk prediction result, and the risk prediction model adopts a multimodal fusion deep learning model.

2. The method according to claim 1, characterized in that The fundus plane feature group includes at least one of the following: cup-to-disc ratio of left and right eyes, vertical cup-to-disc ratio, optic cup area, optic cup volume, optic disc area, average thickness of retinal nerve fiber layer, foveal avascular zone area, foveal avascular zone perimeter and near-circularity index.

3. The method according to claim 1 or 2, characterized in that: The fundus stereoscopic feature group includes at least one of the following: total vessel diameter, arterial diameter, venous diameter, total vessel fractal dimension, arterial fractal dimension, venous fractal dimension, total vessel tortuosity, arterial tortuosity, venous tortuosity, total vessel tortuosity density, arterial tortuosity density, venous tortuosity density and total vessel branching angle.

4. The method according to claim 3, characterized in that The risk prediction model includes a feature fusion layer, a deep residual network layer, a bidirectional long short-term memory network layer, a multi-head self-attention layer and a fully connected output layer; The feature fusion layer is used to fuse the fundus plane feature group and the fundus stereo feature group: F fusion =[F p ,F s ] In the formula, F p represents the fundus plane feature group, F s represents the fundus stereo feature group, and F fusion represents the fundus fusion feature group; The deep residual network layer includes multiple residual blocks, each residual block includes multiple convolutional layers and short connections between different convolutional layers; the deep residual network layer is used to extract the deep convolutional features corresponding to the fundus fusion feature group: H=F fusion +F(F fusion ,{W i }) Where F represents the convolution operation in the residual block, W i represents the weight of the i-th convolutional layer in the residual block, and H represents the deep convolutional features; The bidirectional long short-term memory network layer is used to determine the bidirectional temporal dependency features corresponding to the deep convolution features: In the formula, H BiLSTM Represents the bidirectional timing dependency feature, represents the output features of the forward LSTM of H, Represents the output features of the backward LSTM of H; The multi-head self-attention layer is used to process the bidirectional temporal dependency features to obtain the corresponding global attention features: H attention =MultiHead(Q,K,V)=[head1,head2,...,head h ]W O Q=H BiLSTM W Q K=H BiLSTM W K V=H BiLSTM W V Where W Q , W K and W V denote the query weight matrix, key weight matrix, and value weight matrix, respectively; Q, K, and V denote the query matrix, key matrix, and value matrix, respectively; softmax denotes the softmax function, Indicates the dimension of the key vector, head i represents the output of the i-th attention head; W O represents the multi-head output weight matrix, h represents the total number of attention heads, H attention Represents the global attention feature; The fully connected output layer is used to process the global attention features to obtain the corresponding coronary heart disease risk probability: A FC =ReLU(W FC A attention +b FC ) y=Sigmoid(W y H FC +b y ) Where y represents the final output of the risk probability of coronary heart disease, W FC and b FC Respectively represent the weight matrix and bias term of the fully connected layer; ReLU represents the nonlinear activation function, W y and b y They represent the weight matrix and bias term of the output layer respectively; Sigmoid represents the Sigmoid activation function.

5. The method according to claim 4, characterized in that The loss function L of the risk prediction model is total for: L total =λ1·L focal +λ2·L bayesian Where, L focal represents the focal loss term, L bayesian represents the Bayesian optimization loss term, λ1 and λ2 represent the importance coefficients of the corresponding loss terms; N represents the total number of samples in the data sample set; β t represents the category weight of the tth sample, p t is the model's predicted probability that the tth sample belongs to the true category, γ represents the adjustment factor; σ t is the prediction uncertainty of the t-th sample, y t is the true label of the t-th sample, is the predicted value of the tth sample.

6. The method according to claim 5, characterized in that The multi-scale attention model comprises a plurality of cascaded attention modules and a combination module; each of the attention modules comprises a convolution layer and an attention layer respectively, so as to extract attention features at corresponding scales; the combination module is used to determine a corresponding multi-scale attention feature map by combining the attention features at each scale, so as to obtain a fundus plane feature group; Extracting multi-scale image features corresponding to the first fundus image through the convolutional layers in each of the attention modules: F c =Conv c (I1) Where c = 1, 2, ..., K, K represents the total number of attention modules, and the convolutional layers in each attention module have corresponding convolution kernel sizes; I1 represents the first fundus image; Conv c represents the convolution operation of the convolutional layer in the cth attention module, F c Represents the feature map extracted by the cth convolutional layer; Based on the first-ranked first-scale attention module among the attention modules, a self-attention feature operation is performed to obtain a corresponding self-attention feature: G1=[g 1,1 ,g 1,2 ,...,g 1,C ] α1=softmax(W a1 ·G1+b a1 ) A1=α1·F1 Where G1 represents the global feature vector extracted by the first scale attention module, W a1 and b a1 They represent the weight matrix and bias term of the first-scale attention module respectively; α1 represents the attention weight corresponding to the first-scale attention module, and A1 represents the self-attention feature output by the first-scale attention module; f u,v,q represents the value of the qth channel of the feature map F1 at position (u, v), g 1,q represents the value of the qth channel after global average pooling, H, W and C represent the height, width and number of channels of F1 respectively; Based on the non-first-ranked attention modules, the calculated attention features are fused with the attention features output by the previous attention module to establish the connection between features of different scales: Q e =F e W Qe ,K e-1 =A e-1 W Ke ,V e-1 =A e-1 W Ve In the formula, Q e represents the query matrix output by the e-th attention module, K e-1 and V e-1 They represent the key matrix and value matrix output by the e-1th attention module respectively; W Qe , W Ke and W Ve Respectively represent the query weight matrix, key weight matrix and value weight matrix of the e-th attention module; α e,g represents the attention weight of the e-th attention module to the g-th position, e>1; The attention weight is determined by calculating the cosine similarity: Calculate the attention features based on the attention weights: In the formula, A e represents the attention feature of the e-th scale; After pooling all scales of attention features, we can get a complete multi-scale attention feature map: P e =Pooling(A e ) F MSA =Concat(P1,P2,…,P K ) Where P e represents the pooling result of the attention feature of the e-th scale, F MSA Represents a multi-scale attention feature map.

7. The method according to claim 6, characterized in that The regional guided attention model includes a plurality of regional attention modules and a fusion module; each of the regional attention modules includes a cascaded regional convolution layer and a regional guided attention layer to extract the regional guided attention features of the corresponding region; the fusion module is used to fuse the regional guided attention features of each region to obtain a corresponding fundus stereo feature group; Through the regional convolution layer in each regional attention module, the regional convolution feature map of the second fundus image in each specific area is extracted: R l =Conv l (I2) Where, l = 1, 2, ..., U, U is the total number of regional attention modules, each of which is used to process the corresponding demarcated pixel area; I2 represents the second fundus image; R l represents the regional convolution feature map extracted by the regional convolution layer in the lth regional attention module, Conv l represents the convolution operation of the regional convolution layer in the l-th regional attention module; Through the regional guided attention layer in each regional attention module, the regional convolution feature map is weighted using the membrane layer regional image to obtain the corresponding regional guided feature map: R lM =R l ·M Where M represents the film area image, R lM represents the region-guided feature map determined by the region-guided attention layer in the l-th region-attention module; The region guides the feature map R lM Divide into multiple sub-regions to guide feature maps Where s represents the sth sub-region; For each sub-region guided feature map The corresponding local feature vector is obtained by local average pooling Calculate the local feature vector through the fully connected layer The corresponding local attention weight In the formula, They represent the weight matrix and bias term calculated by the fully connected layer for the attention weight of the s-th sub-region of the l-th region, respectively. represents the attention weight of the s-th sub-region of the l-th region; Guide feature maps for each sub-region Weighted, we get the sub-region attention-guiding features: In the formula, represents the sub-region guided attention feature of the s-th sub-region of the l-th region; The attention features of all sub-regions are concatenated to obtain a regional guided attention feature map, and then fused through the fusion module: F RGA =Concat(F RGA,1 ,F RGA,2 ,…,F RGA,U ) In the formula, F RGA,l represents the regional attention feature map of the lth region, S represents the total number of sub-regions contained in the lth region, and F RGA Represents the fused features based on the region-guided attention features of each region.

8. The method according to claim 7, characterized in that The film layer area image is determined by: For each pixel in the second fundus image, calculate the local mean within a window centered at the pixel: Where (i, j) represents the position of the pixel in the image, μ(i, j) represents the local average pixel value at (i, j), I2(i+m, j+n) represents the pixel value of the second fundus image I2 at the position (i+m, j+n), and w represents the window size; Compute the local standard deviation: Where σ(i,j) represents the local standard deviation at (i,j); Compute an adaptive threshold using the local mean and standard deviation: Where r represents the adjustment parameter, T(i,j) represents the adaptive threshold at (i,j); The second fundus image is binarized according to the adaptive threshold T(i,j) to obtain the membrane area image: Where M(i, j) represents the pixel value of the film region image M at (i, j), and I2(i, j) represents the pixel value of the image I2 at (i, j).

9. A coronary heart disease risk prediction system based on multimodal fundus images, characterized in that: The system comprises: A multimodal data acquisition unit, used to acquire a multimodal fundus image group corresponding to a target user; the multimodal fundus image group includes a first fundus image and a second fundus image; the first fundus image is obtained by color fundus photography, and the second fundus image is obtained by optical coherence tomography blood flow imaging; A plane feature extraction unit, configured to extract a fundus plane feature group corresponding to the first fundus image based on a multi-scale attention model; A stereo feature extraction unit, configured to extract a fundus stereo feature group corresponding to the second fundus image based on a region-guided attention model; The risk prediction unit is used to input the fundus plane feature group and the fundus stereoscopic feature group into a risk prediction model to determine a corresponding coronary heart disease risk prediction result, wherein the risk prediction model adopts a multimodal fusion deep learning model.

Images