Eye fundus image feature-based diabetes mellitus great vasculopathy prediction method and device

Abstract

The invention provides a diabetes mellitus great vasculopathy prediction method and device based on fundus image features, and belongs to the technical field of image processing, and the method comprises the steps: constructing a diabetes mellitus carotid intima middle layer thickness bilateral fundus data set; performing blood vessel segmentation processing on the diabetic carotid intima middle layer thickness bilateral fundus data set by using a semantic segmentation model based on visual stimulation self-adaption to obtain a binocular blood vessel data set; training the carotid intima middle-layer thickness prediction model by using the diabetic carotid intima middle-layer thickness bilateral fundus data set and the binocular blood vessel data set to obtain a trained carotid intima middle-layer thickness prediction model; and performing diabetes mellitus great vasculopathy prediction by using the trained carotid intima middle layer thickness prediction model. According to the method, the pressure of medical resources can be effectively relieved, the cost is reduced, and the detection rate of the great vasculopathy marker of the diabetic patient is increased, so that the application of precise medical treatment in screening of the great vasculopathy of the diabetic patient is promoted.

Description

Background Art

[0002] In my country, the number of diabetes patients has reached about 141 million, making it the country with the largest number of diabetes patients in the world. Diabetic patients often suffer from macrovascular disease, which is the primary cause of disability and death caused by diabetes. Early identification and prevention of macrovascular disease is crucial to improving the long-term health status and quality of life of diabetic patients.

[0003] Carotid Intima-Media Thickness (CIMT) is widely considered to be an important indicator for evaluating macrovascular disease, especially closely related to cardiovascular risk. Studies have shown that for every 0.1 mm increase in CIMT, the patient's risk of death increases by 1.12 times. However, although CIMT is an effective biomarker, conventional B-mode ultrasound measurement methods still have multiple limitations in practical applications, such as high equipment cost, high professional technical requirements, long measurement time, and poor accessibility. These limit the widespread application of CIMT in the general screening of diabetic macrovascular disease. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present invention provides a method and device for predicting diabetic macroangiopathy based on fundus image features.

[0005] According to a first aspect of an embodiment of the present invention, a method for predicting diabetic macroangiopathy based on fundus image features is provided, the method comprising:

[0006] Construct a bilateral fundus dataset of diabetic carotid intima-media thickness;

[0007] Using a semantic segmentation model based on visual stimulus adaptation, the bilateral fundus dataset of diabetic carotid intima-media thickness is processed for vascular segmentation to obtain a binocular vascular dataset;

[0008] Using the diabetic carotid intima-media thickness bilateral fundus data set and the binocular blood vessel data set to train a carotid intima-media thickness prediction model, to obtain a trained carotid intima-media thickness prediction model;

[0009] The trained carotid intima-media thickness prediction model is used to predict diabetic macrovascular disease.

[0010] In some exemplary embodiments of the present invention, based on the above scheme, constructing a bilateral fundus dataset of diabetic carotid intima-media thickness includes:

[0011] Access medical images and clinical data related to diabetic patients;

[0012] Screening the medical images and the clinical data based on inclusion criteria and exclusion criteria to obtain bilateral fundus data;

[0013] The bilateral fundus data are cleaned and standardized to obtain the diabetic carotid intima-media thickness bilateral fundus data set.

[0014] In some exemplary embodiments of the present invention, based on the aforementioned scheme, the inclusion criteria include: patients hospitalized in the endocrinology department, patients who have undergone fundus imaging and carotid artery ultrasound examination during hospitalization;

[0015] The exclusion criteria included: images that could not provide valid research data, patients lacking binocular images, and patients without detailed carotid intima-media thickness measurement records.

[0016] In some exemplary embodiments of the present invention, based on the above scheme, a semantic segmentation model based on visual stimulus adaptation is used to perform vascular segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness, and the obtained vascular segmentation dataset includes:

[0017] Performing blood vessel segmentation processing on each data in the bilateral fundus data set of diabetic carotid intima-media thickness to obtain a segmented data set;

[0018] Performing data enhancement processing on each data in the segmented data set to obtain an enhanced image data set;

[0019] Using a semantic segmentation model based on visual stimulus adaptation to perform vascular feature extraction processing on each image data in the enhanced image data set to obtain a vascular feature data set;

[0020] Merging the features belonging to the same eyeball in the blood vessel feature data set to obtain eyeball blood vessel features;

[0021] Performing color transformation on the eyeball blood vessel features to obtain a blood vessel segmentation map;

[0022] All blood vessel segmentation images are packaged to obtain the blood vessel segmentation dataset.

[0023] In some exemplary embodiments of the present invention, based on the above scheme, the semantic segmentation model based on visual stimulus adaptation includes:

[0024] A downsampling module, a stimulus adaptive feature extraction module, a first encoder, a second encoder, a third encoder, a first decoder, a second decoder, a third decoder, a fourth decoder and an upsampling module;

[0025] Wherein, each image data in the enhanced image data set enters the stimulus adaptive feature extraction module after being sampled by the downsampling module. The stimulus adaptive feature extraction module includes a first feature extraction unit and a second feature extraction unit arranged in parallel, and a feature fusion layer. The first feature extraction unit is a feature extraction unit based on a convolutional network, and the second feature extraction unit is a stimulus-guided adaptive feature extraction unit. The first feature extraction unit and the second feature extraction unit respectively extract features from the output of the downsampling module, and perform feature fusion through the feature fusion layer. The output of the feature fusion layer serves as the input of the first encoder. The first encoder, the second encoder and the third encoder are arranged in sequence. The output of the third encoder and the output of the second encoder are simultaneously used as the input of the first decoder. The output of the first encoder and the output of the first decoder are simultaneously used as the input of the second decoder. The output of the second decoder and the output of the stimulus adaptive feature extraction module are simultaneously used as the input of the third decoder. The output of the third decoder and the input of the downsampling module are simultaneously used as the input of the fourth decoder. The output of the fourth decoder serves as the input of the upsampling module. The output of the upsampling module is vascular feature data.

[0026] In some exemplary embodiments of the present invention, based on the above scheme, the carotid intima-media thickness prediction model includes:

[0027] A first preprocessing module, a second preprocessing module, a third preprocessing module, a fourth preprocessing module, a first twin neural network based on an attention mechanism, a second twin neural network based on an attention mechanism, a third twin neural network based on an attention mechanism, a fourth twin neural network based on an attention mechanism, a feature fusion module and an output module;

[0028] Among them, the first preprocessing module, the second preprocessing module, the third preprocessing module and the fourth preprocessing module are arranged in parallel to preprocess the left eye fundus data, the right eye fundus data, the left eye vascular data and the right eye vascular data in the multimodal fundus image fusion data set respectively, the first attention mechanism-based twin neural network, the second attention mechanism-based twin neural network, the third attention mechanism-based twin neural network and the fourth attention mechanism-based twin neural network are arranged in parallel to extract features of the preprocessed left eye fundus data, the preprocessed right eye fundus data, the preprocessed left eye vascular data and the preprocessed right eye vascular data respectively, and generate a normal carotid intima-media thickness group and a carotid intima-media thickness thickening group after feature fusion using the feature fusion module; the first attention mechanism-based twin neural network and the second attention mechanism-based twin neural network share weights, and the third attention mechanism-based twin neural network and the fourth attention mechanism-based twin neural network share weights.

[0029] In some exemplary embodiments of the present invention, based on the above scheme, the loss function H of the carotid intima-media thickness prediction model is:

[0030]

[0031] Among them, x i represents the label of the i-th sample, p(x i ) represents the probability of the true value distribution of the i-th label, q(x i ) represents the probability value of the i-th label predicting a certain category, and n represents the total number of sample labels.

[0032] According to a second aspect of an embodiment of the present invention, there is provided a device according to the above-mentioned method for predicting diabetic macroangiopathy based on fundus image features, comprising:

[0033] Dataset construction module, used to construct bilateral fundus dataset of diabetic carotid intima-media thickness;

[0034] A blood vessel segmentation module is used to perform blood vessel segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness using a semantic segmentation model based on visual stimulation adaptation to obtain a binocular blood vessel dataset;

[0035] A network training module, used to train a carotid intima-media thickness prediction model using the diabetic carotid intima-media thickness bilateral fundus data set and the binocular vascular data set to obtain a trained carotid intima-media thickness prediction model;

[0036] The result prediction module is used to predict diabetic macrovascular lesions using the trained carotid intima-media thickness prediction model.

[0037] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method for predicting diabetic macrovascular lesions based on fundus image features in the first aspect is implemented.

[0038] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for predicting diabetic macroangiopathy based on fundus image features in the first aspect is implemented.

[0039] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0040] The present invention uses a semantic segmentation model based on visual stimulation adaptation to perform vascular segmentation on the bilateral fundus dataset of diabetic carotid intima-media thickness, which can reduce the errors and human intervention of traditional manual annotation, accurately extract vascular information in fundus images, and improve the accuracy and stability of segmentation effect; combining the bilateral fundus dataset of diabetic carotid intima-media thickness and the vascular dataset, training a carotid intima-media thickness prediction model, and through effective data fusion, it can ensure the prediction ability of the training model for diabetic macrovascular lesions, so that based on the vascular features in the fundus images, it can automatically identify and predict the changes in the carotid intima-media, thereby effectively predicting macrovascular lesions.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 A schematic diagram showing a system architecture of an exemplary application environment of a method and device for predicting diabetic macroangiopathy based on fundus image features according to an embodiment of the present invention is applicable;

[0044] Figure 2 Schematically showing a flow chart of a method for predicting diabetic macroangiopathy based on fundus image features according to some embodiments of the present invention;

[0045] Figure 3Schematically showing a schematic diagram of a process for constructing a bilateral fundus dataset of diabetic carotid intima-media thickness according to some embodiments of the present invention;

[0046] Figure 4 A schematic diagram of a process of performing vascular segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness using a semantic segmentation model based on visual stimulus adaptation to obtain a vascular segmentation dataset according to some embodiments of the present invention;

[0047] Figure 5 The schematic diagram shows a process of predicting diabetic macrovascular disease using a trained carotid intima-media thickness prediction model according to some embodiments of the present invention.

[0048] Figure 6 A schematic diagram of a device for predicting diabetic macroangiopathy based on fundus image features according to some embodiments of the present invention is schematically shown;

[0049] Figure 7 A schematic diagram schematically shows a structure of a computer system of an electronic device according to some embodiments of the present invention;

[0050] Figure 8 A schematic diagram of a computer-readable storage medium according to some embodiments of the present invention is schematically shown. DETAILED DESCRIPTION

[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0052] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0053] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0054] Figure 1 A schematic diagram of a system architecture of an exemplary application environment of a method and apparatus for predicting diabetic macroangiopathy based on fundus image features, to which an embodiment of the present invention can be applied, is shown.

[0055] like Figure 1 As shown, the system architecture 100 may include one or more terminal devices such as a desktop computer 101, a portable computer 102, a smart phone 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The terminal device may be any electronic device with a data processing function, which has a display screen for displaying the prediction results of diabetic macrovascular disease to the user, including but not limited to the above-mentioned desktop computers, portable computers, smart phones, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a sub-server cluster composed of multiple sub-servers.

[0056] The method for predicting diabetic macroangiopathy based on fundus image features provided in the embodiment of the present invention can generally be executed by a terminal device, and accordingly, the device for predicting diabetic macroangiopathy based on fundus image features is generally disposed in the terminal device. However, it is easy for a person skilled in the art to understand that the method for predicting diabetic macroangiopathy based on fundus image features provided in the embodiment of the present invention can also be executed by the server 105, and accordingly, the device for predicting diabetic macroangiopathy based on fundus image features can also be disposed in the server 105, which is not particularly limited in this exemplary embodiment.

[0057] In addition, it should be understood that the method for predicting diabetic macroangiopathy based on fundus image features of the embodiment of the present invention can be configured as a software module. In some implementation scenarios, the diabetic macroangiopathy prediction scheme based on fundus image features of the present invention can be deployed separately to achieve macroangiopathy prediction in different diabetic patients. In other implementation scenarios, the diabetic macroangiopathy prediction scheme based on fundus image features of the present invention can be deployed in other software as a functional module of the software, such as deployed in analysis software for underground pipelines. The present invention does not place any special restrictions on the application of the diabetic macroangiopathy prediction method based on fundus image features.

[0058] Next, the embodiments of the present invention are described in detail.

[0059] like Figure 1 As shown, Figure 1 The present invention is a flowchart of a method for predicting diabetic macroangiopathy based on fundus image features according to an exemplary embodiment of the present invention, comprising the following steps:

[0060] S210: Construct a bilateral fundus dataset of diabetic carotid intima-media thickness;

[0061] S220: performing vascular segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness using a semantic segmentation model based on visual stimulation adaptation to obtain a binocular vascular dataset;

[0062] S230: training a carotid intima-media thickness prediction model using the diabetic carotid intima-media thickness bilateral fundus dataset and the binocular vascular dataset to obtain a trained carotid intima-media thickness prediction model;

[0063] S240: Predicting diabetic macrovascular disease using the trained carotid intima-media thickness prediction model.

[0064] In S210, a bilateral fundus dataset of diabetic carotid intima-media thickness is constructed.

[0065] Diabetic patients often have vascular lesions such as arteriosclerosis. Carotid Intima-Media Thickness (CIMT) is an important indicator of arteriosclerosis. CIMT is an important predictive biomarker for assessing atherosclerosis and cardiovascular risk, and its changes reflect changes in vascular health. Studies have shown that for every 0.1 mm increase in CIMT, the risk of death increases by 1.12 times. Accurate measurement and monitoring of CIMT is essential for early detection of cardiovascular disease and guiding clinical intervention.

[0066] The structure and function of fundus vessels are significantly affected by systemic diseases such as atherosclerosis, hypertension, and diabetic nephropathy, so fundus vessels are an important sensitive indicator of systemic arterial health. Anatomically, the central retinal artery (a branch of the internal carotid artery) is closely related to the carotid artery. Therefore, hemodynamic changes in the carotid artery can directly affect the morphology and function of retinal vessels, manifested as typical pathological changes, including retinal artery stenosis, reduced microvascular density, venous dilatation, arteriovenous cross compression, cotton spots, punctate hemorrhages, and hard exudates. These manifestations are particularly significant in the case of carotid atherosclerosis or stenosis. Therefore, the abnormal microvascular structure observed in fundus images can be used as an imaging biomarker of carotid artery lesions and provide early warning for carotid and other cardiovascular diseases.

[0067] Fundus images are rich in details and complex in features. Due to their high dimensionality, lesion identification and analysis are challenging. Traditional diagnostic methods have limitations in detecting small lesions (such as retinal artery stenosis and reduced microvascular density), especially when multiple lesions coexist.

[0068] To this end, the present invention first constructs a bilateral fundus dataset of carotid intima-media thickness in patients with diabetes, including high-quality binocular fundus images, CIMT measurement data, and comprehensive clinical information (including age and gender). The bilateral fundus dataset of carotid intima-media thickness in patients with diabetes can make up for the lack of public datasets in cardiovascular disease prediction, providing a valuable multimodal resource that supports the comprehensive analysis of fundus images and CIMT data, and promotes the discovery of early markers of cardiovascular disease. Secondly, the dataset provides a solid foundation for the validation and generalization of artificial intelligence models, enabling researchers to evaluate the performance, stability, and generalization ability of the model, thereby accelerating the application of AI technology in cardiovascular risk prediction. Finally, the dataset has great potential to promote the development of non-invasive, efficient, and low-cost early screening tools for cardiovascular disease based on fundus images, and has important practical application value, especially in areas with limited resources.

[0069] refer to Figure 3 As shown in the figure, the construction process of the bilateral fundus dataset of diabetic carotid intima-media thickness includes:

[0070] Access medical images and clinical data related to diabetic patients;

[0071] Screening the medical images and the clinical data based on inclusion criteria and exclusion criteria to obtain bilateral fundus data;

[0072] The bilateral fundus data are cleaned and standardized to obtain the diabetic carotid intima-media thickness bilateral fundus data set.

[0073] The present invention first conducted a retrospective analysis of patient data from the Department of Endocrinology of the Second Affiliated Hospital of Anhui Medical University and the Department of Endocrinology of Hefei People's Hospital. The data collection time was from April 2025 to February 2024, and the data content included medical images and clinical data.

[0074] Afterwards, the medical images and clinical data were screened using the inclusion and exclusion criteria. Here, the inclusion criteria included: 1) patients hospitalized in the endocrinology department; 2) patients who underwent fundus imaging and carotid ultrasound during hospitalization. The exclusion criteria were: 1) images that could not provide valid research data; 2) patients who lacked binocular images; 3) patients without detailed carotid intima-media thickness measurement records.

[0075] Finally, the bilateral fundus data are cleaned and standardized to obtain the diabetic carotid intima-media thickness bilateral fundus data set.

[0076] Cleaning refers to the removal or correction of possible errors or invalid parts in fundus image data, which may include noise, blur, missing image parts, abnormal data, etc., to ensure that the data is accurate and valid, thereby ensuring the quality of subsequent analysis.

[0077] Standardization is the process of converting data into a unified format so that different data are comparable. In an embodiment of the present invention, standardization may include unifying image sizes, rotating at random angles, and mirror flipping and upside-down flipping of left and right eye images to ensure the diversity and representativeness of the data set.

[0078] In S220, a visual stimulus-based adaptive semantic segmentation model is used to perform vascular segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness to obtain a binocular vascular dataset.

[0079] In some embodiments, using a semantic segmentation model based on visual stimulus adaptation to perform vascular segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness to obtain a binocular vascular dataset may include:

[0080] Performing blood vessel segmentation processing on each data in the bilateral fundus data set of diabetic carotid intima-media thickness to obtain a segmented data set;

[0081] Performing data enhancement processing on each data in the segmented data set to obtain an enhanced image data set;

[0082] Using a semantic segmentation model based on visual stimulus adaptation to perform vascular feature extraction processing on each image data in the enhanced image data set to obtain a vascular feature data set;

[0083] Merging the features belonging to the same eyeball in the blood vessel feature data set to obtain eyeball blood vessel features;

[0084] Performing color transformation on the eyeball blood vessel features to obtain a blood vessel segmentation map;

[0085] All blood vessel segmentation images are packaged to obtain the blood vessel segmentation dataset.

[0086] In the embodiment of the present invention, after constructing the bilateral fundus dataset of diabetic carotid intima-media thickness, each data is first segmented into 16 equal parts, and the segmentation results of all data are summarized as the segmented dataset. Of course, the data can be segmented into 14 equal parts, 12 equal parts, 10 equal parts, 8 equal parts, etc., and the present invention does not make specific limitations.

[0087] On this basis, data enhancement processing is performed on each data in the segmented data set. The data enhancement processing method can be one or more combinations of rotation, translation, scaling, flipping, color change, and noise addition. Those skilled in the art can set it according to the situation. Then, the data after data enhancement processing of all data are summarized to obtain an enhanced image data set.

[0088] Finally, the enhanced image dataset is fed into the visual stimulus guided adaptive

[0089] The trained Transformer-UNet segmentation network is trained to obtain the adaptive Transformer-UNet segmentation network guided by visual stimulation, which is the semantic segmentation model based on visual stimulation adaptation of the present invention.

[0090] Structural reference for semantic segmentation models based on visual stimulus adaptation Figure 4 As shown, including:

[0091] A downsampling module, a stimulus adaptive feature extraction module, a first encoder, a second encoder, a third encoder, a first decoder, a second decoder, a third decoder, a fourth decoder and an upsampling module;

[0092] Wherein, each image data in the enhanced image data set enters the stimulus adaptive feature extraction module after being sampled by the downsampling module. The stimulus adaptive feature extraction module includes a first feature extraction unit and a second feature extraction unit arranged in parallel, and a feature fusion layer. The first feature extraction unit is a feature extraction unit based on a convolutional network, and the second feature extraction unit is a stimulus-guided adaptive feature extraction unit. The first feature extraction unit and the second feature extraction unit respectively extract features from the output of the downsampling module, and perform feature fusion through the feature fusion layer. The output of the feature fusion layer serves as the input of the first encoder. The first encoder, the second encoder and the third encoder are arranged in sequence. The output of the third encoder and the output of the second encoder are simultaneously used as the input of the first decoder. The output of the first encoder and the output of the first decoder are simultaneously used as the input of the second decoder. The output of the second decoder and the output of the stimulus adaptive feature extraction module are simultaneously used as the input of the third decoder. The output of the third decoder and the input of the downsampling module are simultaneously used as the input of the fourth decoder. The output of the fourth decoder serves as the input of the upsampling module. The output of the upsampling module is vascular feature data.

[0093] The fused features will go through 3 downsampling and 3 upsampling processes to accurately capture the detailed features of the fundus blood vessels. During the model training process, an iterative optimization strategy is adopted and the network parameters are continuously adjusted based on the loss function in order to obtain a high-performance semantic segmentation model based on visual stimulus adaptation.

[0094] In S230, a carotid intima-media thickness prediction model is trained using the diabetic carotid intima-media thickness bilateral fundus dataset and the binocular vascular dataset to obtain a trained carotid intima-media thickness prediction model.

[0095] After obtaining the binocular vascular dataset, the carotid intima-media thickness prediction model was trained using the diabetic carotid intima-media thickness bilateral fundus dataset and binocular vascular dataset.

[0096] Here, reference Figure 5 As shown in Figure 2, the carotid intima-media thickness prediction model includes:

[0097] A first preprocessing module, a second preprocessing module, a third preprocessing module, a fourth preprocessing module, a first twin neural network based on an attention mechanism, a second twin neural network based on an attention mechanism, a third twin neural network based on an attention mechanism, a fourth twin neural network based on an attention mechanism, a feature fusion module and an output module;

[0098] Among them, the first preprocessing module, the second preprocessing module, the third preprocessing module and the fourth preprocessing module are arranged in parallel to preprocess the left eye fundus data, the right eye fundus data, the left eye vascular data and the right eye vascular data in the multimodal fundus image fusion data set respectively, the first attention mechanism-based twin neural network, the second attention mechanism-based twin neural network, the third attention mechanism-based twin neural network and the fourth attention mechanism-based twin neural network are arranged in parallel to extract features of the preprocessed left eye fundus data, the preprocessed right eye fundus data, the preprocessed left eye vascular data and the preprocessed right eye vascular data respectively, and generate a normal carotid intima-media thickness group and a carotid intima-media thickness thickening group after feature fusion using the feature fusion module; the first attention mechanism-based twin neural network and the second attention mechanism-based twin neural network share weights, and the third attention mechanism-based twin neural network and the fourth attention mechanism-based twin neural network share weights.

[0099] The present invention uses CIMT and 0.9 mm to distinguish the thickened group, the normal group and the thickened group of the carotid intima-media thickness. CIMT<0.9 mm is the normal group, and CIMT≥0.9 mm is the thickened group.

[0100] This framework trains the bilateral fundus CIMT dataset and binocular vascular dataset by fusing two single-modal features and sharing parameters. The preprocessed and enhanced left and right fundus images and left and right eye vascular data are fed into four parallel ResNeXt50 networks to extract image features. Subsequently, these features are merged in the feature fusion module, and the spatial attention mechanism is used to accurately locate and fuse key vascular features.

[0101] The first twin neural network based on the attention mechanism, the second twin neural network based on the attention mechanism, the third twin neural network based on the attention mechanism and the fourth twin neural network based on the attention mechanism have the same structure, all of which are ResNeXt50 neural networks based on the attention mechanism. The ResNeXt50 neural network based on the attention mechanism is a model that combines the ResNeXt50 architecture and the attention mechanism. The attention mechanism here can be spatial attention or channel attention. Since the attention mechanism and the ResNeXt50 neural network are both prior art, the present invention will not be repeated.

[0102] The process of S230 uses multiple rounds of iterative training, guided by the loss function, and continuously optimizes the parameters of the carotid intima-media thickness prediction model through back propagation to obtain a trained carotid intima-media thickness prediction model. The loss function is:

[0103]

[0104] Among them, x i represents the i-th sample label, p(x i ) represents the probability of the true value distribution of the i-th label, q(x i ) represents the probability value of the i-th label predicting a certain category, and n represents the total number of sample labels.

[0105] The weight function of the carotid intima-media thickness prediction model is:

[0106]

[0107] Among them, w i represents the weight value of the normal group and the thickening group, n1 represents the number of normal groups, n2 represents the number of thickening groups, and N represents the sum of the number of normal and thickening groups.

[0108] The optimizer of the carotid intima-media thickness prediction model is a gradient-based optimizer named Adam. The batch size is 32 for training, the initial learning rate is set to 0.001, regularization and dropout strategies are applied to reduce overfitting, and dynamic learning rate adjustment and early stopping mechanisms are used to ensure the efficiency and stability of training. Two GT1080ti graphics cards and an 8-core CPU are used to accelerate training, and a total of 500 rounds are performed. After each round, the classification accuracy is evaluated on the validation set and the model parameters are adjusted. Finally, the model with the best performance on the validation set is saved.

[0109] The Adam optimization algorithm is a popular gradient descent method for optimizing the parameters of machine learning models. The algorithm updates the value of each parameter of the neural network by calculating the weighted moving average of the gradient of each parameter and the weighted moving average of the square of the gradient. The algorithm uses momentum parameters and learning rates to adjust the speed and stability of the algorithm, which can better solve the problem of oscillation and stagnation of parameter updates. Its parameter update formula is:

[0110] m t =β1m t-1 +(1-β1)g t

[0111]

[0112] Among them, m t and v t are the weighted moving average of the gradient and the weighted moving average of the square of the gradient, θ i is the parameter update formula. β1 and β2 are momentum parameters, g t is the current gradient, α is the learning rate, and ∈ is a very small constant to prevent the denominator from being zero.

[0113] In addition, refer to Figure 5 As shown in the figure, the historical prediction results of diabetic macrovascular disease can also be used as training data for training the carotid intima-media thickness prediction model, and can be combined with the diabetic carotid intima-media thickness bilateral fundus dataset and binocular vascular dataset to jointly train the carotid intima-media thickness prediction model.

[0114] In S240, the trained carotid intima-media thickness prediction model is used to predict diabetic macroangiopathy.

[0115] According to a second aspect of an embodiment of the present invention, a device for predicting diabetic macroangiopathy based on fundus image features is also provided. Figure 6 As shown, the device for predicting diabetic macroangiopathy based on fundus image features comprises:

[0116] The data set construction module 610 is used to construct a bilateral fundus data set of carotid intima-media thickness in diabetes mellitus;

[0117] A blood vessel segmentation module 620 is used to perform blood vessel segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness by using a semantic segmentation model based on visual stimulation adaptation to obtain a binocular blood vessel dataset;

[0118] A network training module 630 is used to train a carotid intima-media thickness prediction model using the diabetic carotid intima-media thickness bilateral fundus dataset and the binocular vascular dataset to obtain a trained carotid intima-media thickness prediction model;

[0119] The result prediction module 640 is used to predict diabetic macroangiopathy using the trained carotid intima-media thickness prediction model.

[0120] In an exemplary embodiment of the present invention, based on the above solution, the data set construction module 610 may further include:

[0121] A data acquisition unit, used for acquiring medical images and clinical data related to diabetic patients;

[0122] A data screening unit, used for screening the medical images and the clinical data based on inclusion criteria and exclusion criteria to obtain bilateral fundus data;

[0123] The data cleaning unit is used to clean and standardize the bilateral fundus data to obtain the diabetic carotid intima-media thickness bilateral fundus data set.

[0124] In an exemplary embodiment of the present invention, based on the above solution, the blood vessel segmentation module 620 may include:

[0125] A data segmentation unit, used for performing a blood vessel segmentation process on each data in the bilateral fundus data set of the diabetic carotid intima-media thickness to obtain a segmented data set;

[0126] A data enhancement unit, used for performing data enhancement processing on each data in the segmented data set to obtain an enhanced image data set;

[0127] A feature extraction unit, configured to perform a vascular feature extraction process on each image data in the enhanced image data set by using a semantic segmentation model based on visual stimulus adaptation to obtain a vascular feature data set;

[0128] A feature merging unit, used for merging the features belonging to the same eyeball in the blood vessel feature data set to obtain eyeball blood vessel features;

[0129] A color conversion unit, used for performing color conversion on the eyeball blood vessel features to obtain a blood vessel segmentation map;

[0130] The data integration unit is used to package all the blood vessel segmentation images to obtain the blood vessel segmentation data set.

[0131] It should be noted that, although several modules and modules of the underground pipeline detection device are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0132] In addition, in an exemplary embodiment of the present invention, an electronic device capable of implementing the above-mentioned method for predicting diabetic macroangiopathy based on fundus image features is also provided.

[0133] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0134] Refer to the following Figure 7 An electronic device 700 according to such an embodiment of the present invention is described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0135] like Figure 7As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0136] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps of various exemplary embodiments of the present invention described in the above “Exemplary Method” section. For example, the processing unit 710 can perform the following steps: Figure 2 S210 shown in: constructing a bilateral fundus dataset of diabetic carotid intima-media thickness; S220: using a semantic segmentation model based on visual stimulation adaptation to perform vascular segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness to obtain a binocular vascular dataset; S230: using the bilateral fundus dataset of diabetic carotid intima-media thickness and the binocular vascular dataset to train a carotid intima-media thickness prediction model to obtain a trained carotid intima-media thickness prediction model; S240: using the trained carotid intima-media thickness prediction model to predict diabetic macrovascular lesions.

[0137] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 721 and / or a cache memory unit 722 , and may further include a read-only memory unit (ROM) 723 .

[0138] The storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0139] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0140] The electronic device 700 may also communicate with one or more external devices 770 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0141] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0142] In an exemplary embodiment of the present invention, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present invention is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps of various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present invention.

[0143] refer to Figure 8 As shown, a program product 800 for implementing the above-mentioned method for predicting diabetic macroangiopathy based on fundus image features according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In the present invention, the readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in combination with an instruction execution system, an apparatus or a device.

[0144] The program product may employ any combination of one or more readable storage media. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0145] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0146] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0147] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0148] Other embodiments of the invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0149] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for predicting diabetic macroangiopathy based on fundus image features, characterized in that: include: Construct a bilateral fundus dataset of diabetic carotid intima-media thickness; Using a semantic segmentation model based on visual stimulus adaptation, the bilateral fundus dataset of diabetic carotid intima-media thickness is processed for vascular segmentation to obtain a binocular vascular dataset; Using the diabetic carotid intima-media thickness bilateral fundus data set and the binocular blood vessel data set to train a carotid intima-media thickness prediction model, to obtain a trained carotid intima-media thickness prediction model; The trained carotid intima-media thickness prediction model is used to predict diabetic macrovascular disease.

2. The method for predicting diabetic macroangiopathy based on fundus image features according to claim 1, characterized in that: The construction of a bilateral fundus data set of diabetic carotid intima-media thickness includes: Access medical images and clinical data related to diabetic patients; Screening the medical images and the clinical data based on inclusion criteria and exclusion criteria to obtain bilateral fundus data; The bilateral fundus data are cleaned and standardized to obtain the diabetic carotid intima-media thickness bilateral fundus data set.

3. The method for predicting diabetic macroangiopathy based on fundus image features according to claim 2, characterized in that: The inclusion criteria described included: patients admitted to the endocrinology department, patients who underwent fundus imaging and carotid artery ultrasound during hospitalization; The exclusion criteria included: images that could not provide valid research data, patients lacking binocular images, and patients without detailed carotid intima-media thickness measurement records.

4. The method for predicting diabetic macroangiopathy based on fundus image features according to claim 1, characterized in that: The diabetic carotid intima-media thickness bilateral fundus dataset is processed for vascular segmentation using a semantic segmentation model based on visual stimulation adaptation, and the obtained vascular segmentation dataset includes: Performing blood vessel segmentation processing on each data in the bilateral fundus data set of diabetic carotid intima-media thickness to obtain a segmented data set; Performing data enhancement processing on each data in the segmented data set to obtain an enhanced image data set; Using a semantic segmentation model based on visual stimulus adaptation to perform vascular feature extraction processing on each image data in the enhanced image data set to obtain a vascular feature data set; Merging the features belonging to the same eyeball in the blood vessel feature data set to obtain eyeball blood vessel features; Performing color transformation on the eyeball blood vessel features to obtain a blood vessel segmentation map; All blood vessel segmentation images are packaged to obtain the blood vessel segmentation dataset.

5. The method for predicting diabetic macroangiopathy based on fundus image features according to claim 4, characterized in that: The semantic segmentation model based on visual stimulus adaptation includes: A downsampling module, a stimulus adaptive feature extraction module, a first encoder, a second encoder, a third encoder, a first decoder, a second decoder, a third decoder, a fourth decoder and an upsampling module; Wherein, each image data in the enhanced image data set enters the stimulus adaptive feature extraction module after being sampled by the downsampling module. The stimulus adaptive feature extraction module includes a first feature extraction unit and a second feature extraction unit arranged in parallel, and a feature fusion layer. The first feature extraction unit is a feature extraction unit based on a convolutional network, and the second feature extraction unit is a stimulus-guided adaptive feature extraction unit. The first feature extraction unit and the second feature extraction unit respectively extract features from the output of the downsampling module, and perform feature fusion through the feature fusion layer. The output of the feature fusion layer serves as the input of the first encoder. The first encoder, the second encoder and the third encoder are arranged in sequence. The output of the third encoder and the output of the second encoder are simultaneously used as the input of the first decoder. The output of the first encoder and the output of the first decoder are simultaneously used as the input of the second decoder. The output of the second decoder and the output of the stimulus adaptive feature extraction module are simultaneously used as the input of the third decoder. The output of the third decoder and the input of the downsampling module are simultaneously used as the input of the fourth decoder. The output of the fourth decoder serves as the input of the upsampling module. The output of the upsampling module is vascular feature data.

6. The method for predicting diabetic macroangiopathy based on fundus image features according to claim 1, characterized in that: The carotid intima-media thickness prediction model includes: A first preprocessing module, a second preprocessing module, a third preprocessing module, a fourth preprocessing module, a first twin neural network based on an attention mechanism, a second twin neural network based on an attention mechanism, a third twin neural network based on an attention mechanism, a fourth twin neural network based on an attention mechanism, a feature fusion module and an output module; Among them, the first preprocessing module, the second preprocessing module, the third preprocessing module and the fourth preprocessing module are arranged in parallel to preprocess the left eye fundus data, the right eye fundus data, the left eye vascular data and the right eye vascular data in the multimodal fundus image fusion data set respectively, the first attention mechanism-based twin neural network, the second attention mechanism-based twin neural network, the third attention mechanism-based twin neural network and the fourth attention mechanism-based twin neural network are arranged in parallel to extract features of the preprocessed left eye fundus data, the preprocessed right eye fundus data, the preprocessed left eye vascular data and the preprocessed right eye vascular data respectively, and generate a normal carotid intima-media thickness group and a carotid intima-media thickness thickening group after feature fusion using the feature fusion module; the first attention mechanism-based twin neural network and the second attention mechanism-based twin neural network share weights, and the third attention mechanism-based twin neural network and the fourth attention mechanism-based twin neural network share weights.

7. The method for predicting diabetic macroangiopathy based on fundus image features according to claim 6, characterized in that: The loss function H of the carotid intima-media thickness prediction model is: Among them, x i represents the i-th sample label, p(x i ) represents the probability of the true value distribution of the i-th label, q(x i ) represents the probability value of the i-th label predicting a certain category, and n represents the total number of sample labels.

8. A device for predicting diabetic macroangiopathy based on fundus image features according to any one of claims 1 to 7, characterized in that: include: Dataset construction module, used to construct bilateral fundus dataset of diabetic carotid intima-media thickness; A blood vessel segmentation module is used to perform blood vessel segmentation processing on the bilateral fundus dataset of diabetic carotid intima-media thickness using a semantic segmentation model based on visual stimulation adaptation to obtain a binocular blood vessel dataset; A network training module, used to train a carotid intima-media thickness prediction model using the diabetic carotid intima-media thickness bilateral fundus data set and the binocular vascular data set to obtain a trained carotid intima-media thickness prediction model; The result prediction module is used to predict diabetic macrovascular lesions using the trained carotid intima-media thickness prediction model.

9. An electronic device, comprising: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method for predicting diabetic macroangiopathy based on fundus image features as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for predicting diabetic macroangiopathy based on fundus image features according to any one of claims 1 to 7 is implemented.

Images