A method and system for diabetes prediction based on multi-modal data

Abstract

A diabetes prediction method and system based on multi-modal data, comprising the steps of: constructing a meta-classifier model for processing multi-modal data, the multi-modal data at least containing three types of data of fundus images, fundus blood vessel features and basic information features; extracting fundus blood vessel features and basic information features according to original data; processing the extracted features through the constructed meta-classifier model to perform early screening or prediction. The meta-classifier model comprises a first input processing module, a second input processing module and a third input processing module, wherein the first input processing module adopts a convolutional neural network and is used for processing input fundus images; the second input processing module and the third input processing module both adopt a three-layer structure of a multi-layer perceptron and are distributed for processing input fundus blood vessel features and basic information features. The model can efficiently process multi-modal data at the same time, greatly improving the prediction accuracy.

Description

Technical Field

[0001] This invention relates to a digital image processing and application technology, specifically to a method and system for predicting diabetes based on multimodal data. The multimodal data includes at least fundus image features, fundus vascular features, and basic information features, and belongs to the field of medical digital image processing technology. Background Technology

[0002] The fundus is one of the few parts of the human body where arteries, veins, and capillaries can be directly and closely observed with the naked eye. These blood vessels reflect the dynamics and health status of the body's overall blood circulation. Fundus examination is not only an important method for examining diseases of the retina, choroid, and optic nerve, but also a "window" for monitoring many systemic diseases. Changes in fundus blood vessels, to a certain extent, reflect changes in the body's overall blood vessels. These changes can be non-invasively displayed using fundus imaging, allowing doctors to analyze and assess the severity of related diseases.

[0003] Currently, obtaining retinal images through fundus cameras is the most effective and basic way to screen for common eye diseases. It can also identify fundus lesions caused by nephritis, leukemia, anemia, heart disease, etc. However, retinal images diagnosed using neural networks generally need to have obvious lesion characteristics, and professional doctors can directly judge the disease based on such images. Moreover, these lesions only occur when the disease is very serious, and the disease has already caused long-term and irreversible damage to the human body. Therefore, the medical community hopes not only to use retinal images for disease diagnosis and screening, but also to use retinal images for early prediction of diseases.

[0004] In contrast, in the early stages of systemic diseases, retinal images often do not show obvious lesions, making it difficult even for experienced physicians to diagnose or predict related diseases based on such images. While some explorations have been undertaken using neural networks and machine learning algorithms to screen and predict diseases early based on retinal images, the rich information in these images leads to redundant features learned by the models, often including interfering or irrelevant features. Furthermore, the learned features may not be key features, resulting in suboptimal performance and even failure to achieve early disease screening. Moreover, these models are prone to overfitting during training.

[0005] Meanwhile, most existing studies are cross-sectional studies, and the images used mostly show obvious lesions, which are of little clinical significance. However, for chronic diseases like diabetes, which have a long early course and no obvious symptoms, early screening or even prediction is much more clinically important, which is precisely what current technologies rarely address. Summary of the Invention

[0006] Therefore, in order to overcome the many shortcomings and defects of the prior art, the present invention provides a data processing method based on multimodal data, comprising the following steps:

[0007] Step S100: Construct a meta-classifier model for processing multimodal data;

[0008] Step S200: Extract features based on the original data;

[0009] Step S300: The features extracted in step S200 are processed by the meta-classifier model constructed in step S100 to perform early screening or prediction.

[0010] The multimodal data includes at least fundus images, fundus vascular features, and basic information features; the meta-classifier model is the ResNet-Meta-Classifier Model.

[0011] In the above technical solution, the early screening or prediction in step S300 is aimed at diabetes.

[0012] In the above technical solution, the meta-classifier model includes an input layer and a fully connected output layer. The input layer includes a first input processing module, a second input processing module, and a third input processing module. The first input processing module uses a deep residual convolutional neural network to process the input fundus image. The second input processing module uses a three-layer multilayer perceptron to process the input fundus vascular features. The third input processing module also uses a three-layer multilayer perceptron to process the input basic information features. The fully connected output layer includes a Scores module and a Softmax module. The outputs of the first, second, and third input processing modules are weighted and concatenated in the Scores module, and the processed result is input into the Softmax module to obtain the final result.

[0013] In the above technical solution, the deep residual convolutional neural network is ResNet50, which includes five processing stages: stage 0, stage 1, stage 2, stage 3, and stage 4. Stage 0 is used for preprocessing the input data, including convolutional layers and max pooling layers, wherein the kernel size of the convolutional layers is 7×7. Stage 1 consists of a 3-layer structure composed of residual units. Stage 2 consists of a 4-layer structure composed of residual units. Stage 3 consists of a 6-layer structure composed of residual units. Stage 4 consists of a 3-layer structure composed of residual units.

[0014] In the above technical solution, the Resnet50 further includes a global average pooling layer and a fully connected layer.

[0015] In the above technical solution, the multilayer perceptron uses three hidden layers, each with 128 neuron nodes, and the activation function is a linear rectified function.

[0016] In the above technical solution, the fundus image is a fused image, specifically a five-channel fundus color image. This five-channel fundus color image is formed by fusing arterial and venous data onto a fundus color image (RGB three channels). The fundus vascular features include multiple or all of the following features: number of arteries, global arterial tortuosity, arterial fractal dimension, arterial bifurcation angle, arterial bifurcation asymmetry, arterial diameter features, number of veins, global venous tortuosity, venous fractal dimension, venous bifurcation angle, venous bifurcation asymmetry, venous diameter features, central arterial equivalent, central venous equivalent, and central arteriovenous equivalent ratio. The basic information features include multiple or all of the following features: height, weight, BMI, age, and gender.

[0017] In the above technical solution, step S200 includes the steps of completing the center line of the fundus blood vessels and extracting the blood vessel features, which specifically includes: step S210: identifying blood vessel breakpoints in the fundus blood vessel image;

[0018] Step S220: Determine the relationship between the two blood vessel breakpoints;

[0019] Step S230: Connect the breakpoints of two blood vessels belonging to the same vessel to complete the central line of the fundus vessels;

[0020] Step S240: Extract fundus vascular features using the completed fundus vascular centerline.

[0021] In the above technical solution, step S240 further includes:

[0022] Step S241: Extract and store the coordinates of the fundus blood vessel centerline;

[0023] Step S242: Extract the diameter features of blood vessels in the fundus;

[0024] Step S243: Extract the tortuosity features of retinal blood vessels;

[0025] Step S244: Extract the branching features of retinal blood vessels;

[0026] Step S245: Extract the fractal dimension features of the fundus blood vessels.

[0027] The present invention also provides a data processing system based on multimodal data, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described method.

[0028] The present invention also provides a computer instruction storage medium storing instructions that cause a computer processor to execute the above-described method.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] (1) The method for connecting the breakpoints of the blood vessel centerline proposed in this invention can automatically identify the breakpoints and determine whether the breakpoints belong to the same blood vessel. It can complete the breakpoints caused by the crossing of arteries and veins. After completion, the blood vessel features can be extracted. It can connect the features of the same blood vessel, ensure the integrity of the blood vessel features, provide quality assurance for the subsequent prediction data, and facilitate the automatic processing of breakpoint information by the machine.

[0031] (2) This invention constructs a ResNet-Meta-Classifier Model based on fundus images, basic information of the subject, and fundus vascular features for early screening and prediction of diabetes. This model can efficiently process multimodal data simultaneously, and by combining vascular features with vascular images, it overcomes the problem that using only vascular images may lead to poor model learning performance, thus greatly improving prediction accuracy.

[0032] The results show that the vascular features extracted after completing the vascular breakpoints in this invention are more accurate, and the proposed Resnet-Meta-Classifier Model has a significant improvement in the performance of diabetes prediction based on images and vascular features. Attached Figure Description

[0033] Figure 1 This is the overall flowchart of the present invention.

[0034] Figure 2 This is a schematic diagram of the meta-classifier model of the present invention.

[0035] Figure 3 This is a flowchart of the method for completing the vascular centerline and extracting vascular features according to the present invention.

[0036] Figure 4 This is a schematic diagram of the bifurcation relationship algorithm of the present invention.

[0037] Figure 5 This is a schematic diagram of the method for calculating vascular equivalents according to the present invention. Detailed Implementation

[0038] The preferred embodiments of the present invention are described below. These specific embodiments are intended to illustrate the present invention in detail, but should not be construed as limiting the present invention. Various modifications and variations can be made without departing from the spirit and scope of the present invention, and all of these should be included within the protection scope of the present invention.

[0039] To address the aforementioned technical problems, this invention provides a diabetes prediction system and method based on multimodal data. The multimodal data includes at least fundus image features, fundus vascular features, and basic information features. Specifically, this invention uses a ResNet-Meta-Classifier Model to process the multimodal data to achieve early screening and prediction of diabetes.

[0040] The overall process of this invention is as follows: Figure 1 As shown, it includes the following steps:

[0041] Step S100: Construct a meta-classifier model for processing multimodal data.

[0042] The multimodal data includes at least three types of data: fundus images, fundus vascular features, and basic information features. The basic information features include, but are not limited to, data such as the age, gender, height, weight, and BMI of the user corresponding to the input fundus image. It may also include all or part of the user's demographic information, physical examination data, blood test data, etc.

[0043] The proposed meta-classifier model (ResNet-Meta-Classifier Model) in this invention has the following specific structure: Figure 2 As shown, it can be generally divided into two layers, namely Figure 2 The input layer on the left side of the middle, Figure 2 The fully connected output layer is on the right side. The input layer consists of three modules: Figure 2The left-hand input layer, from top to bottom, consists of a first input processing module, a second input processing module, and a third input processing module. The first input processing module uses a convolutional neural network (ResNet) to process the input fundus image, preferably a fused image. This fused image is a five-channel fundus color image, specifically fusing arterial and venous data onto a three-channel (RGB) fundus color image. Specifically, the fundus color image is a three-channel RGB image, and the arterial and venous vessel annotations are red-blue dual-channel images. The three-channel image and the dual-channel image are concatenated in parallel to form the five-channel fused image. The second input processing module uses a three-layer Multilayer Perceptron (MLP) to process the one-dimensional fundus vascular features. The third input processing module also uses a three-layer MLP to process the one-dimensional basic information features. The fully connected output layer includes a Scores module and a Softmax module. The outputs from the first, second, and third input processing modules are weighted and scored in the Scores module, and the result is then input into the Softmax module for final processing.

[0044] Among them, the convolutional neural network used to process fundus images is preferably a deep residual network (Resnet). There are five commonly used depth structures of Resnet, namely 18, 34, 50, 101 and 152. This invention preferably uses the 50 depth structure, namely Resnet50. Specifically, ResNet50 mainly includes five processing stages (excluding global average pooling layers and fully connected layers). Stage 0 is for preprocessing the input data, including convolutional layers (Conv) and max pooling layers, where the convolutional kernels (Conv) are 7×7. Stages 1, 2, 3, and 4 are processing layers composed of residual units (ResBlocks). Stage 1 consists of 3 layers of residual units, Stage 2 consists of 4 layers, Stage 3 consists of 6 layers, and Stage 4 consists of 3 layers. ResNet50 may further include global average pooling layers (Avg Pooling) and fully connected layers. The fully connected layers include fully connected layers (FC) and softmax layers, where FC performs full connection and softmax performs output normalization.

[0045] The three-layer perceptron structure described above preferably uses three hidden layers, each with 128 neurons, and the activation function is the rectified linear function (ReLU).

[0046] For the fully connected output layer, the 1×2 dimensional data [a1, b1] output from the last layer of ResNet50 and the 1×2 dimensional data [a2, b2] and [a3, b3] output from the two MLPs are multiplied by the optimal weights obtained during training in the Scores module, and then concatenated into 3×2 dimensional data [[a1, b1]×w1, [a2, b2]×w2, [a3, b3]×w3], which is the scores. This data is then input into the Softmax module to obtain the final output, where w1, w2, and w3 are the weights of the three types of features: fundus image, fundus vascular features, and basic information features, respectively.

[0047] Step S200: Feature extraction is performed based on the original data.

[0048] Step S200 includes extracting fundus vascular features and basic information features based on the original data. Specifically, features obtained from the original fundus image are fused with the extracted fundus vascular features and demographic information to form a fused feature vector. The specific concatenation method is shown in the formula:

[0049] W = [W1, W2, W3]

[0050] Among them, W1, W2, and W3 are feature vectors obtained from three different data objects. W1 is the fundus image feature with a vector dimension of 1×2048, W2 is the extracted fundus blood vessel feature with a vector dimension of 1×27, and W3 is the demographic information feature with a vector dimension of 1×8. The fused feature dimension after being concatenated according to the above formula is 1×2083.

[0051] While any known existing technology can be used to extract fundus vascular features and basic information features from relevant raw data, this invention further provides a method for completing the fundus vascular centerline and extracting vascular features in order to obtain better prediction results. This method specifically includes the following steps:

[0052] Step S210: Identify vascular breakpoints in fundus vascular images;

[0053] Because arteries and veins overlap and cross in fundus vascular images, after distinguishing between arteries and veins and identifying them separately, vascular breakpoints may occur at the overlapping and crossing locations. These vascular breakpoints also appear in fundus vascular images as similar to the endpoints of vascular vessels. However, the difference between a vascular breakpoint and the true endpoint of a complete vessel is that a breakpoint is the endpoint inside the vessel, located between the endpoints of the myopic and hyperopic discs. Therefore, it can be judged based on the distance from the center of the optic disc. If the distance between the endpoint of the vessel in the fundus vascular image and the center of the optic disc is greater than 0.5 times the diameter of the optic disc but less than 2.5 times the diameter of the optic disc, it is initially judged as a vascular breakpoint and saved.

[0054] Step S220: Determine the relationship between the two vessel breakpoints, that is, determine whether the two vessel breakpoints belong to the interruption of the same vessel.

[0055] According to statistical data, the maximum diameter of blood vessels in the fundus is 268 micrometers. Therefore, blood vessel breaks within 300 micrometers of each other can be identified as belonging to the same blood vessel. Furthermore, to prevent the occurrence of blood vessel breaks that are not actually from the same blood vessel but are very close together, resulting in secondary connections, each blood vessel break is restricted to only being identified and connected once.

[0056] Step S230: Connect the breakpoints of two blood vessels belonging to the same vessel to complete the center line of the fundus vessels.

[0057] The operation of connecting retinal vessel breakpoints involves setting the pixel value of the midpoint between two vessel breakpoints from 0 to the same value as the centerline, i.e., 1. Since the vessel breakpoint a(x a y a ) and vessel breakpoint b(x b y b The position of the line connecting two blood vessel breaks is variable. If a direct connection is used, the linear expression of the connection can be calculated based on the coordinates of the two breaks. However, the coordinates of pixels between the two breaks are all integers, with no decimals. Therefore, some points may match the linear expression of the connection but cannot be connected. Based on this, this method uses a right-angled line to connect the two blood vessel breaks, i.e., (x... b y a ) and (x b y b The midpoint of (x) and (x) b y a ) and (x a y a Connect all the midpoints of the given information.

[0058] This method effectively solves the problem of vascular discontinuity caused by vascular breakpoints, laying the foundation for subsequent extraction of fundus vascular features.

[0059] Step S240: Extract fundus vascular features using the completed fundus vascular centerline.

[0060] This step enables the identification and storage of the coordinates of the fundus blood vessel centerline and the bifurcation relationship of the blood vessels. After connecting the breakpoints of the blood vessel centerline, the original PNG format image of the fundus blood vessel data is converted into a data sequence containing location information.

[0061] The above feature extraction steps further include:

[0062] Step S241: Extract and store the coordinates of the center line of the fundus blood vessels.

[0063] Because the blood vessels in the fundus radiate from the center, a growth algorithm is used to save the coordinates of the vessel centerline starting from the myopic disc. However, most blood vessels are bifurcated. For bifurcated scenarios, the method provided in this invention applies a recursive approach to the growth algorithm. That is, when the same blood vessel has multiple key points (key points include endpoints and bifurcation points), the growth algorithm is used between two key points in the order they are encountered. The growth process of a key point stops when it encounters the next key point, and the blood vessel coordinates are counted starting from the next key point. These coordinates are then saved to the corresponding parent branch, where the child branches of the same parent branch are numbered counterclockwise. This invention adopts a branch saving method based on sequence numbers. Since there is almost no extreme case of more than 4 bifurcations at once during blood vessel processing, the maximum number of bifurcations for each node is defined as 3. For a blood vessel labeled n, if a bifurcation occurs later, the labels of its child branches are defined as 3n, 3n+1, and 3n+2. The first main vein found from the center of the optic disc is designated as number 1. Therefore, for any vessel, if its id > 1, then the id of its parent vessel is id / 3 (rounded down). The specific relationship diagram is shown below. Figure 4 As shown.

[0064] Step S242: Extract the diameter features of the blood vessels in the fundus.

[0065] Based on the saved structure of the vessel centerline, the vessel diameter is calculated by segmenting the vessel into segments. The calculated vessel diameter is then stored in a data structure at the same level as the vessel coordinates for subsequent calculations.

[0066] For single blood vessel diameter measurement: the blood vessel is segmented, and every 8 pixels are selected as a blood vessel segment. The least squares method is used to obtain the linear regression model of the blood vessel. Based on the obtained blood vessel segment model, its vertical direction vector is determined. This vertical direction vector is extended to both ends of the blood vessel. The boundary is determined based on the pixel values ​​of the blood vessel and the background. The distance between the two endpoints of the boundary is the diameter.

[0067] Based on vessel diameter, vascular equivalents can be further extracted: although the thickness of vessels varies greatly among different branches, the thickest main vessels have the greatest impact on disease relevance. The six thickest arteries and veins in region B (0.5-1.5DD) of the retinal fundus image are selected for calculation; if there are fewer than six, all are included in the calculation. Vascular equivalents are not simply averaged based on vessel width. Instead, the root mean square of the two largest and smallest vessels is calculated with certain weights, and the result is then added back to the data for the next iteration until the final equivalent is obtained. The process is as follows: Figure 5 As shown.

[0068] Equivalent algorithm for two blood vessels:

[0069]

[0070]

[0071] Among them, W a For a narrower blood vessel width, W b For a wider blood vessel width, W c This is an estimate of the vessel width.

[0072] Because actual vascular images are quite diverse, there are anomalies where the width of branch vessels is greater than that of the main vessels. In such cases, a more effective analysis of "significance" is needed to clarify the impact of the main vessel's location and vessel diameter on the actual correlation. Therefore, the six thickest arteries and veins with a diameter greater than 40 micrometers were used to calculate the equivalent central retinal artery and central retinal vein. If a portion of the main vessel is too short to be used for diameter calculation, the main vessel diameter is calculated using its sub-branches.

[0073] Building upon this, the arteriovenous equivalent ratio (AVR) of the central vessels can be further included. The measured vessel diameter is generally within a range of 0.5 to 1.5 times the optic disc diameter from the edge of the solid disc. The measured AVR value is the ratio of the accompanying arterial equivalent to the venous equivalent, i.e.:

[0074] AVR = CRAE / CRVE

[0075] CRAE and CRVE are the arterial and venous equivalents of the accompanying vessels, respectively.

[0076] Step S243: Extract the tortuosity features of the retinal blood vessels.

[0077] Similar to step S242, blood vessels in region B (0.5-1.5DD) of the retinal fundus image are selected for feature extraction, with the six thickest arteries and veins having a diameter greater than 40 micrometers chosen. The commonly used method for calculating tortuosity in existing technologies is Distance Factor (DF). DF calculation only considers the two ends of the blood vessel and is unrelated to the direction of the internal curvature, resulting in DF not accurately reflecting the tortuosity of fundus blood vessels. Therefore, this step uses the ratio of the vessel's curvature to its arc length to describe the tortuosity characteristics of fundus blood vessels.

[0078] Step S244: Extract the branching features of retinal blood vessels.

[0079] For branching characteristics, the main calculation scope is the first-order bifurcation of blood vessels. The specific characteristics calculated are the bifurcation angle and branch asymmetry. Since the branching blood vessels are not straight vessels, the branching characteristics of the near bifurcation point and the far bifurcation point are calculated. The asymmetry between the daughter branches and the parent branches is calculated to describe the relationship between the daughter branches and the parent branches.

[0080] For the bifurcation angle: Using a method that fits the vessel segment, the optimal linear models for the two branch vessels are obtained. The direction vectors of the two branches are then calculated, with the common starting point being the bifurcation point (x0, y0). The direction vector of the thicker branch is (x1, y1), and the direction vector of the thinner branch is (x2, y2). First, the lengths of the two vectors are calculated, then the cosine value of the included angle is calculated, and finally, the included angle is obtained using the arccos function.

[0081] For branch asymmetry: For the diameter of a vessel near a bifurcation, the asymmetry between the two branches is described by dividing the square of the diameter of the larger branch by the square of the diameter of the smaller branch.

[0082] Step S245: Extract the fractal dimension features of the fundus blood vessels.

[0083] The above descriptions are all based on indicators calculated from a single fundus blood vessel. In order to describe the global complexity of fundus blood vessels, the fractal dimension feature is introduced.

[0084] Fractal dimension is a statistical measure used to describe the degree of space filling by a fractal. Methods for defining fractal dimension include Hausdorff dimension, box-counting dimension, and distributive dimension. This invention uses fractal dimension to reflect the overall complexity of blood vessels, preferably employing the box-counting dimension method. The principle is as follows: Using a number of cells of a given size to completely cover the retinal blood vessel object, calculate how many cells are needed to cover the object, and repeat this process for cells of different sizes. The scaling factor of the number of cells covering the object with the cell size provides an estimate of the object's fractal dimension.

[0085] Assuming that when the side length of the grid is ε, N grids are needed to cover the retinal blood vessels, then the box dimension is:

[0086] The extraction of the above-mentioned fundus vascular features can all be extracted separately by distinguishing between veins and arteries.

[0087] In step S300, the features extracted in step S200 are processed by the meta-classifier model constructed in step S100 to perform early screening or prediction.

[0088] This early screening or prediction targets diabetes. Although the specific embodiments of this invention use diabetes as an example for feature selection and model construction, it is foreseeable that the method of this invention can be applied to the early screening or prediction of other systemic diseases.

[0089] Test results:

[0090] To verify the method and model of this invention, tests were conducted using a dataset provided by a collaborating institution, and all results were reviewed by professional physicians. The collaborating institution is a renowned hospital, and the dataset consists of three sets of data: a fundus image dataset, a vascular image dataset, and two sets of demographic information from the participants, spaced approximately three years apart, used for disease prediction research. All three datasets were derived from participants recruited through annual health checkups, including 6202 male participants with an average age of 46.33 years and 3995 female participants with an average age of 44.05 years. Among them, 996 participants had diabetes, and 495 had newly diagnosed diabetes. The dataset has been anonymized to remove any information related to individual patients.

[0091] The test used a total of 10,000 images from the entire dataset for validation. The selected input features specifically included: for demographic information features, height, weight, BMI, age, and gender; and for fundus vascular features, the number of arteries, global arterial tortuosity, arterial fractal dimension, arterial bifurcation angle, arterial bifurcation asymmetry, and arterial diameter; for veins, the number of veins, global vein tortuosity, vein fractal dimension, vein bifurcation angle, vein bifurcation asymmetry, and vein diameter; central arterial equivalent; central vein equivalent; and central arteriovenous equivalent ratio. The model preferably employs a Stacking ensemble strategy. Through model training and testing, the information gain of each data feature, i.e., the weight of each typical data feature in relation to the classification result, can be obtained. For early screening of diabetes, the information gain was 0.15 for age, 0.06 for BMI, and 0.03 for gender; the highest information gain was 0.040 for global arterial tortuosity; the information gain for arterial fractal dimension was 0.034; the information gain for venous fractal dimension was 0.0012; and the information gain for central arteriovenous equivalent ratio was 0.019. For predicting diabetes, the information gain was 0.07 for BMI, 0.03 for age, and 0.02 for gender; the highest information gain was 0.060 for central venous equivalent; the information gain for central arterial equivalent was 0.033; the information gain for venous fractal dimension was 0.000031; and the information gain for central arteriovenous equivalent ratio was 0.021.

[0092] Experimental results show that incorporating fundus vascular features and demographic information features for prediction is more effective than using only a single fundus vascular image. The Resnet-Meta-Classifier Model proposed in this invention can maximize the role of different modal features in diabetes prediction by processing image and vascular features, thereby achieving the goal of early diabetes screening.

[0093] This invention is not limited to the specific embodiments described above. It is understood that various modifications and variations can be made without departing from the spirit and scope of this invention, and all such modifications and variations should be included within the scope of protection of this invention.

Claims

1. A method for predicting diabetes based on multimodal data, characterized in that... Including the following steps: Step S100: Construct a meta-classifier model for processing multimodal data; Step S200: Extract features based on the original data; The process includes steps for completing the central line of retinal blood vessels and extracting vascular features, specifically including: Step S210: Identify the vascular breakpoints in the fundus vascular image. If the distance between the end point of the vascular vessel in the fundus vascular image and the center of the optic disc is greater than 0.5 times the diameter of the optic disc and less than 2.5 times the diameter of the optic disc, it is preliminarily judged as a vascular breakpoint. Step S220: Determine the relationship between two blood vessel breakpoints, identify blood vessel breakpoints with a distance of less than 300 micrometers as belonging to the same blood vessel, and restrict each blood vessel breakpoint to only one judgment and connection. Step S230: Connect the two blood vessel breaks belonging to the same blood vessel, complete the fundus blood vessel centerline, and connect the two blood vessel breaks using a right-angle line; Step S240: Extract fundus vascular features using the completed fundus vascular centerline; Step S300: The features extracted in step S200 are processed by the meta-classifier model constructed in step S100 to perform early screening or prediction. The multimodal data includes at least fundus images, fundus vascular features, and basic information features; the meta-classifier model is ResNet-Meta-ClassifierModel; this meta-classifier model includes an input layer and a fully connected output layer; The input layer includes a first input processing module, a second input processing module, and a third input processing module. The first input processing module employs a deep residual convolutional neural network to process the input fundus image, which is a fused image, specifically a five-channel fundus color image, fused with arterial and venous data. The second input processing module uses a three-layer multilayer perceptron to process the input one-dimensional fundus vascular features. The third input processing module also uses a three-layer multilayer perceptron to process the input one-dimensional basic information features. The fully connected output layer includes a Scores module and a Softmax module. The outputs from the first, second, and third input processing modules are weighted and concatenated in the Scores module, and the resulting output is then processed by the Softmax module to obtain the final result.

2. The diabetes prediction method based on multimodal data as described in claim 1, characterized in that: The early screening or prediction in step S300 is aimed at diabetes.

3. The diabetes prediction method based on multimodal data as described in claim 2, characterized in that: The deep residual convolutional neural network is ResNet50, which includes five processing stages: stage 0, stage 1, stage 2, stage 3, and stage 4. Stage 0 is used for preprocessing the input data, including convolutional layers and max pooling layers, where the kernel size of the convolutional layers is 7×7. Stage 1 consists of a 3-layer structure composed of residual units. Stage 2 consists of a 4-layer structure composed of residual units. Stage 3 consists of a 6-layer structure composed of residual units. Stage 4 consists of a 3-layer structure composed of residual units.

4. The diabetes prediction method based on multimodal data as described in claim 3, characterized in that: The ResNet50 further includes a global average pooling layer and a fully connected layer.

5. The diabetes prediction method based on multimodal data as described in claim 4, characterized in that: The multilayer perceptron employs three hidden layers, each with 128 neuron nodes, and uses a linear rectified function as the activation function.

6. The diabetes prediction method based on multimodal data as described in claim 5, characterized in that: Fundus vascular features include multiple or all of the following features: number of arteries, global arterial tortuosity, arterial fractal dimension, arterial bifurcation angle, arterial bifurcation asymmetry, arterial diameter characteristics; number of veins, global venous tortuosity, venous fractal dimension, venous bifurcation angle, venous bifurcation asymmetry, venous diameter characteristics, central arterial equivalent, central venous equivalent, and central arteriovenous equivalent ratio; basic information features include multiple or all of the following features: height, weight, BMI, age, and sex.

7. The diabetes prediction method based on multimodal data as described in claim 6, characterized in that: Step S240 further includes: Step S241: Extract and store the coordinates of the fundus blood vessel centerline; Step S242: Extract the diameter characteristics of blood vessels in the fundus; Step S243: Extract the tortuosity features of retinal blood vessels; Step S244: Extract the branching features of retinal blood vessels; Step S245: Extract the fractal dimension features of the fundus blood vessels.

8. A diabetes prediction system based on multimodal data, comprising at least one processor and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which are then executed by the at least one processor to cause the at least one processor to perform a diabetes prediction method based on multimodal data as described in any one of claims 1-7.

Images