A method and system for processing fundus images for diabetes prediction
By combining preprocessing of fundus vascular images with a meta-classifier model, multiple vascular features are extracted, solving the problem of insufficient early screening and prediction in existing technologies, and achieving more efficient early screening and prediction of diabetes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2021-12-07
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies for early screening and prediction of diabetes rely solely on fundus vascular features, which are insufficient to effectively utilize more information. Furthermore, manual or semi-automated vascular extraction methods are inefficient and subject to subjective factors. Current neural network diagnosis requires obvious lesion features and cannot make early predictions.
By preprocessing fundus vascular images, vascular information is identified and converted into coordinate and morphological data. The vascular centerline and key points are extracted and the breakpoints are connected. A meta-classifier model (ResNet-Meta-Classifier Model) is constructed and combined with multimodal data for screening and prediction. Vascular features such as diameter, curvature, and branching features are extracted and processed using a deep residual convolutional neural network and a multilayer perceptron.
It achieves completeness and accuracy of vascular features, improves the accuracy of early diabetes screening and prediction, overcomes the problem of poor learning performance of single vascular image models, and provides more efficient automated processing capabilities.
Smart Images
Figure CN116309235B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a digital image processing and application technology, specifically to a method and system for processing fundus images for diabetes prediction, belonging to the field of medical digital image processing technology. Background Technology
[0002] The fundus is one of the few parts of the human body where arteries, veins, and capillaries can be directly and closely observed with the naked eye. These blood vessels reflect the dynamics and health status of the body's overall blood circulation. Fundus examination is not only an important method for examining diseases of the retina, choroid, and optic nerve, but also a "window" for monitoring many systemic diseases. Changes in fundus blood vessels, to a certain extent, reflect changes in the body's overall blood vessels. These changes can be non-invasively displayed using fundus imaging, allowing doctors to analyze and assess the severity of related diseases.
[0003] Currently, obtaining retinal images through fundus cameras is the most effective and basic way to screen for common eye diseases. It can also identify fundus lesions caused by nephritis, leukemia, anemia, heart disease, etc. However, retinal images diagnosed using neural networks generally need to have obvious lesion characteristics, and professional doctors can directly judge the disease based on such images. Moreover, these lesions only occur when the disease is very serious, and the disease has already caused long-term and irreversible damage to the human body. Therefore, the medical community hopes not only to use retinal images for disease diagnosis and screening, but also to use retinal images for early prediction of diseases.
[0004] In ophthalmology, changes in parameters such as the diameter and tortuosity of retinal vessels are generally considered to reflect the severity of specific diseases. Current algorithms for extracting retinal vessels mainly fall into two categories: manual extraction and semi-automated software extraction. Manual extraction methods for retinal vessel features in medicine primarily rely on instruments combining mechanical and optical techniques. However, this approach is time-consuming and inefficient when measuring a single retinal image, making it unsuitable for large-scale feature extraction. Furthermore, such methods are directly related to the experience of the measurement personnel and are subject to subjective influences. Semi-automated retinal vessel measurement software, such as AVRnet, SIVA (Singapore I Vessel Analyzer), CAIAR (Computer-Assisted Image Analysis of the Retina), and IDX systems, primarily identify the boundaries of retinal vessels from the input retinal image and then perform retinal vessel feature measurement with manual assistance. While this has significantly reduced manual labor, the selection and delineation of retinal vessels still vary from person to person.
[0005] For disease screening and prediction based on fundus vascular features, existing technologies generally only use a few features of fundus images, such as the diameter and curvature of fundus vessels. However, the occurrence of a certain disease may not only affect the morphology of blood vessels, but may also have an impact on the retina itself. Therefore, it is not enough to use fundus vascular features for early screening and prediction of diseases, as a lot of information is missing. In order to screen and predict diseases more accurately, more information is needed to assist in diagnosis. Summary of the Invention
[0006] Therefore, in order to overcome the many shortcomings and defects of the prior art, the present invention provides a method for processing fundus vascular images for diabetes prediction, comprising the following steps:
[0007] Step S100: Preprocess the original fundus blood vessel image;
[0008] Step S200 involves identifying vascular information in the fundus vascular image and converting it into coordinate and morphological data of the fundus vessels for storage; step S200 specifically includes:
[0009] Step S210: Extract the center line of the arteries and veins in the fundus;
[0010] Step S220: Identify key points of retinal blood vessels;
[0011] Step S230: Connect the identified fundus vessel breaks; connect two vessel breaks belonging to the same fundus vessel using a right-angle broken line.
[0012] Step S240: Store the coordinates of the fundus blood vessel centerline and the bifurcation relationship of the fundus blood vessels;
[0013] Step S300: Extract fundus vascular features based on fundus vascular coordinates and morphological data.
[0014] In the above technical solution, step S100 specifically includes:
[0015] Step S110: The original fundus blood vessel image is subjected to size uniformization processing;
[0016] Step S120: Perform color uniformization processing on the fundus blood vessel image that has undergone size uniformization processing;
[0017] Step S130: Perform size correction processing on the fundus blood vessel image after color uniformization processing.
[0018] In the above technical solution, step S110 specifically includes:
[0019] Step S112: Convert the original RGB three-channel color fundus blood vessel image into a single-channel grayscale image;
[0020] Step S114: Obtain the grayscale threshold of the black border region based on the grayscale image, and obtain the effective image region mask on the grayscale image based on the grayscale threshold of the black border region.
[0021] Step S116: Filter the pixels of the three RGB channels of the original color fundus blood vessel image using effective image area masking. After all steps are completed, reassemble the three RGB channels into a color fundus blood vessel image.
[0022] In the above technical solution, step S120 specifically uses Gaussian filtering to adjust brightness and contrast; after correction in step S130, one pixel in the image corresponds to a length of 10 micrometers.
[0023] In the above technical solution, in step S220, key points are extracted by the connectivity of the 8-neighborhood of the center point, thereby determining the type of key points. The formula used is as follows:
[0024]
[0025] Where p represents the center pixel, N(p) represents the type of pixel P, and I t (p) represents the eight neighbor values of the center pixel P; when N(p) is 1, pixel P is the endpoint of the fundus blood vessel; when N(p) is 2, pixel P is a continuous point of the fundus blood vessel; when N(p) is 3, pixel P is the bifurcation point of the fundus blood vessel; when N(p) is 4, pixel P is the intersection point of the fundus blood vessel.
[0026] In the above technical solution, step S300 specifically includes one or more of the following steps:
[0027] Step S310: Extract the diameter characteristics of blood vessels in the fundus;
[0028] Step S320: Extract vascular equivalent features of retinal blood vessels;
[0029] Step S330: Extract the tortuosity features of retinal blood vessels;
[0030] Step S340: Extract the branching features of retinal blood vessels;
[0031] Step S350: Extract the fractal dimension features of the retinal blood vessels.
[0032] In the above technical solution, step S320 includes selecting the six thickest arteries and veins (greater than 40 micrometers) in the region of interest of the retinal fundus image for calculation. If there are fewer than six, all of them are included in the calculation. Each time, the two largest and smallest blood vessels are selected and their root mean squares are calculated according to a certain weight. Then, the calculation results are added back to the data for the next round of iteration until the final equivalent is obtained.
[0033] The specific calculation method for the converted root mean square is as follows:
[0034]
[0035]
[0036] Among them, W a For a narrower blood vessel width, W b For a wider blood vessel width, W c This is an estimate of the vessel width.
[0037] The above technical solution further includes:
[0038] In step S400, the fundus vascular features extracted in step S300 are processed by a meta-classifier model to perform early screening or prediction of diabetes. The meta-classifier model is the Resnet-Meta-Classifier Model.
[0039] In the above technical solution, the meta-classifier model includes an input layer and a fully connected output layer;
[0040] The input layer includes a first input processing module and a second input processing module; wherein, the first input processing module uses a deep residual convolutional neural network to process the input fundus image, and the deep residual convolutional neural network is ResNet50; the second input processing module uses a three-layer multilayer perceptron to process the input fundus vascular features;
[0041] The fully connected output layer includes a Scores module and a Softmax module; the outputs of the first input processing module and the second input processing module are weighted and scored in the Scores module and then concatenated, and the results are input into the Softmax module to obtain the final result.
[0042] The present invention also provides a data processing system, including at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described method.
[0043] The present invention also provides a computer instruction storage medium storing instructions that cause a computer processor to execute the above-described method.
[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0045] (1) The method for connecting the breakpoints of the blood vessel centerline proposed in this invention can automatically identify the breakpoints and determine whether the breakpoints belong to the same blood vessel. It can complete the breakpoints caused by the crossing of arteries and veins. After completion, the blood vessel features can be extracted. It can connect the features of the same blood vessel, ensure the integrity of the blood vessel features, provide quality assurance for the subsequent prediction data, and facilitate the automatic processing of breakpoint information by the machine.
[0046] (2) This invention constructs a ResNet-Meta-Classifier Model based on fundus images and fundus vascular features for early screening and prediction of diabetes. This model can efficiently process multimodal data simultaneously, and by combining vascular features with vascular images, it overcomes the problem that using only vascular images may lead to poor model learning performance, thus greatly improving prediction accuracy.
[0047] The results show that the vascular features extracted after completing the vascular breakpoints in this invention are more accurate, and the proposed Resnet-Meta-Classifier Model has a significant improvement in the performance of diabetes prediction based on images and vascular features. Attached Figure Description
[0048] Figure 1 This is a schematic diagram of the fundus blood vessel feature extraction process of the present invention.
[0049] Figure 2 This is an example diagram illustrating the identification of special key points according to the present invention.
[0050] Figure 3 This is an example diagram illustrating the coordinates of the traversal centerline of the present invention.
[0051] Figure 4 This is an example diagram illustrating the bifurcation relationship of blood vessels according to the present invention.
[0052] Figure 5 This is a schematic diagram of the bifurcation relationship algorithm of the present invention.
[0053] Figure 6 This is a schematic diagram of the method for calculating vascular equivalent according to the present invention.
[0054] Figure 7 This is a schematic diagram of the meta-classifier model of the present invention. Detailed Implementation
[0055] The preferred embodiments of the present invention are described below. These specific embodiments are intended to illustrate the present invention in detail, but should not be construed as limiting the present invention. Various modifications and variations can be made without departing from the spirit and scope of the present invention, and all of these should be included within the protection scope of the present invention.
[0056] To address the aforementioned technical problems, this invention provides a method and system for processing fundus vascular images for diabetes prediction, enabling early screening and prediction of diabetes.
[0057] The present invention provides a method for processing fundus vascular images for diabetes prediction, comprising the following steps:
[0058] Step S100: Preprocess the original fundus blood vessel image.
[0059] The fundus vascular image test dataset used in this invention consists of 20,312 fundus images with a 45° field of view, captured using a Topcon TRC-NW8 mydriatic / non-mydriatic integrated fundus camera or a Cannon CR-2 non-mydriatic digital fundus camera in a non-mydriatic mode. The image resolution is 2464*2248, and the color is 8-bit RGB3. Each subject has one or two fundus images captured; if two are captured, one image each for the left and right eyes. The following description uses the right eye fundus image from the aforementioned test dataset as an example to illustrate the specific technical solution of this invention. A consistent processing method for the right and left eye fundus images can be achieved by horizontally flipping the original fundus vascular image.
[0060] For fundus images, differences in the equipment used for shooting in different environments and the lighting conditions can lead to variations in the overall position and color curve of the images. These differences can affect the feature extraction results and cause an overall shift in the metrics, which in turn affects the effectiveness and accuracy of subsequent models. Therefore, it is necessary to homogenize all images to eliminate the differences caused by the equipment and environment.
[0061] The homogenization process includes two parts: homogenization of positional and size information and homogenization of image color.
[0062] The preprocessing step S100 for the raw fundus vascular image specifically includes:
[0063] Step S110: The original fundus blood vessel image is subjected to size uniformization processing.
[0064] Since the fundus blood vessel image is circular while the background of the photograph is rectangular, black borders may appear around the fundus blood vessel image. Furthermore, the fundus blood vessel image is not necessarily strictly centered on the rectangular background. These deviations not only affect the efficiency of feature extraction but also interfere with feature extraction and create noise. Therefore, it is necessary to remove unnecessary background data for specific images and center the fundus blood vessel image to ensure that the region of interest is fully preserved and used for the next step of processing.
[0065] The specific processing steps of step S110 include:
[0066] Step S112: Convert the original color fundus blood vessel image with RGB three channels into a single-channel grayscale image.
[0067] Step S114: Obtain the grayscale threshold of the black border region, and obtain the effective image region mask based on the grayscale threshold of the black border region.
[0068] Because the black background differs significantly from the actual fundus image's grayscale value, statistical analysis of the original data confirmed that a threshold should be set to perfectly cover the black border area of all photos. After loading the images into memory, all pixels with grayscale values greater than the threshold are saved, thus obtaining a mask of the effective image area.
[0069] Step S116: Filter the pixels of the three RGB channels of the original color fundus blood vessel image using effective image area masking. After all the filtering is completed, stitch the three channels together into a complete image.
[0070] Since the effective area in a fundus vascular image is circular, the saved image naturally ensures that the effective area is centered, and the obtained complete image is the region of interest.
[0071] Step S120: Perform color uniformization processing on the fundus blood vessel image that has undergone size uniformization processing.
[0072] Due to the influence of shooting equipment and environment, fundus images may contain excessively bright or dark images where blood vessels are not clearly visible. In such cases, adjustments to image brightness and contrast are necessary. Because fundus blood vessels are distributed in a divergent pattern, becoming increasingly curved and smaller towards the periphery, and blending more closely with the background, adjusting contrast and brightness can easily lead to value overflow in the large blood vessels at the image center in order to ensure the visibility of blood vessels at the image edges. Conversely, controlling the brightness of the large blood vessels at the image center within acceptable limits makes it difficult to achieve good contrast for the blood vessels at the image edges. Therefore, this invention employs Gaussian filtering to adjust image contrast and brightness. By detecting the variation in neighboring pixels through Gaussian filtering, the color of the region of interest is dynamically adjusted, effectively avoiding the problem of insufficient contrast in edge areas leading to loss of accuracy. The preferred parameter combination of the present invention is as follows: the kernel size of the Gaussian filter is set to (0, 0), and the specific size is calculated according to sigma, wherein the standard deviation of the Gaussian kernel in the X direction, sigmaX, is set to 30; the image processed by Gaussian filtering is weighted and fused with the original image, wherein the weight of the image with the extracted target region is set to 4, the weight of the image after Gaussian processing is set to -3.5, and the offset of the weighted image is set to 100.
[0073] Step S130: Perform size correction processing on the fundus blood vessel image after color uniformization processing.
[0074] Because the size and layout of actual retinal blood vessel images vary, the images themselves cannot directly and accurately represent dimensional indicators such as vessel width and length. Furthermore, these differences can significantly affect scale-sensitive indicators such as curvature. Therefore, more precise dimensional correction of the blood vessel images is necessary to ensure data accuracy in subsequent steps. To improve accuracy, before feature extraction, the image needs to be corrected so that its pixel length is proportional to the actual length, saving subsequent conversion time and improving efficiency. Preferably, after correction, one pixel in the image corresponds to a length of 10 micrometers.
[0075] Step S200: Identify the vascular information in the fundus vascular image and convert it into coordinate and morphological data of the fundus vascular system for storage.
[0076] The main process of this step is as follows: Figure 1 As shown, this method separates arteries and veins in a fundus vascular image and extracts their centerlines separately, saving the coordinates of the centerlines. Specifically, it identifies key points and then uses a growth algorithm combined with recursion to save the centerlines. Based on the saved vessel coordinates, the vessel is segmented, and subsequent calculations can be performed on features such as vessel diameter, tortuosity, bifurcation angle, bifurcation asymmetry, and fractal dimension. Based on this, medical statistical analysis can be performed on these features to analyze their accuracy and rationality, as well as their correlation with diabetes.
[0077] Coordinate data of fundus vessels includes, but is not limited to, the centerline coordinates of arteries and veins, and the coordinates of key points of the vessels; morphological data of fundus vessels includes, but is not limited to, the types of key points of the vessels, and the bifurcation relationships of the vessels.
[0078] Step S200 specifically includes:
[0079] Step S210: Extract the center line of the arteries and veins.
[0080] Physiologically, arteries primarily transport nutrients to the retina, while veins mainly transport metabolic waste products. Morphologically, arteries are bright red and smaller, while veins are dark red and thicker. Arteries and veins differ significantly in both function and morphology, necessitating the extraction of their vascular features separately. This invention separates arteries and veins based on the different pixel colors in retinal vascular images. While the human eye can directly determine the direction of arteries and veins after separation, computers cannot directly identify their direction and network topology. Therefore, it is necessary to obtain the topological structure of arteries and veins and deduce their direction based on this structure.
[0081] To obtain the centerline of arteries and veins, the arteries and veins need to be refined. The method used in this invention is a refinement algorithm (Zhan Suen), which removes boundaries successively, resulting in vessels without bifurcations at both ends.
[0082] Step S220: Identify key points of blood vessels.
[0083] After obtaining the vascular centerline, the computer cannot directly read the location information of the vascular center point; further processing of the vascular centerline is required. The start and end points of the vascular centerline in the fundus vascular image are defined as endpoints. In reality, a complete arteriovenous vessel may have more than two endpoints in the fundus vascular image. The intersection point of the main trunk vessel with two or more branch vessels in the fundus vascular image is defined as the bifurcation point. This point is crucial for recording the relationship between the main trunk and branches. Fundus vessels grow outward from the center, so when performing vascular coordinate position statistics, recording begins from the point closest to the center. Based on this, this invention extracts key points by utilizing the connectivity of the 8-neighborhood of the center point, thereby determining the key point type. The formula is:
[0084]
[0085] Where p represents the center pixel, N(p) represents the type of pixel P, and I t(p) represents the eight-neighbor value of the center pixel P. When N(p) is 1, pixel P is the endpoint of the blood vessel; when N(p) is 2, pixel P is a continuous point of the blood vessel; when N(p) is 3, pixel P is a key point of the blood vessel's bifurcation; when N(p) is 4, pixel P is a key point of the blood vessel's intersection. After processing in step S220, the identified key points are saved in lists according to their proximity to the optic disc endpoint, distance from the optic disc endpoint, and intersection point, respectively, for subsequent breakpoint connection and bifurcation relationship saving.
[0086] Step S230: Connect the broken points of the blood vessels.
[0087] After identifying key points, since arteries and veins cross in the fundus, the crossing points become breakpoints after arteriovenous separation. To ensure the integrity of vascular features, breakpoints belonging to the same vessel need to be connected. Generally, the distance between the two endpoints is used to identify whether they belong to the same vessel. According to statistics, the maximum diameter range of retinal vessels is 268 micrometers, so vascular breakpoints with a distance of less than 300 micrometers can be identified as belonging to the same vessel. Moreover, to prevent the occurrence of vascular breakpoints that are not actually from the same vessel but are very close together, leading to secondary connections, each vascular breakpoint is restricted to only one judgment and connection.
[0088] The operation of connecting retinal vessel breakpoints involves setting the pixel value of the midpoint between two vessel breakpoints from 0 to the same value as the centerline, i.e., 1. Since the vessel breakpoint a(x a y a ) and vessel breakpoint b(x b y b The position of the line connecting two blood vessel breaks is variable. If a direct connection is used, the linear expression of the connection can be calculated based on the coordinates of the two breaks. However, the coordinates of pixels between the two breaks are all integers, with no decimals. Therefore, some points may match the linear expression of the connection but cannot be connected. Based on this, this method uses a right-angled line to connect the two blood vessel breaks, i.e., (x... b y a ) and (x b y b The midpoint of (x) and (x) b y a ) and (x a y a Connect all the midpoints of the given information.
[0089] Step S220 processes arteries and veins separately, effectively solving the problem of discontinuity in blood vessels caused by vascular breakpoints, and laying the foundation for subsequent extraction of fundus vascular features.
[0090] Step S240: Store the coordinates of the vessel centerline and the vessel bifurcation relationship.
[0091] After connecting the breakpoints of the vascular centerline, the original PNG image needs to be converted into a data sequence containing location information. Since the retinal vessels radiate outwards from the center, the coordinates of the vascular centerline must be stored starting from the myopic disc. A common method is a growth algorithm, which uses the 8-neighbor pixel values of the seed point to enqueue and dequeue the data until the last point is traversed. Because the vessels extend from the center of the optic disc outwards, the traversal process also starts from the center of the image coordinates and spirals outwards, as follows... Figure 3 As shown. However, the coordinates of the same blood vessel obtained by the existing growth algorithm are not continuous and cannot be directly input into the computer. Therefore, this invention improves the existing growth algorithm by changing the seed point storage method from a queue to a stack, thereby obtaining ordered blood vessel coordinates.
[0092] After this step, the retinal vessel images stored as PNG format are converted into a set of sequences. Each sequence consists of a list of binary coordinates, preferably stored in JSON format. Simultaneously, the proximal and distal ends of each vessel center point sequence are also saved in separate lists for subsequent calculations.
[0093] Following the above steps, the centerline coordinates of each artery and vein can be saved and used for subsequent calculations of indicators such as diameter / width and classification. However, there is another type of indicator in vascular features, represented by bifurcation angle and symmetry. This requires not only individual vascular indicators but also the parent-child relationship of bifurcated vessels for joint calculation. Since simply saving the vascular coordinate sequence as a list cannot reflect the relationship between vascular vessels, a special data structure needs to be designed to store the relationship between parent and child vessels and to effectively perform bidirectional lookups.
[0094] For bifurcation scenarios, this invention applies the idea of recursion based on the growth algorithm. That is, if there are multiple key points in the same blood vessel, the growth algorithm is used between two key points in the order they are encountered. When the next key point is encountered during the growth process, the growth stops and the blood vessel coordinates are counted from the next key point. The coordinates are saved to the corresponding parent branch, and the child branches of the same parent branch are numbered counterclockwise.
[0095] This invention employs a number-based branching method. Since it's highly unlikely that more than four branches will occur simultaneously during vessel processing, the maximum number of branches for each node is defined as 3. For a vessel labeled n, if subsequent branches occur, the sub-branch labels are defined as 3n, 3n+1, and 3n+2. The first main branch found from the center of the visual disc is defined as number 1. Thus, for any vessel, if its id > 1, then the id of its parent vessel is id / 3 (rounded down). The specific relationship diagram is shown below. Figure 5 As shown above, a bidirectional index from parent node to child node and from child node to parent node can be quickly implemented, while data can be easily exported and imported using JSON.
[0096] Step S300: Extract fundus vascular features based on fundus vascular coordinates and morphological data.
[0097] Step S300 can extract the features of arteries and veins respectively, specifically including:
[0098] Step S310: Extract the diameter features of the blood vessels in the fundus.
[0099] Based on the saved structure of the vessel centerline, the vessel diameter is calculated by segmenting the vessel into segments. The calculated vessel diameter is then stored in a data structure at the same level as the vessel coordinates for subsequent calculations.
[0100] For single blood vessel diameter measurement: the blood vessel is segmented, and every 8 pixels are selected as a blood vessel segment. The least squares method is used to obtain the linear regression model of the blood vessel. Based on the obtained blood vessel segment model, its vertical direction vector is determined. This vertical direction vector is extended to both ends of the blood vessel. The boundary is determined based on the pixel values of the blood vessel and the background. The distance between the two endpoints of the boundary is the diameter.
[0101] Step S320: Extract vascular equivalent features of retinal blood vessels.
[0102] Based on vessel diameter, vascular equivalents can be further extracted: although the thickness of vessels varies greatly among different branches, the thickest main vessels have the greatest impact on disease relevance. The six thickest arteries and veins in region B (0.5-1.5DD) of the retinal fundus image are selected for calculation; if there are fewer than six, all are included in the calculation. Vascular equivalents are not simply averaged based on vessel width. Instead, the root mean square of the two largest and smallest vessels is calculated with certain weights, and the result is then added back to the data for the next iteration until the final equivalent is obtained. The process is as follows: Figure 6 As shown.
[0103] Equivalent algorithm for two blood vessels:
[0104]
[0105]
[0106] Among them, W a For narrower blood vessel width, W b For a wider blood vessel width, W c This is an estimate of the vessel width.
[0107] Because actual vascular images are quite diverse, there are anomalies where the width of branch vessels is greater than that of the main vessels. In such cases, a more effective analysis of "significance" is needed to clarify the impact of the main vessel's location and vessel diameter on the actual correlation. Therefore, the six thickest arteries and veins with a diameter greater than 40 micrometers were used to calculate the equivalent central retinal artery and central retinal vein. If a portion of the main vessel is too short to be used for diameter calculation, the main vessel diameter is calculated using its sub-branches.
[0108] Building upon this, further calculation of the central vessel arteriovenous equivalent ratio (AVR) can be included. The measured vessel diameter is generally within a range of 0.5 to 1.5 times the optic disc diameter from the edge of the solid disc. The measured AVR value is the ratio of the accompanying arterial equivalent to the venous equivalent, i.e.:
[0109] AVR = CRAE / CRVE
[0110] CRAE and CRVE are the arterial and venous equivalents of the accompanying vessels, respectively.
[0111] Step S330: Extract the tortuosity features of the retinal blood vessels.
[0112] Similar to step S320, blood vessels in region B (0.5-1.5DD) of the retinal fundus image are selected for feature extraction, with the six thickest arteries and veins having a diameter greater than 40 micrometers chosen. The commonly used method for calculating tortuosity in existing technologies is Distance Factor (DF). DF calculation only considers the two ends of the blood vessel and is unrelated to the direction of the internal curvature, resulting in DF not accurately reflecting the tortuosity of fundus blood vessels. Therefore, this step uses the ratio of the vessel's curvature to its arc length to describe the tortuosity characteristics of fundus blood vessels.
[0113] Step S340: Extract the branching features of retinal blood vessels.
[0114] For branching characteristics, the main calculation scope is the first-order bifurcation of blood vessels. The specific characteristics calculated are the bifurcation angle and branch asymmetry. Since the branching blood vessels are not straight vessels, the branching characteristics of the near bifurcation point and the far bifurcation point are calculated. The asymmetry between the daughter branches and the parent branches is calculated to describe the relationship between the daughter branches and the parent branches.
[0115] For the bifurcation angle: Using a method that fits the vessel segment, the optimal linear models for the two branch vessels are obtained. The direction vectors of the two branches are then calculated, with the common starting point being the bifurcation point (x0, y0). The direction vector of the thicker branch is (x1, y1), and the direction vector of the thinner branch is (x2, y2). First, the lengths of the two vectors are calculated, then the cosine value of the included angle is calculated, and finally, the included angle is obtained using the arccos function.
[0116] For branch asymmetry: For the diameter of a vessel near a bifurcation, the asymmetry between the two branches is described by dividing the square of the diameter of the larger branch by the square of the diameter of the smaller branch.
[0117] Step S350: Extract the fractal dimension features of the retinal blood vessels.
[0118] The above descriptions are all based on indicators calculated from a single fundus blood vessel. In order to describe the global complexity of fundus blood vessels, the fractal dimension feature is introduced.
[0119] Fractal dimension is a statistical measure used to describe the degree of space filling by a fractal. Methods for defining fractal dimension include Hausdorff dimension, box-counting dimension, and distributive dimension. This invention uses fractal dimension to reflect the overall complexity of blood vessels, preferably employing the box-counting dimension method. The principle is as follows: Using a number of cells of a given size to completely cover the retinal blood vessel object, calculate how many cells are needed to cover the object, and repeat this process for cells of different sizes. The scaling factor of the number of cells covering the object with the cell size provides an estimate of the object's fractal dimension.
[0120] Assuming that when the side length of the grid is ε, N grids are needed to cover the retinal blood vessels, then the box dimension is:
[0121] The fundus vascular features obtained through step S300 above can be used in conjunction with fundus images for early screening or prediction of diabetes, specifically including:
[0122] In step S400, the features extracted in step S300 are processed by the ResNet-Meta-Classifier Model to perform early screening or prediction.
[0123] The proposed meta-classifier model (ResNet-Meta-Classifier Model) in this invention has the following specific structure: Figure 7 As shown, it can be generally divided into two layers, namely Figure 7 The input layer on the left side of the middle, Figure 7 The fully connected output layer is on the right side. The input layer consists of two modules: Figure 7 The left-hand input layer consists of a first input processing module and a second input processing module, from top to bottom. The first input processing module uses a convolutional neural network (ResNet) to process the input image, specifically a five-channel fundus color image, i.e., a fundus image on top of an RGB three-channel fundus color image. This fundus image is preferably a fused image, which integrates arterial and venous data. Specifically, the fundus color image is a three-channel RGB image, and the arterial and venous annotated images are red-blue two-channel images. The three-channel image and the two-channel image are concatenated in parallel to form a five-channel fused image. The second input processing module uses a three-layer multilayer perceptron (MLP) to process the one-dimensional fundus vascular features of the input. The fully connected output layer includes a Scores module and a Softmax module. The outputs from the first and second input processing modules are weighted and scored in the Scores module and then concatenated. The resulting output is then fed into the Softmax module to obtain the final result.
[0124] Among them, the convolutional neural network used to process fundus images is preferably a deep residual network (Resnet). There are five commonly used depth structures of Resnet, namely 18, 34, 50, 101 and 152. This invention preferably uses the 50 depth structure, namely Resnet50. Specifically, ResNet50 mainly includes five processing stages (excluding global average pooling layers and fully connected layers). Stage 0 is for preprocessing the input data, including convolutional layers (Conv) and max pooling layers, where the convolutional kernels (Conv) are 7×7. Stages 1, 2, 3, and 4 are processing layers composed of residual units (ResBlocks). Stage 1 consists of 3 layers of residual units, Stage 2 consists of 4 layers, Stage 3 consists of 6 layers, and Stage 4 consists of 3 layers. ResNet50 may further include global average pooling layers (Avg Pooling) and fully connected layers. The fully connected layers include fully connected layers (FC) and softmax layers, where FC performs full connection and softmax performs output normalization.
[0125] The three-layer perceptron structure described above preferably uses three hidden layers, each with 128 neurons, and the activation function is the rectified linear function (ReLU).
[0126] Test results:
[0127] To verify the method and model of this invention, a total of 10,000 images from the entire dataset were used for testing. The selected input features, specifically for fundus vascular features, mainly included the number of arteries, global arterial tortuosity, arterial fractal dimension, arterial bifurcation angle, arterial bifurcation asymmetry, arterial diameter features, the number of veins, global vein tortuosity, vein fractal dimension, vein bifurcation angle, vein bifurcation asymmetry, vein diameter features, central arterial equivalent, central vein equivalent, and central arteriovenous equivalent ratio. The model preferably employs a Stacking ensemble strategy. Through model training and testing, the information gain of each data feature can be obtained, i.e., the weight of each typical data feature to the classification result.
[0128] Experimental results show that incorporating fundus vascular features and making predictions is more effective than using only a single fundus vascular image. The fundus vascular feature image processing method proposed in this invention extracts more accurate vascular features after completing the vascular breakpoints. It can maximize the predictive effect of diabetes when combined with the Resnet-Meta-Classifier Model, thus achieving the goal of early screening and prediction of diabetes.
[0129] This invention is not limited to the specific embodiments described above. It is understood that various modifications and variations can be made without departing from the spirit and scope of this invention, and all such modifications and variations should be included within the scope of protection of this invention.
Claims
1. A method for processing fundus vascular images for diabetes prediction, characterized in that... include: Step S100: Preprocess the original fundus blood vessel image; Step S200 involves identifying vascular information in the fundus vascular image and converting it into coordinate and morphological data of the fundus vessels for storage; step S200 specifically includes: Step S210: Extract the center line of the arteries and veins in the fundus; Step S220: Identify key points of retinal blood vessels; Step S230: Connect the identified fundus vessel breaks; connect two vessel breaks belonging to the same fundus vessel using a right-angle broken line. Step S240: Store the coordinates of the fundus blood vessel centerline and the bifurcation relationship of the fundus blood vessels; Step S300: Extract fundus vascular features based on fundus vascular coordinates and morphological data, which includes the following steps: Step S310: Extract the diameter characteristics of blood vessels in the fundus; Step S320: Extract vascular equivalent features of retinal blood vessels; Step S330: Extract the tortuosity features of retinal blood vessels; Step S340: Extract the branching features of retinal blood vessels; Step S350: Extract the fractal dimension features of the fundus blood vessels. Using the box-counting dimension method, use a number of cells of a given size to completely cover the fundus blood vessel object. Calculate how many cells are needed to cover the object and repeat the process for cells of different sizes. The scaling factor of the number of cells covering the object with the cell size gives an estimate of the object's fractal dimension. In step S400, the fundus vascular features extracted in step S300 are processed by a meta-classifier model to perform early screening or prediction of diabetes. The meta-classifier model is the Resnet-Meta-Classifier Model. The meta-classifier model includes an input layer and a fully connected output layer; the input layer includes a first input processing module and a second input processing module; wherein, the first input processing module adopts a deep residual convolutional neural network to process the input fundus image; the second input processing module adopts a three-layer multilayer perceptron to process the input fundus vascular features.
2. The fundus vascular image processing method for diabetes prediction as described in claim 1, characterized in that: Step S100 specifically includes: Step S110: The original fundus blood vessel image is subjected to size uniformization processing; Step S120: Perform color uniformization processing on the fundus blood vessel image that has undergone size uniformization processing; Step S130: Perform size correction processing on the fundus blood vessel image after color uniformization processing.
3. The fundus vascular image processing method for diabetes prediction as described in claim 2, characterized in that: Step S110 specifically includes: Step S112: Convert the original RGB three-channel color fundus blood vessel image into a single-channel grayscale image; Step S114: Obtain the grayscale threshold of the black border region based on the grayscale image, and obtain the effective image region mask on the grayscale image based on the grayscale threshold of the black border region. Step S116: Filter the pixels of the three RGB channels of the original color fundus blood vessel image using effective image area masking. After all steps are completed, reassemble the three RGB channels into a color fundus blood vessel image.
4. The fundus vascular image processing method for diabetes prediction as described in claim 3, characterized in that: Specifically, step S120 uses Gaussian filtering to adjust brightness and contrast; after correction in step S130, one pixel in the image corresponds to a length of 10 micrometers.
5. The fundus vascular image processing method for diabetes prediction as described in claim 4, characterized in that: In step S220, key points are extracted based on the connectivity of the 8-neighborhood of the center point to determine the key point type. The formula used is as follows: ； Where p represents the center pixel, N(p) represents the type of pixel p, and I t (p) represents the eight neighbor values of the center pixel p; when N(p) is 1, pixel p is the endpoint of the fundus blood vessel; when N(p) is 2, pixel p is the continuous point of the fundus blood vessel; when N(p) is 3, pixel p is the key point of the bifurcation of the fundus blood vessel; when N(p) is 4, pixel p is the key point of the intersection of the fundus blood vessel.
6. The fundus vascular image processing method for diabetes prediction as described in claim 5, characterized in that: Step S320 includes selecting the six thickest arteries and veins (greater than 40 micrometers) in the region of interest from the retinal fundus image for calculation. If there are fewer than six, all of them are included in the calculation. Each time, the two largest and smallest blood vessels are selected and their root mean squares are calculated according to a certain weight. The calculation results are then added back to the data for the next round of iteration until the final equivalent is obtained. The specific calculation method for the converted root mean square is as follows: ； ； Among them, W a For narrower blood vessel width, W b For a wider blood vessel width, W c This is an estimate of the vessel width.
7. The fundus vascular image processing method for diabetes prediction as described in claim 6, characterized in that: The deep residual convolutional neural network is ResNet50; The fully connected output layer includes a Scores module and a Softmax module; the outputs of the first input processing module and the second input processing module are weighted and scored in the Scores module and then concatenated, and the results are input into the Softmax module to obtain the final result.
8. A fundus vascular image processing system for diabetes prediction, comprising at least one processor and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions executable by the processor, which are executed by the at least one processor to cause the at least one processor to perform a fundus vascular image processing method for diabetes prediction as described in any one of claims 1-7.