Direct recognition method and system for retinal arteriovenous blood vessels based on fundus images

Abstract

A direct recognition method for retinal arteriovenous blood vessels based on fundus images is provided, including: processing an input fundus image using a basic segmentation network to output a feature map with 64 channels; processing the feature map with 64 channels using a blood vessel constraint module to output a first result map; the first result map can generate a 4-channel arteriovenous feature map including background, artery, vein, and unknown blood vessels through a 1×1 convolution; the 4-channel arteriovenous feature map generates a final retinal arteriovenous blood vessel recognition map through a first Sigmoid function. The arteriovenous blood vessel classification result obtained by the present invention has higher accuracy, higher efficiency, and higher robustness.

Description

Technical Field

[0001] The present invention belongs to the field of fundus image detection, and particularly relates to a method and system for directly identifying retinal arteriovenous blood vessels based on fundus images. Background Art

[0002] Retinal blood vessels are the only internal blood vessel tissues that can be observed non-invasively in the human body. Many systemic diseases such as diabetes, hypertension, and cardiovascular diseases can cause changes in the structure and morphology of retinal blood vessels. Different diseases and stages of progression have different effects on arteries and veins. For example, artery narrowing is considered a phenomenon related to hypertension, while vein widening is related to stroke and cardiovascular diseases. In order to analyze the morphological characteristics of retinal arteriovenous vessels, it is first necessary to accurately segment and obtain retinal arteriovenous blood vessels from fundus images. Traditional retinal blood vessel segmentation methods rely on the manual operations of professional physicians with a large amount of professional knowledge and experience accumulation, and have defects such as difficult standardization among different operators, poor repeatability, and low efficiency, making it difficult to meet the huge diagnosis and treatment needs. Therefore, realizing automatic arteriovenous identification of retinal blood vessels and obtaining relevant quantitative parameters can greatly reduce medical costs, assist medical research, and promote the development and popularization of fundus screening technologies.

[0003] In recent years, many automated technologies for retinal arteriovenous classification have been proposed, and these technologies can generally be classified into two categories: graph-based technologies and feature-based technologies. However, these technologies mainly rely on the results of the previously segmented binary graph of the blood vessel background to extract the blood vessel centerline to generate a blood vessel graph or extract features. On the one hand, the entire classification process is relatively slow, and on the other hand, the results of arteriovenous classification seriously depend on the accuracy of blood vessel segmentation. If the quality of blood vessel segmentation in the first stage is low, the accuracy of arteriovenous classification in the second stage will naturally decrease. Currently, all the proposed retinal arteriovenous automatic analysis software based on fundus images adopt a two-stage method of first performing blood vessel segmentation and then classifying arteriovenous vessels according to the color, brightness, morphological characteristics, etc. of arteriovenous vessels, and can only analyze a limited fundus image dataset, with a narrow application range, low blood vessel segmentation accuracy and efficiency.

[0004] In addition, the extraction of retinal blood vessel morphological geometric parameters by existing analysis software is also limited, and most only include blood vessel diameter and curvature measurement.

[0005] Therefore, it is necessary to study a method for directly identifying retinal arteriovenous blood vessels based on fundus images to solve one or more of the above technical problems. Summary of the Invention

[0006] To solve at least one of the above technical problems, according to one aspect of the present invention, there is provided a direct recognition method of retinal arteriovenous blood vessels based on fundus images, which adopts a semantic segmentation network for direct artery-vein-background classification without relying on the result of the binary map of blood vessel segmentation. The input of the network is the original fundus image taken, and the output is directly the arteriovenous segmentation result.

[0007] Furthermore, the present invention introduces the measurement of more geometric parameters of the retinal blood vessel morphology, making the application scope and popularity of the system wider. Moreover, the present invention only extracts the parameters of several main blood vessels within the region of interest, which improves the working efficiency of the system while meeting the clinical requirements.

[0008] Specifically, the direct recognition method of retinal arteriovenous blood vessels based on fundus images is characterized by including the following steps:

[0009] Process the input fundus image using the basic segmentation network to output a 64-channel feature map;

[0010] Process the 64-channel feature map using the blood vessel constraint module to output a first result map;

[0011] The first result map can generate a 4-channel arteriovenous feature map containing background, artery, vein, and unknown blood vessels through a 1×1 convolution;

[0012] The 4-channel arteriovenous feature map generates the final retinal arteriovenous blood vessel recognition map through the first Sigmoid function;

[0013] Among them, the blood vessel constraint module includes two parallel first branches and second branches. The first branch includes: two 3×3 convolution modules and a 1×1 convolution module for generating a blood vessel segmentation feature map according to the 64-channel feature map; a second Sigmoid function module for converting the blood vessel segmentation feature map into a probability map; and a Gaussian activation function module for generating a blood vessel activation map according to the probability map;

[0014] The second branch includes: two 3×3 convolution modules for generating an arteriovenous feature map according to the 64-channel feature map; a first matrix multiplication module for multiplying the probability map and the arteriovenous feature map and outputting, and a second matrix multiplication module for multiplying the output of the first matrix multiplication module and the blood vessel activation map and outputting the first result map.

[0015] According to another aspect of the present invention, the basic segmentation network is a U-shaped segmentation network.

[0016] According to another aspect of the present invention, the U-shaped segmentation network includes a left downsampling part and a right upsampling part. The left downsampling part performs two 3×3 convolutions on the fundus image to obtain a feature map, and then downsamples the size of the feature map to half of the original through a 2×2 max pooling layer. Next, multi-scale feature extraction is continuously performed using multi-scale feature modules in 3 feature layers, and the size of the input feature map is halved through a 3×3 convolution operation with a stride of 2. Finally, after passing through the multi-scale feature module and the 3×3 convolution operation, the obtained feature map is output to the right upsampling part; the right upsampling part includes 3 upsampling and feature map merging modules. Each upsampling and feature map merging module doubles the size of the feature map through a 2×2 upsampling operation, merges it with the corresponding feature map of the left downsampling part, and then performs two 3×3 convolution operations to output a feature map. Next, after a 2×2 upsampling operation, a feature map with 192 channels is obtained, and then a 64-channel feature map is output through a 3×3 convolution operation.

[0017] According to another aspect of the present invention, the multi-scale feature module uses a pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets according to the number of channels. The i-th subset is x i , i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, denoted by F i (), and the output y i is expressed as:

[0018]

[0019] Thus, outputs with different receptive field sizes are obtained. Finally, the k outputs, namely y1 to y k are fused and passed through a 1×1 convolution operation.

[0020] According to another aspect of the present invention, a system for directly identifying retinal arteriovenous blood vessels based on fundus images is further provided, which is characterized by including:

[0021] A basic segmentation network module for processing the input fundus image and outputting a 64-channel feature map;

[0022] A blood vessel constraint module for processing the 64-channel feature map and outputting a first result map;

[0023] A 1×1 convolution module for generating a 4-channel arteriovenous feature map including background, artery, vein, and unknown blood vessels based on the first result map; and

[0024] A first Sigmoid function module for generating a final retinal arteriovenous blood vessel identification map based on the 4-channel arteriovenous feature map;

[0025] Among them, the blood vessel constraint module includes two parallel first branches and second branches. The first branch includes: two 3×3 convolution modules and a 1×1 convolution module, which are used to generate a blood vessel segmentation feature map according to the 64-channel feature map; a second Sigmoid function module, which is used to convert the blood vessel segmentation feature map into a probability map; and a Gaussian activation function module, which is used to generate a blood vessel activation map according to the probability map.

[0026] The second branch includes: two 3×3 convolution modules, which are used to generate an arteriovenous feature map according to the 64-channel feature map; a first matrix multiplication module, which is used to multiply the probability map by the arteriovenous feature map and output, and a second matrix multiplication module, which is used to multiply the output of the first matrix multiplication module by the blood vessel activation map and output the first result map.

[0027] According to another aspect of the present invention, the basic segmentation network module is a U-shaped segmentation network module.

[0028] According to another aspect of the present invention, the U-shaped segmentation network module includes a left downsampling part and a right upsampling part. The left downsampling part performs two 3×3 convolutions on the fundus image to obtain a feature map, and then reduces the size of the feature map to half of the original through a 2×2 max pooling layer. Next, multi-scale feature extraction is continuously performed using a multi-scale feature module in 3 consecutive feature layers, and the size of the input feature map is halved through a 3×3 convolution operation with a stride of 2. Finally, after passing through the multi-scale feature module and a 3×3 convolution operation, the obtained feature map is output to the right upsampling part; the right upsampling part includes 3 upsampling and merging feature map modules. Each upsampling and merging feature map module doubles the size of the feature map through a 2×2 upsampling operation and merges it with the corresponding feature map of the left downsampling part, and then performs two 3×3 convolution operations to output a feature map. Next, after a 2×2 upsampling operation, a 192-channel feature map is obtained, and then a 64-channel feature map is output through a 3×3 convolution operation.

[0029] According to another aspect of the present invention, the multi-scale feature module uses a pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets according to the number of channels. The i-th subset is x i , i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, which is represented by F i (), then the output y i is expressed as:

[0030]

[0031] Outputs of different perceived visual field sizes are obtained, and finally the k outputs, namely y1 to y k are fused and subjected to a 1x1 convolution operation.

[0032] According to another aspect of the present invention, there is also provided a method for automatically analyzing retinal arteriovenous blood vessels based on fundus images, which is characterized by including the following steps:

[0033] Obtaining the fundus image to be analyzed;

[0034] Automatically identifying retinal arteriovenous blood vessels;

[0035] Post-processing and repairing arteriovenous blood vessels;

[0036] Extracting the centerline and boundary of blood vessels;

[0037] Identifying the intersection points of the centerlines of blood vessels;

[0038] Detecting the optic disc in the fundus image;

[0039] Locating the region of interest;

[0040] Selecting the arteriovenous blood vessels to be analyzed within the region of interest;

[0041] Obtaining morphological geometric parameters;

[0042] Among them, the automatic identification of retinal arteriovenous blood vessels is specifically to directly identify using the aforementioned method.

[0043] According to another aspect of the present invention, there is also provided a system for automatically analyzing retinal arteriovenous blood vessels based on fundus images, which is characterized by including:

[0044] A first module for obtaining the fundus image to be analyzed;

[0045] A second module for automatically identifying retinal arteriovenous blood vessels;

[0046] A third module for post-processing and repairing arteriovenous blood vessels;

[0047] A fourth module for extracting the centerline and boundary of blood vessels;

[0048] A fifth module for identifying the intersection points of the centerlines of blood vessels;

[0049] A sixth module for detecting the optic disc in the fundus image;

[0050] A seventh module for locating the region of interest;

[0051] An eighth module for selecting the arteriovenous blood vessels to be analyzed within the region of interest;

[0052] The ninth module is used for obtaining morphological geometric parameters;

[0053] Among them, the second module is specifically the aforementioned system for directly identifying retinal arteriovenous blood vessels based on fundus images.

[0054] The present invention can achieve one or more of the following technical effects:

[0055] 1. The present invention realizes the direct identification of retinal arteriovenous blood vessels (one-step direct identification) through the blood vessel constraint module. This blood vessel constraint module alleviates the problem of imbalance between positive and negative samples in retinal blood vessel segmentation and enhances the feature expression of the edges of microvessels and main blood vessels at the same time;

[0056] 2. The classification result of arteriovenous blood vessels obtained by the present invention has higher accuracy, is more efficient, and has higher robustness;

[0057] 3. Selecting the annular region within 0.5 - 2 times the disc diameter around the optic disc as the region of interest for extracting morphological parameters further improves the working efficiency of the system and has clinical significance;

[0058] 4. The design of the multi-scale feature module can adapt to and automatically analyze fundus photos collected by cameras with different resolutions and different models, improving the compatibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The following further describes the present invention in detail with reference to the drawings and specific embodiments.

[0060] Figure 1 It is a schematic diagram of the blood vessel constraint module according to a preferred embodiment of the present invention.

[0061] Figure 2 It is a schematic diagram of the U-shaped segmentation network module according to another preferred embodiment of the present invention.

[0062] Figure 3 It is a schematic diagram of the arteriovenous recognition results of fundus images with different resolutions according to another preferred embodiment of the present invention.

[0063] Figure 4 It is a flowchart of the method for automatically analyzing retinal arteriovenous blood vessels based on fundus images according to another preferred embodiment of the present invention.

[0064] Figure 5 It is a schematic diagram of the optic disc detection and the positioning of the region of interest (the white circular ring is the region of interest) according to another preferred embodiment of the present invention.

[0065] Figure 6 It is Figure 5 a schematic diagram of the extraction results of arteriovenous blood vessels within the region of interest in DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] The following will describe the best implementation mode of the present invention in conjunction with the accompanying drawings through preferred embodiments. The specific implementation mode here is to describe the present invention in detail, and should not be construed as a limitation of the present invention. Without departing from the spirit and essence scope of the present invention, various deformations and modifications can be made, and these should be included within the protection scope of the present invention.

[0067] Embodiment 1

[0068] According to a preferred implementation mode of the present invention, referring to Figure 1 , a direct recognition method for retinal arteriovenous blood vessels based on fundus images is provided, which is characterized by including the following steps:

[0069] Processing the input fundus image by using a basic segmentation network to output a feature map with 64 channels;

[0070] Processing the feature map with 64 channels by using a blood vessel constraint module to output a first result map;

[0071] The first result map can generate a 4-channel arteriovenous feature map including background, artery, vein, and unknown blood vessels through 1×1 convolution;

[0072] The 4-channel arteriovenous feature map generates a final retinal arteriovenous blood vessel recognition map through the first Sigmoid function.

[0073] Among them, the blood vessel constraint module includes two parallel first branches and second branches. The first branch includes: two 3×3 convolution modules and one 1×1 convolution module, which are used to generate a blood vessel segmentation feature map according to the 64-channel feature map; a second Sigmoid function module, which is used to convert the blood vessel segmentation feature map into a probability map; and a Gaussian activation function module, which is used to generate a blood vessel activation map according to the probability map.

[0074] Preferably, the second branch includes: two 3×3 convolution modules, which are used to generate an arteriovenous feature map according to the 64-channel feature map; a first matrix multiplication module, which is used to multiply the probability map and the arteriovenous feature map and output, and a second matrix multiplication module, which is used to multiply the output of the first matrix multiplication module and the blood vessel activation map and output the first result map.

[0075] Preferably, referring to Figure 3 , which shows the arteriovenous recognition results of the method of the present invention for fundus images with different resolutions.

[0076] According to another preferred implementation mode of the present invention, the basic segmentation network is a U-shaped segmentation network.

[0077] According to another preferred implementation mode of the present invention, referring toFigure 2 The U-shaped segmentation network includes a left downsampling part and a right upsampling part. The left downsampling part performs two 3×3 convolutions on the fundus image to obtain a feature map, and then reduces the size of the feature map to half of the original through a 2×2 max pooling layer. Next, multi-scale feature extraction is continuously performed using multi-scale feature modules in three feature layers, and the size of the input feature map is halved through a 3×3 convolution operation with a stride of 2. Finally, after passing through the multi-scale feature module and the 3×3 convolution operation, the obtained feature map is output to the right upsampling part; the right upsampling part includes three upsampling and merging feature map modules. Each upsampling and merging feature map module doubles the size of the feature map through a 2×2 upsampling operation, merges it with the corresponding feature map in the left downsampling part, and then performs two 3×3 convolution operations to output a feature map. Next, after a 2×2 upsampling operation, a feature map with 192 channels is obtained, and then a 64-channel feature map is output through a 3×3 convolution operation.

[0078] According to another preferred embodiment of the present invention, the multi-scale feature module uses a pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets according to the number of channels. The i-th subset is x i , i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, denoted by F i (), and the output y i is expressed as:

[0079]

[0080] Thus, outputs with different receptive field sizes are obtained. Finally, the k outputs, namely y1 to y k are fused and passed through a 1×1 convolution operation.

[0081] According to another preferred embodiment of the present invention, a system for directly identifying retinal arteriovenous blood vessels based on fundus images is further provided, which is characterized by including:

[0082] A basic segmentation network module for processing the input fundus image and outputting a 64-channel feature map;

[0083] A blood vessel constraint module for processing the 64-channel feature map and outputting a first result map;

[0084] A 1×1 convolution module for generating a 4-channel arteriovenous feature map including background, artery, vein, and unknown blood vessels according to the first result map; and

[0085] The first Sigmoid function module is used to generate the final retinal arteriovenous blood vessel recognition map according to the 4-channel arteriovenous feature map;

[0086] Among them, the blood vessel constraint module includes two parallel first branches and second branches. The first branch includes: two 3×3 convolution modules and a 1×1 convolution module, which are used to generate a blood vessel segmentation feature map according to the 64-channel feature map; the second Sigmoid function module is used to convert the blood vessel segmentation feature map into a probability map; and the Gaussian activation function module is used to generate a blood vessel activation map according to the probability map;

[0087] The second branch includes: two 3×3 convolution modules, which are used to generate an arteriovenous feature map according to the 64-channel feature map; the first matrix multiplication module is used to multiply the probability map and the arteriovenous feature map and output, and the second matrix multiplication module is used to multiply the output of the first matrix multiplication module and the blood vessel activation map and output the first result map.

[0088] According to another preferred embodiment of the present invention, the basic segmentation network module is a U-shaped segmentation network module.

[0089] According to another preferred embodiment of the present invention, the U-shaped segmentation network module includes a left downsampling part and a right upsampling part. The left downsampling part performs two 3×3 convolutions on the fundus image to obtain a feature map, and then passes through a 2×2 max pooling layer to downsample the feature map size to half of the original. Next, multi-scale feature extraction is continuously performed using the multi-scale feature module in 3 feature layers, and the input feature map size is halved through a 3×3 convolution operation with a stride of 2. Finally, after passing through the multi-scale feature module and a 3×3 convolution operation, the obtained feature map is output to the right upsampling part; the right upsampling part includes 3 upsampling and merging feature map modules. Each upsampling and merging feature map module doubles the feature map size to 2 times the input feature map through a 2×2 upsampling operation, merges it with the corresponding feature map in the left downsampling part, and then performs two 3×3 convolution operations to output a feature map. Next, after performing a 2×2 upsampling operation, a 192-channel feature map is obtained, and then a 64-channel feature map is output through a 3×3 convolution operation.

[0090] According to another preferred embodiment of the present invention, the multi-scale feature module uses a pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets according to the number of channels. The i-th subset is x i , i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, which is represented by F i (), then the output y i is expressed as:

[0091]

[0092] Thus, outputs with different perceived visual field sizes are obtained, and finally, the k outputs, namely y1 to y k are fused and subjected to a 1x1 convolution operation.

[0093] According to another preferred embodiment of the present invention, referring to Figures 4-6 , a method for automatically analyzing retinal arteriovenous blood vessels based on fundus images is further provided, which is characterized by including the following steps:

[0094] Obtaining the fundus image to be analyzed;

[0095] Automatically identifying retinal arteriovenous blood vessels;

[0096] Post-processing and repairing arteriovenous blood vessels;

[0097] Extracting the centerline and boundary of blood vessels;

[0098] Identifying the intersection points of the centerlines of blood vessels;

[0099] Detecting the optic disc in the fundus image;

[0100] Locating the region of interest;

[0101] Selecting the arteriovenous blood vessels to be analyzed within the region of interest;

[0102] Obtaining morphological geometric parameters;

[0103] Among them, the automatic identification of retinal arteriovenous blood vessels is specifically to directly identify using the aforementioned method.

[0104] Each step will be described in detail below.

[0105] Obtaining the fundus image to be analyzed. The doctor takes pictures of the patient's eye images through a fundus camera, and then conducts quality analysis. If the photo quality does not meet the standard, the collection continues until the quality of the collected fundus pictures meets the standard before they can be used for the next analysis and processing. There is no requirement for the camera model for collecting fundus photos here.

[0106] Identifying retinal arteriovenous blood vessels. Conduct arteriovenous identification on the fundus pictures collected in the first step. The present invention proposes a vessel-constraint network (VC-Net) for classifying retinal arteriovenous blood vessels. Preferably, a U-shaped network can be used as the basic network architecture for arteriovenous identification, such as Figure 2As shown. At the same time, a multi-scale feature module (Multi-scale feature) is introduced in the downsampling process of the present invention. This module is used to learn the features of retinal blood vessels at different scales. Since there are retinal blood vessels of different scales in fundus images, such as the diameter of venous blood vessels being larger than that of arterial blood vessels, and the diameter of main blood vessels being larger than that of capillaries. Preferably, the multi-scale feature module uses pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets by the number of channels. The i-th subset is x i , i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, which is represented by F i (), then the output y i is expressed as:

[0107]

[0108] In this way, outputs with different receptive field sizes will be obtained. Finally, for example, four (k is taken as 4, but not limited to this) outputs are fused and passed through a 1x1 convolution. Preferably, k is set as a control parameter, that is, the input channel number can be evenly divided into multiple feature channels. The larger k is, the stronger the multi-scale ability is.

[0109] Advantageously, a vessel-constraint (VC) module is designed at the end of the basic segmentation network. After the fundus image passes through the basic segmentation network, two parallel branches will be output. One branch is used to generate a blood vessel segmentation feature map, and the other branch is used to generate an arteriovenous feature map. The blood vessel segmentation feature map is transformed into a probability map through the Sigmoid function, and then multiplied by the arteriovenous feature map for further arteriovenous classification. This part combines local and global blood vessel features to generate a highly reasonable blood vessel activation map to constrain the arteriovenous features, that is, to suppress the features tending to the background (negative samples) and pay more attention to the blood vessel (positive sample) features. This can well alleviate the problem of imbalance between positive and negative samples. In a fundus image, blood vessels only account for 15% of the whole image, and the corresponding arterial and venous blood vessels each account for about 7.5%. This is extremely challenging for directly classifying the fundus image into background, artery and vein. The designed VC module can focus the arteriovenous classification task on blood vessels and pay more attention to blood vessel features, so as to directly perform arteriovenous classification. At the same time, after the blood vessel segmentation probability map, a Gaussian kernel function is designed to map the probability to increase the feature weights of the blood vessel edge region and capillaries, so as to enhance the feature expression of the microvascular and thick blood vessel edges. The Gaussian activation function in the present invention is defined as: F(x) = α(e -|x-0.5| -e -0.5) + 1, where x is the probability map of the entire blood vessel segmentation, x ∈ [0, 1], and α is a fixed parameter greater than 0 (set to 1 in this study). Based on existing research and experimental observations, the probabilities of capillary and boundary pixels are basically concentrated around 0.5, while the main blood vessels and background pixels are close to 1 and 0 respectively. F(x) constrains the activation weight values within the range of [1, α(1 - e -0.5 ) + 1]. After passing through F(x), pixels with probabilities close to 0.5 will be assigned higher weights (close to (α(1 - e -0.5 ) + 1)), while the background and main blood vessels will be assigned lower weights (close to 1), and potential capillaries will be activated during this process. Then, the result after multiplying the blood vessel activation map by the blood vessel probability map and the arteriovenous feature map is multiplied, and a 4-channel (background, artery, vein, unknown blood vessel) arteriovenous feature map can be generated through a 1×1 convolution. Finally, the final retinal blood vessel arteriovenous recognition probability map is generated through the Sigmoid function. The network structure of the entire retinal blood vessel arteriovenous recognition is as Figure 1 shown.

[0110] The input of this network is the original fundus image collected in the first step, and the output after passing through the network is the classified artery-vein-background feature map. The retinal blood vessel arteriovenous recognition network proposed by the present invention is applicable to fundus images with different resolutions, such as the DRIVE, LES, and HRF fundus image datasets, whose resolutions are 584×565, 1444×1620, and 3304×2336 respectively. The arteriovenous recognition results of these three datasets with different resolutions are as Figure 3 shown. The first row is the original fundus images corresponding to the three datasets (the input of the network), and the second row is the arteriovenous recognition results using the method proposed by the present invention (the output of the network). Compared with the two-stage method that first performs blood vessel segmentation and then recognizes arteriovenous based on features such as the color and morphology of the blood vessels, the direct arteriovenous recognition method proposed by the present invention is more efficient, more robust, and has stronger model generalization ability. However, even the most advanced deep learning frameworks currently cannot achieve completely accurate arteriovenous blood vessel recognition, and there will always be certain flaws in the results. Therefore, after the results of passing through the deep learning network, we can repair them through a series of post-processing algorithms to make the prediction results more perfect. The post-processing methods mentioned here mainly refer to some traditional algorithms based on network topologies, etc.

[0111] Vascular centerline and boundary extraction. After identifying the arteries and veins in fundus images, the vascular centerline and boundary are extracted. The vascular centerline and boundary information play an auxiliary role in obtaining vascular parameters. For example, for the measurement of vascular diameter, an edge detector based on information fusion can be used to obtain the vascular edge information. After identifying the vascular path of interest, the initial diameter of the vascular of interest can be calculated by automatically generating cross-lines perpendicular to the vascular centerline. The distance between the two cross-lines of each cross-line and the vascular edge is used as the vascular diameter. For a certain section of blood vessels, the average value of the distances between multiple cross-lines and the two cross-lines of the vascular edge can be taken as the diameter of this blood vessel.

[0112] Identification of the intersection points of the vascular centerline. The intersection points can be the intersection points of arterial blood vessels and arterial blood vessels, the branch points of arterial blood vessels, the intersection points of venous blood vessels and venous blood vessels, the branch points of venous blood vessels, or the intersection points of arterial blood vessels and venous blood vessels. To reduce the influence of the classification error between arteries and veins, the vascular parameters at the intersection points are not considered.

[0113] Detection of the optic disc in fundus images. For the fundus images collected in the first step, some mainstream deep learning network models for object detection can be used for rapid and accurate detection of the optic disc. Since the blood vessels in the optic disc are complexly intertwined and not conducive to analysis, and the calculation of vascular parameters mainly focuses on the main blood vessels, the farther away from the optic disc, the lower the calculation value. Therefore, information acquisition and quantification of relevant morphological parameters are only carried out on the blood vessels within a certain range around the optic disc.

[0114] Localization of the region of interest. Taking the center of the optic disc detected in the previous step as the origin, an annular region within 0.5 - 2 times the diameter of the optic disc is selected as the region of interest for extracting morphological parameters, as Figure 5 shown.

[0115] Selection of the arteries and veins to be analyzed in the region of interest. According to clinical needs, usually K of the widest arteries and veins within the annular region are selected for parameter extraction. The number of blood vessels selected can be determined by the doctor according to clinical needs, and K can take integers such as 4, 5, 6, 7, etc. Compared with calculating the morphological geometric parameters of all blood vessels within the annular region, this method can greatly reduce the workload and improve the working efficiency of the system while meeting the requirements.

[0116] Obtaining morphological geometric parameters. Quantify the morphological geometric parameters of retinal blood vessels. In this step, the morphological geometric parameters of the K pairs of arteries and veins selected in the previous step are obtained. The parameters mainly include: vessel diameter, curvature, artery-vein ratio, fractal dimension, branch angle, branch coefficient, etc. Compared with extracting vascular geometric parameters from the entire fundus image, the method proposed by the present invention can meet clinical needs while reducing the computational complexity.

[0117] According to another preferred embodiment of the present invention, there is also provided a retinal arteriovenous vessel automatic analysis system based on fundus images, which is characterized by including:

[0118] A first module for obtaining the fundus image to be analyzed;

[0119] A second module for automatically identifying retinal arteriovenous vessels;

[0120] A third module for post-processing and repairing arteriovenous vessels;

[0121] A fourth module for extracting the centerline and boundary of blood vessels;

[0122] A fifth module for identifying the intersection points of the centerlines of blood vessels;

[0123] A sixth module for detecting the optic disc in the fundus image;

[0124] A seventh module for positioning the region of interest;

[0125] An eighth module for selecting the arteriovenous vessels to be analyzed within the region of interest;

[0126] A ninth module for obtaining morphological and geometric parameters;

[0127] Among them, the second module is specifically the aforementioned system for directly identifying retinal arteriovenous vessels based on fundus images.

[0128] The present invention can achieve one or more of the following technical effects:

[0129] 1. The present invention realizes the direct identification of retinal arteriovenous vessels (one-step direct identification) through the blood vessel constraint module, which alleviates the problem of unbalanced positive and negative samples in retinal blood vessel segmentation and enhances the feature expression of the edges of microvessels and main blood vessels;

[0130] 2. The classification result of arteriovenous vessels obtained by the present invention has higher accuracy, is more efficient, and has higher robustness;

[0131] 3. Selecting the annular region within 0.5 - 2 times the optic disc diameter around the optic disc as the region of interest for extracting morphological parameters further improves the working efficiency of the system and has clinical significance;

[0132] 4. The design of the multi-scale feature module can adapt to and automatically analyze fundus photos collected by cameras with different resolutions and different models, improving the compatibility of the system.

[0133] Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A direct recognition method for retinal arteriovenous blood vessels based on fundus images, characterized in that It includes the following steps: Process the input fundus image using the basic segmentation network to output a 64-channel feature map; Process the 64-channel feature map using the vascular constraint module to output a first result map; The first result map can generate a 4-channel arteriovenous feature map including background, artery, vein, and unknown blood vessels through 1×1 convolution; The 4-channel arteriovenous feature map generates the final retinal arteriovenous blood vessel recognition map through the first Sigmoid function; Among them, the vascular constraint module includes two parallel first branches and second branches. The first branch includes: two 3×3 convolution modules and a 1×1 convolution module for generating a vascular segmentation feature map according to the 64-channel feature map; a second Sigmoid function module for converting the vascular segmentation feature map into a probability map; and a Gaussian activation function module for generating a vascular activation map according to the probability map; The second branch includes: two 3×3 convolution modules for generating an arteriovenous feature map according to the 64-channel feature map; a first matrix multiplication module for multiplying the probability map and the arteriovenous feature map and outputting, and a second matrix multiplication module for multiplying the output of the first matrix multiplication module and the vascular activation map and outputting the first result map.

2. The direct recognition method of retinal arteriovenous blood vessels based on fundus images according to claim 1, characterized in that The basic segmentation network is a U-shaped segmentation network.

3. The method for directly identifying retinal arteriovenous blood vessels based on fundus images according to claim 2, wherein The U-shaped segmentation network includes a left downsampling part and a right upsampling part. The left downsampling part performs two 3×3 convolutions on the fundus image to obtain a feature map, and then reduces the size of the feature map to half of the original through a 2×2 max pooling layer. Next, multi-scale feature extraction is continuously performed using the multi-scale feature module in 3 feature layers, and the size of the input feature map is halved through a 3×3 convolution operation with a stride of 2. Finally, after the multi-scale feature module and 3×3 convolution operation, the obtained feature map is output to the right upsampling part; the right upsampling part includes 3 upsampling and merging feature map modules. Each upsampling and merging feature map module expands the size of the feature map to 2 times the input feature map through a 2×2 upsampling operation, merges it with the corresponding feature map in the left downsampling part, and then performs two 3×3 convolution operations to output a feature map. Next, after a 2×2 upsampling operation, a 192-channel feature map is obtained, and then a 64-channel feature map is output through a 3×3 convolution operation.

4. The method for directly identifying retinal arteriovenous blood vessels based on fundus images according to claim 3, wherein The multi-scale feature module uses pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets by the number of channels, and the i-th subset is x i , where i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, denoted by F i (), and the output y i is expressed as: Outputs of different perceived field sizes are obtained, and finally the k outputs, namely y1 to y k are fused and passed through a 1x1 convolutional operation.

5. A system for directly identifying retinal arteriovenous blood vessels based on fundus images, characterized in that It includes: A basic segmentation network module for processing the input fundus image and outputting a 64-channel feature map; A vascular constraint module for processing the 64-channel feature map and outputting a first result map; A 1×1 convolution module for generating a 4-channel arteriovenous feature map including background, artery, vein, and unknown blood vessels according to the first result map; and A first Sigmoid function module for generating the final retinal arteriovenous blood vessel recognition map according to the 4-channel arteriovenous feature map; Among them, the blood vessel constraint module includes two parallel first branches and second branches. The first branch includes: two 3×3 convolution modules and a 1×1 convolution module, which are used to generate a blood vessel segmentation feature map according to the 64-channel feature map; a second Sigmoid function module, which is used to convert the blood vessel segmentation feature map into a probability map; and a Gaussian activation function module, which is used to generate a blood vessel activation map according to the probability map. The second branch includes: two 3×3 convolution modules, which are used to generate an arteriovenous feature map according to the 64-channel feature map; a first matrix multiplication module, which is used to multiply the probability map and the arteriovenous feature map and output; a second matrix multiplication module, which is used to multiply the output of the first matrix multiplication module and the blood vessel activation map and output the first result map.

6. The system for directly identifying retinal arteriovenous blood vessels based on fundus images according to claim 5, characterized in that The basic segmentation network module is a U-shaped segmentation network module.

7. The system for directly identifying retinal arteriovenous blood vessels based on fundus images according to claim 6, wherein The U-shaped segmentation network module includes a left downsampling part and a right upsampling part. The left downsampling part performs two 3×3 convolutions on the fundus image to obtain a feature map, and then passes through a 2×2 max pooling layer to downsample the feature map size to half of the original. Next, multi-scale feature extraction is continuously performed using a multi-scale feature module in 3 feature layers, and the input feature map size is halved through a 3×3 convolution operation with a stride of 2. Finally, after passing through the multi-scale feature module and a 3×3 convolution operation, the obtained feature map is output to the right upsampling part; the right upsampling part includes 3 upsampling and merging feature map modules. Each upsampling and merging feature map module enlarges the feature map size to 2 times the input feature map through a 2×2 upsampling operation, merges it with the corresponding feature map in the left downsampling part, and then performs two 3×3 convolution operations to output a feature map. Next, after performing a 2×2 upsampling operation, a 192-channel feature map is obtained, and then a 64-channel feature map is output through a 3×3 convolution operation.

8. The system for direct recognition of retinal arteriovenous blood vessels based on fundus images according to claim 7, characterized in that The multi-scale feature module uses pre-trained Res2Net. First, the feature map after 1×1 convolution is evenly divided into k subsets by the number of channels, and the i-th subset is x i , where i ∈ {1, 2,..., k}. Except for x1, each x i will have a corresponding 3×3 convolution, denoted by F i (), and the output y i is expressed as: Outputs of different perceived field sizes are obtained accordingly, and finally the k outputs, namely y1 to y k are fused and passed through a 1x1 convolutional operation.

9. An automatic analysis method for retinal arteriovenous blood vessels based on fundus images, characterized in that It includes the following steps: Obtaining the fundus image to be analyzed; Automatic arteriovenous recognition of retinal blood vessels; Post-processing and repair of arteriovenous blood vessels; Extracting the blood vessel centerline and boundary; Identifying the intersection points of the blood vessel centerline; Detecting the optic disc in the fundus image; Locating the region of interest; Selecting the arteriovenous blood vessels to be analyzed within the region of interest; Obtaining the morphological geometric parameters; Among them, the automatic arteriovenous recognition of retinal blood vessels is specifically direct recognition using the method described in any one of claims 1-4.

10. A retinal arteriovenous vessel automatic analysis system based on fundus images, characterized in that It includes: A first module, which is used for obtaining the fundus image to be analyzed; A second module, which is used for automatic arteriovenous recognition of retinal blood vessels; A third module, which is used for post-processing and repair of arteriovenous blood vessels; A fourth module, which is used for extracting the blood vessel centerline and boundary; A fifth module, which is used for identifying the intersection points of the blood vessel centerline; A sixth module, which is used for detecting the optic disc in the fundus image; A seventh module, which is used for locating the region of interest; An eighth module, which is used for selecting the arteriovenous blood vessels to be analyzed within the region of interest; A ninth module, which is used for obtaining the morphological geometric parameters; Among them, the second module is specifically the system for direct recognition of retinal arteriovenous blood vessels based on fundus images described in any one of claims 5-8.

Images