A method for automatically labeling retinal arteries, veins and capillaries in fundus color photos

Abstract

The present invention provides a method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs, comprising: using matching fundus color photographs and retinal angiography images from the same eye, first extracting all retinal blood vessels based on the retinal angiography images, registering retinal angiography images of different stages with fundus color photographs, automatically distinguishing arteries and veins using the difference between the development of the venous phase and the arterial phase of fluorescent angiography, and after registering the fundus color photography with the retinal angiography images, completely labeling the blood vessels, especially the capillaries, in the fundus color photography. The present invention automatically labels retinal arteries, veins and capillaries in fundus color photographs based on retinal angiography images, and the method can achieve more accurate, faster and more complete labeling of the contours and boundaries of retinal blood vessels in fundus color photography, while having the effect of distinguishing retinal arteries and veins.

Description

Technical Field

[0001] The invention relates to the technical field of retinal blood vessel annotation, and in particular to a method for automatically annotating retinal arteries, veins and capillaries in fundus color photographs. Background Art

[0002] Blood vessels are important structures for maintaining life and health. The retina is one of the organs with the highest oxygen consumption and therefore one of the organs with the highest blood supply. Normal vascular structure is essential for maintaining normal visual health. At the same time, the eye is the only part where blood vessels can be directly and clearly observed through equipment. The health of the microvessels in the eye is closely related to the health of the microvessels in important organs throughout the body, such as the heart, brain, and kidneys.

[0003] Fundus color photography is the most commonly used method for retinal examination. In order to explore the relationship between the characteristics of retinal blood vessels and eye diseases and systemic diseases, it is necessary to manually annotate the contours and boundaries of retinal blood vessels in fundus color photography. This manual annotation method has the following disadvantages: First, it is slow and inefficient. It takes a lot of time to manually annotate a large number of retinal blood vessels in a picture. Second, it is low in accuracy. It is difficult to accurately mark the complex paths, contours and boundaries of retinal blood vessels using manual methods. There is a huge annotation variation, poor repeatability, and it is difficult to avoid omissions in annotation. Third, it is difficult to complete the annotation of capillaries. The capillaries in the retinal blood vessels are only a few pixels and have complex paths, making them difficult to manually annotate. Fourth, it is difficult to accurately distinguish between arteries and veins. Traditional manual annotation distinguishes arteries and veins based on the color of the blood vessels (bright red for arteries and dark red for veins), thickness and path, but it is difficult to distinguish small blood vessels based on these characteristics. Summary of the invention

[0004] The purpose of the present invention is to provide a method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs. Based on retinal angiography images, the retinal arteries, veins and capillaries in fundus color photographs are automatically labeled, which can achieve more accurate, faster and more complete labeling of the contours and boundaries of retinal blood vessels in fundus color photography, while distinguishing the effect of retinal arteries and veins.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs comprises the following steps:

[0007] S1, based on the matching fundus color photographs and retinal angiography images from the same eye, first extract the entire retinal blood vessels from the retinal angiography images, and automatically distinguish arteries and veins based on the difference in fundus angiography between the venous phase and the arterial phase;

[0008] S2, registering the fundus color photography image with the retinal angiography image, and completing the complete labeling of the contours and boundaries of all retinal blood vessels in the fundus color photography, wherein the retinal blood vessels include arteries, veins and capillaries.

[0009] Furthermore, the step S1, extracting the entire retinal blood vessels based on the retinal angiography image, specifically includes: extracting a binary retinal vascular structure from the fluorescence angiography image using a morphological processing method or a deep learning model.

[0010] Furthermore, in step S1, the arteries and veins are automatically distinguished by the difference between the venous phase and the arterial phase of fundus angiography, which specifically includes:

[0011] The retinal angiography images of different stages were registered with the fundus color photographs, and the arteries and veins were automatically distinguished using the pixel intensity, color gradient, and morphological differences of the venous and arterial phases of fluorescence angiography.

[0012] Furthermore, the step S2, registering the fundus color photography image with the retinal angiography image, specifically includes:

[0013] Rigid registration was used to roughly match the green channel of the color photograph and the angiographic image of the same eye;

[0014] When the images are too different to match, the blood vessels in the fundus color photo are extracted and matched with the blood vessels in the fluorescence angiography image;

[0015] When rigid registration cannot match the peripheral small vessels well, further non-rigid matching is performed to fine-tune the misaligned small vessels.

[0016] Furthermore, the rigid registration method is used to roughly match the green channel of the color photo of the same eye with the angiography image, specifically:

[0017] Image features are obtained through feature point detection, mutual information, and similarity methods, the features in the two images are matched, abnormal features are filtered out, the homography matrix is ​​calculated, and the mapping of the two images and the alignment of one image to another are completed.

[0018] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs provided by the present invention extracts retinal blood vessels in fundus angiography images, and automatically distinguishes arteries and veins through the difference in the development of the venous phase and the arterial phase of fundus angiography; and aligns the fundus color photography with the retinal angiography image, thereby completing the complete labeling of the contours and boundaries of the arteries, veins and capillaries in the fundus color photographs. This method can label the contours and boundaries of the retinal blood vessels in the fundus color photography more accurately, quickly and completely. The obtained fundus color photographs with retinal blood vessel labeling can be used for quantitative and qualitative measurement of retinal blood vessels, and can also be used for the development and verification of artificial intelligence automatic blood vessel analysis systems, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 Schematic diagram of the process of the method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the automatic annotation process of microvessels in fundus color photography according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the artery and vein labeling process according to an embodiment of the present invention;

[0023] Figure 4 This is a comparison chart of the effects of the labeling method of the present invention and the manual labeling method. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The purpose of the present invention is to provide a method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs. Based on retinal angiography images, the retinal arteries, veins and capillaries in fundus color photographs are automatically labeled, which can achieve more accurate, faster and more complete labeling of the contours and boundaries of retinal blood vessels in fundus color photography, while distinguishing the effect of retinal arteries and veins.

[0026] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, the method for automatically marking retinal arteries, veins and capillaries in fundus color photographs provided by the present invention comprises the following steps:

[0028] S1, S1, based on matching fundus color photographs and retinal angiography images from the same eye, first extract the entire retinal blood vessels in the retinal angiography image, and automatically distinguish arteries and veins through the difference between the fundus angiography venous phase and arterial phase development, such as Figure 3 As shown;

[0029] S2, aligning the fundus color photography image with the retinal angiography image, so that the blood vessel positions in the fundus color photography correspond one to one with the blood vessel positions in the retinal angiography, using the fine blood vessel segmentation map extracted from the angiography as a label, completing the complete annotation of the contours and boundaries of all retinal blood vessels in the fundus color photography, the retinal blood vessels including arteries, veins and capillaries.

[0030] Wherein, the step S1, extracting the whole retinal blood vessels based on the retinal angiography image, specifically includes: using morphological processing methods (such as adaptive threshold, Frangi filter, Sato tubeness filter, etc.) or pre-trained deep learning models (such as U-net, W-net, etc.) to extract the binary retinal blood vessel structure from the fluorescence angiography image. When the original image registration is difficult, the above method can be used to extract blood vessels from the fundus color photograph.

[0031] In step S1, the arteries and veins are automatically distinguished by the difference between the venous phase and the arterial phase of fundus angiography, which specifically includes:

[0032] The retinal angiography images of different stages are registered with the fundus color photographs, and the arteries and veins are automatically distinguished using the difference in the fluorescence angiography venous phase and arterial phase (such as pixel intensity, color gradient, morphology). The registration method of the fluorescence angiography images of different stages adopts the same registration method as step S2.

[0033] Among them, Figure 2As shown, the step S2, registering the fundus color photography image with the retinal angiography image, specifically includes:

[0034] The green channel of the color photo of the same eye is roughly matched with the angiography image by using the rigid registration method. Specifically, the image features are obtained by feature point detection, mutual information, similarity and other methods, the features in the two images are matched, abnormal features are filtered, the homography matrix is ​​calculated, the mapping of the two images and the alignment of one image to the other are completed;

[0035] When the images are too different to match, the blood vessels in the fundus color photo can be extracted and matched with the blood vessels in the fluorescent angiography image;

[0036] When rigid registration cannot match the peripheral small blood vessels well, further non-rigid matching is performed to map, deform and register the two blood vessel images at the pixel level to fine-tune the misaligned small blood vessels.

[0037] Among them, the fundus photography examination method includes but is not limited to color fundus photography and ultra-wide-angle laser scanning ophthalmoscope.

[0038] In summary, the method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs provided by the present invention uses retinal angiography images to automatically label retinal arteries, veins and capillaries in fundus color photographs, such as Figure 4 As shown, compared with the traditional manual labeling method, this method can achieve more accurate, faster and more complete labeling of the contours and boundaries of retinal blood vessels in fundus color photography images, and distinguish retinal arteries and veins; specifically, the method includes extracting all retinal blood vessels based on retinal angiography images, registering retinal angiography images of different stages with fundus color photography images, automatically distinguishing arteries and veins using the difference between the venous phase and the arterial phase of fluorescent angiography, and after registering the fundus color photography with the retinal angiography images, completely labeling the blood vessels in the fundus color photography. Based on such a segmentation map, a more comprehensive quantitative and qualitative analysis of various retinal blood vessels (especially capillaries) in fundus color photography can be performed, and fundus color photography images with accurate blood vessel labeling can be used for the development of artificial intelligence automatic blood vessel analysis systems, etc.

[0039] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for automatically labeling retinal arteries, veins and capillaries in fundus color photographs. It is characterized in that The following steps are involved: S1, based on the matching fundus color photographs and retinal angiography images from the same eye, first extract the whole retinal blood vessels from the retinal angiography images, and automatically distinguish the arteries and veins by the difference between the fundus angiography venous phase and the arterial phase, specifically including: extracting the binary retinal vascular structure from the fluorescence angiography images using morphological processing methods or deep learning models; aligning the retinal angiography images of different stages with the fundus color photographs, and automatically distinguishing the arteries and veins by using the pixel intensity, color gradient, and morphological differences between the fluorescence angiography venous phase and the arterial phase; S2, registering the fundus color photography image with the retinal angiography image, and completing the complete annotation of the contours and boundaries of all retinal blood vessels in the fundus color photography, wherein the retinal blood vessels include arteries, veins and capillaries; The registration of the fundus color photography image and the retinal angiography image specifically includes: The green channel of the color photo of the same eye is roughly matched with the angiography image by using the rigid registration method. Specifically, the image features are obtained by feature point detection, mutual information, and similarity methods, the features in the two images are matched, abnormal features are filtered, the homography matrix is ​​calculated, and the mapping of the two images and the alignment of one image to the other are completed. When the images are too different to match, the blood vessels in the fundus color photo are extracted and matched with the blood vessels in the fluorescence angiography image; When rigid registration cannot match the peripheral small vessels well, further non-rigid matching is performed to fine-tune the misaligned small vessels.

Images