Method and apparatus for processing retinal images, and computer-readable storage medium

Abstract

The present disclosure relates to a method and apparatus for processing retinal images, a computer-readable storage medium, and relates to the technical field of image processing. The retinal images include fundus photographs and optical coherence tomography angiography (OCTA) images. The method for processing the retinal images includes: determining candidate feature points in the fundus photographs and OCTA images; selecting landmark feature points in the fundus photographs and OCTA images from the candidate feature points based on ocular anatomical information; and aligning the vascular structures in the fundus photographs and OCTA images based on the landmark feature points in the fundus photographs and OCTA images.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and particularly to a method and apparatus for processing retinal images, and a computer-readable storage medium. Background Art

[0002] Retinal images include fundus photographs, optical coherence tomography angiography (OCTA), etc. Fundus photographs can provide two-dimensional color images of the retina and are commonly used to examine the overall structure and vascular distribution of the retina. OCTA images, on the other hand, can provide three-dimensional structures and blood flow information of retinal blood vessels for evaluating retinal blood flow density and vascular morphology.

[0003] Data of retinal blood vessels is of great significance for health assessment. In related technologies, the analysis of retinal blood vessels is mainly based on a single retinal image, for example, a fundus photograph or an OCTA image. Summary of the Invention

[0004] According to a first aspect of the present disclosure, there is provided a method for processing a retinal image, wherein the retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image, and the method for processing the retinal image includes: determining candidate feature points in the fundus photograph and the OCTA image; selecting landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information; and aligning the vascular structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image.

[0005] In some embodiments, the step of selecting landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information includes: determining regions of interest (ROIs) in the fundus photograph and the OCTA image based on eye anatomical information, where the ROIs in the fundus photograph and the OCTA image include at least one of the optic disc region, the macula region, and the vascular region; and selecting landmark feature points from the candidate feature points located in the ROIs of the fundus photograph and the OCTA.

[0006] In some embodiments, the step of selecting landmark feature points from the candidate feature points located in the ROIs of the fundus photograph and the OCTA includes: selecting at least one of the candidate feature points of the optic disc center, the candidate feature points of the macula center, vascular crossing points, and vascular branching points from the candidate feature points located in the ROIs of the fundus photograph and the OCTA as landmark feature points.

[0007] In some embodiments, the method for processing a retinal image further includes at least one of the following: eliminating candidate feature points caused by illumination; eliminating candidate feature points caused by noise; eliminating non-significant candidate feature points on the blood vessel curve; eliminating candidate feature points in the background area.

[0008] In some embodiments, aligning the blood vessel structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image includes: matching the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matched marker feature points; based on the multiple pairs of matched marker feature points, mapping the fundus photograph and the OCTA image to the same coordinate system to align the blood vessel structures in the fundus photograph and the OCTA image.

[0009] In some embodiments, the method for processing a retinal image further includes: fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image.

[0010] In some embodiments, the step of fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: in the same coordinate system, performing weighted summation on the pixel values corresponding to the same position in the fundus photograph and the OCTA image to obtain the pixel value at that position in the fused image.

[0011] In some embodiments, the step of fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: normalizing the gray scale values of the pixels of the OCTA image; based on a pseudo-color mapping table, converting the normalized gray scale values into red, green, and blue primary color values to obtain a pseudo-color OCTA image; fusing the pseudo-color OCTA image and the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of blood vessels in the fused image represent different blood flow densities.

[0012] In some embodiments, the method for processing a retinal image further includes: converting the fundus photograph into a color space with brightness; segmenting the arteries and veins therein according to the brightness of the converted fundus photograph; determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph.

[0013] In some embodiments, the step of segmenting the arteries and veins therein according to the brightness of the converted fundus photograph includes: preliminarily segmenting the arteries and veins therein according to the brightness of the converted fundus photograph; based on a first machine learning model, segmenting the arteries and veins in the fundus photograph according to the preliminary segmentation.

[0014] In some embodiments, determining the arteries and veins in the fused image based on the arteries and veins in the fundus photograph includes: determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph and the colors of the blood vessels in the fused image.

[0015] In some embodiments, the method for processing a retinal image further includes: calculating the fractal dimensions of the arteries and veins respectively according to the fused image.

[0016] In some embodiments, the method for processing a retinal image further includes: determining the blood flow density of the arteries and veins according to the colors of the arteries and veins in the fused image.

[0017] In some embodiments, aligning the blood vessel structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image includes: selecting the landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on a second machine learning model.

[0018] In some embodiments, determining the candidate feature points in the fundus photograph and the OCTA image includes: determining the candidate feature points in the fundus photograph and the OCTA image based on a feature point detection algorithm.

[0019] According to a second aspect of the present disclosure, there is provided a processing apparatus for a retinal image, wherein the retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image, and the processing apparatus for the retinal image includes: a determination module configured to determine candidate feature points in the fundus photograph and the OCTA image; a selection module configured to select landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information; and an alignment module configured to align the blood vessel structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image.

[0020] According to a third aspect of the present disclosure, there is provided a processing apparatus for a retinal image, including: a memory; and a processor coupled to the memory, the processor being configured to execute the method for processing a retinal image according to any of the embodiments of the present disclosure based on instructions stored in the memory.

[0021] According to a fourth aspect of the present disclosure, there is provided a processing system for a retinal image, including: the processing apparatus for a retinal image according to any of the embodiments of the present disclosure; a fundus camera; and an optical coherence tomography angiography system.

[0022] According to a fifth aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, which when executed by a processor, implement the method for processing a retinal image according to any of the embodiments of the present disclosure.

[0023] According to a sixth aspect of the present disclosure, a computer program product includes computer program instructions, which when executed by a processor, implement the method for processing a retinal image according to any of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings forming a part of the specification illustrate embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.

[0025] Referring to the drawings, the present disclosure can be more clearly understood from the following detailed description, wherein:

[0026] Figure 1 A flowchart showing a method for processing a retinal image according to some embodiments of the present disclosure;

[0027] Figure 2 Showing original pictures of fundus photographs and OCTA images;

[0028] Figure 3 Showing redundant feature points identified according to some embodiments of the present disclosure;

[0029] Figure 4 Showing marker feature points according to some embodiments of the present disclosure;

[0030] Figure 5 Showing a schematic diagram of OCTA different layer contrasts according to some embodiments of the present disclosure;

[0031] Figure 6 Showing a schematic diagram of image fusion according to some embodiments of the present disclosure;

[0032] Figure 7 Showing a schematic diagram of extracting features from a fused image according to some embodiments of the present disclosure;

[0033] Figure 8 Showing a block diagram of a device for processing a retinal image according to some embodiments of the present disclosure;

[0034] Figure 9 Showing a block diagram of a device for processing a retinal image according to other embodiments of the present disclosure;

[0035] Figure 10 Showing a block diagram of a computer system for implementing some embodiments of the present disclosure;

[0036] Figure 11A block diagram of a processing system for a retinal image according to some embodiments of the present disclosure is shown. Detailed Description

[0037] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0038] At the same time, it should be understood that, for the sake of convenience of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationships.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way a limitation on the present disclosure or its application or use.

[0040] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered as part of the specification.

[0041] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that: like reference numerals and letters denote like items in the following drawings, and thus, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0043] Abbreviations related to the present disclosure are explained as follows.

[0044] Hue, Saturation, Value color space (HSV)

[0045] Luminance, a-axis of green - red, b-axis of blue - yellow color space (LAB), where L represents the luminance of the color, A represents the change from green to red, and B represents the change from blue to yellow

[0046] Convolutional Neural Network (CNN)

[0047] Scale-Invariant Feature Transform (SIFT)

[0048] Features from Accelerated Segment Test FAST

[0049] Foveal Avascular Zone (FAZ)

[0050] Radial Peripapillary Capillary Plexus (RPCP)

[0051] Superficial Vascular Complex (SVC)

[0052] Intermediate Capillary Plexus (ICP)

[0053] Deep Capillary Plexus (DCP)

[0054] Tortuosity (T)

[0055] Branching Angle (θ)

[0056] Flow Density (F)

[0057] Fractal Dimension (FD), a measure used to describe the complexity and irregularity of geometric shapes and used to analyze the structural complexity of retinal blood vessels

[0058] Radial Peripapillary Capillary Plexus (RPCP), a capillary plexus located around the optic disc, responsible for supplying blood around the optic nerve. The blood flow density of RPCP is often used to evaluate the health of the optic nerve.

[0059] Superficial Vascular Complex (SVC), a vascular network located in the inner layer of the retina, mainly including the nerve fiber layer and the ganglion cell layer, responsible for supplying blood to the superficial retina

[0060] Intermediate Capillary Plexus (ICP), a capillary network located in the middle layer of the retina, mainly including the inner nuclear layer and the inner plexiform layer, responsible for supplying blood to the middle layer of the retina

[0061] Deep Capillary Plexus (DCP), a capillary network located in the deep layer of the retina, mainly including the outer plexiform layer and the outer nuclear layer, responsible for supplying blood to the deep layer of the retina.

[0062] In the related art, the analysis of retinal blood vessels is mainly based on a single retinal image. However, fundus photographs can only provide information on the surface structure of the retina and cannot reflect the details of different layers of the retina and blood flow dynamics, which limits their application in comprehensively evaluating the health status of the retina.

[0063] At the same time, simply relying on the color and morphological segmentation of fundus photographs may perform poorly under poor lighting conditions or in the presence of noise. Especially in the early stage or mild stage of retinal diseases, the arteriovenous segmentation may be inaccurate.

[0064] OCTA images are generated based on the change of coherent signals of red blood cell movement. This method cannot directly measure the oxygen content of blood or other biometric features that can clearly distinguish arteries and veins. Whether the blood vessels in OCTA images are arteries or veins, they all appear as the same high-signal region. In short, due to the imaging principle of OCTA based on red blood cell movement, it cannot distinguish arteries and veins, and the image quality is easily affected by artifacts, which also limits its application in comprehensive health assessment.

[0065] Figure 1 A flowchart showing a method for processing retinal images according to some embodiments of the present disclosure is shown.

[0066] As Figure 1 shown, the method for processing retinal images includes steps S1 - S3, and the retinal images include fundus photographs and optical coherence tomography angiography (OCTA) images. In some embodiments, the method for processing retinal images is executed by a processing device for retinal images.

[0067] In step S1, candidate feature points in the fundus photograph and the OCTA image are determined. In step S2, based on the ocular anatomical information, marker feature points in the fundus photograph and the OCTA image are selected from the candidate feature points. In step S3, based on the marker feature points in the fundus photograph and the OCTA image, the blood vessel structures in the fundus photograph and the OCTA image are aligned.

[0068] In steps S1 - S2, the fundus photograph and the OCTA image can be processed separately. For example, first, the feature points in the fundus photograph and the feature points in the OCTA image are respectively identified. Then, marker feature points are further selected from the fundus photograph and irrelevant points are removed. Among them, the marker feature points are, for example, some feature points with anatomical significance. Similar processing is also performed on the OCTA image.

[0069] In step S3, the blood vessel structures in the fundus photograph and the OCTA image are aligned, and the fundus photograph and the OCTA image are superimposed.

[0070] According to embodiments of the present disclosure, alignment of the vascular structures in fundus photographs and OCTA images is achieved. By aligning the two types of images, the advantages of each modality can be fully utilized to provide more comprehensive structural and blood flow information of the retina.

[0071] Figure 2 Original pictures showing fundus photographs and OCTA images are presented.

[0072] As Figure 2 shown, it is difficult to directly align a fundus photograph with an OCTA image. For this reason, some embodiments of the present disclosure adopt a feature point registration method to align the vascular structures in the fundus photograph and the OCTA image.

[0073] In some embodiments, candidate feature points in the fundus photograph and the OCTA image are determined, including: based on a feature point detection algorithm, determining candidate feature points in the fundus photograph and the OCTA image.

[0074] For example, using the SIFT or FAST algorithm, candidate feature points in the fundus photograph and the OCTA image are respectively extracted.

[0075] In some embodiments, based on ocular anatomical information, marker feature points in the fundus photograph and the OCTA image are selected from the candidate feature points, including: based on ocular anatomical information, determining regions of interest (ROIs) in the fundus photograph and the OCTA image, where the ROIs in the fundus photograph and the OCTA image include at least one of the optic disc region, the macula region, and the vascular region; selecting marker feature points from the candidate feature points located in the ROIs of the fundus photograph and the OCTA.

[0076] The SIFT and FAST algorithms can detect points with significant features in an image (such as local extrema, corner points, etc.). In anatomy, markers such as the optic disc usually have obvious image features (such as brightness, contrast, gradient changes), which can be detected by these algorithms.

[0077] Based on the anatomically based region of interest (ROI), feature points not in important regions (such as near the optic disc, the center of the macula, and vascular crossing points) are initially filtered out, thereby reducing redundant points in the background region and irrelevant regions. For example, the radius ranges of the optic disc and the macula can be defined, and only feature points near these regions are extracted.

[0078] In some embodiments, selecting marker feature points from the candidate feature points located in the ROIs of the fundus photograph and the OCTA includes: selecting at least one of the candidate feature points of the optic disc center, the candidate feature points of the macula center, vascular crossing points, and vascular branch points from the candidate feature points located in the ROIs of the fundus photograph and the OCTA as marker feature points.

[0079] The brightness contrast between the center of the optic disc and the optic disc is related to its circular structure. The center of the optic disc in fundus photographs is usually a prominent bright spot area.

[0080] The foveal center is located in the Foveal Avascular Zone (FAZ) and is usually a dark area in the retinal image, with no obvious vascular structure around it.

[0081] The vascular crossing point is where two or more blood vessels cross, and the brightness and morphology of the blood vessels change at this point, which is commonly found in the crossing area of the retinal vascular network.

[0082] The arteriovenous bifurcation point is, for example, the starting point of the vascular branch. At this point, the blood vessel divides into two or more smaller branches.

[0083] The key nodes on the blood vessel curve are, for example, the points where the blood vessel bends or turns sharply, which is the area where the geometric shape of the blood vessel changes significantly.

[0084] Further screening can be carried out according to anatomical information to make the distribution of feature points conform to the structural characteristics of the optic disc, macula, and vascular crossing points. For example, the feature points of the center of the optic disc should be distributed around the circular area; the center of the macula should be near the avascular area; the vascular crossing points should be at the intersection.

[0085] For example, the optic disc usually has an obvious gray-scale contrast in fundus images, especially at the edge of the optic disc. This high contrast is manifested as a significant gradient change. SIFT locates the feature points near the edge of the optic disc by detecting the local extrema of these changes.

[0086] There are local extreme points of brightness near the center of the optic disc, especially when the contrast with the surrounding background is large, which enables SIFT to detect the feature points of the optic disc. FAST identifies the position of the optic disc by detecting the corner points at the edge of the optic disc.

[0087] The foveal center (especially the fovea centralis) is an avascular area in the fundus image and usually appears as a smooth and avascular area. Although there are no blood vessels in the fovea centralis itself, there are significant gradient or contrast changes in the surrounding area.

[0088] SIFT can detect feature points through the brightness gradient difference between the area around the fovea centralis and the external vascular area. Although the fovea centralis itself is smooth, the significant gradient changes around it enable SIFT to find feature points in the macular edge area.

[0089] FAST identifies the position of the macula by detecting the corner points at the edge of the macular area. Although there are no significant corner points in the fovea centralis itself, the edge of the fovea centralis forms corner points with the surrounding vascular area, and FAST can detect these corner points.

[0090] Vascular intersections are anatomical structures with significant features. Since there are obvious brightness and gradient changes at the intersections of blood vessels, these areas can be detected by SIFT and FAST algorithms.

[0091] SIFT locates feature points by detecting gradient changes in images at different scales. The gradient changes at vascular intersections are very obvious, which enables SIFT to detect the features of these areas.

[0092] FAST identifies vascular intersections by detecting corner points in images. Since vascular intersections usually form multiple edge intersections, generating multiple corner points, FAST can effectively detect these features.

[0093] The center of the optic disc, the center of the macula, and vascular intersections all have their own significant local image features (such as brightness contrast, gradient changes, corner points, etc.). These features match the detection mechanisms of SIFT and FAST, enabling these algorithms to identify these markers. Based on principles such as image gray level, gradient, and contrast changes, SIFT and FAST can detect key anatomical structures of the retina.

[0094] That is to say, the algorithm can detect feature points such as the optic disc, the center of the macula, and vascular intersections. However, at the same time, it will also detect redundant feature points that are anatomically irrelevant (for example, extreme points caused by noise). Therefore, it is necessary to eliminate redundant feature points.

[0095] In some embodiments, the method for processing retinal images further includes at least one of the following: eliminating candidate feature points caused by illumination; eliminating candidate feature points caused by noise; eliminating non-significant candidate feature points on the blood vessel curve; eliminating candidate feature points in the background area.

[0096] Figure 3 Shows redundant feature points identified according to some embodiments of the present disclosure.

[0097] Based on morphological principles, redundant candidate feature points can be eliminated. As Figure 3 shown, the redundant feature points in the image are, for example, points without anatomical significance. The following gives specific examples.

[0098] Local extreme points caused by image noise: Irrelevant extreme points caused by noise in the imaging device, uneven illumination, etc.

[0099] False brightness gradients caused by illumination changes: Locally enhanced contrast areas caused by light refraction or device reflection.

[0100] Meaningless texture points in the background area: Points in the texture or spots in the retina background that are detected by the algorithm due to local contrast changes.

[0101] Non-key points of blood vessel curves: At some small bends of blood vessels, although there are corner points or gradient changes, they are not related to important intersections or bifurcations.

[0102] Pseudo feature points (light spots or reflections): False feature points caused by light spots or reflections during imaging, which have nothing to do with the actual anatomical structure.

[0103] Non-significant (i.e., non-key) points on the blood vessel curve are, for example, corner points at slight bends. Candidate feature points in the background area are, for example, noise points in the background area.

[0104] In some embodiments, based on the marker feature points in the fundus photograph and the OCTA image, align the blood vessel structures in the fundus photograph and the OCTA image, including: based on a second machine learning model, select the marker feature points in the fundus photograph and the OCTA image from the candidate feature points.

[0105] For example, the marker feature points can also be determined with the assistance of a machine learning model. For example, annotate the data set, and train machine learning models (such as convolutional neural networks) for the fundus photograph and the OCTA image respectively, so that the models can distinguish marker feature points from redundant feature points.

[0106] After selecting the marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on the eye anatomical information, further screen the marker feature points through a machine learning model, so as to more accurately select the marker feature points.

[0107] Figure 4 Show marker feature points according to some embodiments of the present disclosure.

[0108] As Figure 4 shown, the set of marker feature points selected through the foregoing steps has high anatomical significance, and only retains the feature points related to markers such as the center of the optic disc, the center of the macula, and the blood vessel intersection points.

[0109] In some embodiments, based on the marker feature points in the fundus photograph and the OCTA image, align the blood vessel structures in the fundus photograph and the OCTA image, including: match the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matching marker feature points; based on the multiple pairs of matching marker feature points, map the fundus photograph and the OCTA image to the same coordinate system to align the blood vessel structures in the fundus photograph and the OCTA image.

[0110] For example, the feature points in the fundus photograph can be matched with the feature points in the OCTA image by methods such as the Euclidean distance. Based on the matching feature point pairs, calculate the affine transformation matrix, which is used to transform the OCTA image into the coordinate system of the fundus photograph.

[0111] The OCTA image is transformed using a transformation matrix so that the OCTA image is aligned with the fundus photograph in the geometric space. After transformation, the vascular structures in the OCTA image coincide with those in the fundus photograph at the same position.

[0112] The following is an example to illustrate the alignment method of the fundus photograph and the OCTA image.

[0113] Ten marker feature points are detected around the optic disc in the fundus photograph, and ten marker feature points at the position of the optic disc are detected in the OCTA image. By using the Euclidean distance or other matching algorithms, the marker feature points in the fundus photograph are matched with those in the OCTA image to obtain eight pairs of matching marker feature points, which are used to calculate the transformation matrix.

[0114] Using the pair of matching feature points, the following affine transformation matrix is calculated.

[0115]

[0116] The transformation matrix indicates that the OCTA image needs to be rotated, scaled, and / or translated to align with the fundus photograph.

[0117] In some embodiments, the method for processing the retinal image further includes: fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image. That is, the spatially aligned images are fused so that the fused image includes both the information of the fundus photograph and the information of the OCTA image.

[0118] In some embodiments, fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: in the same coordinate system, the pixel values corresponding to the same position in the fundus photograph and the OCTA image are weighted and summed to obtain the pixel value at that position in the fused image.

[0119] Weighted fusion takes into account the importance of information in different images and realizes image fusion by performing weighted processing on the two images. The calculation formula is as follows.

[0120] I fused (x,y) = w retina ·I retina (x,y) + w OCTA ·I OCTA (x,y)

[0121] w retina + W OCTA = 1 (2)

[0122] Wherein, I fused (x,y) represents the pixel value of the fused image. Iretina (x, y) represents the pixel values of the fundus photograph. I OCTA (x, y) represents the pixel values of the OCTA image. w retina and w OCTA represent the weights of the fundus photograph and the OCTA image. These weights can be adjusted according to the image quality and analysis requirements. For example, if you want to highlight the structural information of the retinal blood vessels, you can set w retina > w OCTA .

[0123] It is also possible to adopt the method of transparency adjustment (Alpha Blending) and use linear interpolation to fuse the two images. The calculation formula is as follows.

[0124] I fused (x, y) = α · I retina (x, y) + (1 - α) · I OCTA (x, y) (3)

[0125] Among them, I fused (x, y) represents the pixel values of the fused image. I retina (x, y) represents the pixel values of the fundus photograph. I OCTA (x, y) represents the pixel values of the OCTA image. α is the transparency coefficient (the value range is between 0 and 1). For example, when α = 0.6 is set, the structural information of the fundus photograph is more prominent, and the information of the OCTA image is displayed as an auxiliary level.

[0126] In some embodiments, the fundus photograph and the OCTA image in the same coordinate system are fused to obtain a fused image, including: normalizing the gray values of the pixels of the OCTA image; based on the pseudo-color mapping table, converting the normalized gray values into red, green, and blue primary color values to obtain a pseudo-color OCTA image; fusing the pseudo-color OCTA image and the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of blood vessels in the fused image represent different blood flow densities.

[0127] For example, pseudo-color mapping can be performed on the OCTA image to map the pixel values in the original gray image to a specific color range, so as to more intuitively display the blood flow density information in the OCTA image. The method of pseudo-color mapping is introduced in detail below.

[0128] First, normalize the gray values of the OCTA image to the range of [0, 1]. The calculation formula is as follows.

[0129]

[0130] Then, color mapping processing is performed. Using common pseudo-color mapping tables (such as Jet, HSV, etc.), the normalized grayscale values are converted into RGB colors. For example:

[0131] I norm (x, y) = 0 is mapped to blue

[0132] I norm (x, y) = 1 is mapped to red (5)

[0133] The pseudo-color OCTA image after mapping is superimposed on the fundus photograph.

[0134] In the fused image, the fundus photograph retains the arteriovenous structure information, and the pseudo-color image of OCTA shows the blood flow density.

[0135] For example, taking the fundus photograph as the base layer, clear arterial and venous structures are shown. After pseudo-color mapping, the OCTA image is shown as red, yellow, and green areas, representing different levels of blood flow density.

[0136] It is also possible to layer the OCTA image and extract the blood flow density data of each layer. For example, using thresholding or histogram equalization to process the blood flow density image to make the difference in blood flow density more obvious. The method for extracting blood flow density information from the OCTA image is introduced below.

[0137] # Read the grayscale image

[0138] octa image = cv2.imread('octa image .png', 0)

[0139] Figure 5 Shows a schematic diagram of different layer contrasts of OCTA according to some embodiments of the present disclosure.

[0140] As Figure 5 shown, through layering processing, the blood flow density of different layers in the OCTA image can be separated. After separation, the data of each layer can be processed separately.

[0141] The following introduces the specific implementation of pseudo-color mapping.

[0142] First, use the pseudo-color mapping function of OpenCV to map different blood flow density ranges to different colors. According to the blood flow density value, the higher density area is mapped to a bright color (such as yellow), and the lower density area is mapped to a dark color (such as blue).

[0143] # Apply pseudo-color mapping

[0144]

[0145] After pseudo-color mapping, a fused image is generated. For example, the arteriovenous annotation result is overlaid on the pseudo-color mapped OCTA blood flow density image to generate a comprehensive analysis image. Arteries and veins are labeled with different colors respectively, while the blood flow density information is displayed in the background through pseudo-color mapping. The specific implementation of image fusion is introduced below.

[0146] #Overlay the arteriovenous segmentation result on the pseudo-color OCTA image

[0147]

[0148] final image = cv2.addWeighted(final image , 0.5, vein segment , 0.5, 0)

[0149] Through the above steps, arteries and veins can be distinguished in the registered fused image, and the blood flow density information of the OCTA image can be visualized. The arteriovenous annotation information combined with the blood flow density information of the OCTA image can provide a more comprehensive multimodal analysis result for the health assessment of the retina.

[0150] In the fused image, the vascular structure in the fundus photo layer is clearly visible, while the blood flow density information of the OCTA image is overlaid in pseudo-color at the corresponding position. By adjusting the transparency or weighted fusion, while highlighting the structural information, the display effect of the blood flow information can be retained. Through transparency adjustment, weighted fusion and pseudo-color mapping, the overlay and fusion of the fundus photo and the OCTA image can be realized, so as to effectively combine the retinal structural information and the blood flow density information and provide a more comprehensive analysis for the retinal health assessment.

[0151] In some embodiments, the method for processing retinal images further includes: converting the fundus photo to a color space with brightness; segmenting the arteries and veins therein according to the brightness of the converted fundus photo; and determining the arteries and veins in the fused image according to the arteries and veins in the fundus photo.

[0152] The color space with brightness is, for example, the LAB color space, the HSV color space, etc. The method for distinguishing arteries and veins in the fundus photo based on color thresholds is introduced below.

[0153] In the fundus photo, arteries are usually brighter in color (reddish), and veins are darker in color (purplish or dark red). This color difference can be used as an important basis for distinguishing arteriovenous.

[0154] First, use color space conversion to convert the RGB image to the HSV or LAB color space to more accurately separate colors.

[0155] Threshold segmentation is performed according to the hue (H) and value (V) in the HSV color space or the luminance (L) in the LAB color space: Arteries have a higher luminance, and a higher threshold can be set to segment out the arteries; veins have a lower luminance, and a lower threshold can be set to segment out the veins.

[0156] The specific implementation is as follows.

[0157] Convert the image from RGB to HSV (python):

[0158] # Read the image

[0159] import cv2

[0160] image = cv2.imread('retinal image .png')

[0161] # Convert the image from BGR to HSV color space

[0162] hsv image = cv2.cvtColor(image, cv2.COLOR BGR2HSV )

[0163] # Set the color threshold for arteries (higher luminance)

[0164] # H (hue) range: 0 - 10

[0165] # S (saturation) range: 50 - 255

[0166] # V (value) range: 150 - 255

[0167] lower_artery = (0, 50, 150)

[0168] upper_artery = (10, 255, 255)

[0169] # Create a mask for arteries according to the set threshold

[0170] artery_mask = cv2.inRange(hsv_image, lower_artery,

[0171] upper_artery)

[0172] # Set the color threshold for veins (lower luminance)

[0173] # H (hue) range: 0 - 10

[0174] # S (saturation) range: 50 - 255

[0175] # V (Brightness) range: 50 - 150

[0176] lower_vein = (0, 50, 50)

[0177] upper_vein = (10, 255, 150)

[0178] # Create a mask of the veins according to the set threshold

[0179] vein_mask = cv2.inRange(hsv_image, lower_vein, upper_vein)

[0180] # Apply the artery mask to the original image to extract the artery part

[0181] artery_segment = cv2.bitwise_and(image, image,

[0182] mask = artery_mask)

[0183] # Apply the vein mask to the original image to extract the vein part

[0184] vein_segment = cv2.bitwise_and(image, image,

[0185] mask = vein_mask)

[0186] In some embodiments, the arteries and veins in the converted fundus photograph are segmented according to the brightness of the converted fundus photograph, including: preliminarily segmenting the arteries and veins in the converted fundus photograph according to the brightness; and segmenting the arteries and veins in the fundus photograph based on a first machine learning model according to the preliminary segmentation.

[0187] The color threshold method classifies based on the color difference between the retinal arteries and veins (arteries are brighter in color and veins are darker). This method is simple and fast, and has good results in scenarios with obvious and highly consistent image features, being simple and having low computational overhead. However, in images with high variability or a large amount of noise, the color threshold method may not be able to accurately distinguish arteries and veins.

[0188] For example, for areas in the image with a large brightness difference, segmenting arteries and veins based on brightness has a higher confidence level, so preliminary segmentation is first performed based on brightness. However, in some areas of the image, the brightness information is insufficient to distinguish arteries and veins. At this time, using machine learning-assisted segmentation can make the segmentation result more accurate.

[0189] The method of machine learning-assisted segmentation is introduced below.

[0190] Use publicly available retinal image datasets (such as DRIVE or CHASE-DB1) for model training. The datasets contain the annotation results of arteries and veins. Perform model pre-training through the annotated retinal datasets to enable the model to learn the characteristics of arteries and veins, such as color, texture, and morphology.

[0191] Use the pre-trained convolutional neural network model for artery and vein segmentation. For example, input fundus photos into the trained model to generate the segmentation results of arteries and veins.

[0192] Deep learning models can be trained with large-scale data to capture complex features in images (such as morphology, texture, etc.), not relying solely on color information. Deep learning models can have higher accuracy under complex backgrounds and different imaging conditions, be able to handle complex image features, and adapt to different changes in images.

[0193] During the artery and vein annotation process, color thresholds and deep learning models can be used separately. For example, if the image conditions in the artery and vein annotation scenario are consistent and mainly rely on color information, the color threshold method can be selected; if the images are complex or higher accuracy is required, deep learning models can be used.

[0194] The two can also be used in combination, which will be introduced in detail below.

[0195] First, use the color threshold method to perform a preliminary classification of retinal arteries and veins. Arteries are usually brighter in color, and veins are darker. By setting different color thresholds, a preliminary separation of arteries and veins is carried out. Color thresholds can quickly perform a rough annotation, but errors may occur in some details or complex areas (such as intersections or areas with similar colors).

[0196] Then, input the preliminary annotation results into the deep learning model for further fine annotation and correction. The deep learning model can identify areas that cannot be accurately classified by the color threshold method, especially for correction and optimization at artery-vein intersections or areas with unclear images, improving the overall annotation accuracy. Therefore, first quickly generate preliminary results through color thresholds to reduce the computational overhead of the deep learning model, making the entire process more efficient. At the same time, the subsequent deep learning model can handle areas that cannot be accurately recognized by the color threshold method, improving the accuracy.

[0197] In some embodiments, determine the arteries and veins in the fused image according to the arteries and veins in the fundus photo, including: determine the arteries and veins in the fused image according to the arteries and veins in the fundus photo and the colors of the blood vessels in the fused image.

[0198] In the fused image, the high-resolution anatomical structure of the fundus photograph can be used to help identify the origin, course, and morphology of blood vessels. The morphological features of arteries and veins are clearly visible in the fundus photograph, which provides strong support for the basic segmentation of arteries and veins.

[0199] After fusing the fundus photograph and the OCTA image, the effect of artery-vein segmentation will be significantly improved, especially after integrating the structural information and blood flow information. The blood flow density information in the OCTA image can be used to verify whether the segmented arteries and veins in the fundus photograph conform to the actual blood flow situation. For example, in a healthy retina, the blood flow densities of arteries and veins are different. In the fused image, by combining the information of the OCTA image layer, the artery-vein segmentation in the fundus photograph image layer can be finely adjusted and optimized to improve the accuracy of artery-vein segmentation.

[0200] After fusing the artery and vein segmentation information in the fundus photograph with the different-level blood flow information provided by OCTA, the blood vessels can be further subdivided from the perspective of blood flow. The fused image allows the combination of the morphological information of arteries and veins (extracted from the fundus photograph) and the blood flow function information (extracted from the OCTA image). For example, areas with higher blood flow density are more likely to be arteries, while areas with lower blood flow density may be veins. This can accurately distinguish whether a blood vessel is an artery or a vein, rather than relying solely on color or morphological features. The fusion of this information can optimize and correct the artery-vein segmentation that relies only on morphological analysis. For example, at the artery-vein crossing point, the fused image can further confirm the blood vessel type through blood flow density, reducing errors.

[0201] Therefore, by aligning the OCTA image and the fundus photograph, the characteristics of arteries and veins can be identified and extracted more precisely.

[0202] In some embodiments, the method for processing retinal images further includes: calculating the fractal dimension of arteries and veins respectively according to the fused image.

[0203] For example, calculate the fractal dimension (Fractal Dimension) separately from the distinguished arteries and veins. The fractal dimension reflects the complexity and branching structure of blood vessels. The box-counting method or other fractal geometry methods can be used to calculate the fractal dimension.

[0204] In some embodiments, the method for processing retinal images further includes: determining the blood flow density of arteries and veins according to the colors of arteries and veins in the fused image. Based on the distinguishing information of arteries and veins and the color information of blood vessels in the fused image, calculating the fractal dimension, blood flow density, etc. can be more accurate.

[0205] For example, different levels of blood flow density (such as SVC, ICP, DCP) can be extracted from the fused image. For each vascular plexus, the blood flow density is calculated separately, that is, the amount of flowing blood in the blood vessels per unit area.

[0206] Figure 6 FIG. shows a schematic diagram of image fusion according to some embodiments of the present disclosure.

[0207] As Figure 6 shown, the arteriovenous vessels are labeled in the fundus photograph, the blood flow density levels are distinguished in the OCTA image, and the processed fundus photograph and OCTA image are registered based on the landmark feature points, thereby realizing image fusion.

[0208] Information such as fractal dimension and blood flow density can be extracted from the fused image through image processing techniques.

[0209] Figure 7 FIG. shows a schematic diagram of extracting features from a fused image according to some embodiments of the present disclosure.

[0210] As Figure 7 shown, features can be extracted through basic mathematical operations, or through big data, neural networks, etc.

[0211] The fused image can also provide more information for disease prediction, which will be introduced in detail below.

[0212] 1. Structure - function consistency check: The fused image can provide a structure - function consistency check after arteriovenous segmentation. For example, if the structural morphology of a certain section of artery is normal, but the blood flow density decreases, it may indicate the presence of potential functional disorders (such as vascular stenosis or insufficient blood supply) in this area.

[0213] 2. Local complexity analysis: The combination of the morphological characteristics of arteriovenous vessels and blood flow density can assist in more complex local analysis. For example, the combination of the fractal dimension of an artery (reflecting the complexity of blood vessels) and its blood flow density can reveal the health status of a certain blood vessel. Separate fractal dimension or blood flow density may not be able to reflect this complexity, but the combination can obtain a more accurate assessment.

[0214] By fusing arteriovenous segmentation with OCTA blood flow information, some new features that cannot be obtained from individual fundus photographs or OCTA can be extracted. For example:

[0215] (1) Arteriovenous blood flow function index: By combining the fractal dimension of arteriovenous vessels with their corresponding blood flow density, the blood flow function index of each arteriovenous vessel can be calculated. This index can comprehensively evaluate the morphology and function of arteries / veins, and further evaluate the health status of retinal blood vessels;

[0216] (2) Detection of arteriovenous structure - function inconsistency: The fused image allows simultaneous assessment of the anatomical structure and function of blood vessels. Through this assessment, it can be found that the arteries or veins in some areas are morphologically normal but may be functionally abnormal. For example, the artery has a normal morphology but a reduced blood flow, which may indicate early lesions. In contrast, such abnormalities cannot be detected from fundus photographs alone, and separate OCTA images cannot distinguish which blood vessels (arteries or veins) have functional abnormalities.

[0217] (3) Compared with the fact that early blood flow function abnormalities may not be recognized by separate fundus photographs alone, and the blood flow states of arteries and veins may not be accurately separated by separate OCTA images, the fused image can provide more accurate assessment information in early lesions by combining arteriovenous structure and blood flow density information.

[0218] (4) Precise assessment of arteriovenous crossing points: Arteriovenous crossing points are often high - incidence sites of lesions. Through the fused image, the structural and functional changes in these areas can be better evaluated. By combining the morphological characteristics of arteries and veins with blood flow function information, it is possible to more accurately assess whether there are problems such as blood flow restriction or abnormal dilation of arteries / veins at the crossing points.

[0219] In summary, from the perspective of arteriovenous segmentation, after fusing fundus photographs and OCTA images, the morphological and blood flow characteristics of arteries and veins can be more precisely extracted. This combination makes arteriovenous segmentation more accurate and can detect some function - structure inconsistency problems that cannot be detected in individual images. This provides more comprehensive support for the detection of early lesions, the assessment of complex lesion areas (such as arteriovenous crossing points), and the assessment of the overall retinal health status.

[0220] In summary, compared with using OCTA images or fundus photographs alone, after fusing fundus photographs and OCTA images, the morphological and blood flow characteristics of arteries and veins can be more precisely extracted, and arteriovenous segmentation can be more accurate. Moreover, from the fused image, some function - structure inconsistency problems that cannot be detected in individual images can be found, providing more comprehensive support for the assessment of retinal health status.

[0221] The following introduces a method for processing retinal images according to some embodiments of the present disclosure.

[0222] First, pre - process and register the fundus photograph and the OCTA image to obtain a fused image.

[0223] Then, use a deep - learning model to perform blood vessel segmentation on the fused image to extract arteries and veins respectively. The segmentation results of arteries and veins are denoted as Va and Vv respectively.

[0224] According to the retinal anatomical structure, the fused image is divided into the optic disc area and the macular area, and features are extracted respectively.

[0225] According to the hierarchical information of the fused image, the vascular segmentation results are classified into different vascular plexuses. For example, the radial peripapillary capillary plexus, the superficial vascular plexus, the intermediate capillary plexus, and the deep capillary plexus. For each vascular plexus C, the arterial and venous features are denoted as X a,C and X v,C .

[0226] Features such as vascular tortuosity, branching angle, and blood flow density are extracted from arteries and veins, the optic disc and macular areas, and different vascular plexuses.

[0227] Figure 8 The block diagram of a processing device for retinal images according to some embodiments of the present disclosure is shown.

[0228] As Figure 8 shown, the processing device 8 for retinal images includes a determination module 81, a selection module 82, and an alignment module 83. The retinal images include fundus photographs and optical coherence tomography angiography (OCTA) images.

[0229] The determination module 81 is configured to determine candidate feature points in the fundus photograph and the OCTA image, for example, perform step S1 as Figure 1 shown.

[0230] The selection module 82 is configured to select landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information, for example, perform step S2 as Figure 1 shown.

[0231] The alignment module 83 is configured to align the vascular structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image, for example, perform step S3 as Figure 1 shown.

[0232] In some embodiments, the selection module 82 is further configured to: determine regions of interest (ROIs) in the fundus photograph and the OCTA image based on eye anatomical information, where the ROIs in the fundus photograph and the OCTA image include at least one of the optic disc area, the macular area, and the vascular area; select landmark feature points from the candidate feature points in the ROIs of the fundus photograph and the OCTA.

[0233] In some embodiments, the selection module 82 is further configured to select at least one of the candidate feature points of the optic disc center, the candidate feature points of the fovea center, the vascular crossing points, and the vascular bifurcation points from the candidate feature points located in the ROI of the fundus photograph and the OCTA as the marker feature points.

[0234] In some embodiments, the processing device for retinal images further includes an elimination module, configured to: eliminate the candidate feature points caused by illumination; eliminate the candidate feature points caused by noise; eliminate the non-significant candidate feature points on the vascular curve; eliminate the candidate feature points in the background area.

[0235] In some embodiments, the alignment module 83 is further configured to: match the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matched marker feature points; based on the multiple pairs of matched marker feature points, map the fundus photograph and the OCTA image to the same coordinate system to align the vascular structures in the fundus photograph and the OCTA image.

[0236] In some embodiments, the processing device for retinal images further includes: a fusion module, configured to fuse the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image.

[0237] In some embodiments, the fusion module is further configured to: perform weighted summation on the pixel values corresponding to the same position in the fundus photograph and the OCTA image in the same coordinate system to obtain the pixel value at this position in the fused image.

[0238] In some embodiments, the fusion module is further configured to: perform normalization processing on the gray scale values of the pixels of the OCTA image; based on the pseudo-color mapping table, convert the normalized gray scale values into red, green, and blue primary color values to obtain a pseudo-color OCTA image; fuse the pseudo-color OCTA image and the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of the blood vessels in the fused image represent different blood flow densities.

[0239] In some embodiments, the processing device for retinal images further includes: a segmentation module, configured to convert the fundus photograph into a color space with brightness; segment the arteries and veins therein according to the brightness of the converted fundus photograph; determine the arteries and veins in the fused image according to the arteries and veins in the fundus photograph.

[0240] In some embodiments, the segmentation module is further configured to: perform preliminary segmentation on the arteries and veins therein according to the brightness of the converted fundus photograph; segment the arteries and veins in the fundus photograph based on the first machine learning model according to the preliminary segmentation.

[0241] In some embodiments, the segmentation module is further configured to determine arteries and veins in the fused image based on the arteries and veins in the fundus photograph and the colors of the blood vessels in the fused image.

[0242] In some embodiments, the processing apparatus for retinal images further includes a calculation module configured to calculate the fractal dimensions of the arteries and veins respectively based on the fused image.

[0243] In some embodiments, the processing apparatus for retinal images further includes a calculation module configured to determine the blood flow densities of the arteries and veins based on the colors of the arteries and veins in the fused image.

[0244] In some embodiments, the alignment module 83 is further configured to select marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on a second machine learning model.

[0245] In some embodiments, the determination module 81 is further configured to determine candidate feature points in the fundus photograph and the OCTA image based on a feature point detection algorithm.

[0246] Figure 9 The block diagram of a processing apparatus for retinal images according to some other embodiments of the present disclosure is shown.

[0247] As Figure 9 shown, the processing apparatus 9 for retinal images includes a memory 91; and a processor 92 coupled to the memory 91. The memory 91 is used to store instructions for executing the method for processing retinal images. The processor 92 is configured to execute the method for processing retinal images in any of the embodiments of the present disclosure based on the instructions stored in the memory 91.

[0248] Figure 10 The block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0249] As Figure 10 shown, the computer system 100 may be presented in the form of a general-purpose computing device. The computer system 100 includes a memory 1010, a processor 1020, and a bus 1000 connecting different system components.

[0250] The memory 1010 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory stores, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium stores, for example, instructions for executing the method for processing retinal images in any of the embodiments of the present disclosure. The non-volatile storage medium includes but is not limited to a disk memory, an optical memory, a flash memory, etc.

[0251] The processor 1020 can be implemented in the form of a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Correspondingly, each module such as a judgment module and a determination module can be implemented by a central processing unit (CPU) running instructions for executing corresponding steps in a memory, or by dedicated circuitry for executing the corresponding steps.

[0252] The bus 1000 can use any bus structure among a variety of bus structures. For example, the bus structure includes but is not limited to an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0253] The computer system 100 may further include an input / output interface 1030, a network interface 1040, a storage interface 1050, etc. These interfaces 1030, 1040, 1050, the memory 1010, and the processor 1020 can be connected via the bus 1000. The input / output interface 1030 can provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 1040 provides a connection interface for various networking devices. The storage interface 1050 provides a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0254] Figure 11 A block diagram showing a processing system for retinal images according to some embodiments of the present disclosure.

[0255] As Figure 11 shown, the processing system 1100 for retinal images includes: a processing device 8 for retinal images, a fundus camera 12, and an optical coherence tomography angiography system 13.

[0256] The fundus camera 12 is used to generate fundus photos. The optical coherence tomography angiography system 13 is used to generate OCTA images.

[0257] Here, various aspects of the present disclosure have been described with reference to the flowcharts and / or block diagrams of a method, an apparatus, and a computer program product for processing retinal images according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of the blocks, can be implemented by computer-readable program instructions.

[0258] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable devices to generate a machine such that the device implementing the functions specified in one or more blocks in the flowcharts and / or block diagrams is generated by the processor executing the instructions.

[0259] These computer-readable program instructions can also be read and stored in a computer-readable memory, and these instructions cause a computer to operate in a specific manner, thereby producing a manufactured article that includes instructions for implementing the functions specified in one or more boxes in the flowchart and / or block diagram.

[0260] The present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0261] Through the method and apparatus for processing retinal images, computer-readable storage medium in the above embodiments, the alignment of the vascular structures in fundus photographs and OCTA images is achieved.

[0262] Thus far, the method and apparatus for processing retinal images, computer-readable storage medium according to the present disclosure have been described in detail. To avoid obscuring the concept of the present disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. A method for processing a retinal image, wherein, The retinal images include fundus photographs and optical coherence tomography angiography (OCTA) images, and the processing method of the retinal images includes: Determining candidate feature points in the fundus photographs and OCTA images; Selecting landmark feature points in the fundus photographs and OCTA images from the candidate feature points based on eye anatomical information; Aligning the vascular structures in the fundus photographs and OCTA images based on the landmark feature points in the fundus photographs and OCTA images, including: matching the landmark feature points in the fundus photographs and OCTA images to obtain multiple pairs of matched landmark feature points; based on the multiple pairs of matched landmark feature points, mapping the fundus photographs and OCTA images to the same coordinate system to align the vascular structures in the fundus photographs and OCTA images; Fusing the fundus photograph and OCTA image in the same coordinate system to obtain a fused image, including: normalizing the gray values of the pixels of the OCTA image; converting the normalized gray values into red, green, and blue primary color values based on a pseudo-color mapping table to obtain a pseudo-color OCTA image; fusing the pseudo-color OCTA image and the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of the blood vessels in the fused image represent different blood flow densities, and the blood flow density information in the OCTA image is used to verify whether the arteries and veins segmented in the fundus photograph conform to the actual blood flow situation; Converting the fundus photograph to a color space with brightness; Segmenting the arteries and veins therein according to the brightness of the converted fundus photograph; Determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph and the blood flow density information in the fused image.

2. The method for processing a retinal image according to claim 1, wherein, The selecting, from the candidate feature points, the landmark feature points in the fundus photographs and OCTA images based on eye anatomical information includes: Determining regions of interest (ROIs) in the fundus photographs and OCTA images based on eye anatomical information, wherein the ROIs in the fundus photographs and OCTA images include at least one of the optic disc region, the macula region, and the vascular region; Selecting landmark feature points from the candidate feature points of the fundus photographs and OCTA located in the ROI.

3. The method for processing a retinal image according to claim 2, wherein, The selecting, from the candidate feature points of the fundus photographs and OCTA located in the ROI, the landmark feature points includes: Selecting at least one of the candidate feature points of the optic disc center, the candidate feature points of the macula center, vascular crossing points, and vascular branching points from the candidate feature points of the fundus photographs and OCTA located in the ROI as the landmark feature points.

4. The processing method of the retinal image according to claim 1 further includes at least one of the following: Eliminating candidate feature points caused by illumination; Eliminating candidate feature points caused by noise; Eliminating non-significant candidate feature points on the vascular curve; Eliminating candidate feature points in the background region.

5. The method for processing a retinal image according to claim 1, wherein, The fusing the fundus photograph and OCTA image in the same coordinate system to obtain a fused image includes: Weighted summing the pixel values at the same position in the fundus photograph and OCTA image in the same coordinate system to obtain the pixel value at this position in the fused image.

6. The method for processing a retinal image according to claim 1, wherein, Segmenting arteries and veins in the converted fundus photograph according to its brightness includes: Preliminarily segmenting the arteries and veins in the converted fundus photograph according to its brightness; Based on a first machine learning model, segmenting the arteries and veins in the fundus photograph according to the preliminary segmentation.

7. The method for processing a retinal image according to claim 1 further includes: Calculating the fractal dimensions of the arteries and veins respectively according to the fused image.

8. The method for processing a retinal image according to claim 1 further includes: Determining the blood flow densities of the arteries and veins according to the colors of the arteries and veins in the fused image.

9. The method for processing a retinal image according to claim 1, wherein, Aligning the vascular structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image includes: Based on a second machine learning model, selecting the landmark feature points in the fundus photograph and the OCTA image from the candidate feature points.

10. The method for processing a retinal image according to claim 1, wherein, Determining the candidate feature points in the fundus photograph and the OCTA image includes: Based on a feature point detection algorithm, determining the candidate feature points in the fundus photograph and the OCTA image.

11. A processing device for a retinal image, wherein, A retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image. The apparatus for processing a retinal image includes: A determination module configured to determine candidate feature points in the fundus photograph and the OCTA image; A selection module configured to select the landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information; An alignment module configured to align the vascular structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image, including: matching the landmark feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matched landmark feature points; based on the multiple pairs of matched landmark feature points, mapping the fundus photograph and the OCTA image to the same coordinate system to align the vascular structures in the fundus photograph and the OCTA image; fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image; converting the fundus photograph to a color space with brightness; segmenting the arteries and veins in the converted fundus photograph according to its brightness; determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph and the blood flow density information in the fused image. Wherein, fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: normalizing the gray values of the pixels of the OCTA image; based on a pseudo-color mapping table, converting the normalized gray values into red, green, and blue primary color values to obtain a pseudo-color OCTA image; fusing the pseudo-color OCTA image and the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of the blood vessels in the fused image represent different blood flow densities, and the blood flow density information in the OCTA image is used to verify whether the arteries and veins segmented in the fundus photograph conform to the actual blood flow situation.

12. An apparatus for processing a retinal image includes: A memory; And A processor coupled to the memory, the processor being configured to execute the method for processing a retinal image according to any one of claims 1 to 10 based on instructions stored in the memory.

13. A system for processing a retinal image, comprising: The apparatus for processing a retinal image according to claim 11 or 12; A fundus camera; And An optical coherence tomography angiography system.

14. A computer-readable storage medium having computer program instructions stored thereon, which when executed by a processor, implement the method for processing a retinal image according to any one of claims 1 to 10.

15. A computer program product comprising computer program instructions, which when executed by a processor, implement the method for processing a retinal image according to any one of claims 1 to 10.

Images