Retina image processing method and device and computer readable storage medium

Abstract

The invention relates to a retina image processing method and device and a computer readable storage medium, and relates to the technical field of image processing. The retina image comprises a fundus photo and an optical coherence tomography angiography (OCTA) image, and the processing method of the retina image comprises the following steps: determining candidate feature points in the fundus photo and the OCTA image; selecting marker feature points in the fundus photo and the OCTA image from the candidate feature points based on eye anatomy information; and on the basis of the marker feature points in the fundus photo and the OCTA image, aligning the vascular structures in the fundus photo and the OCTA image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a retinal image processing method and device, and a computer-readable storage medium. Background Art

[0002] Retinal images include fundus photography, optical coherence tomography angiography (OCTA), etc. Fundus photography can provide two-dimensional color images of the retina, which are often used to examine the overall structure and vascular distribution of the retina. OCTA images can provide three-dimensional structure and blood flow information of retinal blood vessels, which are used to evaluate retinal blood flow density and vascular morphology.

[0003] The data of retinal blood vessels is of great significance for health assessment. In related technologies, retinal blood vessels are mainly analyzed based on a single retinal image, such as a fundus photograph or OCTA image. Summary of the invention

[0004] According to a first aspect of the present disclosure, a method for processing a retinal image is provided, wherein the retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image, and the method for processing the retinal image includes: determining candidate feature points in the fundus photograph and the OCTA image; selecting marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information; and aligning the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image.

[0005] In some embodiments, the method of selecting marker feature points in the fundus photograph and the OCTA image from candidate feature points based on the eye anatomical information includes: determining the region of interest ROI in the fundus photograph and the OCTA image based on the eye anatomical information, wherein the ROI in the fundus photograph and the OCTA image includes at least one of the optic disc area, the macular area, and the vascular area; and selecting marker feature points from the candidate feature points of the fundus photograph and the OCTA located in the ROI.

[0006] In some embodiments, the step of selecting marker feature points from candidate feature points located in the ROI in the fundus photograph and OCTA includes: selecting at least one of candidate feature points of the optic disc center, candidate feature points of the macula center, vascular intersection points, and vascular branch points from candidate feature points located in the ROI in the fundus photograph and OCTA as marker feature points.

[0007] In some embodiments, the method for processing retinal images further includes at least one of the following: eliminating candidate feature points caused by illumination; eliminating candidate feature points caused by noise; eliminating non-significant candidate feature points on blood vessel curves; and eliminating candidate feature points in background areas.

[0008] In some embodiments, the aligning of the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image includes: matching the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matched marker feature points; based on the multiple pairs of matched marker feature points, mapping the fundus photograph and the OCTA image to the same coordinate system, and aligning the vascular structures in the fundus photograph and the OCTA image.

[0009] In some embodiments, the method for processing retinal images further includes: fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image.

[0010] In some embodiments, fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: in the same coordinate system, weighted summing pixel values ​​corresponding to the same position in the fundus photograph and the OCTA image to obtain the pixel value of the position in the fused image.

[0011] In some embodiments, the fusing of the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: normalizing the grayscale values ​​of the pixels of the OCTA image; converting the normalized grayscale values ​​into red, green and blue primary color values ​​based on a pseudo-color mapping table to obtain a pseudo-color OCTA image; fusing the pseudo-color OCTA image with the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of blood vessels in the fused image represent different blood flow densities.

[0012] In some embodiments, the method for processing retinal images further includes: converting the fundus photograph into a color space with brightness; segmenting the arteries and veins therein according to the brightness of the converted fundus photograph; and determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph.

[0013] In some embodiments, segmenting the arteries and veins in the converted fundus photograph according to the brightness of the converted fundus photograph includes: performing preliminary segmentation of the arteries and veins in the fundus photograph according to the brightness of the converted fundus photograph; and segmenting the arteries and veins in the fundus photograph according to the preliminary segmentation based on a first machine learning model.

[0014] In some embodiments, determining the arteries and veins in the fused image based on the arteries and veins in the fundus photograph includes: determining the arteries and veins in the fused image based on the arteries and veins in the fundus photograph and the color of the blood vessels in the fused image.

[0015] In some embodiments, the method for processing retinal images further includes: calculating the fractal dimensions of the arteries and veins respectively based on the fused image.

[0016] In some embodiments, the method for processing retinal images further includes: determining the blood flow density of the arteries and veins according to the colors of the arteries and veins in the fused image.

[0017] In some embodiments, aligning the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image includes: selecting the marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on the second machine learning model.

[0018] In some embodiments, determining the candidate feature points in the fundus photograph and the OCTA image includes: determining the candidate feature points in the fundus photograph and the OCTA image based on a feature point detection algorithm.

[0019] According to a second aspect of the present disclosure, a retinal image processing device is provided, wherein the retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image, and the retinal image processing device includes: a determination module configured to determine candidate feature points in the fundus photograph and the OCTA image; a selection module configured to select marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on eye anatomical information; and an alignment module configured to align the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image.

[0020] According to a third aspect of the present disclosure, a retinal image processing device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the retinal image processing method according to any of the embodiments of the present disclosure based on instructions stored in the memory.

[0021] According to a fourth aspect of the present disclosure, a retinal image processing system is provided, comprising: a retinal image processing device according to any of the embodiments of the present disclosure; a fundus camera; and an optical coherence tomography vascular imaging system.

[0022] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the instructions are executed by a processor, the method for processing retinal images according to any of the embodiments of the present disclosure is implemented.

[0023] According to a sixth aspect of the present disclosure, a computer program product includes computer program instructions, which, when executed by a processor, implement the retinal image processing method according to any of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0026] Figure 1 A flowchart showing a method for processing a retinal image according to some embodiments of the present disclosure;

[0027] Figure 2 Original pictures showing fundus photographs and OCTA images;

[0028] Figure 3 The redundant feature points identified according to some embodiments of the present disclosure are shown;

[0029] Figure 4 Showing landmark feature points according to some embodiments of the present disclosure;

[0030] Figure 5 A schematic diagram showing different stratified comparisons of OCTA according to some embodiments of the present disclosure;

[0031] Figure 6 A schematic diagram showing image fusion according to some embodiments of the present disclosure;

[0032] Figure 7 A schematic diagram showing a method of extracting features from a fused image according to some embodiments of the present disclosure;

[0033] Figure 8 A block diagram showing a device for processing retinal images according to some embodiments of the present disclosure;

[0034] Fig. 9 A block diagram showing a device for processing retinal images according to other embodiments of the present disclosure;

[0035] Fig.10 A block diagram showing a computer system for implementing some embodiments of the present disclosure;

[0036] Fig.11A block diagram of a system for processing retinal images according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0037] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] The abbreviations used in the present disclosure are explained as follows.

[0044] Hue, Saturation, Value (HSV)

[0045] Brightness, green-red axis, blue-yellow axis color space (LAB), where L represents the brightness of the color, A represents the change from green to red, and B represents the change from blue to yellow

[0046] Convolutional Neural Network (CNN)

[0047] Scale-Invariant Feature Transform (SIFT)

[0048] Fast corner detection algorithm (Features from Accelerated Segment Test FAST)

[0049] Foveal Avascular Zone (FAZ)

[0050] Radial Peripapillary Capillary Plexus (RPCP)

[0051] Superficial Vascular Complex (SVC)

[0052] Intermediate Capillary Plexus (ICP)

[0053] Deep Capillary Plexus (DCP)

[0054] Tortuosity (T)

[0055] Branching Angle (θ)

[0056] Blood flow density (Flow Density, F)

[0057] Fractal Dimension (FD), a measure used to describe the complexity and irregularity of geometric shapes, is used to analyze the structural complexity of retinal blood vessels.

[0058] The radial peripapillary capillary plexus (RPCP) is a capillary plexus located around the optic disc that is responsible for supplying blood around the optic nerve. The blood flow density of the RPCP is often used to assess the health of the optic nerve.

[0059] The Superficial Vascular Complex (SVC) is a network of blood vessels located in the inner layer of the retina, mainly including the nerve fiber layer and the ganglion cell layer, responsible for the blood supply to the superficial retina.

[0060] Intermediate Capillary Plexus (ICP) is a capillary network located in the middle layer of the retina, mainly including the inner nuclear layer and the inner plexiform layer, responsible for the blood supply of the middle layer of the retina.

[0061] The deep capillary plexus (DCP) is a capillary network located in the deep layer of the retina, mainly including the outer plexiform layer and the outer nuclear layer, which is responsible for the blood supply to the deep layer of the retina.

[0062] In related technologies, retinal blood vessels are mainly analyzed based on a single retinal image. However, fundus photographs can only provide information about the surface structure of the retina, and cannot reflect the details and blood flow dynamics of different retinal layers, which limits its application in comprehensively evaluating the health of the retina.

[0063] At the same time, color and morphological segmentation that relies solely on fundus photographs may perform poorly in poor lighting conditions or in the presence of noise, especially in the early stages of retinal disease or mild lesions, where arteriovenous segmentation may be inaccurate.

[0064] OCTA images are generated based on the coherent signal changes of red blood cell movement. This method cannot directly measure the oxygen content of the blood or other biological characteristics that can clearly distinguish between arteries and veins. The blood vessels in OCTA images, whether arteries or veins, appear as the same high signal area. In short, because OCTA is based on the imaging principle of red blood cell movement, it cannot distinguish between arteries and veins, and the image quality is easily affected by artifacts, which also limits its application in comprehensive health assessment.

[0065] Figure 1 A flowchart showing a method for processing a retinal image according to some embodiments of the present disclosure is shown.

[0066] like Figure 1 As shown, the retinal image processing method includes steps S1 to S3, and the retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image. In some embodiments, the retinal image processing method is performed by a retinal image processing device.

[0067] In step S1, candidate feature points in the fundus photograph and the OCTA image are determined. In step S2, based on the ocular anatomical information, the marker feature points in the fundus photograph and the OCTA image are selected from the candidate feature points. In step S3, based on the marker feature points in the fundus photograph and the OCTA image, the vascular structures in the fundus photograph and the OCTA image are aligned.

[0068] In steps S1-S2, the fundus photograph and the OCTA image can be processed separately. For example, the feature points in the fundus photograph and the feature points in the OCTA image are first identified separately. Then, the marker feature points are further selected from the fundus photograph and irrelevant points are eliminated, wherein the marker feature points are, for example, some feature points with anatomical significance. Similar processing is also performed on the OCTA image.

[0069] In step S3, the fundus photograph and the vascular structures in the OCTA image are aligned, and the fundus photograph and the OCTA image are superimposed.

[0070] According to the disclosed embodiment, the alignment of the vascular structure of the fundus photograph and the OCTA image is achieved. By aligning the two images, the advantages of each modality can be fully utilized to provide more comprehensive structural and blood flow information of the retina.

[0071] Figure 2 Fundus photographs and original pictures of OCTA images are shown.

[0072] like Figure 2 As shown, it is difficult to align the fundus photograph with the OCTA image directly. To this end, some embodiments of the present disclosure use a feature point registration method to align the fundus photograph and the vascular structure in the OCTA image.

[0073] In some embodiments, determining candidate feature points in the fundus photograph and the OCTA image includes: determining the candidate feature points in the fundus photograph and the OCTA image based on a feature point detection algorithm.

[0074] For example, SIFT or FAST algorithm is used to extract candidate feature points in fundus photographs and OCTA images, respectively.

[0075] In some embodiments, based on the eye anatomical information, marker feature points in the fundus photograph and the OCTA image are selected from the candidate feature points, including: determining the region of interest ROI in the fundus photograph and the OCTA image based on the eye anatomical information, wherein the ROI in the fundus photograph and the OCTA image includes at least one of the optic disc area, the macular area, and the vascular area; and selecting marker feature points from the candidate feature points of the fundus photograph and the OCTA located in the ROI.

[0076] SIFT and FAST algorithms can detect points with significant features in an image (such as local extrema, corners, etc.). In anatomy, landmarks such as the optic disc usually have obvious image features (such as brightness, contrast, gradient changes) that can be detected by these algorithms.

[0077] Based on the anatomical region of interest (ROI), feature points that are not in important areas (such as the optic disc, the center of the macula, and near the intersection of blood vessels) are initially filtered out, thereby reducing redundant points in background areas and irrelevant areas. For example, the radius range of the optic disc and macula can be limited to extract only feature points near these areas.

[0078] In some embodiments, marker feature points are selected from candidate feature points located in the ROI in the fundus photograph and OCTA, including: selecting at least one of candidate feature points of the optic disc center, candidate feature points of the macula center, vascular intersection points, and vascular branch points from candidate feature points located in the ROI in the fundus photograph and OCTA as marker feature points.

[0079] The optic disc center is related to the brightness contrast and circular structure of the optic disc. The optic disc center is usually a prominent bright spot area in fundus photographs.

[0080] The center of the macula is located in the foveal avascular zone (FAZ), which is usually a dark area in the retinal image with no obvious vascular structure around it.

[0081] A vascular intersection is a point where two or more blood vessels intersect, where the brightness and morphology of the blood vessels change, and is commonly seen in the intersection area of ​​the retinal vascular network.

[0082] An arteriovenous bifurcation is, for example, the starting point of a blood vessel branch, where the blood vessel divides into two or more smaller branches.

[0083] The key nodes on the blood vessel curve are, for example, points where the blood vessel sharply bends or turns, and are areas where the blood vessel geometry changes significantly.

[0084] It is also possible to further filter based on anatomical information so that the distribution of feature points conforms to the structural characteristics of the optic disc, macula, and vascular intersections. For example, the feature points at the center of the optic disc should be distributed around a circular area; the center of the macula should be near the avascular area; and the vascular intersection should be at the intersection.

[0085] For example, the optic disc usually has obvious grayscale contrast in fundus images, especially at the edge of the optic disc. This high contrast is manifested as significant gradient changes, and SIFT locates feature points near the edge of the optic disc by detecting the local extreme values ​​of these changes.

[0086] There are local extreme points of brightness near the center of the optic disc, especially when the contrast with the surrounding background is large, which enables SIFT to detect the feature points of the optic disc. FAST identifies the position of the optic disc by detecting the corner points at the edge of the optic disc.

[0087] The center of the macula (especially the fovea) is an avascular area in fundus images, which usually appears as a smooth and avascular area. Although the fovea itself has no blood vessels, there are large gradient or contrast changes in the surrounding area.

[0088] SIFT can detect feature points by the difference in brightness gradient between the fovea and the blood vessel area outside it. Although the fovea itself is smooth, the significant gradient changes around it enable SIFT to find feature points in the macula edge area.

[0089] FAST identifies the location of the macula by detecting corner points at the edge of the macular area. Although the fovea itself does not have significant corner points, its edge and the surrounding area with blood vessels form corner points, which FAST can detect.

[0090] Blood vessel intersections are anatomical structures with distinctive features. Since blood vessels have obvious brightness and gradient changes at intersections, these areas can be detected by SIFT and FAST algorithms.

[0091] SIFT locates feature points by detecting gradient changes in images at different scales. The gradient changes at the intersection of blood vessels are very obvious, which enables SIFT to detect the features of these areas.

[0092] FAST identifies blood vessel intersections by detecting corners in the image. Since blood vessel intersections usually form multiple edge intersections and generate multiple corners, FAST can effectively detect these features.

[0093] The center of the optic disc, the center of the macula, and the intersection of blood vessels all have their own significant local image features (such as brightness contrast, gradient changes, corners, etc.), which match the detection mechanisms of SIFT and FAST, allowing these algorithms to identify these landmarks. SIFT and FAST are based on the principles of image grayscale, gradient, contrast changes, etc., and can detect the key anatomical structures of the retina.

[0094] That is to say, the algorithm can detect feature points such as the optic disc, macular center, and vascular intersections, but it will also detect redundant feature points that are not related to anatomy (for example, extreme points caused by noise). Therefore, redundant feature points need to be eliminated.

[0095] In some embodiments, the method for processing retinal images further includes at least one of the following: eliminating candidate feature points caused by illumination; eliminating candidate feature points caused by noise; eliminating non-significant candidate feature points on blood vessel curves; and eliminating candidate feature points in background areas.

[0096] Figure 3 Redundant feature points identified according to some embodiments of the present disclosure are shown.

[0097] Based on morphological principles, redundant candidate feature points can be eliminated. Figure 3 As shown, redundant feature points in an image are, for example, points without anatomical significance, which will be described in detail below with examples.

[0098] Local extreme points caused by image noise: irrelevant extreme points caused by noise of imaging equipment, uneven lighting, etc.

[0099] Spurious brightness gradients caused by lighting changes: local areas of increased contrast due to light refraction or device reflections.

[0100] Meaningless texture points in background areas: points where texture or spots in the retinal background are detected by the algorithm due to local contrast changes.

[0101] Non-critical points of vascular curves: In some small bends of the blood vessels, although there are corners or gradient changes, they are not related to important intersections or bifurcations.

[0102] Pseudo feature points (light spots or reflections): False feature points caused by light spots or reflections during imaging, which have nothing to do with the actual anatomical structure.

[0103] The non-significant (ie, non-critical) points on the blood vessel curve are, for example, corner points at slight bends. The candidate feature points of the background region are, for example, noise points in the background region.

[0104] In some embodiments, aligning vascular structures in the fundus photograph and the OCTA image based on marker feature points in the fundus photograph and the OCTA image includes: selecting marker feature points in the fundus photograph and the OCTA image from candidate feature points based on a second machine learning model.

[0105] For example, a machine learning model can also be used to assist in determining the marker feature points. For example, the data set is annotated, and a machine learning model (e.g., a convolutional neural network) is trained for fundus photos and OCTA images, respectively, so that the model can distinguish between marker feature points and redundant feature points.

[0106] After selecting the marker feature points in the fundus photographs and OCTA images from the candidate feature points based on the ocular anatomical information, the marker feature points are further screened through the machine learning model to more accurately select the marker feature points.

[0107] Figure 4 Marker feature points according to some embodiments of the present disclosure are shown.

[0108] like Figure 4 As shown, the set of landmark feature points selected through the aforementioned steps has a high anatomical significance, and only the feature points related to landmarks such as the optic disc center, the macula center, and the vascular intersection are retained.

[0109] In some embodiments, based on the marker feature points in the fundus photograph and the OCTA image, the vascular structures in the fundus photograph and the OCTA image are aligned, including: matching the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matched marker feature points; based on the multiple pairs of matched marker feature points, mapping the fundus photograph and the OCTA image to the same coordinate system, and aligning the vascular structures in the fundus photograph and the OCTA image.

[0110] For example, the feature points in the fundus photograph can be matched with the feature points in the OCTA image by using methods such as Euclidean distance, etc. Based on the matched feature point pairs, an affine transformation matrix is ​​calculated, which is used to transform the OCTA image into the coordinate system of the fundus photograph.

[0111] The OCTA image is transformed using a transformation matrix so that the OCTA image is aligned with the fundus photograph in geometric space. After the transformation, the vascular structure in the OCTA image coincides with the vascular structure in the fundus photograph at the same position.

[0112] The following example illustrates the alignment method between the fundus photograph and the OCTA image.

[0113] Ten landmark feature points are detected around the optic disc in the fundus photograph, and ten landmark feature points at the optic disc position are detected in the OCTA image. The landmark feature points in the fundus photograph are matched with the landmark feature points in the OCTA image by Euclidean distance or other matching algorithms to obtain eight pairs of matching landmark feature points, which are used to calculate the transformation matrix.

[0114] Using the matched feature point pairs, the following affine transformation matrix is ​​calculated.

[0115]

[0116] The transformation matrix indicates that the OCTA image needs to be rotated, scaled, and / or translated so that it is aligned with the fundus photograph.

[0117] In some embodiments, the method for processing retinal images further includes: fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image. In other words, the spatially aligned images are fused so that the fused image includes both information of the fundus photograph and information of the OCTA image.

[0118] In some embodiments, the fundus photograph and the OCTA image in the same coordinate system are fused to obtain a fused image, including: in the same coordinate system, weighted summing the pixel values ​​corresponding to the same position in the fundus photograph and the OCTA image to obtain the pixel value of the position in the fused image.

[0119] Weighted fusion takes into account the importance of information in different images and realizes image fusion by weighted processing of two images. The calculation formula is as follows.

[0120] I fused (x,y)=w retina I retina (x,y)+w OCTA I OCTA (x,y)

[0121] w retina +W OCTA =1 (2)

[0122] Among them, I fused (x,y) represents the pixel value of the fused image.retina (x, y) represents the pixel value of the fundus photograph. OCTA (x,y) represents the pixel value of the OCTA image. retina and w OCTA Indicates the weights of fundus photographs and OCTA images. These weights can be adjusted according to the image quality and analysis requirements. For example, if you want to highlight the structural information of retinal vessels, you can set w retina >w OCTA .

[0123] Alternatively, the transparency adjustment (Alpha Blending) method can be used to fuse the two images using a linear interpolation method. The calculation formula is as follows.

[0124] I fused (x,y)=α·I retina (x,y)+(1-α)·I OCTA (x,y) (3)

[0125] Among them, I fused (x,y) represents the pixel value of the fused image. retina (x, y) represents the pixel value of the fundus photograph. OCTA (x, y) represents the pixel value of the OCTA image. α is the transparency coefficient (the value range is between 0 and 1). For example, when α is set to 0.6, the structural information of the fundus photo is more prominent, and the information of the OCTA image is displayed as an auxiliary layer.

[0126] In some embodiments, the fundus photograph and the OCTA image in the same coordinate system are fused to obtain a fused image, including: normalizing the grayscale values ​​of the pixels of the OCTA image; converting the normalized grayscale values ​​into red, green and blue primary color values ​​based on a pseudo-color mapping table to obtain a pseudo-color OCTA image; fusing the pseudo-color OCTA image and the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of blood vessels in the fused image represent different blood flow densities.

[0127] For example, pseudo color mapping can be performed on the OCTA image to map the pixel values ​​in the original grayscale image to a specific color range, thereby more intuitively displaying the blood flow density information in the OCTA image. The pseudo color mapping method is described in detail below.

[0128] First, the grayscale value of the OCTA image is normalized to the range of [0,1], and the calculation formula is as follows.

[0129]

[0130] Then perform color mapping and use common pseudo color mapping tables (such as Jet, HSV, etc.) to convert the normalized grayscale values ​​into RGB colors, for example:

[0131] I norm (x,y)=0 is mapped to blue

[0132] I norm (x,y)=1 is mapped to red (5)

[0133] The mapped pseudo-color OCTA images were superimposed on the fundus photographs.

[0134] In the fused image, the fundus photograph retains the structural information of arteries and veins, and the pseudo-color image of OCTA shows the blood flow density.

[0135] For example, fundus photographs are used as the base layer to show clear arterial and venous structures. OCTA images are displayed as red, yellow, and green areas after pseudo-color mapping, indicating different levels of blood flow density.

[0136] OCTA images can also be layered to extract blood flow density data for each layer. For example, the blood flow density image can be processed using a threshold method or histogram equalization to make the difference in blood flow density more obvious. The following describes a method for extracting blood flow density information from OCTA images.

[0137] #Read grayscale image

[0138] octa image =cv2.imread(′octa image .png′,0)

[0139] Figure 5 Schematic diagram showing different stratification comparisons of OCTA according to some embodiments of the present disclosure.

[0140] like Figure 5 As shown in FIG. 1 , the blood flow density at different levels in the OCTA image can be separated by layered processing. After separation, the data at each level can be processed separately.

[0141] The specific implementation of pseudo color mapping is introduced below.

[0142] First, use the pseudo color mapping function of OpenCV to map different blood flow density ranges to different colors. According to the blood flow density value, the higher density area is mapped to a bright color (such as yellow) and the lower density area is mapped to a dark color (such as blue).

[0143] # Apply pseudo color mapping

[0144]

[0145] After pseudo color mapping, a fused image is generated. For example, the arterial and venous annotation results are superimposed on the pseudo color mapped OCTA blood flow density image to generate a comprehensive analysis image. Arteries and veins are annotated with different colors, while blood flow density information is displayed in the background through pseudo color mapping. The specific implementation of image fusion is introduced below.

[0146] #Overlay the arteriovenous segmentation results onto the pseudo-color OCTA image

[0147]

[0148] final image =cv2.addWeighted(final image ,0.5,vein segment ,0.5,0)

[0149] Through the above steps, arteries and veins can be distinguished in the registered fusion image, and the blood flow density information of the OCTA image can be visualized. The arterial and venous annotation information combined with the blood flow density information of the OCTA image can provide a more comprehensive multimodal analysis result for retinal health assessment.

[0150] In the fused image, the vascular structure in the fundus photo layer is clearly visible, while the blood flow density information of the OCTA image is superimposed in the corresponding position with pseudo-color. Through transparency adjustment or weighted fusion, the display effect of blood flow information can be retained while highlighting the structural information. Through transparency adjustment, weighted fusion and pseudo-color mapping, the superposition and fusion of fundus photos and OCTA images can be achieved, thereby effectively combining retinal structural information and blood flow density information, providing a more comprehensive analysis for retinal health assessment.

[0151] In some embodiments, the method for processing retinal images further includes: converting the fundus photograph into a color space with brightness; segmenting the arteries and veins therein according to the brightness of the converted fundus photograph; and determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph.

[0152] The color space with brightness is, for example, the LAB color space, the HSV color space, etc. The following describes a method for distinguishing arteries and veins in fundus photographs based on color thresholds.

[0153] In fundus photographs, arteries are usually brighter (reddish) and veins are darker (purple or dark red). This color difference can serve as an important basis for distinguishing arteries and veins.

[0154] First use color space conversion to convert the RGB image to HSV or LAB color space in order to separate the colors more accurately.

[0155] Threshold segmentation is performed based on the hue (H) and brightness (V) in the HSV space or the brightness (L) in the LAB space: the brightness of the artery is higher, so a higher threshold can be set to segment the artery; the brightness of the vein is lower, so a lower threshold can be set to segment the vein.

[0156] The specific implementation is as follows.

[0157] Convert image from RGB to HSV (python):

[0158] #Read the image

[0159] import cv2

[0160] image = cv2.imread('retinal image .png′)

[0161] #Convert the image from BGR to HSV color space

[0162] hsv image =cv2.ctvColor(image,cv2.COLOR BGR2HSV )

[0163] #Set the color threshold of the artery (higher brightness)

[0164] #H (hue) range: 0-10

[0165] #S (saturation) range: 50-255

[0166] #V (brightness) range: 150-255

[0167] lower_artery=(0,50,150)

[0168] upper_artery=(10,255,255)

[0169] #Create an artery mask based on the set threshold

[0170] artery_mask=cv2.inRange(hsv_image,lower_artery,

[0171] upper_artery)

[0172] #Set the color threshold of veins (lower brightness)

[0173] #H (hue) range: 0-10

[0174] #S (saturation) range: 50-255

[0175] #V (brightness) range: 50-150

[0176] lower_vein=(0,50,50)

[0177] upper_vein = (10, 255, 150)

[0178] #Create a vein mask based on the set threshold

[0179] vein_mask=cv2.inRange(hsv_image,lower_vein,upper_vein)

[0180] #Apply the artery mask to the original image to extract the arterial part

[0181] artery_segment=cv2.bitwise_and(image,image,

[0182] mask = artery_mask)

[0183] #Apply the vein mask to the original image to extract the vein part

[0184] vein_segment=cv2.bitwise_and(image,image,

[0185] mask = vein_mask)

[0186] In some embodiments, segmenting the arteries and veins in the converted fundus photograph according to the brightness of the converted fundus photograph includes: performing preliminary segmentation of the arteries and veins in the fundus photograph according to the brightness of the converted fundus photograph; and segmenting the arteries and veins in the fundus photograph according to the preliminary segmentation based on a first machine learning model.

[0187] The color threshold method is based on the color difference between the retinal arteries and veins (the arteries are brighter and the veins are darker). This method is simple and fast, and works well in scenes with obvious image features and high consistency. It is simple and has low computational overhead. However, in images with high variability or high noise, the color threshold method may not be able to accurately distinguish between arteries and veins.

[0188] For example, for areas with large brightness differences in the image, the confidence level is high when performing artery and vein segmentation based on brightness, so we first perform preliminary segmentation based on brightness. However, in some areas of the image, there is less brightness information, which is not enough to distinguish between arteries and veins. In this case, using machine learning to assist segmentation can make the segmentation results more accurate.

[0189] The following introduces the method of using machine learning to assist segmentation.

[0190] Use a public retinal image dataset (such as DRIVE or CHASE-DB1) for model training, which contains the annotated results of arteries and veins. Use the annotated retinal dataset to pre-train the model and let the model learn the characteristics of arteries and veins, such as color, texture, and morphology.

[0191] Use a pre-trained convolutional neural network model to perform arteriovenous segmentation. For example, input a fundus photo into the trained model to generate arteriovenous segmentation results.

[0192] Deep learning models can capture complex features in images (such as morphology, texture, etc.) through large-scale data training, not just relying on color information. Deep learning models can have higher accuracy in complex backgrounds and different imaging conditions, can handle complex image features, and adapt to different changes in images.

[0193] In the process of arteriovenous labeling, color thresholding and deep learning models can be used separately. For example, if the image conditions in the arteriovenous labeling scene are consistent and mainly rely on color information, the color thresholding method can be selected; if the image is complex or higher accuracy is required, the deep learning model can be used.

[0194] The two can also be used in combination, as described in detail below.

[0195] First, use the color threshold method to perform a preliminary classification of retinal arteries and veins. Arteries are usually brighter in color, while veins are darker. Preliminary separation of arteries and veins is performed by setting different color thresholds. Color thresholds can quickly perform rough labeling, but errors may occur in some details or complex areas (such as intersections or areas with similar colors).

[0196] The preliminary annotation results are then input into the deep learning model for further fine-grained annotation and correction. The deep learning model can identify areas that the color threshold method cannot accurately classify, especially at the intersection of arteries and veins or in areas where the image is not clear, and perform corrections and optimizations to improve the accuracy of the overall annotation. Therefore, the color threshold is used to quickly generate preliminary results, reducing the computational overhead of the deep learning model and making the entire process more efficient. At the same time, the subsequent deep learning model can process areas that the color threshold method cannot accurately identify and improve accuracy.

[0197] In some embodiments, determining the arteries and veins in the fused image based on the arteries and veins in the fundus photograph includes: determining the arteries and veins in the fused image based on the arteries and veins in the fundus photograph and the color of the blood vessels in the fused image.

[0198] In the fused image, the high-resolution anatomical structure of the fundus photograph can be used to help identify the origin, direction and morphology of the blood vessels. The morphological features of arteries and veins are clearly visible in the fundus photograph, which provides solid support for the basic segmentation of arteries and veins.

[0199] After fusing the fundus photograph and OCTA image, the effect of arteriovenous segmentation will be significantly improved, especially after fusing the structural information and blood flow information. The blood flow density information in the OCTA image can be used to verify whether the segmented arteries and veins in the fundus photograph are consistent with the actual blood flow situation. For example, in a healthy retina, the blood flow density of arteries and veins will be different. In the fused image, by combining the information of the OCTA image layer, the arteriovenous segmentation in the fundus photograph image layer can be finely adjusted and optimized to improve the accuracy of arteriovenous segmentation.

[0200] After the segmentation information of arteries and veins in fundus photographs is fused with the different levels of blood flow information provided by OCTA, blood vessels can be further subdivided from the perspective of blood flow. The fused image allows the morphological information of arteries and veins (extracted from fundus photographs) to be combined with the blood flow function information (extracted from OCTA images). For example, areas with higher blood flow density are more likely to be arteries, while areas with lower blood flow density may be veins. This allows accurate distinction between whether a blood vessel is an artery or a vein, rather than relying solely on color or morphological features. This fusion of information can optimize and correct arteriovenous segmentation that relies solely on morphological analysis. For example, at the intersection of arteries and veins, the fused image can further confirm the type of vessel by blood flow density to reduce errors.

[0201] Therefore, by aligning OCTA images and fundus photographs, the features of arteries and veins can be more accurately identified and extracted.

[0202] In some embodiments, the method for processing retinal images further includes: calculating the fractal dimensions of the arteries and veins respectively based on the fused image.

[0203] For example, the fractal dimension is calculated from the differentiated arteries and veins, and the fractal dimension reflects the complexity and branching structure of the blood vessels. The fractal dimension can be calculated using the box-counting method or other fractal geometry methods.

[0204] In some embodiments, the retinal image processing method further includes: determining the blood flow density of the arteries and veins according to the colors of the arteries and veins in the fused image. The fractal dimension, blood flow density, etc. can be calculated more accurately based on the information of distinguishing the arteries and veins and the color information of the blood vessels in the fused image.

[0205] For example, different levels of blood flow density (such as SVC, ICP, DCP) can be extracted from the fused image. For each vascular plexus, the blood flow density, that is, the amount of blood flowing per unit area of ​​the blood vessel, is calculated separately.

[0206] Figure 6 A schematic diagram showing image fusion according to some embodiments of the present disclosure.

[0207] like Figure 6 As shown, the arteries and veins are marked on the fundus photograph, the blood flow density layers are distinguished on the OCTA image, and the processed fundus photograph and OCTA image are registered based on the landmark feature points to achieve image fusion.

[0208] Image processing technology can be used to extract information such as fractal dimension and blood flow density from the fused image.

[0209] Figure 7 A schematic diagram showing feature extraction from a fused image according to some embodiments of the present disclosure.

[0210] like Figure 7 As shown, features can be extracted through basic mathematical operations or through big data, neural networks, etc.

[0211] The fused images can also provide more information for disease prediction, which is described in detail below.

[0212] 1. Structure-function consistency check: Fusion images can provide structure-function consistency check after arteriovenous segmentation. For example, if the structure and morphology of a certain artery are normal, but the blood flow density decreases, it may indicate that there is a potential functional disorder in the area (such as vascular stenosis or insufficient blood supply).

[0213] 2. Local complexity analysis: The combination of morphological characteristics and blood flow density of arteries and veins can assist in more complex local analysis. For example, the combination of the fractal dimension of an artery (reflecting the complexity of the blood vessel) and its blood flow density can reveal the health of a particular blood vessel. The fractal dimension or blood flow density alone may not reflect this complexity, but a more accurate assessment can be obtained when combined.

[0214] By fusing arteriovenous segmentation with OCTA blood flow information, some new features that cannot be obtained from fundus photos or OCTA alone can be extracted. For example:

[0215] (1) Arteriovenous blood flow function index: By combining the fractal dimension of the artery and vein with its corresponding blood flow density, the blood flow function index of each artery and vein can be calculated. This index can comprehensively consider the morphology and function of the artery / vein and further evaluate the health of the retinal blood vessels;

[0216] (2) Detection of arteriovenous structure-function inconsistency: Fusion images allow for simultaneous assessment of vascular anatomy and function. This assessment can reveal that arteries or veins in certain areas are morphologically normal but may be functionally abnormal. For example, an artery may be morphologically normal but have reduced blood flow, which may indicate an early lesion. In contrast, such abnormalities cannot be detected from fundus photographs alone, and OCTA images alone cannot distinguish which vessel (artery or vein) has a functional abnormality.

[0217] (3) Compared with fundus photographs alone, which may not be able to identify early blood flow dysfunction, and OCTA images alone, which may not be able to accurately separate the blood flow status of arteries and veins, fused images can provide more accurate assessment information in early lesions by combining arteriovenous structure and blood flow density information.

[0218] (4) Accurate assessment of arteriovenous intersections: Arteriovenous intersections are often high-risk sites for lesions. By fusing images, the structural and functional changes in these areas can be better assessed. Combining the morphological characteristics of arteries and veins with blood flow function information can more accurately assess whether there are problems such as blood flow restriction or abnormal dilatation of arteries / veins at the intersection.

[0219] In summary, from the perspective of arteriovenous segmentation, the fusion of fundus photographs and OCTA images can more accurately extract the morphology and blood flow characteristics of arteries and veins. This combination makes arteriovenous segmentation more accurate and can detect some functional-structural inconsistencies that cannot be detected in some separate images. This provides more comprehensive support for the detection of early lesions, the evaluation of complex lesion areas (such as arteriovenous intersections), and the evaluation of overall retinal health.

[0220] In summary, compared with using OCTA images or fundus photos alone, the fusion of fundus photos and OCTA images can more accurately extract the morphology and blood flow characteristics of arteries and veins, and more accurately segment arteries and veins. In addition, from the fused images, some functional-structural inconsistencies that cannot be detected in individual images can be found, providing more comprehensive support for the assessment of retinal health status.

[0221] The following describes a method for processing retinal images according to some embodiments of the present disclosure.

[0222] Firstly, the fundus photographs and OCTA images are preprocessed and registered to obtain the fused image.

[0223] Then, the deep learning model is used to segment the fused image into blood vessels and extract arteries and veins respectively. The segmentation results of arteries and veins are recorded as Va and Vv respectively.

[0224] According to the anatomical structure of the retina, the fused image is divided into the optic disc area and the macular area, and features are extracted from each area.

[0225] According to the hierarchical information of the fused image, the blood vessel segmentation results are classified into different vascular plexuses, such as radial paraoptic capillary plexus, superficial vascular plexus, intermediate capillary plexus, and deep capillary plexus. For each vascular plexus C, the arterial and venous features are recorded as X a,C and X v,C .

[0226] Features such as vascular tortuosity, branching angles, and blood flow density were extracted from arteries and veins, optic disc and macular regions, and different vascular plexuses.

[0227] Figure 8 A block diagram showing a device for processing retinal images according to some embodiments of the present disclosure.

[0228] like Figure 8 As shown, the retinal image processing device 8 includes a determination module 81, a selection module 82, and an alignment module 83. The retinal image includes a fundus photograph and an optical coherence tomography angiography OCTA image.

[0229] The determination module 81 is configured to determine candidate feature points in the fundus photograph and the OCTA image, for example, by performing the following steps: Figure 1 Step S1 is shown.

[0230] The selection module 82 is configured to select the marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on the ocular anatomical information, for example, by performing the following steps: Figure 1 Step S2 is shown.

[0231] The alignment module 83 is configured to align the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image, for example, by performing the following steps: Figure 1 Step S3 shown.

[0232] In some embodiments, the selection module 82 is further configured to: determine the region of interest ROI in the fundus photograph and the OCTA image based on the eye anatomical information, wherein the ROI in the fundus photograph and the OCTA image includes at least one of the optic disc area, the macular area, and the vascular area; and select marker feature points from the candidate feature points of the fundus photograph and the OCTA located in the ROI.

[0233] In some embodiments, the selection module 82 is further configured to select at least one of the candidate feature points of the optic disc center, the candidate feature points of the macula center, the vascular intersection point, and the vascular branch point from the candidate feature points located in the ROI of the fundus photograph and the OCTA as the marker feature point.

[0234] In some embodiments, the retinal image processing device also includes a rejection module configured to: reject candidate feature points caused by illumination; reject candidate feature points caused by noise; reject non-significant candidate feature points on blood vessel curves; and reject candidate feature points in the background area.

[0235] In some embodiments, the alignment module 83 is further configured to: match the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matching marker feature points; based on the multiple pairs of matching marker feature points, map the fundus photograph and the OCTA image to the same coordinate system, and align the vascular structures in the fundus photograph and the OCTA image.

[0236] In some embodiments, the retinal image processing device further includes: a fusion module configured to fuse the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image.

[0237] In some embodiments, the fusion module is further configured to: in the same coordinate system, perform weighted summation of pixel values ​​corresponding to the same position in the fundus photograph and the OCTA image to obtain the pixel value of the position in the fused image.

[0238] In some embodiments, the fusion module is further configured to: normalize the grayscale values ​​of the pixels of the OCTA image; convert the normalized grayscale values ​​into red, green and blue primary color values ​​based on a pseudo-color mapping table to obtain a pseudo-color OCTA image; fuse the pseudo-color OCTA image with the fundus photograph in the same coordinate system to obtain a fused image, wherein different colors of blood vessels in the fused image represent different blood flow densities.

[0239] In some embodiments, the retinal image processing device also includes: a segmentation module configured to convert the fundus photograph into a color space with brightness; segment the arteries and veins therein according to the brightness of the converted fundus photograph; and determine the arteries and veins in the fused image according to the arteries and veins in the fundus photograph.

[0240] In some embodiments, the segmentation module is further configured to: perform preliminary segmentation of the arteries and veins in the converted fundus photograph according to the brightness of the fundus photograph; and segment the arteries and veins in the fundus photograph according to the preliminary segmentation based on the first machine learning model.

[0241] In some embodiments, the segmentation module is further configured to determine the arteries and veins in the fused image according to the arteries and veins in the fundus photograph and the color of the blood vessels in the fused image.

[0242] In some embodiments, the retinal image processing device further includes: a calculation module configured to calculate the fractal dimensions of the arteries and veins respectively based on the fused image.

[0243] In some embodiments, the retinal image processing device further includes: a calculation module configured to determine the blood flow density of the arteries and veins according to the colors of the arteries and veins in the fused image.

[0244] In some embodiments, the alignment module 83 is further configured to select, from the candidate feature points, marker feature points in the fundus photograph and the OCTA image based on a second machine learning model.

[0245] In some embodiments, the determination module 81 is further configured to: determine candidate feature points in the fundus photograph and the OCTA image based on a feature point detection algorithm.

[0246] Fig. 9 A block diagram showing a device for processing retinal images according to other embodiments of the present disclosure.

[0247] like Fig. 9 As shown, the retinal image processing device 9 includes a memory 91; and a processor 92 coupled to the memory 91, and the memory 91 is used to store instructions for executing the retinal image processing method. The processor 92 is configured to execute the retinal image processing method in any of some embodiments of the present disclosure based on the instructions stored in the memory 91.

[0248] Fig.10 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0249] like Fig.10 As shown, the computer system 100 may be embodied in the form of a general-purpose computing device. The computer system 100 includes a memory 1010, a processor 1020, and a bus 1000 that connects various system components.

[0250] The memory 1010 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing the method for processing a retinal image in any of the embodiments of the present disclosure. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, etc.

[0251] The processor 1020 can be implemented by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0252] The bus 1000 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0253] The computer system 100 may also include an input / output interface 1030, a network interface 1040, a storage interface 1050, etc. These interfaces 1030, 1040, 1050, the memory 1010, and the processor 1020 may be connected via a bus 1000. The input / output interface 1030 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 1040 may provide a connection interface for various networked devices. The storage interface 1050 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0254] Fig.11 A block diagram of a system for processing retinal images according to some embodiments of the present disclosure is shown.

[0255] like Fig.11 As shown, the retinal image processing system 1100 includes: a retinal image processing device 8, a fundus camera 12, and an optical coherence tomography vascular imaging system 13.

[0256] The fundus camera 12 is used to generate fundus photographs. The optical coherence tomography angiography system 13 is used to generate OCTA images.

[0257] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products for processing retinal images according to embodiments of the present disclosure. It should be understood that each block of the flowchart and / or block diagram and the combination of blocks can be implemented by computer-readable program instructions.

[0258] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0259] These computer-readable program instructions can also be readable and stored in a computer-readable memory, which cause the computer to work in a specific manner to produce a product, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0260] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0261] By means of the retinal image processing method and device and the computer-readable storage medium in the above-mentioned embodiments, alignment of the vascular structures of the fundus photograph and the OCTA image is achieved.

[0262] So far, the retinal image processing method and device and computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

Claims

1. A method for processing retinal images, wherein: The retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image, and the retinal image processing method includes: Determine candidate feature points in fundus photographs and OCTA images; Based on the ocular anatomical information, the landmark feature points in the fundus photograph and the OCTA image are selected from the candidate feature points; Based on the landmark feature points in the fundus photograph and OCTA image, the vascular structures in the fundus photograph and OCTA image are aligned.

2. The method for processing retinal images according to claim 1, wherein: The method of selecting the marker feature points in the fundus photograph and the OCTA image from the candidate feature points based on the ocular anatomical information includes: Based on the ocular anatomical information, determining a region of interest (ROI) in the fundus photograph and the OCTA image, wherein the ROI in the fundus photograph and the OCTA image includes at least one of an optic disc area, a macula area, and a vascular area; The marker feature points are selected from the candidate feature points located in the ROI of the fundus photograph and OCTA.

3. The method for processing retinal images according to claim 1, wherein: The step of selecting a marker feature point from candidate feature points located in the ROI in the fundus photograph and the OCTA includes: From the candidate feature points located in the ROI of the fundus photograph and OCTA, at least one of the candidate feature points of the optic disc center, the candidate feature points of the macula center, the blood vessel intersection point, and the blood vessel branch point is selected as the marker feature point.

4. The method for processing retinal images according to claim 1, further comprising at least one of the following: Eliminate candidate feature points caused by lighting; Eliminate candidate feature points caused by noise; Eliminate non-significant candidate feature points on the blood vessel curve; Eliminate candidate feature points in the background area.

5. The method for processing retinal images according to claim 1, wherein: The aligning of the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image includes: Matching the marker feature points in the fundus photograph and the OCTA image to obtain multiple pairs of matching marker feature points; Based on multiple pairs of matching landmark feature points, the fundus photograph and OCTA image are mapped to the same coordinate system, and the vascular structures in the fundus photograph and OCTA image are aligned.

6. The method for processing retinal images according to claim 5, further comprising: The fundus photograph and OCTA image in the same coordinate system are fused to obtain a fused image.

7. The method for processing retinal images according to claim 6, wherein: The method of fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: In the same coordinate system, the pixel values ​​corresponding to the same position in the fundus photograph and the OCTA image are weighted summed to obtain the pixel value of the position in the fused image.

8. The method for processing retinal images according to claim 6, wherein: The method of fusing the fundus photograph and the OCTA image in the same coordinate system to obtain a fused image includes: Normalize the grayscale values ​​of pixels in OCTA images; Based on the pseudo-color mapping table, the normalized grayscale value is converted into the red, green and blue primary color values ​​to obtain the pseudo-color OCTA image; The pseudo-color OCTA image and the fundus photograph in the same coordinate system are fused to obtain a fused image, wherein different colors of blood vessels in the fused image represent different blood flow densities.

9. The method for processing retinal images according to claim 8, further comprising: Convert fundus photographs to color space with brightness; According to the brightness of the converted fundus photograph, the arteries and veins are segmented; Arteries and veins in the fused image are determined based on the arteries and veins in the fundus photograph.

10. The method for processing retinal images according to claim 9, wherein: The step of segmenting the arteries and veins in the converted fundus photograph according to the brightness of the converted fundus photograph comprises: According to the brightness of the converted fundus photograph, the arteries and veins are preliminarily segmented; Based on the first machine learning model, arteries and veins in the fundus photograph are segmented according to the preliminary segmentation.

11. The method for processing retinal images according to claim 9, wherein: The step of determining the arteries and veins in the fused image according to the arteries and veins in the fundus photograph includes: The arteries and veins in the fused image are determined according to the arteries and veins in the fundus photograph and the colors of the blood vessels in the fused image.

12. The method for processing retinal images according to claim 11, further comprising: Based on the fused images, the fractal dimensions of arteries and veins are calculated separately.

13. The method for processing retinal images according to claim 11, further comprising: The blood flow density of arteries and veins was determined based on the colors of the arteries and veins in the fused images.

14. The method for processing retinal images according to claim 1, wherein: The aligning of the vascular structures in the fundus photograph and the OCTA image based on the marker feature points in the fundus photograph and the OCTA image includes: Based on the second machine learning model, marker feature points in the fundus photograph and OCTA image are selected from the candidate feature points.

15. The method for processing retinal images according to claim 1, wherein: The step of determining candidate feature points in the fundus photograph and the OCTA image includes: Based on the feature point detection algorithm, candidate feature points in fundus photographs and OCTA images are determined.

16. A retinal image processing device, wherein: The retinal image includes a fundus photograph and an optical coherence tomography angiography (OCTA) image, and the retinal image processing device includes: A determination module, configured to determine candidate feature points in the fundus photograph and the OCTA image; A selection module is configured to select landmark feature points in the fundus photograph and the OCTA image from the candidate feature points based on ocular anatomical information; The alignment module is configured to align the vascular structures in the fundus photograph and the OCTA image based on the landmark feature points in the fundus photograph and the OCTA image.

17. A retinal image processing device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the method for processing a retinal image according to any one of claims 1 to 15 based on instructions stored in the memory.

18. A retinal image processing system, comprising: The retinal image processing device according to claim 16 or 17; Fundus camera; as well as Optical coherence tomography vascular imaging system.

19. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the retinal image processing method according to any one of claims 1 to 15.

20. A computer program product, comprising computer program instructions, which, when executed by a processor, implement the method for processing retinal images according to any one of claims 1 to 15.

Images