Retina three-dimensional reconstruction method fusing fundus color image and OCT image

Abstract

The invention relates to a retina three-dimensional reconstruction method fusing a fundus color image and an OCT image, and the method comprises the steps: firstly segmenting a retina layer structure in an ophthalmology OCT image, and determining a retina reconstruction region; secondly, converting all data in the retina reconstruction region into sparse three-dimensional point clouds; secondly, densifying the sparse three-dimensional point cloud by adopting a three-dimensional space filling algorithm, and realizing matching between the OCT image data and the fundus color image spatial resolution; and finally, dyeing the point clouds of the retina NFL layer by using RGB colors extracted from the fundus color image, and performing pseudo-color transformation on the point clouds of other layers of the retina by using a color transformation transfer function group to obtain a final retina color three-dimensional image. The method is mainly used for realizing fusion of the fundus color image and the OCT image, constructing a retina color three-dimensional image, and providing a medical image with more real visual effect and richer detail information for medical personnel.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and relates to a method for three-dimensional reconstruction of the retina by fusing fundus color images and OCT images, especially for medical image data generated by a fundus color camera and an ophthalmic OCT device. Background Art

[0002] The physiological structure of the human eye is extremely complex. Retinal examination can not only accurately diagnose various eye diseases, but also assist in the evaluation of systemic diseases. At present, the widely used fundus imaging techniques in clinical practice mainly include fundus color cameras and optical coherence tomography (OCT). Fundus color cameras can provide colored and wide-angle retinal images, intuitively showing the overall structure of the retina, blood vessel distribution, and lesion areas. However, this technique has low sensitivity to subtle structural changes, and the generated two-dimensional images cannot reflect the true three-dimensional spatial relationship of the retina, which to a certain extent limits the diagnostic accuracy of doctors. In contrast, OCT can provide high-resolution cross-sectional images of the retina, clearly showing the layered structure and subtle lesions of the retina, making up for the deficiency of fundus color cameras in the recognition of structural details. However, OCT images are presented in grayscale, unable to reflect the true color information of the retina and difficult to conduct a holistic assessment.

[0003] In the actual clinical diagnosis process, the limitations of a single imaging mode require doctors to comprehensively analyze various image data to improve the accuracy and comprehensiveness of diagnosis. The combined application of fundus color cameras and ophthalmic OCT has played an important role in the early screening and disease monitoring of diseases such as diabetic retinopathy, macular degeneration, and retinal vein occlusion. It not only effectively reduces the risks of missed diagnosis and misdiagnosis, but also provides more comprehensive information for the assessment of fundus lesions. However, at present, when medical staff use fundus color images and ophthalmic OCT images for diagnosis, they can only rely on manual comparison of two-dimensional fundus color images and OCT images, and speculate on their spatial correspondence through experience and association. This diagnostic method can neither truly and comprehensively display the spatial relationship of the fundus retina, nor is it conducive to doctors' intuitive and rapid identification of lesions and complex physiological structures. It also increases the difficulty for doctors to identify lesions, seriously affecting the diagnostic efficiency and accuracy.

[0004] If the fundus color image can be fused with the OCT image to construct a three-dimensional color retinal image, the visual expression and detailed information of the image can be significantly enhanced, helping medical staff to identify lesions and normal tissues more efficiently and accurately. In addition, this image can provide a richer sense of spatial hierarchy and multi-parameter comparison information, further improving the ability to detect diseases at an early stage, while supporting preoperative planning and intraoperative navigation and optimizing treatment plans. At the same time, the three-dimensional color retinal image can also reduce the fatigue of reading images, improve the diagnostic efficiency, promote the communication and cooperation of the medical staff team, thereby improving the overall medical quality and the treatment effect of patients.

[0005] In summary, the three-dimensional color retinal image has significant advantages in terms of visual effects, spatial perception, diagnostic accuracy, teaching and training, and doctor-patient communication. Therefore, developing a method for three-dimensional retinal reconstruction that fuses fundus color images and OCT images has important clinical significance and application value. Summary of the Invention

[0006] The object of the present invention is: for the medical image data generated by a color fundus camera and an ophthalmic OCT device, to provide a method for three-dimensional retinal reconstruction that fuses fundus color images and OCT images to accurately present the three-dimensional color spatial structure of the retina and improve the accuracy of clinical diagnosis and treatment. To achieve the above technical object, the technical solution adopted by the present invention is as follows:

[0007] A method for three-dimensional retinal reconstruction that fuses fundus color images and OCT images includes the following steps:

[0008] S1. Segment the retinal layer structure in the ophthalmic OCT image to determine the retinal reconstruction area;

[0009] S2. Convert all the data within the retinal reconstruction area into a sparse three-dimensional point cloud;

[0010] S3. Densify the sparse three-dimensional point cloud;

[0011] S4. Extract the RGB colors in the fundus color image and perform color rendering on the point cloud of the retinal NFL layer;

[0012] S5. Perform pseudo-color transformation on the point cloud of other retinal layers to obtain the final three-dimensional color retinal image.

[0013] Further, the segmentation steps in step S1 are as follows:

[0014] S1.1. Preprocess the OCT image to construct an algorithm environment;

[0015] S1.2. Solve the gray-scale gradient equation to obtain the gray-scale gradient value of the image;

[0016] S1.3. Correct the gray-scale gradient value through the gradient correction equation and segment the true boundary of the image.

[0017] Further, step S2 is specifically as follows:

[0018] In the Figure 3 OCT image shown, the scanning laser can achieve one-dimensional depth imaging (A-Scan) at a certain point on the retina. By continuously acquiring A-Scans in the fast axis direction, tomographic imaging of biological tissues (B-Scan) can be achieved, and continuous acquisition of B-scans in the slow axis direction is three-dimensional tomographic imaging. In the present invention, a three-dimensional rectangular coordinate system is established with the A-Scan as the X-axis, the B-Scan as the Y-axis, and the depth direction as the Z-axis, and the data within all the retinal reconstruction regions is converted into three-dimensional point clouds in the established three-dimensional rectangular coordinate system.

[0019] Further, step S3 is specifically as follows:

[0020] The sparse three-dimensional point cloud is densified by using a three-dimensional space filling algorithm to achieve the matching of the spatial resolution of OCT image data and fundus color images. The calculation formula of the three-dimensional space filling algorithm is:

[0021]

[0022] In the formula, I(P0) is the gray-scale value of the point P0 to be filled, I(P i ) is the gray-scale value of the point P i adjacent to P0, w i is the weight value of the point P i , and N is the total number of points adjacent to P0 (as shown in Figure 4 ). The core idea of the three-dimensional space filling algorithm is that the gray-scale value of the point to be filled is jointly determined by the gray-scale values of all adjacent points, and different weight values are assigned to different adjacent points according to the distance from the point P0. The calculation formula of the weight value w i is:

[0023]

[0024] In the formula, L i is the Euclidean distance between the point P0 and the point P i . Among them, It can be seen from the calculation formula of the weight value w i that the closer the Euclidean distance between the adjacent point P i and the point P0 to be filled, the greater the weight value of this point, and vice versa, the smaller the weight value. Such a design can make the point cloud closer to the point to be filled have a greater impact on the gray-scale value of the filled point, and the point cloud farther from the point to be filled has a smaller impact on the gray-scale value of the filled point, so as to achieve the point cloud densification effect closest to the true state.

[0025] Further, step S4 is specifically as follows:

[0026] Extract the RGB three channels of the two-dimensional reconstruction region in the fundus color image to obtain the RGB color values of each pixel point, and then use the RGB color to stain the NFL layer point cloud in the dense three-dimensional point cloud, without staining other point cloud data.

[0027] Further, step S5 is specifically as follows:

[0028] The RGB colors extracted from the fundus color image are only used to stain the point cloud of the retinal NFL layer. For the point clouds of other layers of the retina (from the GCL-IPL layer to the RPE layer), a pseudo-color transformation is performed using a set of color transformation transfer functions. The set of color transformation transfer functions can use different transformation functions to calculate the corresponding red, green, and blue channel values from the gray value of the point cloud, and then synthesize the values of the three channels into a corresponding color, thereby changing the gray point cloud into a multi-colored point cloud. The calculation formula of the set of color transformation transfer functions is:

[0029]

[0030] In the formula, I represents the gray value of the point cloud, and R, G, and B respectively represent the values of the red, green, and blue channels. The value ranges of I, R, G, and B are all 0 to 255. According to the color mapping diagram drawn by the set of color transformation transfer functions ( Figure 5 ), it can be seen that during the process of transforming the gray point cloud into a multi-colored point cloud, low gray values are transformed into black-blue, medium gray values are transformed into green-yellow, and high gray values are transformed into red-white, and the color change is continuous and gradual. Such a design can enhance tiny structures and medium-low contrast regions, make structures that were originally difficult to distinguish more obvious, reduce the visual fatigue of doctors when observing images, and improve the accuracy and efficiency of diagnosis.

[0031] Compared with the prior art, the present patent application has the following beneficial effects:

[0032] 1. Accurately present the color three-dimensional structure of the retina. The prior art usually relies on a single modality for reconstruction and it is difficult to provide both the precise three-dimensional structure and color information of the retina at the same time. This method can more comprehensively present the anatomical structure and color characteristics of the retina by fusing the depth information of OCT images and the color information of fundus images, which helps doctors observe the lesion area more intuitively, thereby providing a more reliable basis for the selection of treatment plans.

[0033] 2. Enhance the density and continuity of 3D reconstruction. Use a 3D space filling algorithm to densify the sparse 3D point cloud, achieve the matching of the spatial resolution between OCT image data and fundus color images, and fill the data missing problem in OCT images caused by a large sampling interval. Compared with the existing technology, the 3D model generated by this method is smoother and more continuous, and can more accurately reflect the microscopic structure of the retina.

[0034] 3. Achieve real sensory color reconstruction and effective fusion of color information. Use real RGB color rendering and pseudo-color transformation for color reconstruction of the NFL layer and other layers of the retina respectively, and effectively fuse various color information skillfully. Compared with the existing technology, the color reconstruction of the 3D image achieved by this method has a more intuitive and realistic visual effect, which helps to improve the recognition rate and the doctor's diagnosis efficiency.

[0035] 4. Reduce manual intervention and improve the degree of automation. This method reduces the need for manual intervention and the possibility of subjective errors through an automated segmentation, interpolation, and rendering process. Compared with the traditional method that requires manual parameter adjustment or region annotation, this method has a higher degree of automation and can significantly improve work efficiency.

[0036] 5. Applicable to various ophthalmic diseases. This method is not only applicable to the 3D reconstruction of healthy retinas but also can effectively handle the reconstruction problems of diseased retinas. Compared with the existing technology, this method has stronger robustness in complex diseased situations and can provide support for the diagnosis and treatment of various ophthalmic diseases. Brief Description of the Drawings

[0037] Figure 1 It is a flowchart of the method in an embodiment of a method for 3D reconstruction of the retina by fusing fundus color images and OCT images according to the present invention;

[0038] Figure 2 It is a schematic diagram of the structure of the human eye retina layer in an embodiment of a method for 3D reconstruction of the retina by fusing fundus color images and OCT images according to the present invention;

[0039] Figure 3 It is a schematic diagram of the construction of a sparse 3D point cloud in an embodiment of a method for 3D reconstruction of the retina by fusing fundus color images and OCT images according to the present invention;

[0040] Figure 4 It is a schematic diagram of a 3D space filling algorithm in an embodiment of a method for 3D reconstruction of the retina by fusing fundus color images and OCT images according to the present invention;

[0041] Figure 5 It is a schematic diagram of retina staining and pseudo-color transformation in an embodiment of a method for 3D reconstruction of the retina by fusing fundus color images and OCT images according to the present invention; Detailed implementation mode

[0042] The process of the present invention is as follows Figure 1 shown. First, the retinal layer structure in the ophthalmic OCT image is segmented to determine the retinal reconstruction area; secondly, all the data within the retinal reconstruction area is converted into a sparse three-dimensional point cloud; then, a three-dimensional space filling algorithm is used to densify the sparse three-dimensional point cloud to achieve the matching of the spatial resolution between the OCT image data and the fundus color image; finally, the RGB colors extracted from the fundus color image are used to stain the point cloud of the retinal NFL layer, and the pseudo-color transformation is performed on the point cloud of other retinal layers using the color transformation transfer function group, and the final retinal color three-dimensional image can be obtained. In order to enable those skilled in the art to better understand the present invention, the specific implementation process of the technical solution of the present invention will be described below with reference to the accompanying drawings.

[0043] As Figures 1-5 shown, a three-dimensional retinal reconstruction method that fuses the fundus color image and the OCT image includes the following steps:

[0044] S1. Segment the retinal layer structure in the ophthalmic OCT image to determine the retinal reconstruction area;

[0045] A fast image segmentation method is used to quickly and accurately segment the 7-layer structure of the retina. The 7-layer structure of the retina from top to bottom is: nerve fiber layer (NFL), ganglion cell layer of the retina and inner plexiform layer complex (GCL-IPL), inner nuclear layer (INL), outer plexiform layer (OPL), outer nuclear layer and inner photoreceptor layer complex (ONL-IS), outer photoreceptor layer (OS), retinal pigment epithelium layer (RPE). The 7-layer structure of the retina can be recognized by 8 boundaries: ILM, NFL-GCI, GCI-INL, INL-OPL, OPL-ONL, IS-OS, IRPE, and OBM. The retinal reconstruction area described in the present invention is the retinal area between ILM and OBM, that is, the area between the NFL layer and the RPE layer.

[0046] The segmentation steps are as follows:

[0047] S1.1. Preprocess the OCT image to construct an algorithm environment;

[0048] In S1.1, in order to make the gray-scale gradient attribute of the image more prominent and enhance the adaptive ability of the algorithm to various modal images, multi-scale transformation and Gaussian low-pass filtering need to be performed on the image. Gaussian low-pass filtering helps to enhance the gray-scale potential energy attribute of the image to be detected, making the edge gray-scale distribution closer to the requirements of the algorithm. Multi-scale transformation can not only reveal the structural information of biological tissues at different resolution levels, but also effectively improve the universality of the algorithm.

[0049] S1.2. Solve the gray-scale gradient equation to obtain the gray-scale gradient value of the image;

[0050] In S1.2, the gray-scale gradient equation is where PE i is the gray-scale gradient value of pixel point i, V i is the gray-scale gradient trend value of pixel point i, δ is the trend regulation factor, is the gradient trend base number. According to the gray-scale gradient equation, the gray-scale gradient values of all pixel points in the OCT image can be obtained. The larger the gray-scale gradient value, the greater the probability that the point is a boundary point.

[0051] S1.3. Correct the gray-scale gradient value through the gradient correction equation and segment the true boundary of the image.

[0052] In S1.3, what the image obtains after being processed by the gray-scale gradient equation is a set of point sets containing the boundary line. If the boundary line is to be extracted from this set of point sets, the gradient correction equation needs to be used. The gradient correction equation is: where EDGE i is the final segmentation result. When EDGE i = 1, pixel point i is a boundary point. When EDGE i = 0, pixel point i is not a boundary point. K is the morphological determination factor, and its value is the ratio of the slope at the back side of the point to be measured to the slope at the front side of the point to be measured; C is the gray-scale determination factor, and its value is the ratio of the average gray-scale value at the front side of the point to be measured to the average gray-scale value at the back side of the point to be measured; only when both K and C are greater than 1, pixel point i can be regarded as a true boundary point.

[0053] S2. Convert all data within the retinal reconstruction area into a sparse three-dimensional point cloud;

[0054] As Figure 3 shown in the OCT image, the scanning laser can achieve one-dimensional depth imaging (A-Scan) at a certain point on the retina. By continuously collecting A-Scans in the fast axis direction, tomographic imaging of biological tissues (B-Scan) can be achieved, and continuous collection of B-scans in the slow axis direction is three-dimensional tomographic imaging. In the present invention, the A-Scan is used as the X axis, the B-Scan is used as the Y axis, and the depth direction is used as the Z axis to establish a three-dimensional rectangular coordinate system, and all data within the retinal reconstruction area are converted into a three-dimensional point cloud in the established three-dimensional rectangular coordinate system. Since the OCT image is scanned line by line, neither between A-Scans nor between B-Scans is continuous, so the generated three-dimensional point cloud is relatively sparse.

[0055] S3. Densify the sparse three-dimensional point cloud;

[0056] Fundus color images are dense two-dimensional color images, and OCT images are sparse three-dimensional point cloud data. There is a problem of mismatched spatial resolution between fundus color images and OCT images. For example, when the OCT image and the fundus color image image the same area, the resolution of the fundus color image is 1000×1000 (length×width), while the resolution of the OCT image is 500×500×1000 (A-Scan×B-Scan×depth). At this time, the resolution of the OCT image is significantly lower than that of the fundus color image. In order to make the spatial resolutions of the two image data consistent, it is necessary to increase the resolution of the OCT image to 1000×1000×1000. The process of increasing the resolution of the OCT image is the process of densifying the sparse three-dimensional point cloud. The present invention uses a three-dimensional space filling algorithm to densify the sparse three-dimensional point cloud and achieve the matching of the spatial resolution of the OCT image data and the fundus color image. The calculation formula of the three-dimensional space filling algorithm is:

[0057]

[0058] In the formula, I(P0) is the gray value of the point P0 to be filled, and I(P i ) is the gray value of the point P i adjacent to P0, w i is the weight value of the point P i , and N is the total number of points adjacent to P0 (as Figure 4 shown). The core idea of the three-dimensional space filling algorithm is that the gray value of the point to be filled is jointly determined by the gray values of all adjacent points, and different weight values will be assigned to different adjacent points according to the distance from the point P0. The calculation formula of the weight value w i is:

[0059]

[0060] In the formula, L i is the Euclidean distance between the point P0 and the point P i , where From the calculation formula of the weight value w i , it can be seen that the closer the Euclidean distance between the adjacent point P i and the point P0 to be filled, the greater the weight value of this point, and vice versa, the smaller the weight value. Such a design can make the point cloud closer to the point to be filled have a greater influence on the gray value of the filled point, and the point cloud farther from the point to be filled has a smaller influence on the gray value of the filled point, so as to achieve the point cloud densification effect closest to the real state.

[0061] S4. Extract the RGB colors in the fundus color image and perform color rendering on the point cloud of the retinal NFL layer;

[0062] Extract the RGB three color channels of the two-dimensional reconstruction area in the fundus color image to obtain the RGB color values of each pixel point, and then use the RGB color to stain the NFL layer point cloud in the dense three-dimensional point cloud, without staining other point cloud data. Such a design can not only truly display the color distribution of the retinal surface layer, but also highlight the subtle differences in the retinal surface layer, helping doctors more easily identify lesions and abnormal areas and assisting doctors in making more accurate diagnoses.

[0063] S5. Perform pseudo-color transformation on the point clouds of other retinal layers to obtain the final three-dimensional color image of the retina.

[0064] The RGB colors extracted from the fundus color image are only used to stain the NFL layer point cloud of the retina. For the point clouds of other retinal layers (from the GCL-IPL layer to the RPE layer), the present invention uses a set of color transformation transfer functions for pseudo-color transformation. The set of color transformation transfer functions can use different transformation functions to calculate the corresponding red, green, and blue channel values from the gray value of the point cloud, and then synthesize the values of the three channels into a corresponding color, thereby changing the gray point cloud into a multi-colored point cloud. The calculation formula of the set of color transformation transfer functions is:

[0065]

[0066]

[0067] In the formula, I represents the gray value of the point cloud, and R, G, and B respectively represent the values of the red, green, and blue channels. The value ranges of I, R, G, and B are all 0 to 255. According to the color mapping diagram ([ Figure 5 ) obtained from the set of color transformation transfer functions, during the process of transforming the gray point cloud into a multi-colored point cloud, low gray values are transformed into black-blue, medium gray values are transformed into green-yellow, and high gray values are transformed into red-white, and the color change is continuous and gradual. Such a design can enhance small structures and medium-low contrast areas, make structures that were originally difficult to distinguish more obvious, reduce the visual fatigue of doctors when observing images, and improve the accuracy and efficiency of diagnosis.

[0068] In summary, the present invention realizes three-dimensional color modeling of the retina through operations such as retinal layer segmentation, establishment of a sparse three-dimensional point cloud of the retina, densification of the sparse three-dimensional point cloud, staining of the NFL layer point cloud of the retina, and pseudo-color transformation of the point clouds of other layers, and obtains a three-dimensional color image of the retina that can be directly observed in three dimensions. The three-dimensional color image of the retina obtained by the present invention can visually display the three-dimensional anatomical structure of the retina to doctors, making it easier for doctors to identify lesions and normal biological tissue structures, improving the ability to identify complex diseases, reducing the subjective judgment error of doctors, and having important clinical significance and value.

[0069] The above embodiments are only illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A three-dimensional retinal reconstruction method that fuses fundus color images and OCT images, characterized in that, It includes the following steps: S1. Segment the retinal layer structure in the ophthalmic OCT image to determine the retinal reconstruction area; S2. Convert all data within the retinal reconstruction area into a sparse three-dimensional point cloud; S3. Densify the sparse three-dimensional point cloud; S4. Extract the RGB colors in the fundus color image and perform color rendering on the point cloud of the retinal NFL layer; S5. Perform pseudo-color transformation on the point cloud of other retinal layers to obtain the final retinal color three-dimensional image.

2. The three-dimensional retinal reconstruction method integrating fundus color images and OCT images as claimed in claim 1, wherein The segmentation steps in step S1 are as follows: S1.

1. Preprocess the OCT image to construct the algorithm environment; S1.

2. Solve the gray-scale gradient equation to obtain the gray-scale gradient value of the image; S1.

3. Correct the gray-scale gradient value through the gradient correction equation and segment the true boundary of the image.

Images