A method for detecting the boundary of macular hole in fundus based on multimodal images

Abstract

The present invention discloses a method for detecting the boundary of fundus macular hole based on multimodal images, specifically relating to the technical field of medical image processing, including data preparation, splitting the original image into an IR image and an OCT B-scan image; center positioning, detecting the circumferential scan line and determining the position of the circumferential scan center through a circumferential scan center point positioning algorithm for the IR image; cross-modal mapping, obtaining the coordinates of the highest point on the edge of the macular hole in the OCT B-scan image through the method of key point detection, and then mapping the highest point on the edge of the macular hole in the OCT B-scan image to the IR image through a cross-modal mapping algorithm; boundary fitting, after being processed by the cross-modal mapping algorithm, the highest points on the edge of the macular hole in the OCT B-scan image corresponding to all scan lines in the IR image are mapped to the IR image, and connecting these points with a straight line, the boundary of the macular hole can be seen in the IR image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly relates to a method for detecting the boundary of a macular hole in the fundus based on multi-modal images. Background Art

[0002] A macular hole (MH) refers to a localized full-thickness defect in the neurosensory retina of the macula. According to the cause of onset, it can be divided into idiopathic and secondary macular holes. The cause of the former is unknown, and the latter is mostly secondary to eye trauma, macular degeneration, high myopia and other diseases. Idiopathic full-thickness macular hole (IFTMH) is a complete retinal defect from the inner limiting membrane to the retinal pigment epithelium layer at the fovea of the macula, which is one of the more common fundus diseases in ophthalmic clinics. The symptoms of this disease mainly include decreased vision, metamorphopsia, central scotoma, and visual field defect in patients, which have an extremely serious impact on the quality of life of patients. Vitrectomy combined with internal limiting membrane peeling is the surgical method for treating macular holes. The anatomical closure rate after MH surgery can reach more than 90%, but the postoperative visual acuity recovery still varies from person to person.

[0003] Clinicians measure the size of macular holes by means of OCT (Optical Coherence Tomography) scans. As Figure 1 shown in the image collected by the Heidelberg OCT fundus examination device in Germany, the left side is an infrared (IR) image, and the right side is an OCT B-scan image. During the acquisition process, first, the center position of the macular hole is determined on the IR image, and then a series of OCT B-scan images are obtained by circular scanning. This image is a cross-sectional view of the macular hole, which can express the morphology and size information of the macular hole and is an important basis for clinical diagnosis and evaluation of the postoperative recovery of macular holes.

[0004] During the internal limiting membrane peeling operation, the real-time detection of the boundary of the macular hole can assist doctors in locating the position of the hole. However, there is currently no relevant research on real-time positioning of the boundary of macular holes during macular hole surgery, nor is there any relevant research on cross-modal detection of the boundary of macular holes.

[0005] In 2023, a method for detecting the area of central serous chorioretinopathy was proposed. This method first calculates the endpoint coordinates at both ends of each circular scan line, constructs the linear equation of each circular scan line, and then maps the boundary points of the central serous chorioretinopathy area detected in the OCT B-scan image to the corresponding circular scan line according to the proportional relationship between the size of the OCT B-scan image and the length of the corresponding circular scan line, so as to obtain the boundary of the central serous chorioretinopathy area. However, this method has the following deficiencies: (1) This method cannot be applied when there are incomplete scan lines in the circular scan image. When any endpoint of the scan line is missing, the boundary points of the central serous chorioretinopathy area detected in the OCT B-scan image cannot be correctly projected. (2) The algorithm is relatively complex. This algorithm needs to perform mapping operations on each scan line and the corresponding OCT B-scan image, and positioning errors are easily introduced during the operation process. Summary of the Invention

[0006] Therefore, in view of the precise treatment goal during macular hole surgery and the deficiencies of the above scheme, the present invention proposes a method for detecting the boundary of fundus macular hole based on multimodal images, which maps the highest point coordinates of the macular hole edge detected on the OCT B-scan image to the IR image, and displays the boundary of the macular hole on the IR image. The method proposed by the present invention first detects the circular scan center of the macular hole, then obtains the endpoint positions of the vertical scan lines, and according to the proportional relationship between the length of the circular scan line and the size of the OCT B-scan image, maps all the highest point coordinates of the macular hole edge detected on the OCT B-scan image to the vertical scan line, and then rotates the mapped coordinates according to the angle of the corresponding scan line, so as to obtain the boundary of the macular hole. This method has the following advantages: 1) Since the vertical scan line is generally complete during the image acquisition process, this method is not restricted when the length of some scan lines is incomplete. 2) The algorithm is simple. This method does not need to locate each scan line, but maps all the highest points of the macular hole edge on the OCT B-scan image to the vertical scan line, and obtains the mapped boundary through rotation operation. Since the calculation of this method is relatively simple, it avoids the error caused by the square root operation of calculating the scan line endpoint coordinates.

[0007] Specifically as follows: A method for detecting the boundary of fundus macular hole based on multimodal images, comprising:

[0008] Data preparation, splitting the original image into an IR image and an OCT B-scan image;

[0009] Center positioning, detecting the circular scan line and determining the position of the circular scan center through the circular scan center point positioning algorithm for the IR image;

[0010] Cross-modal mapping obtains the coordinates of the highest point on the edge of the macular hole in the OCT B-scan image through the method of key point detection, and then maps the highest point on the edge of the macular hole in the OCT B-scan image to the IR image through the cross-modal mapping algorithm. The method of key point detection adopted in the present invention is the YOLOv8 algorithm.

[0011] Boundary fitting: After being processed by the cross-modal mapping algorithm, the highest points on the edge of the macular hole in the OCT B-scan image corresponding to all the scan lines on the IR image will be correctly mapped to the IR image. Connecting these points with straight lines can show the boundary of the macular hole on the IR image.

[0012] Preferably, the IR image and the OCT B-scan image are split, and the split images are named according to the corresponding relationship. The infrared images are named IR1 to IR12, and the corresponding OCT B-scan images are named OCT1 to OCT12. In the IR image, the green straight lines with arrows indicate the positions of the 12 circular scan lines. The highlighted green color indicates the circular scan line corresponding to the current OCT B-scan image. The scanning direction of the OCT B-scan image corresponding to the straight line from the tail to the head of the arrow is from left to right. The resolution of the unsplit image is 1264*596. After splitting, the resolution of the IR image is 496*496, and the resolution of the OCT B-scan image is 750*496, with the unit being pixels. The image splitting formula is as follows:

[0013]

[0014] where I (x,y) represents the coordinates of the pixels of the image to be split, I IR represents the pixel coordinates belonging to the IR image, I OCT represents the pixel coordinates belonging to the OCT B-scan image. Through the splitting operation, the image collected by the OCT scanning device can be split into the IR image and the OCT B-scan image and they are corresponding one by one.

[0015] To map the highest point of the macular hole edge in the OCTB-scan image to the IR image, it is necessary to establish coordinate systems in the two modalities respectively, find their corresponding proportional relationships, so that any point on the OCTB-scan image can find the corresponding point on the IR image. The highlighted green scan line on the IR image corresponds to the scanning direction of the OCTB-scan image from left to right from the arrow tail to the arrow head, that is, the starting point of the scan line corresponds to the first column of pixels of the OCTB-scan image, and the end point of the scan line corresponds to the last column of pixels of the OCTB-scan image. In the IR image, all scan lines are green and intersect at the center of the circular scan. Therefore, the present invention proposes a circular scan center point positioning algorithm for the IR image. First, convert the color space and create a mask, then find all green circular scan lines through the edge detection algorithm, and then find the circular scan center through the Hough transform. The specific steps are as follows:

[0016] a. In the IR image, the morphology of the fundus blood vessels will introduce extra interference during edge detection. By converting the color space and creating a mask, it is ensured that only the edges presented in green are detected in the circular scan line detection step, laying a foundation for subsequent research;

[0017] b. Select the Canny algorithm for circular scan line detection; the Canny algorithm is one of the currently most excellent and commonly used edge detection algorithms. This algorithm is not easily affected by noise and can identify strong edges and weak edges in the image, playing an important role in edge detection applications, including the following steps:

[0018] b1. Image denoising: Use Gaussian filtering to smooth the image, and use a 5*5 Gaussian filter of the following formula (2);

[0019]

[0020] b2. Calculate the image gradient: Calculate the gradient direction and amplitude of the gray value of each pixel in the image to detect possible image edges; specifically: Detect the edges in the x direction and y direction of the image through the Sobel operator respectively, and then calculate the direction θ and amplitude G of the gradient using formula (3) and formula (4). In the formula, I x represents the gray value of the pixel in the x direction, and I y represents the gray value of the pixel in the y direction; the Sobel operator extracts the image edge through the discrete differential method;

[0021]

[0022]

[0023] b3. Non-maximum suppression: The spurious responses brought about by edge detection can be eliminated through the non-maximum suppression algorithm. The regions with gray-scale changes are usually concentrated. Among the pixels in the local range along the gradient direction, only the pixels with the maximum gray-scale gradient are retained, and the others are not, which can remove a large number of points and turn the edge with multiple pixel widths into an edge with a single pixel width.

[0024] b4. Double-threshold screening: After non-maximum suppression, there are still many possible edge points. The double-threshold method is applied to divide strong edges and weak edges. The pixels with gray-scale gradients greater than the high threshold are retained as strong edges, and the pixels with gray-scale gradients less than the low threshold are removed. The pixels with gray-scale gradients between the high and low thresholds are screened as weak edges. Search in the neighborhood of the weak edges. If there is a strong edge, the weak edge is retained; if not, the weak edge is deleted to complete edge detection.

[0025] All the circumferential scan lines can be detected through the Canny algorithm, laying a foundation for subsequent detection of the circumferential scan center.

[0026] c. After the circumferential scan line detection is completed, a series of scan lines intersecting at the circumferential scan center are transformed into the parameter space through the Hough transform, ensuring that each circumferential scan line is detected as a straight line, and the intersection points of multiple straight lines are obtained, thereby detecting the circumferential scan center. The Hough transform is an effective image processing technique, especially suitable for detecting geometric shapes in images, such as straight lines. It maps the points in the image to the parameter space (Hough space) and searches for peaks in this space to determine the position of the geometric shape. The Hough transform has high robustness to noise and local defects. The Hough transform searches for the most likely straight lines through an accumulator. Even if the straight lines in the image are discontinuous or contaminated by noise, the Hough transform can effectively detect these straight lines. Moreover, the process of finding the intersection points of straight lines in the parameter space is simpler and more direct mathematically.

[0027] Through the Hough transform, calculating the intersection points of straight lines is simplified to solving a system of linear equations, rather than performing complex calculations directly in the image space.

[0028] The Hough line transform formula is as follows:

[0029] y i =a i x + b i (5)

[0030] b i = -a i x + y i (6)

[0031] Formula (5) is the equation of the i-th straight line in the rectangular coordinate system; formula (6) is to transform the equation of the i-th straight line in the rectangular coordinate system into the equation of a straight line in the (Hough space) parameter space.

[0032] After the Hough transform, a straight line in the x-y coordinate system is converted into a point in the a-b coordinate system;

[0033] The basic principle of Hough line detection is to utilize the duality between points and lines. A straight line in the image space corresponds one-to-one with a point in the parameter space, and a straight line in the parameter space also corresponds one-to-one with a point in the image space. Therefore, the Hough line detection method transforms the problem of straight line detection in the image space into the problem of point detection in the parameter space, and completes the straight line detection task by finding peaks in the parameter space. After the Hough transform, all the straight lines intersecting at a point in the x-y coordinate system are a series of points located on the same straight line in the parameter space. In the present invention, after the Hough transform, a series of scan lines intersecting at the center of the circular scan are converted into multiple collinear points in the parameter space.

[0034] In the parameter space in polar coordinate form, each straight line is represented as shown in formula (7):

[0035] ρ i = xcos(θ i ) + ysin(θ i ) (7)

[0036] where ρ i , θ i are respectively the parameters of the i-th straight line in the polar coordinate system; ρ i is the distance parameter, representing the distance between the straight line and the origin; θ i is the angle parameter, which is the angle between the perpendicular line from the straight line to the origin and the positive half of the x-axis;

[0037] The intersection coordinates of any two straight lines are obtained using formula (8), and then the average value of all the obtained intersection coordinates is calculated to reduce the error, obtaining the intersection coordinates (x0, y0), and this coordinate is the coordinate of the center of the circular scan; the meanings of the parameters in formula (8) are the same as those in formula (7).

[0038]

[0039] Preferably, the cross-modal mapping algorithm includes the following steps:

[0040] d. Coordinate preprocessing: The abscissas of the leftmost and rightmost highest points on the edge of the macular hole in the OCT B-scan image are stored as an array in the following form:

[0041] (i, X_Left i , X_Right i ) (9)

[0042] Among them, i refers to the scanning line order. The vertical scanning line is defined as the first scanning line, and the second scanning line is obtained by rotating 15 degrees clockwise, and so on; X_Left i , X_Right i respectively refer to the abscissas of the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the i-th scanning line;

[0043] e. Cross-modal mapping: All scanning lines on the IR image intersect at the same point, that is, the circumferential scanning center, which is denoted as point O with coordinates (x0, y0); Since some scanning lines are incomplete at both ends, and the circumferential scanning center must be on the IR image, the mapping algorithm adopted in this method is based on the center point, and the algorithm flow of cross-modal mapping is as follows:

[0044] e1. Calculate the proportional relationship: Calculate the ratios of the abscissas of the two highest points to the width of the OCT B-scan image, that is:

[0045]

[0046] Among them, Ratio_Left i , Ratio_Right i respectively represent the results of normalizing the abscissas of the left and right highest points on the edge of the macular hole on the i-th scanning line relative to the width of the OCT B-scan image, X_Left i , X_Right i respectively represent the values of the abscissas of the left and right highest points on the edge of the macular hole on the i-th scanning line, OCT_Width represents the width of the OCT B-scan image, and this value is 750 pixels in the present invention.

[0047] e2. Calculate the length of the scanning line: Since the lengths of all scanning lines are the same, and the midpoints of the scanning lines intersect at the same point, that is, the center point, as long as one end point of one scanning line can be determined, the lengths of all scanning lines can be obtained. In the previous step, all circumferential scanning lines and the circumferential scanning center (x0, y0) have been detected by the Hough transform. In this step, the length of the scanning line is calculated according to the distance from the end point of the vertical scanning line to the circumferential scanning center in the y direction. The length of the vertical scanning line is calculated using formula (11):

[0048] L = 2 * (y top - y0) (11)

[0049] Among them, L is the length of the scanning line, y top is the ordinate of the highest point pixel of the vertical scanning line, and y0 is the ordinate of the center point; The traditional algorithm uses formula (12) to solve the length of the scanning line:

[0050]

[0051] where x i1 , x i2 , y i1 , y i2 are the abscissa and ordinate of the starting point 1 and the ending point 2 of the i-th scan line respectively;

[0052] Compared with the traditional algorithm, when solving the length of the scan line according to formula (11) in this step, the square root operation is not used, which reduces the error caused thereby. And since the lengths of each scan line are the same, it is only necessary to calculate the length of one scan line, which has great convenience in terms of operation.

[0053] e3. Vertical mapping: According to the proportional relationship calculated in step e1, map the highest point on the edge of the macular hole in the OCT B-scan image to the vertical scan line of the IR image, using formula (13):

[0054]

[0055] where x0, y0 represent the abscissa and ordinate of the center point of the circumferential scan; x oct_left_i , x oct_right_i are the abscissas of the corresponding points on the vertical scan line of the IR image after mapping the abscissas of the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line, and their values are the same as x0; y oct_left_i , y oct_right_i are the ordinates of the corresponding points on the vertical scan line of the IR image after mapping the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line; L is the length of the scan line; in this step, all the highest points on the edge of the macular hole in the OCT B-scan image are mapped to the vertical scan line;

[0056] f. Rotation: In the previous steps, all the highest points on the edge of the macular hole in the OCT B-scan image have been mapped to the vertical scan line. In this step, the points mapped to the vertical scan line need to be rotated to the scan line at the corresponding angle; for the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line, after the previous steps, the abscissa and ordinate on the vertical scan line of the IR image are obtained, in the form shown in formula (14):

[0057] (i, x oct_left_i , y oct_left_i , x oct_right_i , y oct_right_i )(14)

[0058] where i is the i-th scan line, x oct_left_i , yoct_left_i are the horizontal and vertical coordinates of the corresponding point after mapping the highest point on the left side of the macular hole to the vertical scan line of the IR image; x oct_right_i , y oct_right_i are the horizontal and vertical coordinates of the corresponding point after mapping the highest point on the right side of the macular hole to the vertical scan line of the IR image;

[0059] According to the rotation angle of formula (15), the points mapped on the vertical scan line are rotated around the ring scan center O point (x0, y0), so that the highest point on the edge of the macular hole on the OCT B-scan image corresponding to each scan line is rotated from the vertical scan line to the corresponding scan line;

[0060] θ i =(i - 1)*180 / n (15)

[0061] In formula (15), θ i is the angle that the i-th scan line needs to rotate around the ring scan center O point, and n is the total number of scan lines;

[0062] Perform a rotation operation on the points mapped on the vertical scan line according to formula (16) to make the points fall on the scan lines corresponding to the corresponding angles;

[0063]

[0064] Among them, x′, y′ are the horizontal and vertical coordinates of the rotated point, x, y are the horizontal and vertical coordinates of the point before rotation, x0, y0 are the horizontal and vertical coordinates of the rotation center, and θ is the rotation angle;

[0065] According to formula (16), let the point with coordinates (x, y) rotate clockwise by θ degrees around the rotation center (x0, y0).

[0066] The present invention has the following advantages:

[0067] In terms of application, the present invention realizes mapping the coordinates of the highest point on the edge of the macular hole on the OCT B-scan image to the IR image, and displaying the boundary of the macular hole on the IR image. The purpose of the present invention is to make up for the deficiencies of the existing fundus lesion area detection scheme in the detection of the macular hole boundary, assist ophthalmologists in quickly and accurately locating the macular hole boundary, and promote the precision medicine of fundus macular holes.

[0068] In terms of the algorithm, the present invention has the following advantages:

[0069] 1. This method makes up for the deficiency that the existing mapping method cannot be used when there are missing scan lines; when the images collected by the device have incomplete partial scan lines, this method can still be normally applied;

[0070] 2. Simple algorithm: This method does not require positioning each scan line. Instead, all the highest points on the edge of the macular hole in the OCT B-scan image are mapped onto the vertical scan lines of the IR image, and the mapped boundary is obtained through rotation operations. Since the calculation of this method is relatively simple, it avoids the error caused by the square root operation for calculating the endpoint coordinates of the scan line.

[0071] 3. The core of this method lies in detecting the center point of the circular scan rather than the endpoints of the circular scan lines, which greatly reduces the computational complexity. It can be applied to the cases with different numbers of scan lines by simply modifying the parameter of the number of circular scan lines. The algorithm has good robustness and generality. Brief Description of the Drawings

[0072] Figure 1 is an image collected by the Heidelberg OCT fundus examination device provided by the present invention;

[0073] Figure 2 is a flowchart of the method for detecting the boundary of the macular hole in the fundus based on multi-modal images provided by the present invention;

[0074] Figure 3 is a flowchart of the algorithm for locating the center point of the circular scan for the IR image provided by the present invention;

[0075] Figure 4 is a schematic diagram of the color space conversion and the effect of the green mask provided by the present invention;

[0076] Figure 5 is a schematic diagram of the Hough transform provided by the present invention;

[0077] Figure 6 is a schematic diagram of the straight line detection by the Hough transform provided by the present invention;

[0078] Figure 7 is a schematic diagram of the highest point on the edge of the macular hole in the OCT B-scan image provided by the present invention;

[0079] Figure 8 is a schematic diagram of mapping the highest point on the edge of the macular hole in the OCT B-scan image to the IR image provided by the present invention;

[0080] Figure 9 is a flowchart of the mapping algorithm provided by the present invention;

[0081] Figure 10 is a flowchart of the rotation operation provided by the present invention;

[0082] Figure 11 is a schematic diagram of the boundary of the macular hole on the IR image provided by the present invention;

[0083] Figure 12 is the original image collected by the device provided by the present invention;

[0084] Figure 13 Implementation diagrams provided for the present invention;

[0085] Figure 14(a) is the IR image provided for the present invention;

[0086] Figure 14(b) is the OCT B-scan image provided for the present invention;

[0087] Figure 15 IR image circumferential scan center point diagram provided for the present invention;

[0088] Figure 16 Cross-modal mapping schematic diagram provided for the present invention;

[0089] Figure 17 Schematic diagram of the position change before and after the rotation of the mapping point provided for the present invention;

[0090] Figure 18 Boundary fitting diagram provided for the present invention;

[0091] Figure 19(a) is 12 circumferential scan images provided for the present invention;

[0092] Figure 19(b) is the boundary detection result diagram on 48 circumferential scan images provided for the present invention. Detailed implementation manners

[0093] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0094] The technical solutions adopted by this embodiment to solve its technical problems are as Figure 2 shown. A method for detecting the boundary of fundus macular hole based on multi-modal images described in this embodiment includes four steps: data preparation, center positioning, cross-modal mapping, and boundary fitting. The specific methods of each step will be described in detail below.

[0095] 1. Data preparation

[0096] As Figure 1As shown, the image collected by the Heidelberg OCT fundus examination device in Germany consists of two parts. The left side is the IR image, and the right side is the OCT B-scan image. For the smooth progress of subsequent steps, it is necessary to split the IR image and the OCT B-scan image, name the split images according to the corresponding relationship, name the infrared images as IR1~IR12, and name the corresponding OCT B-scan images as OCT1~OCT12. In the IR image, the green straight line with an arrow indicates the position of 12 circular scan lines. The highlighted green color indicates the circular scan line corresponding to the current OCT B-scan image. The scanning direction of the OCT B-scan image corresponding to the straight line from the tail to the head of the arrow is from left to right. The resolution of the unsplit image is 1264*596. After splitting, the resolution of the IR image is 496*496, and the resolution of the OCT B-scan image is 750*496, with the unit being pixels. The image splitting formula is as follows:

[0097]

[0098] Among them, I (x,y) represents the coordinates of the pixels of the image to be split, I IR represents the pixel coordinates belonging to the IR image, I OCT represents the pixel coordinates belonging to the OCT B-scan image. Through the splitting operation, the image collected by the OCT scanning device can be split into an IR image and an OCT B-scan image, and they are corresponding one by one.

[0099] 2. Center positioning

[0100] To map the highest point of the edge of the macular hole in the OCT B-scan image to the IR image, it is necessary to establish coordinate systems in two modalities respectively, find their corresponding proportional relationships, so that any point on the OCT B-scan image can find a corresponding point on the IR image. The highlighted green scan line on the IR image, the scanning direction of the OCT B-scan image corresponding to the straight line from the tail to the head of the arrow is from left to right, that is, the starting point of the scan line corresponds to the first column of pixels of the OCT B-scan image, and the end point of the scan line corresponds to the last column of pixels of the OCT B-scan image. In the IR image, all scan lines are green and intersect at the circular scan center. Therefore, the present invention proposes a circular scan center point positioning algorithm for the IR image, and the algorithm flow is as Figure 3 shown. First, convert the color space and create a mask, then all green circular scan lines can be found through the edge detection algorithm, and then the circular scan center can be found through the Hough transform.

[0101] 2.1 Convert the color space and create a mask

[0102] In the IR image, the morphology of fundus blood vessels will introduce redundant interference during edge detection. By converting the color space and creating a mask, it is ensured that only the edges presented in green are detected in the circumferential scan line detection step, laying a foundation for subsequent research.

[0103] Converting the color space means converting the image from the RGB (R represents red, G represents green, B represents blue) color space to the HSV (H represents hue, S represents saturation, V represents value) color space. Through such a conversion, subsequent steps can more effectively detect and segment specific color regions.

[0104] A mask is a binary image used to separate specific colors or regions in image processing and computer vision. To detect the green region in the image, the range of green in the HSV color space needs to be defined. The range of green in the HSV space is from (35, 100, 100) to (85, 255, 255), and the parameters in the parentheses represent hue, saturation, and value respectively. Figure 4 The effect of the IR image containing the green scan line after color space conversion and the green mask is shown. It can be seen that after these two steps, the green scan line is extracted, preparing for subsequent image processing.

[0105] 2.2 Circumferential Scan Line Detection

[0106] In the circumferential scan line detection step, the Canny algorithm is selected for edge detection in this embodiment. The Canny algorithm is one of the currently most excellent and commonly used edge detection algorithms. This algorithm is not easily affected by noise and can identify strong edges and weak edges in the image, playing an important role in edge detection applications. The detection process of this algorithm is divided into the following 4 steps:

[0107] (1) Image denoising. Noise is where the gray level changes greatly and is easily recognized as a false edge. Therefore, Gaussian filtering is used to smooth the image and reduce the noise in the image. Generally, a 5*5 Gaussian filter as shown in formula (2) is used.

[0108]

[0109] (2) Calculate the image gradient and detect possible edges. This step requires calculating the gradient direction and magnitude of the gray value of each pixel in the image. First, the Sobel operator is used to detect the edges in the x and y directions of the image respectively, and then the gradient direction θ and magnitude G are calculated using formula (3) and formula (4). Here, I x represents the gray level of the pixel in the x direction, and I y Similarly. The Sobel operator can effectively extract image edges through the discrete differential method.

[0110]

[0111]

[0112] (3) Non-maximum suppression. The spurious responses brought by edge detection can be eliminated through the non-maximum suppression algorithm. The regions with gray level changes are usually concentrated. Among the pixels in the local range along the gradient direction, only the pixels with the maximum gray level gradient are retained, and the others are not. A large number of points can be removed, and the edge with multiple pixel widths can be turned into an edge with a single pixel width.

[0113] (4) Double-threshold screening. After non-maximum suppression, there are still many possible edge points. The double-threshold method is applied to divide strong edges and weak edges. The pixels with gray level gradient greater than the high threshold are retained as strong edges, and the pixels with gray level gradient less than the low threshold are removed. The pixels with gray level gradient between the high and low thresholds are screened as weak edges. Search in the neighborhood of the weak edges. If there is a strong edge, the weak edge is retained; if not, the weak edge is deleted, and the edge detection is completed.

[0114] All the circumferential scan lines can be detected through the Canny algorithm in this step, laying a foundation for subsequent detection of the circumferential scan center.

[0115] 2.3 Detection of the circumferential scan center

[0116] After the circumferential scan line detection is completed, this method converts a series of scan lines intersecting at the circumferential scan center into the parameter space through the Hough transform, ensuring that each circumferential scan line is detected as a straight line, and finding the intersection points of multiple straight lines, thereby detecting the circumferential scan center. The Hough transform is an effective image processing technology, especially suitable for detecting geometric shapes in images, such as straight lines. It maps the points in the image to the parameter space (Hough space) and finds the peaks in this space to determine the positions of geometric shapes. The Hough transform has high robustness to noise and local defects. The Hough transform finds the most likely straight lines in the parameter space through an accumulator. Even if the straight lines in the image are discontinuous or contaminated by noise, the Hough transform can effectively detect these straight lines. Moreover, the process of finding the intersection points of straight lines in the parameter space is simpler and more direct mathematically. Through the Hough transform, calculating the intersection points of straight lines can be simplified to solving a system of linear equations, rather than performing complex calculations directly in the image space.

[0117] The formula for the Hough line transform is as follows:

[0118] y i =a i x + b i (5)

[0119] b i = -a i x + y i (6)

[0120] Equation (5) is the equation of the i-th straight line in the rectangular coordinate system; Equation (6) is the conversion of the equation of the i-th straight line in the rectangular coordinate system into the equation of a straight line in the Hough space (parameter space). After the Hough transform, a straight line in the x-y coordinate system is converted into a point in the a-b coordinate system, which is shown visually in Figure 5 the figure below.

[0121] The basic principle of the Hough line detection lies in using the duality of points and lines. The straight lines in the image space and the points in the parameter space are in one-to-one correspondence, and the straight lines in the parameter space and the points in the image space are also in one-to-one correspondence. Therefore, the Hough line detection method converts the problem of straight line detection in the image space into the problem of point detection in the parameter space, and completes the straight line detection task by finding the peak value in the parameter space. After the Hough transform, all the straight lines intersecting at a point in the x-y coordinate system are a series of points located on the same straight line in the parameter space, as shown in Figure 6 the figure below. In this case, after the Hough transform, a series of scan lines intersecting at the center of the circular scan are converted into multiple collinear points in the parameter space.

[0122] In the parameter space in polar coordinate form, each straight line is represented by Equation (7):

[0123] ρ i =xcos(θ i )+ysin(θ i ) (7)

[0124] where ρ i , θ i are the parameters of the i-th straight line in the polar coordinate system respectively. ρ i is the distance parameter, representing the distance between the straight line and the origin; θ i is the angle parameter, which is the angle between the perpendicular line from the straight line to the origin and the positive half of the x-axis. The intersection coordinates of any two straight lines are obtained using Equation (8), and then the average value of all the obtained intersection coordinates is calculated to reduce the error, obtaining the intersection coordinates (x0, y0), which are the coordinates of the center of the circular scan. The meanings of the parameters in Equation (8) are the same as those in Equation (7).

[0125]

[0126] 3. Cross-modal mapping

[0127] Before the cross-modal mapping operation, the coordinates of the highest point on the edge of the macular hole in the OCT B-scan image have been obtained through the key point detection method, as shown in Figure 7As shown, the highest point on the edge of the macular hole is represented by a red dot. The red vertical line in the middle is the center line in the horizontal direction of the OCT B-scan image, corresponding to the center point of the circular scan on the IR image.

[0128] As Figure 8 shown, the two highest points on the edge of the macular hole in the OCT B-scan image have been mapped to the vertical circular scan line in the IR image and marked with red dots. The arrow of the vertical circular scan line points upward, indicating that the scanning direction is from bottom to top, and the corresponding scanning direction of the OCT B-scan image is from left to right. Therefore, on the IR image, the points below the circular scan center are mapped from the highest point on the left edge of the macular hole in the OCT B-scan image, and the points above the circular scan center are mapped from the highest point on the right edge of the macular hole in the OCT B-scan image. The mapping algorithm is divided into the following steps, as Figure 9 shown.

[0129] 3.1 Coordinate preprocessing

[0130] The abscissas of the highest points on the left and right edges of the macular hole in the OCT B-scan image are stored as arrays in the following form:

[0131] (i, X_Left i , X_Right i )(9)

[0132] where i refers to the scanning line order. The vertical scanning line is defined as the first scanning line, and the second scanning line is obtained by rotating 15 degrees clockwise, and so on. X_Left i , X_Right i respectively refer to the abscissas of the highest points on the left and right edges of the macular hole in the i-th scanning line of the OCT B-scan image.

[0133] 3.2 Cross-modal mapping

[0134] All the scanning lines on the IR image intersect at the same point, that is, the circular scan center, which is denoted as point O with coordinates (x0, y0). Since some scanning lines are incomplete at both ends, and the circular scan center must be on the IR image, the mapping algorithm adopted in this method is based on the center point. The algorithm flow of cross-modal mapping is as follows:

[0135] 1) Calculate the proportional relationship: Calculate the ratios of the abscissas of the two highest points to the width of the OCT B-scan image. That is

[0136]

[0137] where Ratio_Left i , Ratio_Righti respectively represent the normalized results of the abscissas of the left and right highest points of the macular hole edge on the i-th scan line with respect to the width of the OCT B-scan image, X_Left i , X_Right i respectively represent the values of the abscissas of the left and right highest points of the macular hole edge on the i-th scan line. OCT_Width represents the width of the OCT B-scan image, and this value is 750 pixels in this embodiment.

[0138] 2) Calculate the length of the scan line: Since the lengths of all scan lines are the same and the midpoints of the scan lines intersect at the same point, i.e., the center point, as long as one end point of one scan line can be determined, the lengths of all scan lines can be obtained. In the previous step, all circular scan lines and the circular scan center (x0, y0) have been detected by the Hough transform. In this step, the length of the scan line is calculated based on the distance from the end point of the vertical scan line to the circular scan center in the y direction. The following formula is used to calculate the length of the vertical scan line:

[0139] L = 2 * (y top - y0) (11)

[0140] where L is the length of the scan line, y top is the ordinate of the highest point pixel of the vertical scan line, and y0 is the ordinate of the center point. When the traditional algorithm solves the length of the scan line, the following formula is used:

[0141]

[0142] where x i1 , x i2 , y i1 , y i2 are the abscissa and ordinate of the starting point 1 and the ending point 2 of the i-th scan line respectively.

[0143] Compared with the traditional algorithm, when solving the length of the scan line according to formula (11) in this step, the square root operation is not used, reducing the error caused thereby. And since the lengths of each scan line are the same, only the length of one scan line needs to be calculated, which has great convenience in terms of operation.

[0144] 3) Vertical mapping: According to the proportional relationship calculated in step 1), the highest point of the macular hole edge on the OCT B-scan image is mapped to the vertical scan line of the IR image. The formula used is as follows:

[0145]

[0146] where x0, y0 represent the abscissa and ordinate of the circular scan center point; x oct_left_i , x oct_right_iThey are the abscissas of the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line, mapped to the abscissas of the corresponding points on the vertical scan line of the IR image, and their values are the same as x0; y oct_left_i ,y oct_right_i They are the ordinates of the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line, mapped to the corresponding points on the vertical scan line of the IR image; L is the length of the scan line. In this step, the highest points on the edge of the macular hole in all OCT B-scan images are mapped to the vertical scan line of the IR image.

[0147] 3.3 Rotation

[0148] In the previous step, the highest points on the edge of the macular hole in all OCT B-scan images have been mapped to the vertical scan line of the IR image. In this step, the points mapped to the vertical scan line need to be rotated to the scan line at the corresponding angle. For the two highest points of the i-th scan line, after the previous step, the abscissa and ordinate on the vertical scan line of the IR image are obtained, in the form shown in formula (14):

[0149] (i, x oct_left_i , y oct_left_i , x oct_right_i , y oct_right_i ) (14)

[0150] Among them, i is the i-th scan line, x oct_left_i , y oct_left_i are the abscissa and ordinate of the corresponding point after the left highest point of the macular hole is mapped to the vertical scan line of the IR image; x oct_right_i , y oct_right_i are the abscissa and ordinate of the corresponding point after the right highest point of the macular hole is mapped to the vertical scan line of the IR image. According to the rotation angle of formula (15), the points mapped on the vertical scan line are rotated around the circumferential scan center O point (x0, y0), so that the highest points on the edge of the macular hole in the OCT B-scan image corresponding to each scan line are rotated from the vertical scan line to the corresponding scan line. In formula (15), θ i is the angle that the i-th scan line needs to rotate around the circumferential scan center O point, and n is the total number of scan lines.

[0151] θ i =(i - 1)*180 / n (15)

[0152] Perform a rotation operation on the points mapped on the vertical scan line according to formula (16) so that the points fall on the scan lines corresponding to the corresponding angles. Among them, x′ and y′ are the horizontal and vertical coordinates of the rotated points, x and y are the horizontal and vertical coordinates of the points before rotation, x0 and y0 are the horizontal and vertical coordinates of the rotation center, and θ is the rotation angle. According to formula (16), a point with coordinates (x, y) can be rotated clockwise by θ degrees around the rotation center (x0, y0).

[0153]

[0154] As Figure 10 shown is the flowchart of the rotation operation, taking the boundary points corresponding to the second scan line as an example. The red points represent the positions where the highest points on the edge of the macular hole in the OCT B-scan image are mapped to the vertical scan line, and the yellow points represent the positions of the mapped points after rotating 15 degrees around the circumferential scan center, that is, the positions of the boundary points on the corresponding circumferential scan line. The operations for the remaining scan lines are similar and will not be elaborated here.

[0155] 4. Boundary fitting

[0156] After being processed by the mapping algorithm, the highest points of the macular holes on the OCT B-scan images corresponding to all the scan lines on the IR image will be correctly mapped onto the IR image. Connecting these points with straight lines, the boundary of the macular hole can be seen on the IR image, as Figure 11 shown.

[0157] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the described embodiments.

[0158] The image collected by the Heidelberg OCT fundus examination device in Germany consists of two parts. The left side is the IR image, and the right side is the OCT B-scan image. In order to ensure the smooth progress of the subsequent steps, it is necessary to split the IR image and the OCT B-scan image. The green line in the IR image is the scan line, and the highlighted green line is the scan line corresponding to the current OCT B-scan image. The direction of the scan line is from the tail of the arrow to the head of the arrow, corresponding to the pixels of each column of the OCT B-scan image from left to right. The specific shape of the macular hole is shown in the OCT B-scan image. As Figure 12 shown is the original image collected by the device.

[0159] The implementation steps of the present invention are as Figure 13 shown.

[0160] Data preparation: Split the original image into an IR image and an OCT B-scan image, as shown in Figures 14(a) and 14(b).

[0161] Central positioning: Detect the intersection point of the scanning lines on the IR image, which is the center point of the circumferential scan. As Figure 15 shown, the red word "center" indicates that the red dot is the detected center point of the circumferential scan.

[0162] Cross-modal mapping: First map the highest points on the left and right sides of the macular hole edge on the OCT B-scan image to the vertical scanning lines of the IR image, and then rotate them according to the order of the scanning lines. As Figure 16 shown, the highest points on the left and right sides of the macular hole edge on the OCT B-scan image are mapped to the vertical scanning lines. The L and R marked on the two images represent the highest points on the left and right sides respectively.

[0163] As Figure 17 shown, first map the highest points on the left and right sides of the macular hole edge on the OCT B-scan image corresponding to the second scanning line to the vertical scanning line, as shown by the red dots in the figure, and then rotate them by the corresponding angle around the center point. The rotated points are shown by the yellow dots in the figure. The operation of the i-th scanning line is similar, and it can be rotated according to the corresponding angle, which will not be elaborated here.

[0164] Boundary fitting: After being processed by the mapping algorithm, the highest points of the macular hole on the OCT B-scan image corresponding to all the scanning lines on the IR image will be correctly mapped onto the IR image. Connect these points with straight lines, and the boundary of the macular hole can be seen on the IR image, as Figure 18 shown.

[0165] The method proposed by the present invention is applied to 12 circumferential scan images and 48 circumferential scan images, and good detection effects of the macular hole boundary can be achieved. As Figure 19(a) and 19(b) shown, the boundary detection results on 12 circumferential scan images and 48 circumferential scan images are respectively shown.

[0166] Although the present invention has been described in detail with general descriptions and specific embodiments above, based on the present invention, some modifications or improvements can be made, which are obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection required by the present invention.

Claims

1. A method for detecting the boundary of fundus macular hole based on multimodal images, characterized in that: Including: Data preparation: splitting the original image into an IR image and an OCT B-scan image; Center positioning: detecting the circumferential scan lines and determining the position of the circumferential scan center through a circumferential scan center positioning algorithm for the IR image; Cross-modal mapping: obtaining the coordinates of the highest point on the edge of the macular hole in the OCT B-scan image through the method of key point detection, and then mapping the highest point on the edge of the macular hole in the OCT B-scan image to the IR image through the cross-modal mapping algorithm; Boundary fitting: after being processed by the cross-modal mapping algorithm, the highest points on the edge of the macular hole in the OCT B-scan image corresponding to all scan lines on the IR image are mapped to the IR image, and these points are connected by straight lines, so that the boundary of the macular hole can be seen on the IR image.

2. The method for detecting the boundary of fundus macular hole based on multimodal images according to claim 1, wherein: The circumferential scan center positioning algorithm for the IR image first converts the color space, creates a mask, then finds all green circumferential scan lines through the edge detection algorithm, and then finds the circumferential scan center through the Hough transform. The specific steps are as follows: S21: By converting the color space and creating a mask, ensure that only the edges presented as green are detected in the circumferential scan line detection step; S22: Select the Canny algorithm for circumferential scan line detection; S23: After the circumferential scan line detection is completed, convert a series of scan lines intersecting at the circumferential scan center to the parameter space through the Hough transform, ensure that each circumferential scan line is detected as a straight line, find the intersection points of multiple straight lines, and thus detect the circumferential scan center; Through the Hough transform, calculating the intersection points of straight lines is simplified to solving a system of linear equations; The Hough line transform formula is as follows: y i =a i x + b i （1） b i =-a i x + y i （2） Equation (1) is the equation of the i-th straight line in the rectangular coordinate system; Equation (2) is to transform the equation of the i -th straight line in the rectangular coordinate system into the equation of a straight line in the parameter space; After the Hough transform, a straight line in the x-y coordinate system is converted into a point in the a-b coordinate system; In the parameter space in polar coordinate form, each straight line is represented as shown in formula (3): （3） Among them, , are respectively the parameters of the i-th straight line in the polar coordinate system; is the distance parameter, representing the distance between the straight line and the origin; is the angle parameter, which is the included angle between the perpendicular line from the straight line to the origin and the positive half part of the x-axis; Use formula (4) to find the intersection point coordinates of any two straight lines, and then average all the obtained intersection point coordinates to reduce errors, and obtain the coordinates (x0, y0) of the intersection point. This coordinate is the coordinate of the circumferential scan center; the meanings of the parameters in formula (4) are the same as those in formula (3); （4）。 3. A method for detecting the boundary of fundus macular hole based on multimodal images according to claim 1, wherein: The cross-modal mapping algorithm includes the following steps: S31: Coordinate preprocessing: storing the abscissas of the highest points on the left and right sides of the edge of the macular hole in the OCT B-scan image as an array in the form of formula (5): （5） Among them, i refers to the scanning line order. The vertical scanning line is defined as the first scanning line, and the second scanning line is obtained by rotating 15 degrees clockwise, and so on; i X_Left i respectively refer to the i abscissas of the left and right highest points on the edge of the macular hole in the OCT B-scan image corresponding to the scanning line; S32. Cross-modal mapping: All scan lines on the IR image intersect at the same point, i.e., the circumferential scanning center, which is denoted as O point, with coordinates (x0, y0); The mapping algorithm used is based on the center point, and the algorithm flow of cross-modal mapping is as follows: , X_Right （6） wherein , respectively represent the results of normalizing the abscissas of the left and right highest points of the macular hole edge on the i th scan line with respect to the width of the OCT B-scan image, , respectively represent the values of the abscissas of the left and right highest points of the macular hole edge on the i th scan line, represents the width of the OCT B-scan image; S321: Calculate the proportional relationship: calculate the ratio of the abscissas of the two highest points to the width of the OCT B-scan image, that is: （7） where L is the length of the scanning line, is the ordinate of the pixel at the highest point of the vertical scanning line, is the ordinate of the center point; S322: Calculate the length of the scan line: calculate the length of the scan line according to the distance from the endpoints of the vertical scan line to the circumferential scan center in the y direction, and use formula (7) to calculate the length of the vertical scan line: （8） Among them, , represent the horizontal and vertical coordinates of the center point of the annular scan; are respectively the abscissas of the left and right highest points of the edge of the macular hole on the OCT B-scan image corresponding to the i th scan line mapped to the corresponding points on the vertical scan line of the IR image, and their values are the same as those of ; , are respectively the ordinates of the corresponding points on the vertical scan line of the IR image where the left and right highest points of the edge of the macular hole on the OCT B-scan image corresponding to the i th scan line are mapped; L is the length of the scan line; in this step, the highest points of the edges of the macular holes on all OCT B-scan images are mapped onto the vertical scan line; S33. Rotation: Rotate the points mapped to the vertical scan line to the scan lines at corresponding angles; the left and right highest points of the margin of the macular hole on the OCT B-scan image corresponding to the i first scan line, after the foregoing steps, obtain the horizontal and vertical coordinates of the vertical scan line on the IR image, in the form shown in formula (9): （9） Among them, i is the i th scan line, are the horizontal and vertical coordinates of the corresponding point after mapping the highest point on the left side of the macular hole to the vertical scan line of the IR image; are the horizontal and vertical coordinates of the corresponding point after mapping the highest point on the right side of the macular hole to the vertical scan line of the IR image; According to the rotation angle in formula (10), rotate the points mapped on the vertical scan line around the center of the annular scan O the point (x0, y0), so that the highest point on the edge of the macular hole on the OCT B-scan image corresponding to each scan line rotates from the vertical scan line to the corresponding scan line; （10） In formula (10), is the i angle that the O th scan line needs to rotate around the loop scan center point, and n is the total number of scan lines; S323: Vertical mapping: according to the proportional relationship calculated in step S421, map the highest points on the edge of the macular hole in the OCT B-scan image to the vertical scan lines on the IR image, using formula (8): Perform a rotation operation on the points mapped on the vertical scan line according to formula (11) so that the points fall on the scan lines corresponding to the corresponding angles; （11） Among them, , are the abscissa and ordinate of the rotated point, , are the abscissa and ordinate of the point before rotation, , are the abscissa and ordinate of the rotation center, is the rotation angle; According to formula (11), let the point with coordinates ( , ) rotate clockwise by , ) around the rotation center, and rotate clockwise by degrees.

Images