Eye fundus macular pore boundary detection method based on multi-modal image

Abstract

The invention discloses a fundus macular pore boundary detection method based on a multi-modal image, and particularly relates to the technical field of medical image processing, and the method comprises the steps: data preparation: splitting an original image into an IR image and an OCT B-scan image; center positioning: detecting a ring scanning line and determining the position of a ring scanning center through a ring scanning center point positioning algorithm facing the IR image; performing cross-modal mapping, obtaining coordinates of the highest point of the edge of the macular pore on the OCT B-scan image through a key point detection method, and mapping the highest point of the edge of the macular pore on the OCT B-scan image to the IR image through a cross-modal mapping algorithm; and boundary fitting: after cross-modal mapping algorithm processing, the highest points of the edges of the macular holes on the OCT B-scan image corresponding to all scanning lines on the IR image are mapped to the IR image, and the points are connected by straight lines, so that the boundary of the macular holes can be seen on the IR image.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for detecting a macular hole boundary of a fundus based on multimodal images. Background Art

[0002] Macular Hole (MH) refers to the localized full-thickness defect of the neuroepithelial layer of the macula. It can be divided into idiopathic and secondary macular holes according to the cause of the disease. The cause of the former is unknown, while the latter is mostly secondary to eye trauma, macular degeneration, high myopia and other diseases. Idiopathic Full Thickness Macular Hole (IFTMH) is the loss of the full thickness of the retina from the internal limiting membrane to the retinal pigment epithelium of the fovea. It is one of the more common fundus diseases in ophthalmology. The symptoms of this disease are mainly manifested as decreased vision, visual distortion, central scotoma, and visual field defects, which have an extremely serious impact on the patient's quality of life. Vitrectomy combined with internal limiting membrane peeling is a surgical method for treating macular holes. The anatomical closure rate of MH after surgery can reach more than 90%, but its postoperative visual recovery still varies from person to person.

[0003] Clinicians measure the size of the macular hole by optical coherence tomography (OCT) scanning. Figure 1 The images shown are collected by the OCT fundus examination equipment in Heidelberg, Germany. The left side is an infrared (IR) image, and the right side is an OCT B-scan image. During the acquisition process, the center of the macular hole is first determined on the IR image, and then a series of OCT B-scan images are obtained through circular scanning. This image is a cross-sectional view of the macular hole, which can express the morphology and size information of the macular hole and is an important basis for clinical diagnosis and evaluation of the recovery of macular holes after surgery.

[0004] During internal limiting membrane peeling surgery, real-time detection of the macular hole boundary can assist doctors in locating the hole. However, there is currently no research on real-time localization of the macular hole boundary during macular hole surgery, nor is there any research on cross-modality detection of the macular hole boundary.

[0005] In 2023, a method for detecting the mid-plasma lesion area was proposed. This method first calculates the endpoint coordinates of each ring scan line and constructs the straight line equation of each ring scan line. Then, according to the proportional relationship between the OCTB-scan image size and the corresponding ring scan line length, the boundary points of the mid-plasma lesion area detected in the OCTB-scan image are mapped to the corresponding ring scan line, thereby obtaining the boundary of the mid-plasma lesion area. However, this method has the following shortcomings: (1) This method cannot be applied when there are incomplete scan lines in the ring scan image. When any endpoint of the scan line is missing, the boundary points of the mid-plasma lesion area detected in the OCT B-scan image cannot be projected correctly. (2) The algorithm is relatively complex. The algorithm requires mapping operations for each scan line and the corresponding OCT B-scan image, which easily introduces positioning errors during the operation. Summary of the invention

[0006] To this end, the present invention focuses on the precise treatment goal of macular hole surgery and the shortcomings of the above-mentioned schemes, and proposes a fundus macular hole boundary detection method based on multimodal images, which maps the coordinates of the highest point of the macular hole edge detected on the OCT B-scan image to the IR image, and displays the boundary of the macular hole on the IR image. The method proposed in the present invention first detects the ring scan center of the macular hole, and then obtains the end point position of the vertical scan line. According to the proportional relationship between the ring scan line length and the OCT B-scan image size, the coordinates of the highest point of the macular hole edge detected on all OCT B-scan images are mapped to the vertical scan line, and then the mapped coordinates are rotated according to the angle of the corresponding scan line to obtain the boundary of the macular hole. This method has the following advantages: 1) Since the vertical scan line is generally complete during the image acquisition process, this method is not limited when the length of some scan lines is incomplete. 2) The algorithm is simple. This method does not need to locate each scan line, but maps all the highest points of the macular hole edge on the OCT B-scan image to the vertical scan line, and obtains the mapped boundary through the rotation operation. Since the method is relatively simple to calculate, the error caused by first calculating the square root of the scan line endpoint coordinates is avoided.

[0007] Specifically as follows: A method for detecting the boundary of a fundus macular hole based on a multimodal image, comprising:

[0008] Data preparation: split the original image into IR image and OCT B-scan image;

[0009] Center positioning: through the ring scan center point positioning algorithm for IR images, the ring scan line is detected and the position of the ring scan center is determined;

[0010] Cross-modal mapping, the coordinates of the highest point of the edge of the macular hole on the OCT B-scan image are obtained by the key point detection method, and then the highest point of the edge of the macular hole on the OCT B-scan image is mapped to the IR image through the cross-modal mapping algorithm; the key point detection method adopted in the present invention is the YOLOv8 algorithm.

[0011] Boundary fitting, after being processed by the cross-modal mapping algorithm, the highest point of the macular hole edge on the OCT B-scan image corresponding to all scan lines on the IR image will be correctly mapped to the IR image. By connecting these points with a straight line, the boundary of the macular hole can be seen on the IR image.

[0012] Preferably, the IR image and the OCT B-scan image are separated, and the separated images are named according to the corresponding relationship. The infrared images are named IR1 to IR12, and the corresponding OCT B-scan images are named OCT1 to O CT12. In the IR image, the green straight line with an arrow indicates the position of the 12 ring scan lines, and the highlighted green indicates the ring scan line corresponding to the current OCT B-scan image. The straight line from the tail of the arrow to the direction of the arrow corresponds to the scanning direction of the OCTB-scan image from left to right. The resolution of the unsplit image is 1264*596. After splitting, the resolution of the IR image is 496*496, and the resolution of the OCT B-scan image is 750*496, and the units are all pixels. The image splitting formula is as follows:

[0013]

[0014] Among them I (x,y) Represents the coordinates of the pixels of the image to be split, I IR represents the pixel coordinates belonging to the IR image, I OCT Represents the pixel coordinates belonging to the OCTB-scan image. After the splitting operation, the image acquired by the OCT scanning device can be split into an IR image and an OCTB-scan image, and they correspond one to one.

[0015] In order to map the highest point of the edge of the macular hole in the OCTB-scan image to the IR image, it is necessary to establish coordinate systems in two modes respectively and find the corresponding proportional relationship so that any point on the OCTB-scan image can find the corresponding point on the IR image. The highlighted green scan line on the IR image, from the tail of the arrow to the direction of the arrow, corresponds to the OCTB-scan image scanning direction from left to right, that is, the starting point of the scan line corresponds to the first column of pixels in the OCTB-scan image, and the end point of the scan line corresponds to the last column of pixels in the OCTB-scan image. In the IR image, all scan lines are green and intersect at the center of the ring scan. Therefore, the present invention proposes a ring scan center point positioning algorithm for IR images. First, convert the color space and create a mask, then find all green ring scan lines through the edge detection algorithm, and then find the ring scan center through the Hough transform. The specific steps are as follows:

[0016] a. In IR images, the morphology of fundus blood vessels will introduce unnecessary interference during edge detection. By converting the color space and creating a mask, we can ensure that only the green edges are detected in the ring scan line detection step, laying the foundation for subsequent research;

[0017] b. Select the Canny algorithm for ring scan line detection; the Canny algorithm is one of the most excellent and commonly used edge detection algorithms. The algorithm is not easily affected by noise and can identify strong and weak edges in images. It plays an important role in edge detection applications, including the following steps:

[0018] b1. Image denoising: Use Gaussian filtering to smooth the image. Use a 5*5 Gaussian filter as shown in formula (2).

[0019]

[0020] b2. Calculate the image gradient: Calculate the gradient direction and magnitude of the grayscale value of each pixel in the image to detect possible image edges. Specifically, use the Sobel operator to detect the edges in the x and y directions of the image, and then use formulas (3) and (4) to calculate the gradient direction θ and magnitude G, where I x Indicates the grayscale of the pixel in the x direction, I y Represents the grayscale of the pixel in the y direction; the Sobel operator extracts the image edge through discrete differential method;

[0021]

[0022]

[0023] b3. Non-maximum suppression: The non-maximum suppression algorithm can eliminate the stray response caused by edge detection. The grayscale change area is usually concentrated. The pixel with the largest grayscale gradient in the gradient direction within the local range is retained, and the others are not retained. A large part of the points can be eliminated, and the edge with multiple pixel widths can be turned into an edge with a single pixel width;

[0024] b4. Double threshold screening: After non-maximum suppression, there are still many possible edge points. The double threshold method is used to divide strong edges and weak edges. Pixels with grayscale gradient greater than the high threshold are retained as strong edges, and pixels with grayscale gradient less than the low threshold are removed. Pixels with grayscale gradient between the high and low thresholds are screened as weak edges. Search in the neighborhood of the weak edge. If a strong edge exists, the weak edge is retained. If not, the weak edge is deleted to complete the edge detection.

[0025] All ring scan lines can be detected by the Canny algorithm, laying the foundation for the subsequent detection of the ring scan center;

[0026] c. After the ring scan line detection is completed, a series of scan lines intersecting at the center of the ring scan are converted into the parameter space through the Hough transform to ensure that each ring scan line is detected as a straight line, and the intersection of multiple straight lines is found to detect the center of the ring scan; Hough transform is an effective image processing technology, especially suitable for detecting geometric shapes in images, such as straight lines. It determines the position of the geometric shape by mapping the points in the image to the parameter space (Hough space) and finding the peak in the space. The Hough transform has high robustness to noise and local defects. The Hough transform uses an accumulator to find the most likely straight line in the parameter space. Even if the straight lines in the image are discontinuous or contaminated by noise, the Hough transform can effectively detect these straight lines. In addition, the process of finding the intersection of straight lines in the parameter space is mathematically simpler and more direct.

[0027] Through the Hough transform, calculating the intersection of straight lines is simplified to solving a system of linear equations instead of performing complex calculations directly in the image space;

[0028] The Hough linear transformation formula is as follows:

[0029] y i =a i x+b i (5)

[0030] b i =-a i x+y i (6)

[0031] Formula (5) is the equation of the ith straight line in the rectangular coordinate system; Formula (6) converts the equation of the ith straight line in the rectangular coordinate system into the equation of the straight line in the (Hough space) parameter space;

[0032] After the Hough transform, a straight line in the xy coordinate system is converted into a point in the ab coordinate system;

[0033] The basic principle of Hough line detection is to use the duality of points and lines. There is a one-to-one correspondence between the lines in the image space and the points in the parameter space, and there is also a one-to-one correspondence between the lines in the parameter space and the points in the image space. Therefore, the Hough line detection method converts the line detection problem in the image space into the point detection problem in the parameter space, and completes the line detection task by finding the peak in the parameter space. After the Hough transform, all the lines that intersect at a point in the xy coordinate system are a series of points located on the same line in the parameter space. In the present invention, after the Hough transform, a series of scan lines that intersect at the center of the ring scan are converted into multiple collinear points in the parameter space.

[0034] In the parameter space in polar coordinate form, each straight line is expressed as shown in formula (7):

[0035] ρ i =xcos(θ i )+ysin(θ i ) (7)

[0036] Among them, ρ i ,θ i are the parameters of the i-th straight line in the polar coordinate system; ρ i is the distance parameter, which indicates the distance between the straight line and the origin; θ i is the angle parameter, which is the angle between the perpendicular line from the straight line to the origin and the positive half of the x-axis;

[0037] Formula (8) is used to calculate the coordinates of the intersection of any two straight lines, and then the average value of all the intersection coordinates is calculated to reduce the error, and the coordinates of the intersection (x0, y0) are obtained. This coordinate is the coordinate of the center of the ring scan. The meaning of each parameter in formula (8) is the same as that in formula (7).

[0038]

[0039] Preferably, the cross-modal mapping algorithm comprises the following steps:

[0040] d. Coordinate preprocessing: The horizontal coordinates of the highest points on the left and right sides of the macular hole edge in the OCT B-scan image are stored as an array in the following format:

[0041] (i,X_Left i ,X_Right i )(9)

[0042] Here, i refers to the scan line order. Define the vertical scan line as the first scan line, rotate 15 degrees clockwise to get the second scan line, and so on; X_Left i ,X_Right i They refer to the horizontal coordinates of the highest points on the left and right sides of the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line;

[0043] e. Cross-modal mapping: All scan lines on the IR image intersect at the same point, the center of the ring scan, which is denoted as point O and has coordinates (x0, y0). Since some scan lines are incomplete at both ends and the center of the ring scan must be on the IR image, the mapping algorithm used in this method is based on the center point. The algorithm flow of cross-modal mapping is as follows:

[0044] e1. Calculate the proportional relationship: Calculate the ratio of the horizontal coordinates of the two highest points to the width of the OCT B-scan image, that is:

[0045]

[0046] Where Ratio_Left i , Ratio_Right i X_Left represents the normalized result of the horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the i-th scan line relative to the width of the OCT B-scan image. i , X_Right i They respectively represent the values ​​of the horizontal coordinates of the highest points on the left and right sides of the edge of the macular hole on the i-th scanning line, and OCT_Width represents the width of the OCT B-scan image, which is 750 pixels in the present invention.

[0047] e2. Calculate the length of the scan line: Since all scan lines have the same length and the midpoints of the scan lines intersect at the same point, i.e., the center point, as long as one endpoint of a scan line can be determined, the length of all scan lines can be obtained. In the previous step, all the ring scan lines and the ring scan center (x0, y0) have been detected by Hough transform. In this step, the length of the scan line is calculated based on the distance from the endpoint of the vertical scan line to the ring scan center in the y direction. Formula (11) is used to calculate the length of the vertical scan line:

[0048] L = 2*(y top -y0) (11)

[0049] Where L is the length of the scan line, y top is the ordinate of the highest pixel on the vertical scan line, and y0 is the ordinate of the center point. The traditional algorithm uses formula (12) to solve the scan line length:

[0050]

[0051] where x i1 ,x i2 ,y i1 ,y i2 are the horizontal and vertical coordinates of the starting point 1 and the end point 2 of the i-th scan line respectively;

[0052] Compared with the traditional algorithm, this step does not use the square root operation when solving the scan line length according to formula (11), which reduces the error caused by it. Moreover, since the length of each scan line is the same, only the length of one scan line is required, which is extremely convenient in terms of calculation.

[0053] e3. Vertical mapping: According to the proportional relationship calculated in step e1, the highest point of the edge of the macular hole on the OCT B-scan image is mapped to the vertical scanning line of the IR image using formula (13):

[0054]

[0055] Among them, x0, y0 represent the horizontal and vertical coordinates of the center point of the ring scan; oct_left_i ,x oct_right_i The horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the OCT B-scan image corresponding to the i-th scan line are mapped to the horizontal coordinates of the corresponding points on the vertical scan line of the IR image, and their values ​​are the same as x0; y oct_left_i ,y oct_right_i are the ordinates of the left and right highest points of the edge of the macular hole on the OCT B-scan image corresponding to the i-th scan line mapped to the corresponding points on the vertical scan line of the IR image; L is the length of the scan line; in this step, the highest points of the edge of the macular hole on all OCT B-scan images are mapped to the vertical scan line;

[0056] f. Rotation: The previous steps have mapped the highest points of the macular hole edge on all OCT B-scan images to the vertical scan line. This step needs to rotate the points mapped to the vertical scan line to the scan line of the corresponding angle. The highest points on the left and right sides of the macular hole edge on the OCT B-scan image corresponding to the i-th scan line, after the above steps, obtain the horizontal and vertical coordinates of the vertical scan line on the IR image, as shown in formula (14):

[0057] (i,x oct_left_i ,y oct_left_i ,x oct_right_i ,y oct_right_i )(14)

[0058] Where i is the i-th scan line, x oct_left_i ,yoct_left_i x is the horizontal and vertical coordinates of the corresponding point after the highest point on the left side of the macular hole is mapped to the vertical scanning line of the IR image; oct_right_i ,y oct_right_i are the horizontal and vertical coordinates of the corresponding point after the highest point on the right side of the macular hole is mapped to the vertical scanning line of the IR image;

[0059] According to the rotation angle of formula (15), the points mapped on the vertical scanning line are rotated around the center of the ring scan point O (x0, y0), so that the highest point of the edge of the macular hole on the OCT B-scan image corresponding to each scanning line is rotated from the vertical scanning line to the corresponding scanning line;

[0060] θ i =(i-1)*180 / n (15)

[0061] In formula (15), θ i is the angle that the i-th scan line needs to rotate around the center of the ring scan, point O, and n is the total number of scan lines;

[0062] According to formula (16), the points mapped on the vertical scan line are rotated so that the points fall on the scan line of the corresponding angle;

[0063]

[0064] Among them, x′, y′ are the horizontal and vertical coordinates of the point after rotation, x, y are the horizontal and vertical coordinates of the point before rotation, x0, y0 are the horizontal and vertical coordinates of the rotation center, and θ is the angle of rotation;

[0065] According to formula (16), let the point with coordinates (x, y) rotate clockwise by θ degrees around the rotation center (x0, y0).

[0066] The present invention has the following advantages:

[0067] In terms of application, the present invention realizes mapping the coordinates of the highest point of the edge of the macular hole on the OCT B-scan image to the IR image, and displays the boundary of the macular hole on the IR image. The present invention aims to make up for the shortcomings of the existing fundus lesion area detection scheme in detecting the boundary of the macular hole, assist ophthalmologists to quickly and accurately locate the boundary of the macular hole, and promote the precision medical treatment of fundus macular holes.

[0068] In terms of algorithm, the present invention has the following advantages:

[0069] 1. This method makes up for the deficiency of the existing mapping method that it cannot be used when there are missing scan lines; when the image collected by the device has some incomplete scan lines, this method can still be used normally;

[0070] 2. Simple algorithm: This method does not need to locate each scan line, but maps all the highest points of the edge of the macular hole on the OCT B-scan image to the vertical scan line of the IR image, and obtains the mapped boundary through rotation. Since this method is relatively simple to calculate, it avoids the error caused by calculating the square root of the scan line endpoint coordinates first.

[0071] 3. The core of this method is to detect the center point of the ring scan rather than the endpoint of the ring scan line, which greatly reduces the complexity of the calculation. It can be applied to situations with different numbers of scan lines by only modifying the number of ring scan lines. The algorithm has good robustness and versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The image collected by the German Heidelberg OCT fundus examination device provided by the present invention;

[0073] Figure 2 A flow chart of a method for detecting fundus macular hole boundaries based on multimodal images provided by the present invention;

[0074] Figure 3 A flow chart of the IR image-oriented circular scanning center point positioning algorithm provided by the present invention;

[0075] Figure 4 A schematic diagram of the color space conversion and green mask effect provided by the present invention;

[0076] Figure 5 A schematic diagram of the Hough transform provided by the present invention;

[0077] Figure 6 A schematic diagram of a straight line detected by Hough transform provided by the present invention;

[0078] Figure 7 A schematic diagram of the highest point of the edge of the macular hole in the OCT B-scan image provided by the present invention;

[0079] Figure 8 A schematic diagram of mapping the highest point of the edge of the macular hole on the OCT B-scan image provided by the present invention to the IR image;

[0080] Fig. 9 A flowchart of the mapping algorithm provided by the present invention;

[0081] Fig.10 A flow chart of the rotation operation provided by the present invention;

[0082] Fig.11 A schematic diagram of the boundary of a macular hole on an IR image provided by the present invention;

[0083] Fig.12 The original image collected by the device provided by the present invention;

[0084] Fig.13 An implementation diagram provided for the present invention;

[0085] FIG. 14( a ) is an IR image provided by the present invention;

[0086] FIG14( b ) is an OCT B-scan image provided by the present invention;

[0087] Fig.15 The IR image ring scan center point diagram provided by the present invention;

[0088] Fig.16 A schematic diagram of cross-modal mapping provided by the present invention;

[0089] Fig.17 A schematic diagram of the position change of the mapping point before and after rotation provided by the present invention;

[0090] Fig.18 A boundary fitting diagram provided by the present invention;

[0091] FIG. 19( a ) shows 12 circular scan images provided by the present invention;

[0092] FIG. 19( b ) is a diagram showing the boundary detection results on 48 ring scan images provided by the present invention. DETAILED DESCRIPTION

[0093] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0094] The technical solution adopted by this embodiment to solve the technical problem is as follows Figure 2 As shown, the present embodiment describes a method for detecting the boundary of a fundus macular hole based on multimodal images, including four steps: data preparation, center positioning, cross-modal mapping, and boundary fitting. The specific method of each step will be described in detail below.

[0095] 1. Data preparation

[0096] like Figure 1As shown in the figure, the image collected by the German Heidelberg OCT fundus examination device consists of two parts, the IR image on the left and the OCT B-scan image on the right. In order to proceed smoothly in the subsequent steps, it is necessary to separate the IR image and the OCT B-scan image, and name the split images according to the corresponding relationship. The infrared images are named IR1~IR12, and the corresponding OCT B-scan images are named OCT1~OCT12. In the IR image, the green straight line with arrows indicates the position of the 12 ring scan lines. The highlighted green indicates the ring scan line corresponding to the current OCT B-scan image. The straight line from the tail of the arrow to the direction of the arrow corresponds to the scanning direction of the OCT B-scan image from left to right. The resolution of the unsplit image is 1264*596. After splitting, the resolution of the IR image is 496*496, and the resolution of the OCT B-scan image is 750*496. The units are all pixels. The image splitting formula is as follows:

[0097]

[0098] Among them I (x,y) Represents the coordinates of the pixels of the image to be split, I IR represents the pixel coordinates belonging to the IR image, I OCT Represents the pixel coordinates belonging to the OCT B-scan image. After the splitting operation, the image acquired by the OCT scanning device can be split into an IR image and an OCT B-scan image, and they correspond one to one.

[0099] 2. Center positioning

[0100] In order to map the highest point of the edge of the macular hole in the OCT B-scan image to the IR image, it is necessary to establish coordinate systems in two modes respectively and find the corresponding proportional relationship so that any point on the OCT B-scan image can find the corresponding point on the IR image. The highlighted green scan line on the IR image, from the tail of the arrow to the direction of the arrow, corresponds to the scanning direction of the OCT B-scan image from left to right, that is, the starting point of the scan line corresponds to the first column of pixels in the OCT B-scan image, and the end point of the scan line corresponds to the last column of pixels in the OCT B-scan image. In the IR image, all scan lines are green and intersect at the center of the ring scan. Therefore, the present invention proposes a ring scan center point positioning algorithm for IR images, and the algorithm flow is as follows: Figure 3 As shown in the figure. First, the color space is converted and a mask is created. Then, all green ring scan lines can be found through the edge detection algorithm, and the ring scan center can be found through the Hough transform.

[0101] 2.1 Convert color space and create mask

[0102] In IR images, the morphology of fundus blood vessels will introduce unnecessary interference during edge detection. By converting the color space and creating a mask, we can ensure that only green edges are detected in the ring scan line detection step, laying the foundation for subsequent research.

[0103] Converting the color space means converting the image from the RGB (R stands for red, G stands for green, and B stands for blue) color space to the HSV (H stands for hue, S stands for saturation, and V stands for brightness) color space. Through such conversion, the subsequent steps can more effectively detect and segment specific color areas.

[0104] A mask is a binary image used to isolate specific colors or regions in image processing and computer vision. In order to detect the green region in an image, it is necessary to define the range of green in the HSV color space. The range of green in the HSV space is from (35,100,100) to (85,255,255), and the parameters in the brackets represent hue, saturation, and brightness, respectively. Figure 4 The figure shows the effect of the IR image containing the green scan line after color space conversion and green mask. It can be seen that after these two steps, the green scan line is extracted and prepared for subsequent image processing.

[0105] 2.2 Circular scanning line detection

[0106] In the scanning line detection step, this embodiment selects the Canny algorithm for edge detection. The Canny algorithm is one of the most excellent and commonly used edge detection algorithms. The algorithm is not easily affected by noise and can identify strong and weak edges in the image. It plays an important role in edge detection applications. The algorithm detection process is divided into the following four steps:

[0107] (1) Image denoising. Noise is where the grayscale changes greatly and is easily identified as a false edge. Therefore, a Gaussian filter is used to smooth the image and reduce the noise in the image. Generally, a 5*5 Gaussian filter is used as shown in formula (2).

[0108]

[0109] (2) Calculate the image gradient and detect possible edges. This step requires calculating the gradient direction and amplitude of the grayscale value of each pixel in the image. First, the edges in the x and y directions of the image are detected by the Sobel operator, and then the direction θ and amplitude G of the gradient are calculated using formulas (3) and (4), where I x Indicates the grayscale of the pixel in the x direction, I y Similarly, the Sobel operator can effectively extract image edges through discrete differential methods.

[0110]

[0111]

[0112] (3) Non-maximum suppression. The non-maximum suppression algorithm can eliminate the stray responses caused by edge detection. The grayscale change area is usually concentrated. The pixels with the largest grayscale gradient in the gradient direction within the local range are retained, and the others are not retained. A large part of the points can be eliminated, and the edge with multiple pixel widths can be turned into an edge with a single pixel width.

[0113] (4) Double threshold screening. After non-maximum suppression, there are still many possible edge points. The double threshold method is applied to divide strong edges and weak edges. Pixels with grayscale gradients greater than the high threshold are retained as strong edges, and pixels with grayscale gradients less than the low threshold are removed. Pixels with grayscale gradients between the high and low thresholds are screened as weak edges. Search within the neighborhood of the weak edge. If a strong edge exists, the weak edge is retained. If not, the weak edge is deleted to complete edge detection.

[0114] In this step, all ring scan lines can be detected through the Canny algorithm, laying the foundation for the subsequent detection of the ring scan center.

[0115] 2.3 Circular scanning center detection

[0116] After the ring scan line detection is completed, this method converts a series of scan lines that intersect at the center of the ring scan into the parameter space through the Hough transform, ensuring that each ring scan line is detected as a straight line, and finding the intersection of multiple straight lines, so as to detect the center of the ring scan. The Hough transform is an effective image processing technology, especially suitable for detecting geometric shapes in images, such as straight lines. It determines the position of the geometric shape by mapping the points in the image to the parameter space (Hough space) and finding the peak in the space. The Hough transform has high robustness to noise and local defects. The Hough transform uses an accumulator to find the most likely straight line in the parameter space. Even if the straight lines in the image are discontinuous or contaminated by noise, the Hough transform can effectively detect these lines. In addition, the process of finding the intersection of straight lines in the parameter space is mathematically simpler and more direct. Through the Hough transform, calculating the intersection of straight lines can be simplified to solving a system of linear equations instead of performing complex calculations directly in the image space.

[0117] The Hough linear transformation formula is as follows:

[0118] y i =a i x+b i (5)

[0119] b i =-a i x+y i (6)

[0120] Formula (5) is the equation of the ith straight line in the rectangular coordinate system; Formula (6) is the equation of the ith straight line in the rectangular coordinate system converted into the straight line equation in the Hough space (parameter space). After the Hough transform, a straight line in the xy coordinate system is converted into a point in the ab coordinate system, which can be represented by an intuitive image as follows: Figure 5 shown.

[0121] The basic principle of Hough line detection is to use the duality of points and lines. There is a one-to-one correspondence between lines in image space and points in parameter space, and there is also a one-to-one correspondence between lines in parameter space and points in image space. Therefore, the Hough line detection method transforms the line detection problem in image space into the point detection problem in parameter space, and completes the line detection task by finding the peak in parameter space. After the Hough transform, all lines that intersect at a point in the xy coordinate system are a series of points on the same line in parameter space, such as Figure 6 In this case, after the Hough transform, a series of scan lines intersecting at the center of the ring scan are converted into multiple collinear points in the parameter space.

[0122] In the parameter space in polar coordinate form, each straight line is expressed as formula (7):

[0123] ρ i =xcos(θ i )+ysin(θ i ) (7)

[0124] Among them, ρ i ,θ i are the parameters of the i-th straight line in the polar coordinate system. i is the distance parameter, which indicates the distance between the straight line and the origin; θ i is the angle parameter, which is the angle between the perpendicular line from the straight line to the origin and the positive half of the x-axis. Use formula (8) to find the coordinates of the intersection of any two straight lines, and then average all the intersection coordinates to reduce the error, and get the coordinates of the intersection (x0, y0), which is the coordinate of the center of the ring scan. The meaning of each parameter in formula (8) is the same as that in formula (7).

[0125]

[0126] 3. Cross-modal mapping

[0127] Before the cross-modal mapping operation, the coordinates of the highest point of the macular hole edge on the OCT B-scan image have been obtained by the key point detection method, such as Figure 7As shown, the highest point of the macular hole edge is represented by a red dot, and the red vertical line in the middle is the horizontal center line of the OCT B-scan image, corresponding to the center point of the ring scan on the IR image.

[0128] like Figure 8 As shown in the figure, the two highest points on the edge of the macular hole in the OCT B-scan image have been mapped to the vertical ring scan line in the IR image, marked by red dots. The vertical ring scan line arrow is upward, indicating that the scanning direction is from bottom to top, and the corresponding OCT B-scan image scanning direction is from left to right. Therefore, in the IR image, the point below the ring scan center is mapped from the highest point on the left edge of the macular hole in the OCTB-scan image, and the point above the ring scan center is mapped from the highest point on the right edge of the macular hole in the OCT B-scan image. The mapping algorithm is divided into the following steps, such as Fig. 9 shown.

[0129] 3.1 Coordinate preprocessing

[0130] The horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the OCT B-scan image are stored as an array in the following format:

[0131] (i,X_Left i ,X_Right i )(9)

[0132] Here, i refers to the scan line order, and the vertical scan line is defined as the first scan line. Rotating 15 degrees clockwise gets the second scan line, and so on. i ,X_Right i They respectively refer to the horizontal coordinates of the highest points on the left and right sides of the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line.

[0133] 3.2 Cross-modal mapping

[0134] All scan lines on the IR image intersect at the same point, the center of the ring scan, which is denoted as point O and has coordinates (x0, y0). Since some scan lines are incomplete at both ends, and the center of the ring scan must be on the IR image, the mapping algorithm used in this method is based on the center point. The algorithm flow of cross-modal mapping is as follows:

[0135] 1) Calculate the proportional relationship: Calculate the ratio of the horizontal coordinates of the two highest points to the width of the OCT B-scan image.

[0136]

[0137] Where Ratio_Left i , Ratio_Righti X_Left represents the normalized result of the horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the i-th scan line relative to the width of the OCT B-scan image. i , X_Right i They respectively represent the values ​​of the horizontal coordinates of the highest points on the left and right sides of the edge of the macular hole on the i-th scanning line, and OCT_Width represents the width of the OCT B-scan image, which is 750 pixels in this embodiment.

[0138] 2) Calculate the length of the scan line: Since all scan lines have the same length and the midpoints of the scan lines intersect at the same point, i.e., the center point, as long as one endpoint of a scan line can be determined, the length of all scan lines can be obtained. In the previous step, all the ring scan lines and the ring scan center (x0, y0) have been detected by Hough transform. This step calculates the length of the scan line based on the distance from the endpoint of the vertical scan line to the ring scan center in the y direction. The following formula is used to calculate the length of the vertical scan line:

[0139] L = 2*(y top -y0) (11)

[0140] Where L is the length of the scan line, y top is the ordinate of the highest pixel on the vertical scan line, and y0 is the ordinate of the center point. The traditional algorithm uses the following formula to solve the scan line length:

[0141]

[0142] where x i1 ,x i2 ,y i1 ,y i2 are the horizontal and vertical coordinates of the starting point 1 and end point 2 of the i-th scan line respectively.

[0143] Compared with the traditional algorithm, this step does not use the square root operation when solving the scan line length according to formula (11), which reduces the error caused by it. Moreover, since the length of each scan line is the same, only the length of one scan line is required, which is extremely convenient in terms of calculation.

[0144] 3) Vertical mapping: According to the proportional relationship calculated in step 1), the highest point of the edge of the macular hole on the OCT B-scan image is mapped to the vertical scanning line of the IR image. The formula used is as follows:

[0145]

[0146] Among them, x0, y0 represent the horizontal and vertical coordinates of the center point of the ring scan; oct_left_i ,x oct_right_iThe horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the OCT B-scan image corresponding to the i-th scan line are mapped to the horizontal coordinates of the corresponding points on the vertical scan line of the IR image, and their values ​​are the same as x0; y oct_left_i ,y oct_right_i are the ordinates of the left and right highest points of the macular hole edge on the OCT B-scan image corresponding to the i-th scan line mapped to the corresponding points on the vertical scan line of the IR image; L is the length of the scan line. In this step, the highest points of the macular hole edge on all OCT B-scan images are mapped to the vertical scan line of the IR image.

[0147] 3.3 Rotation

[0148] The previous steps have mapped the highest points of the macular hole edges on all OCT B-scan images to the vertical scan lines of the IR images. This step requires rotating the points mapped to the vertical scan lines to the scan lines of corresponding angles. After the previous steps, the two highest points of the i-th scan line obtain the horizontal and vertical coordinates of the vertical scan line on the IR image, which are shown in formula (14):

[0149] (i,x oct_left_i ,y oct_left_i ,x oct_right_i ,y oct_right_i ) (14)

[0150] Where i is the i-th scan line, x oct_left_i ,y oct_left_i x is the horizontal and vertical coordinates of the corresponding point after the highest point on the left side of the macular hole is mapped to the vertical scanning line of the IR image; oct_right_i ,y oct_right_i are the horizontal and vertical coordinates of the corresponding point after the highest point on the right side of the macular hole is mapped to the vertical scan line of the IR image. According to the rotation angle of formula (15), the point mapped on the vertical scan line is rotated around the center of the ring scan point O (x0, y0), so that the highest point of the edge of the macular hole on the OCT B-scan image corresponding to each scan line is rotated from the vertical scan line to the corresponding scan line. In formula (15), θ i is the angle that the i-th scan line needs to rotate around the center of the ring scan point O, and n is the total number of scan lines.

[0151] θ i =(i-1)*180 / n (15)

[0152] According to formula (16), the points mapped on the vertical scan line are rotated so that the points fall on the scan line of the corresponding angle. Among them, x′, y′ are the horizontal and vertical coordinates of the point after rotation, x, y are the horizontal and vertical coordinates of the point before rotation, x0, y0 are the horizontal and vertical coordinates of the rotation center, and θ is the rotation angle. According to formula (16), the point with coordinates (x, y) can be rotated clockwise by θ degrees around the rotation center (x0, y0).

[0153]

[0154] like Fig.10 The figure shows the rotation operation flow chart, taking the boundary point corresponding to the second scan line as an example. The red point indicates the position of the highest point of the edge of the macular hole on the OCT B-scan image mapped to the vertical scan line, and the yellow point indicates the position of the mapping point after rotating 15 degrees around the center of the ring scan, that is, the position of the boundary point on the corresponding ring scan line. The operations of the remaining scan lines are similar and will not be repeated here.

[0155] 4. Boundary Fitting

[0156] After the mapping algorithm is used, the highest point of the macular hole on the OCT B-scan image corresponding to all scan lines on the IR image will be correctly mapped to the IR image. By connecting these points with straight lines, the boundary of the macular hole can be seen on the IR image, such as Fig.11 shown.

[0157] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0158] The image captured by the OCT fundus examination device in Heidelberg, Germany, consists of two parts, the IR image on the left and the OCT B-scan image on the right. In order to proceed smoothly in the subsequent steps, the IR image and the OCT B-scan image need to be separated. The green line in the IR image is the scan line, and the highlighted green line is the scan line corresponding to the current OCT B-scan image. The direction of the scan line is from the tail of the arrow to the head of the arrow, corresponding to each column of pixels from left to right in the OCT B-scan image. The specific morphology of the macular hole is shown in the OCT B-scan image. Fig.12 Shown is the original image captured by the device.

[0159] The implementation steps of the present invention are as follows Fig.13 shown.

[0160] Data preparation: The original image is split into IR image and OCT B-scan image, as shown in Figure 14(a) and Figure 14(b).

[0161] Center positioning: Detect the intersection of the scan lines on the IR image, which is the center of the ring scan. Fig.15 As shown, the red word "center" indicates that the red dot is the detected center point of the ring scan.

[0162] Cross-modality mapping: The highest points on the left and right sides of the macular hole edge on the OCT B-scan image are first mapped to the vertical scan line of the IR image, and then rotated according to the order of the scan lines. Fig.16 As shown, the highest points on the left and right sides of the edge of the macular hole on the OCT B-scan image are mapped to the vertical scanning line, and the L and R marked in the two figures represent the highest points on the left and right sides, respectively.

[0163] like Fig.17 As shown in the figure, the highest points on the left and right sides of the edge of the macular hole on the OCT B-scan image corresponding to the second scan line are first mapped to the vertical scan line, as shown by the red dots in the figure, and then rotated around the center point by a corresponding angle. The rotated points are shown as yellow dots in the figure. The operation of the i-th scan line is similar, and it can be rotated according to the corresponding angle, which will not be repeated.

[0164] Boundary fitting: After the mapping algorithm is used, the highest point of the macular hole on the OCT B-scan image corresponding to all scan lines on the IR image will be correctly mapped to the IR image. By connecting these points with straight lines, the boundary of the macular hole can be seen on the IR image, such as Fig.18 shown.

[0165] The method proposed in the present invention can achieve good macular hole boundary detection effect when applied to 12 ring scan images and 48 ring scan images. Figure 19(a) and 19(b) As shown in FIG. 1 , the boundary detection results on 12 ring scan images and 48 ring scan images are shown respectively.

[0166] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for detecting macular hole boundary based on multimodal images, characterized in that: include: Data preparation: split the original image into IR image and OCT B-scan image; Center positioning: through the ring scan center point positioning algorithm for IR images, the ring scan line is detected and the position of the ring scan center is determined; Cross-modal mapping: The coordinates of the highest point of the macular hole edge on the OCT B-scan image are obtained by key point detection, and then the highest point of the macular hole edge on the OCT B-scan image is mapped to the IR image through the cross-modal mapping algorithm; Boundary fitting,After processing by the cross-modal mapping algorithm, the highest point of the macular hole edge on the OCT B-scan image corresponding to all scan lines on the IR image is mapped to the IR image. By connecting these points with a straight line, the boundary of the macular hole can be seen on the IR image.

2. The method for detecting the boundary of a fundus macular hole based on multimodal images according to claim 1, characterized in that: The ring scan center point location algorithm for IR images first converts the color space and creates a mask, then finds all green ring scan lines through the edge detection algorithm, and then finds the ring scan center through the Hough transform. The specific steps are as follows: S21, by converting the color space and creating a mask, ensuring that only the edges that appear green are detected in the ring scan line detection step; S22, selecting the Canny algorithm to perform ring scan line detection; S23, after the ring scan line detection is completed, a series of scan lines intersecting at the ring scan center are converted into the parameter space through Hough transformation, ensuring that each ring scan line is detected as a straight line, and the intersection point of multiple straight lines is obtained, thereby detecting the ring scan center; By using the Hough transform, calculating the intersection of straight lines is simplified to solving a system of linear equations; The Hough linear transformation formula is as follows: y i =a i x+b i (1) b i =-a i x+y i (2) Formula (1) is the equation of the ith straight line in the rectangular coordinate system; Formula (2) converts the equation of the ith straight line in the rectangular coordinate system into the equation of the straight line in the parameter space; After the Hough transform, a straight line in the xy coordinate system is converted into a point in the ab coordinate system; In the parameter space in polar coordinate form, each straight line is expressed as shown in formula (3): r i =xcos(θ i )+ysin(θ i ) (3) Among them, ρ i ,θ i are the parameters of the i-th straight line in the polar coordinate system; ρ i is the distance parameter, which indicates the distance between the straight line and the origin; θ i is the angle parameter, which is the angle between the perpendicular line from the straight line to the origin and the positive half of the x-axis; Use formula (4) to find the coordinates of the intersection of any two straight lines, then find the average value of all the intersection coordinates to reduce the error, and get the coordinates of the intersection (x0, y0), which is the coordinate of the center of the ring scan; the meaning of each parameter in formula (4) is the same as that in formula (3); 3. The method for detecting the boundary of a fundus macular hole based on multimodal images according to claim 1, characterized in that: The cross-modal mapping algorithm includes the following steps: S31, coordinate preprocessing: the horizontal coordinates of the highest points on the left and right sides of the macular hole edge in the OCT B-scan image are stored as an array in the form of formula (5): (i,X_Left i ,X_Right i )(5) Here, i refers to the scan line order. Define the vertical scan line as the first scan line, rotate 15 degrees clockwise to get the second scan line, and so on; X_Left i ,X_Right i They refer to the horizontal coordinates of the highest points on the left and right sides of the edge of the macular hole in the OCT B-scan image corresponding to the i-th scan line; S32, cross-modal mapping: All scan lines on the IR image intersect at the same point, namely the center of the ring scan, which is denoted as point O, with coordinates (x0, y0); the mapping algorithm used is based on the center point, and the algorithm flow of cross-modal mapping is as follows: S321, calculate the proportional relationship: calculate the ratio of the horizontal coordinates of the two highest points to the width of the OCT B-scan image, that is: Where Ratio_Left i , Ratio_Right i X_Left represents the normalized result of the horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the i-th scan line relative to the width of the OCT B-scan image. i , X_Right i Respectively represent the values ​​of the horizontal coordinates of the highest points on the left and right sides of the edge of the macular hole on the i-th scan line, and OCT_Width represents the width of the OCT B-scan image; S322, calculate the length of the scan line: calculate the length of the scan line according to the distance from the end point of the vertical scan line to the center of the ring scan in the y direction, and use formula (7) to calculate the length of the vertical scan line: L=2*(y top -y0) (7) Where L is the length of the scan line, y top is the ordinate of the pixel at the highest point of the vertical scan line, and y0 is the ordinate of the center point; S323, vertical mapping: According to the proportional relationship calculated in step S421, the highest point of the edge of the macular hole on the OCT B-scan image is mapped to the vertical scanning line of the IR image, using formula (8): Among them, x0, y0 represent the horizontal and vertical coordinates of the center point of the ring scan; oct_left_i ,x oct_right_i The horizontal coordinates of the highest points on the left and right sides of the macular hole edge on the OCT B-scan image corresponding to the i-th scan line are mapped to the horizontal coordinates of the corresponding points on the vertical scan line of the IR image, and their values ​​are the same as x0; y oct_left_i ,y oct_right_i are the ordinates of the left and right highest points of the edge of the macular hole on the OCT B-scan image corresponding to the i-th scan line mapped to the corresponding points on the vertical scan line of the IR image; L is the length of the scan line; in this step, the highest points of the edge of the macular hole on all OCT B-scan images are mapped to the vertical scan line; S33, rotation: Rotate the points mapped to the vertical scanning line to the scanning line of the corresponding angle; the highest points on the left and right sides of the edge of the macular hole on the OCT B-scan image corresponding to the i-th scanning line, after the above steps, obtain the horizontal and vertical coordinates of the vertical scanning line on the IR image, in the form shown in formula (9): (i,x oct_left_i ,and oct_left_i ,x oct_right_i ,and oct_right_i )(9) Where i is the i-th scan line, x oct_left_i ,y oct_left_i x is the horizontal and vertical coordinates of the corresponding point after the highest point on the left side of the macular hole is mapped to the vertical scanning line of the IR image; oct_right_i ,y oct_right_i are the horizontal and vertical coordinates of the corresponding point after the highest point on the right side of the macular hole is mapped to the vertical scanning line of the IR image; According to the rotation angle of formula (10), the points mapped on the vertical scanning line are rotated around the center of the ring scan point O (x0, y0), so that the highest point of the edge of the macular hole on the OCT B-scan image corresponding to each scanning line is rotated from the vertical scanning line to the corresponding scanning line; i i =(i-1)*180 / n (10) In formula (10), θ i is the angle that the i-th scan line needs to rotate around the center of the ring scan, point O, and n is the total number of scan lines; According to formula (11), the points mapped on the vertical scan line are rotated so that the points fall on the scan line of the corresponding angle; Among them, x′, y′ are the horizontal and vertical coordinates of the point after rotation, x, y are the horizontal and vertical coordinates of the point before rotation, x0, y0 are the horizontal and vertical coordinates of the rotation center, and θ is the angle of rotation; According to formula (11), let the point with coordinates (x, y) rotate clockwise by θ degrees around the rotation center (x0, y0).

Images