Full-eye image automatic splicing method and system based on frequency sweep OCT (Optical Coherence Tomography)

Abstract

The invention relates to the technical field of image processing, in particular to a full-eye image automatic splicing method and system based on sweep-frequency OCT, and the method comprises the steps: collecting anterior segment images and retina images at different visual angles in combination with manual guidance; pre-processing the anterior segment images of different visual angles to obtain anterior chamber angle information, performing rigid registration on the anterior segment images based on the anterior chamber angle information, and fusing the registered anterior segment images to obtain an anterior segment OCT spliced image; performing transverse and longitudinal registration on the retina images of different visual angles by using the synchronously acquired laser eye fundus images, and performing fusion and curvature correction on the registered retina images to obtain a retina OCT spliced image; and splicing the anterior segment OCT spliced image and the retina OCT spliced image to obtain a full-eye image. According to the method and the device, the anterior segment image and the retina image can be seamlessly spliced while high-resolution detail presentation is ensured, and a more complete full-eye image with a clearer structure is obtained.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to an automatic full-eye image stitching method and system based on swept-source OCT. Background Art

[0002] Optical Coherence Tomography (OCT), as a non-invasive biological tissue imaging technology, has been widely used in ophthalmic clinical diagnosis and scientific research since its introduction into the clinic due to its high-resolution cross-sectional imaging ability. By using a longer-wavelength light source and extremely high instantaneous coherence, swept-source OCT not only improves the tissue penetration depth but also expands the longitudinal imaging range, enabling it to obtain a wider and deeper anterior segment and retina imaging. It performs particularly well in axial biometry and other aspects, providing more comprehensive structural and functional information for the clinic.

[0003] Currently, there are usually two methods for obtaining full-eye images based on swept-source OCT. One is the single-scan composite method. During a single swept-source OCT scan, the anterior segment image and the retina image are collected simultaneously, and then the full-eye image is directly generated through an image composite algorithm. The other is the segmented imaging correction method. First, the anterior segment and retina regions are scanned independently, and then the two images are registered and fused through a curvature or distortion correction algorithm to reconstruct the full-eye image.

[0004] Although the above two methods can obtain full-eye images, the single-scan composite method often needs to make a compromise between resolution and signal-to-noise ratio to balance imaging depth and field of view, resulting in incomplete local structures and unclear details. The segmented imaging correction method can maintain a high resolution in their respective regions, but it is prone to registration errors or structural discontinuities at the junction of the anterior segment and the retina, thus affecting the overall integrity and coherence of the full-eye image. Therefore, how to achieve seamless stitching of the anterior segment image and the retina image while ensuring the presentation of high-resolution details and obtaining a more complete and clearer-structured full-eye image is a difficult problem currently faced. Summary of the Invention

[0005] This application provides an automatic full-eye image stitching method and system based on swept-source OCT, which can achieve seamless stitching of the anterior segment image and the retina image while ensuring the presentation of high-resolution details and obtain a more complete and clearer-structured full-eye image. This application provides the following technical solutions: In a first aspect, this application provides an automatic full-eye image stitching method based on swept-source OCT, and the method includes: Combining manual guidance to collect anterior segment images and retina images from different perspectives; the retina images from different perspectives include corresponding laser fundus images; Preprocess the anterior segment images from different perspectives to obtain anterior chamber angle information, perform rigid registration on the anterior segment images based on the anterior chamber angle information, and fuse the registered anterior segment images to obtain an anterior segment OCT mosaic image; Use the simultaneously acquired fundus images to perform horizontal and vertical registration on the retinal images from different perspectives, fuse the registered retinal images, and perform curvature correction to obtain a retinal OCT mosaic image; Stitch the anterior segment OCT mosaic image and the retinal OCT mosaic image to obtain a full-eye image.

[0006] In a specific feasible implementation, the combining artificial guidance to collect anterior segment images and retinal images from different perspectives includes: The anterior segment images from different perspectives include anterior segment OCT frontal images, anterior segment OCT left images, and anterior segment OCT right images; The retinal images from different perspectives include retinal OCT frontal images and corresponding fundus images, retinal OCT left images and corresponding fundus images, and retinal OCT right images and corresponding fundus images.

[0007] In a specific feasible implementation, the preprocessing the anterior segment images from different perspectives to obtain anterior chamber angle information, performing rigid registration on the anterior segment images based on the anterior chamber angle information, and fusing the registered anterior segment images to obtain an anterior segment OCT mosaic image includes: Perform refractive correction on the anterior segment OCT frontal image in the anterior segment images; Preprocess the anterior segment images to obtain a mask image of the effective region including the cornea and iris; After obtaining the mask image of each anterior segment image, accurately locate the upper and lower boundary point sets of the anterior chamber angle from the mask image, and apply the RANSAC algorithm to the two sets of points on the upper and lower boundaries respectively through random sampling and inlier consistency test to remove the outliers that deviate greatly from the overall trend; Use the remaining candidate points to perform linear fitting on the upper and lower boundaries respectively to obtain two smooth straight lines representing both sides of the anterior chamber angle. After completing the straight line fitting, calculate the intersection point of these two boundary straight lines , the intersection point is the vertex coordinate of the anterior chamber angle, and obtain the inclination angle of any boundary straight line with the horizontal axis , the intersection point and the inclination angle together constitute the registration parameters required for rigid registration.

[0008] In a specific feasible implementation, the preprocessing of the anterior segment images from different perspectives to obtain anterior chamber angle information, the rigid registration of the anterior segment images based on the anterior chamber angle information, and the fusion of the registered anterior segment images to obtain the anterior segment OCT mosaic image further include: Using the anterior segment OCT frontal image as the reference image, and the anterior segment OCT left image and the anterior segment OCT right image as the floating images, and performing rigid registration according to the obtained registration parameters; After completing the rigid registration, perform longitudinal fine-tuning on the floating image. Use normalized cross-correlation matching to measure the matching degree between the corresponding regions of the template image and the floating image. Extract the key region that coincides with the floating image from the reference image as the template image, and form all the pixels in the template image into a row vector in column order , that is, the feature vector of the template image; Select a vertical sliding window in the floating image, slide row by row in its corresponding overlapping region to extract candidate regions, and also convert them into feature vectors , which is called the feature vector of the detection region; Measure the similarity between regions by calculating the cosine value of the angle between the two feature vectors, as follows: ; Wherein, and are the Euclidean norms of the feature vectors and respectively. After completing the entire sliding process, count the cosine values of the angles corresponding to all candidate regions, and select the position corresponding to the maximum cosine value of the angle as the optimal matching position between the template image and the floating image. The vertical offset of this position relative to the initial search starting point of the template image is used as the compensation translation amount, and the floating image is longitudinally translated; Fuse the registered anterior segment images by using a gray-scale weighted method to generate the anterior segment OCT mosaic image.

[0009] In a specific feasible implementation, the registration of the retinal images from different perspectives in the horizontal and vertical directions by using the synchronously acquired fundus laser images, the fusion and curvature correction of the registered retinal images to obtain the retinal OCT mosaic image include: Using the retinal OCT frontal image as the reference image, and the retinal OCT left image and the retinal OCT right image as the floating images. For any one of the retinal OCT frontal image and the retinal OCT left image or the retinal OCT right image, each corresponds to a fundus laser image. Using the two fundus laser images as the registration input, use the scale-invariant feature transform algorithm to extract the feature points of the fundus laser image, and match the extracted feature points through the fast nearest neighbor search package to find similar feature point pairs; Use the RANSAC algorithm to screen the matching points. Based on the filtered valid feature point pairs, calculate the mean of their horizontal distances to obtain the horizontal overlap length of the fundus laser image. Calculate the horizontal overlap length of the retinal image based on the horizontal overlap length of the fundus laser image. As follows: ; Among them, is the image width of the fundus laser image in this registration pair, is the image width of the corresponding retinal OCT image. Based on the horizontal overlap length of the retinal image , determine the horizontal alignment position of the reference image and the floating image, and adjust the position of the floating image horizontally; Determine the corresponding column pixel positions of the reference image and the floating image at the splicing location according to the calculated horizontal overlap length. Based on this column of pixels, calculate the displacement difference between the reference image and the floating image vertically to obtain the vertical translation parameter, and perform a vertical translation transformation operation on the floating image according to the vertical translation parameter to achieve vertical position alignment.

[0010] In a specific feasible implementation, the use of simultaneously acquired fundus laser images to perform horizontal and vertical registration of retinal images from different perspectives, fusing the registered retinal images, and performing curvature correction to obtain a retinal OCT mosaic image further includes: Adopt a method based on gray weighting to fuse the registered retinal images to generate a retinal OCT mosaic image; According to the retinal OCT imaging characteristics and the optical structure model, perform curvature correction on the retinal OCT mosaic image to generate a retinal OCT mosaic image close to the true fundus curvature; The curvature correction maps the mosaic image to a fan-shaped ring image structure from the back end of the vitreous body to the fundus. The center of the circle where the fan-shaped ring is located is the scanning axis point at the pupil. According to the splicing width of the image and the optical parameters of OCT imaging, calculate the opening angle corresponding to the complete mosaic image As follows: ; Among them, is the scanning field of view angle of the retinal OCT image, is the width of the retinal OCT mosaic image, is the image width of the retinal OCT image; For any point in the fan-shaped ring structure, its physical distance to the center of the circle Estimation is performed according to the optical path and the refractive index of the medium, and its calculation formula is as follows: ; Wherein, and are respectively the optical path thicknesses of the lens and the vitreous at the point position, and are respectively the refractive indices of the lens and the vitreous; After the calculation is completed, each pixel point in the retinal OCT mosaic image is combined with its scanning angle determined relative to the opening angle and the corresponding physical distance , transformed from the Cartesian coordinate system to the polar coordinate system with the scanning axis point as the pole to obtain its true spatial position distribution, and then through the inverse transformation from polar coordinates to Cartesian coordinates, the retinal OCT mosaic image is mapped back to the planar image to complete the correction of curvature distortion.

[0011] In a specific feasible implementation manner, the splicing of the anterior segment OCT mosaic image and the retinal OCT mosaic image to obtain the full-eye image includes: Adjust the size of the retinal OCT mosaic image to make its physical resolution consistent with that of the anterior segment OCT mosaic image; When splicing the images, find the intersection point of the visual axis and the connection line of the two anterior chamber angles in the anterior segment OCT mosaic image, and determine the relative positions of the anterior segment OCT mosaic image and the retinal OCT mosaic image according to the coordinates of the intersection point; According to the obtained relative positions of the anterior segment OCT mosaic image and the retinal OCT mosaic image, move and splice the two to obtain the full-eye mosaic image.

[0012] In a second aspect, the present application provides an automatic full-eye image splicing system based on swept-source OCT, adopting the following technical solutions: An automatic full-eye image splicing system based on swept-source OCT, comprising: An image acquisition module, configured to acquire anterior segment images and retinal images from different perspectives in combination with manual guidance; An anterior segment image splicing module, configured to preprocess the anterior segment images from different perspectives to obtain anterior chamber angle information, perform rigid registration on the anterior segment images based on the anterior chamber angle information, and fuse the registered anterior segment images to obtain an anterior segment OCT mosaic image; A retinal image splicing module, configured to perform registration on the retinal images from different perspectives in the horizontal and vertical directions by using the simultaneously acquired laser fundus images, fuse the registered retinal images, and perform curvature correction to obtain a retinal OCT mosaic image; The full-eye image stitching module is used to stitch the anterior segment OCT stitching image and the retinal OCT stitching image to obtain a full-eye image.

[0013] In a third aspect, the present application provides an electronic device, which includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a method for automatically stitching full-eye images based on swept-source OCT as described in the first aspect.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program is stored, and when the program is executed by a processor, it is used to implement a method for automatically stitching full-eye images based on swept-source OCT as described in the first aspect.

[0015] In summary, the beneficial effects of the present application at least include: (1) On the one hand, the multi-view acquisition strategy guided by humans is combined to ensure an overlap of at least 20% between each pair of adjacent images, providing sufficient feature information for subsequent registration; on the other hand, through the extraction of the anterior chamber angle features of the anterior segment images, the rigid registration enhanced by RANSAC, and the fusion algorithm based on gray-weighting, not only the high-resolution details of tiny tissue structures such as the cornea and iris are retained within the region, but also the gray-level mutations and false artifacts at the stitching edges are eliminated, enabling the anterior segment and retinal stitching images to achieve a natural and smooth transition at the detail level, thus solving the trade-off problem between resolution and signal-to-noise ratio in a single scan.

[0016] (2) Aiming at the problems of unclear features and high noise in retinal OCT images, the synchronously acquired fundus laser images are introduced to assist in completing highly robust registration in the horizontal and vertical directions; then, through the curvature correction method, based on the optical path and refractive index of the lens and vitreous body, the spatial morphology of the scanned fan-shaped ring on the real spherical surface is restored; finally, with the scanning axis point at the pupil as the common positioning reference, the physical scales of the anterior segment and retinal images are unified and seamlessly stitched, making the final full-eye image not only visually coherent and natural, but also geometrically conforming to the internal and external curvatures of the real eyeball, providing a highly consistent and reliable data basis for subsequent three-dimensional reconstruction and quantitative analysis.

[0017] By combining manual guidance to first collect anterior segment images and retinal images from different perspectives, the problems of incomplete coverage and difficult registration caused by limited field of view are eliminated at the source; subsequently, the anterior segment images from different perspectives are preprocessed to obtain anterior chamber angle information, and based on the anterior chamber angle information, the anterior segment images are rigidly registered, and the registered anterior segment images are fused to obtain an anterior segment OCT mosaic image; not only high-resolution tissue details are retained, but also natural transitions between regions are achieved, solving the problem of unclear details caused by the trade-off between single-scan resolution and signal-to-noise ratio; then, the laser fundus images obtained synchronously are used to register the retinal images from different perspectives in the horizontal and vertical directions, and the registered retinal images are fused and curvature corrected to obtain a retinal OCT mosaic image. The laser fundus images are introduced to assist in completing high-robustness registration in the horizontal and vertical directions, and the true fundus curvature is restored through spherical curvature correction, effectively compensating for the structural discontinuity and distortion at the splicing junction of the traditional segmented correction method; finally, the anterior segment and retinal mosaic images are seamlessly synthesized on the basis of unified physical scale and scanning axis positioning, achieving both high resolution and overall coherence, thereby obtaining a more complete and clearer full-eye image with more detailed structures.

[0018] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly and implement it in accordance with the content of the specification, the following describes in detail with reference to the preferred embodiments of this application and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the overall process of the full-eye image automatic stitching method based on swept-source OCT in the embodiment of this application.

[0020] Figure 2 It is a schematic diagram of use cases of the anterior segment images and retinal images collected in the embodiment of this application.

[0021] Figure 3 It is a schematic diagram of the process of step S102 in the embodiment of this application.

[0022] Figure 4 It is a schematic diagram of use cases for determining the upper and lower boundary points of the anterior chamber angle in the embodiment of this application.

[0023] Figure 5 It is a schematic diagram of the process of step S103 in the embodiment of this application.

[0024] Figure 6 It is a schematic diagram of use cases for curvature correction to form a fan-shaped ring image from the posterior end of the vitreous body to the fundus in the embodiment of this application.

[0025] Figure 7 It is a schematic diagram of use cases of the images after anterior segment stitching, retinal stitching, and full-eye stitching in the embodiment of this application.

[0026] Figure 8 It is a structural block diagram of an automatic full-eye image stitching system based on swept-source OCT in an embodiment of the present application.

[0027] Figure 9 It is a block diagram of an electronic device for automatic full-eye image stitching based on swept-source OCT in an embodiment of the present application. Specific embodiments

[0028] The following further describes the specific embodiments of the present application in detail in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0029] Optionally, the present application takes the automatic full-eye image stitching method based on swept-source OCT provided in each embodiment and applied to an electronic device as an example for illustration. The electronic device is a terminal or a server. The terminal can be a mobile phone, a computer, a tablet computer, etc. The type of the electronic device is not limited in this embodiment.

[0030] Referring to Figure 1 , it is a schematic flow chart of an automatic full-eye image stitching method based on swept-source OCT provided in an embodiment of the present application. The method at least includes the following steps: Step S101: Combine manual guidance to collect anterior segment images and retinal images from different perspectives.

[0031] In step S101, in a manner combining manual guidance, the images of the anterior segment and the retina of the subject are sequentially collected from different perspectives. In implementation, the full-eye image collection is divided into two parts: anterior segment image collection and retinal image collection. Since the field of view of a single OCT image is limited, the maximum outer eye field of view of anterior segment OCT is usually 39°, and the maximum outer eye field of view of retinal OCT is usually 90°. Considering that the overlapping area for subsequent image stitching requires at least 20% overlap, at least three anterior segment OCT images and three retinal OCT images need to be collected at least in the frontal, left, and right perspectives respectively to generate a full-eye image covering the entire meridian plane.

[0032] Specifically, during the collection of anterior segment images, first, the anterior segment OCT frontal image is collected at the maximum imaging field of view. Subsequently, in the same mode, by adjusting the position of the fixation light, the subject is guided to look horizontally to the left and horizontally to the right respectively to obtain the anterior segment OCT left image and the anterior segment OCT right image. Similarly, during the collection of retinal images, first, the retinal OCT frontal image and the corresponding fundus laser image are synchronously collected at the maximum imaging field of view. Subsequently, in the same mode, by adjusting the position of the fixation light, the subject is guided to look horizontally to the left and horizontally to the right respectively to obtain the retinal OCT left image and the corresponding fundus laser image, as well as the retinal OCT right image and the corresponding fundus laser image. Referring toFigure 2 , from left to right and from top to bottom, are the anterior segment frontal view, anterior segment left lateral view, anterior segment right lateral view, retina frontal view, retina left lateral view, and retina right lateral view images collected by a set of full-eye stitching. Only OCT images are required for the anterior segment, and OCT images and corresponding fundus laser images are required for the retina.

[0033] It should be noted that three images are collected respectively from the frontal, left, and right perspectives mainly to leave sufficient overlapping regions between two adjacent perspective images, providing stable registration feature points. A minimum overlap of 20% can ensure that the stitching algorithm finds sufficient common structures (such as corneal edges, lens contours, or retinal vascular branches) between images, thus achieving seamless connection and maintaining the continuity of the overall structure. Using only two images often makes it difficult to meet both the full-field coverage and overlap requirements simultaneously. Therefore, at least three images are required to cover the entire meridian plane and ensure a stable stitching basis between each pair of adjacent images.

[0034] Step S102: Preprocess the anterior segment images from different perspectives to obtain anterior chamber angle information, perform rigid registration on the anterior segment images based on the anterior chamber angle information, and fuse the registered anterior segment images to obtain an anterior segment OCT stitching image.

[0035] Refer to Figure 3 , which is a flow schematic diagram of step S102 in the embodiment of the present application. This method at least includes the following steps: Step S1021: Perform refractive correction on the anterior segment OCT frontal view image in the anterior segment images.

[0036] Specifically, to eliminate the influence of refractive errors on subsequent measurements and stitching, refractive correction is performed on the frontal view image to ensure more accurate subsequent physiological parameter measurements. In the anterior segment stitching process, the main purpose of refractive correction is to eliminate geometric distortion caused by differences in the refractive index of the eyeball, thereby ensuring the accuracy of measurement results. These physiological parameters are usually extracted only from the frontal view image where the central visual axis lies, and the lateral view images are more used to provide structural overlap information for stitching alignment. Therefore, performing refractive correction only on the frontal view image can not only meet the measurement accuracy requirements but also avoid unnecessary optical correction of the lateral view images, simplifying the process.

[0037] Step S1022: Preprocess the anterior segment images to obtain a mask image of the effective region including the cornea and iris.

[0038] Specifically, methods such as contrast stretching and gamma enhancement are respectively performed on the three collected anterior segment images to increase the contrast of the anterior segment images. Subsequently, the maximum inter-class variance method is used to perform binarization processing on the enhanced images to obtain a mask image of the effective region including the cornea and iris.

[0039] Step S1023: Extract the upper and lower boundary point sets of the anterior chamber angle based on the mask image of each anterior segment image, and obtain the registration parameters by means of line fitting.

[0040] In step S1023, after obtaining the mask image of each anterior segment image, accurately locate the upper and lower boundary point sets of the anterior chamber angle from the mask image. The anterior chamber angle is the angle between the cornea and the iris, and the registration parameters required for subsequent rigid registration are obtained by line fitting.

[0041] In implementation, search for the anterior chamber angle boundary points of the mask image obtained by preprocessing column by column , specifically, referring to Figure 4 , which is a schematic diagram of a use case for determining the upper and lower boundary points of the anterior chamber angle in the embodiments of the present application. Search for the first consecutive pixels with a value of 255 from top to bottom in each column of the mask image, and then the coordinates of the point where the next consecutive pixels are 0 are , then the upper boundary point of the anterior chamber angle in this column is ; then search for the first consecutive pixels with a value of 255 from bottom to top in this column, and the coordinates of the point where the next consecutive n pixels are 0 are , then the lower boundary point of the anterior chamber angle in this column is , and . That is, for the mask image of each anterior segment image, pixel scanning is performed column by column from left to right. First, find the first position in each column that satisfies the condition of having n consecutive foreground pixels followed by n background pixels from top to bottom, and record the upper boundary point of this column as the position offset n - 1 rows downward from the start of this foreground; then, search for the starting position that meets the same condition from bottom to top in the same column, and record the lower boundary point as the position offset n - 1 rows upward from this position. In this way, a set of upper boundary candidate points and lower boundary candidate points distributed in the column direction can be obtained.

[0042] In addition, since there may be noise or local breaks in the mask image, directly using all candidate points may lead to unstable fitting results. Therefore, first apply the RANSAC algorithm to the two sets of points of the upper and lower boundaries respectively through random sampling and inlier consistency testing to effectively eliminate the outliers that deviate significantly from the overall trend. Subsequently, use the remaining candidate points to perform linear fitting on the upper and lower boundaries respectively to obtain two smooth lines representing both sides of the anterior chamber angle. After completing the line fitting, calculate the intersection point of these two boundary lines. This intersection point reflects the vertex coordinates of the anterior chamber angle, and at the same time obtain the inclination angle of any boundary line with the horizontal axis. The above intersection point and inclination angle together constitute the registration parameters required for rigid registration: the intersection point is used as the rotation center for reference positioning, and the inclination angle is used for subsequent image rotation correction.

[0043] Step S1024: Use the anterior segment OCT frontal image as the reference image, and the anterior segment OCT left image and the anterior segment OCT right image as the floating images, and perform rigid registration according to the obtained registration parameters.

[0044] In step S1024, use the anterior segment OCT frontal image among the three anterior segment images as the reference image, and the anterior segment OCT left image and the anterior segment OCT right image as the floating images, and perform rigid registration on the floating images based on the registration parameters obtained in step S1023 to achieve consistent alignment of the three images in spatial position and angle.

[0045] Specifically, the registration parameters include the vertex coordinates of the anterior chamber angle in each anterior segment image and the tilt angle with respect to the horizontal axis . During the registration process, first, by comparing the coordinates of the anterior chamber angle vertices in the floating image and the reference image , calculate the displacement differences in the horizontal and vertical directions between them, and translate the floating image accordingly so that the anterior chamber angle vertices are aligned with the reference image in spatial position. Then, with the aligned anterior chamber angle vertex as the rotation center, calculate the difference in the tilt angles of the anterior chamber angle boundary lines in the floating image and the reference image, and use this difference as the rotation angle to adjust the pose direction of the floating image to achieve alignment in structural angle.

[0046] Through the above translation and rotation operations, perform a rigid transformation on the floating image to make it precisely aligned with the reference image in structural features, ensuring that the three images have a unified geometric reference framework, providing a standardized basis for subsequent image fusion, analysis, and feature extraction.

[0047] In addition, preferably, after completing the above rigid transformation, to further improve the alignment accuracy of the local structures between the images, further perform longitudinal fine-tuning on the floating images. Since there may be slight vertical offsets during the actual image shooting process, rigid registration based only on the anterior chamber angle may not be able to completely eliminate all displacement errors. Therefore, on this basis, introduce a correlation-based template matching method to match the overlapping regions of the reference image and the floating images to obtain an accurate longitudinal translation compensation amount.

[0048] Specifically, use normalized cross-correlation matching to measure the matching degree between the template image and the corresponding regions of the floating image. First, extract the key region overlapping with the floating image from the reference image as the template image, and form all the pixels in the template image into a row vector in column order , that is, the feature vector of the template image. Then, a vertical sliding window is selected in the floating image, and candidate regions are extracted row by row in its corresponding overlapping region and also converted into feature vectors , which is called the feature vector of the detection region. The similarity between regions is measured by calculating the cosine value of the angle between the two feature vectors as follows: ; Among them, and are the Euclidean norms of the feature vectors and respectively. After completing the entire sliding process, the cosine values of the angles corresponding to all candidate regions are statistically calculated, and the position corresponding to the maximum cosine value of the angle is selected as the optimal matching position between the template image and the floating image. The vertical offset of this position relative to the initial search starting point of the template image is the number of pixels that the floating image should be adjusted vertically. Taking this offset as the compensation translation amount, the floating image is vertically translated, and finally the precise alignment of the overlapping region is achieved, thereby further improving the registration quality of the entire image in terms of local details.

[0049] Step S1025: Use a method based on gray-weighting to fuse the registered anterior segment images to generate an anterior segment OCT mosaic image.

[0050] In step S1025, for the reference image and the floating image after rigid registration and translational fine-tuning, a method based on gray-value weighting is used for image fusion to generate a complete, continuous, and naturally transitioning anterior segment OCT mosaic image. This fusion operation is mainly applied to the overlapping regions between the frontal image and its left and right floating images, aiming to eliminate artifacts such as gray-level mutations and edge breaks that may exist at the image splicing locations.

[0051] Specifically, the overlapping region on one side of the frontal image and its corresponding overlapping region of the lateral image, the overlapping region of the fused image can be expressed as: Among them, represents the corresponding overlapping region in the frontal image, represents the corresponding overlapping region in the floating image (left or right), and represent the weight coefficients of the images and respectively, and . In the overlapping region on the other side, the fusion process is carried out in the same weighted manner as above.

[0052] Step S103: Use the simultaneously acquired fundus laser image to perform horizontal and vertical registration on the retinal images from different perspectives, fuse the registered retinal images, and perform curvature correction to obtain the retinal OCT mosaic image.

[0053] Refer to Figure 5 , which is the flow schematic diagram of step S103 in the embodiments of the present application. This method includes at least the following steps: Step S1031: Use the frontal retinal OCT image as the reference image, and the left retinal OCT image and the right retinal OCT image as the floating images, and use the simultaneously acquired fundus laser image to perform horizontal and vertical registration on the retinal images.

[0054] In step S1031, the purpose is to use the simultaneously acquired fundus laser image to assist in the horizontal and vertical registration of the retinal images. Use the frontal retinal OCT image as the reference image, and use the left retinal OCT image and the right retinal OCT image as the floating images. Since the retinal images from different perspectives have problems such as high noise and unclear features, it is relatively difficult to directly perform the registration of the retinal images. Therefore, the registration of the retinal OCT images is indirectly performed through the simultaneously acquired fundus laser image.

[0055] Specifically, for any one of the frontal retinal OCT image and the left or right retinal OCT image, each corresponds to a simultaneously acquired fundus laser image. Use these two fundus laser images as the registration input, use the scale-invariant feature transform algorithm to extract the feature points of the fundus laser image, and then use the fast nearest neighbor search package to match the extracted feature points to find similar feature point pairs. Then, use the RANSAC algorithm to screen the matching points to remove the incorrect matching points to ensure the reliability of the matching point pairs. Based on the screened valid feature point pairs, calculate the mean value of their horizontal distances to obtain the horizontal coincidence length of the fundus laser image , and calculate the horizontal coincidence length of the retinal image according to the horizontal coincidence length of the fundus laser image as follows: ; where is the image width of the fundus laser image in this registration pair (the width of the left fundus laser image or the right fundus laser image), is the image width of the corresponding retinal OCT image (such as the width of the left retinal OCT image or the right retinal OCT image). According to the horizontal coincidence length of the retinal image, determine the alignment position of the reference image and the floating image in the horizontal direction, and adjust the position of the floating image in the horizontal direction to ensure its accurate alignment with the reference image in the horizontal direction.

[0056] After completing the horizontal registration, vertical registration is further performed. Specifically, based on the horizontally overlapping length calculated above, the corresponding column pixel positions of the reference image and the floating image at the splicing position are determined. Based on these column pixels, using the template matching method based on cross-correlation, the displacement difference between the reference image and the floating image in the vertical direction is calculated to obtain the vertical translation parameter. The floating image is subjected to a vertical translation transformation operation according to the vertical translation parameter, thereby achieving vertical position alignment.

[0057] It should be noted that the principle of the template matching method based on cross-correlation is the same as that of step S1024, and will not be elaborated here.

[0058] It should be noted that the above horizontal and vertical registration operations are respectively performed independently on the left image and the right image once, and finally accurate registration of the three retinal images in the unified coordinate system is achieved.

[0059] Step S1032: The registered retinal images are fused using a method based on gray weighting to generate a retinal OCT splicing image.

[0060] It should be noted that the principle here is the same as that of step S1025, and will not be elaborated here.

[0061] Step S1033: According to the imaging characteristics of retinal OCT and the optical structure model, curvature correction is performed on the retinal OCT splicing image to generate a retinal OCT splicing image close to the true fundus curvature.

[0062] In step S1033, during the retinal OCT imaging process, the scanning beam scans and images the fundus in a fan shape with the scanning axis point at the pupil as the center. Therefore, each frame of OCT image can be approximately regarded as an isophase plane perpendicular to the scanning light. In three-dimensional space, these planes are actually spherical slices formed around the scanning axis point. This imaging method causes the retinal structure in the image to show a certain curved shape. Therefore, if directly analyzed or three-dimensionally reconstructed after splicing multiple retinal images, the uncorrected curvature image will not accurately reflect the actual fundus spatial structure. Therefore, it is necessary to perform curvature correction on the spliced image according to the OCT imaging optical model to make its geometric shape fit the true fundus sphere and improve spatial consistency and analysis accuracy.

[0063] Specifically, as Figure 6 shown, the essence of curvature correction is to map the spliced image to a fan-shaped ring image structure from the posterior vitreous to the fundus. The center of the circle where the fan-shaped ring is located is the scanning axis point at the pupil, and the concentric circles of this circle represent the equal optical path surfaces. According to the splicing width of the image and the optical parameters of OCT imaging, the opening angle corresponding to the complete spliced image is calculated As follows: ; Among them, is the scanning field angle of the retinal OCT image, is the width of the retinal OCT mosaic image, is the image width of the retinal OCT image. The opening angle is used to determine the spanning range of the entire mosaic image in the field of view.

[0064] Next, for any point in the fan-shaped ring structure, its physical distance to the center of the circle is estimated according to the optical path and the refractive index of the medium, and its calculation formula is as follows: ; Among them, and are respectively the optical path thicknesses of the lens and the vitreous at the point position, and are respectively the refractive indices of the lens and the vitreous.

[0065] After the calculation is completed, each pixel point in the retinal OCT mosaic image is combined with its specific scanning angle determined relative to the opening angle and the corresponding physical distance , and is transformed from the Cartesian coordinate system to the polar coordinate system with the scanning axis point as the pole to obtain its true spatial position distribution. Subsequently, through the inverse transformation from polar coordinates to Cartesian coordinates, the retinal OCT mosaic image is mapped back to the planar image, thus completing the correction of curvature distortion.

[0066] Among them, in order to determine the specific scanning angle corresponding to each pixel point in the mosaic image, first, based on the calculated opening angle , a correspondence is established between the width range of the mosaic image and the field angle range. That is, the leftmost pixel point in the mosaic image is corresponded to the starting angle of the opening angle range, the rightmost pixel point is corresponded to the ending angle of the opening angle range, and the remaining pixel points are mapped to the opening angle range according to their relative horizontal positions in the mosaic image in a linear proportion. Through the above method, a specific scanning angle can be assigned to each pixel point in the mosaic image, so that in the subsequent conversion process between the Cartesian coordinate system and the polar coordinate system, combined with the corresponding physical distance accurate spatial position reconstruction and curvature correction are realized.

[0067] Step S104: Mosaic the anterior segment OCT mosaic image and the retinal OCT mosaic image to obtain a full-eye image.

[0068] In step S104, since the anterior segment and the retinal images are from different sources, they usually have different physical resolutions. To ensure that the stitched image can be presented at the same scale, it is first necessary to adjust the size of the retinal OCT stitched image to make its physical resolution consistent with that of the anterior segment OCT stitched image. Subsequently, when stitching the images, it is first necessary to determine the relative positions of the anterior segment OCT stitched image and the retinal OCT stitched image. The positioning relationship between the two is calculated based on the scanning axis point at the pupil. Specifically, find the intersection point of the visual axis and the line connecting the two anterior chamber angles in the anterior segment OCT stitched image. This intersection point is the same as the center of the circle where the fan-shaped ring is located after curvature correction in the retinal OCT stitched image. The relative positions of the anterior segment OCT stitched image and the retinal OCT stitched image are determined according to the coordinates of the intersection point. The anterior segment OCT stitched image and the retinal OCT stitched image are moved and stitched according to the relative positions obtained above to obtain a full-eye stitched image. As Figure 7 shown, from top to bottom and from left to right are the anterior segment OCT stitched image, the retinal OCT stitched image, and the stitched full-eye image.

[0069] In summary, the present application provides an automatic full-eye image stitching method based on swept-source OCT, aiming to solve the problem of how to achieve seamless stitching of the anterior segment image and the retinal image while ensuring the presentation of high-resolution details in the prior art. Although the existing swept-source OCT technology can provide high-resolution anterior segment images and retinal images, due to its field of view limitation and image registration accuracy problems, the stitched image cannot achieve the effect of a high-precision full-eye image, and there are often defects such as unnatural stitching and detail loss.

[0070] In view of the deficiencies of the prior art, the present application first collects anterior segment images and retinal images from different perspectives by combining manual guidance, eliminating coverage incompleteness and registration difficulties caused by limited field of view at the source; subsequently, preprocesses the anterior segment images from different perspectives to obtain anterior chamber angle information, performs rigid registration on the anterior segment images based on the anterior chamber angle information, and fuses the registered anterior segment images to obtain an anterior segment OCT mosaic image; not only retains high-resolution tissue details but also achieves natural transitions between regions, solving the problem of unclear details caused by the trade-off between single-scan resolution and signal-to-noise ratio; then uses the simultaneously acquired fundus laser images to perform horizontal and vertical registration on the retinal images from different perspectives, fuses and curvature corrects the registered retinal images to obtain a retinal OCT mosaic image, introduces fundus laser images to assist in achieving highly robust horizontal and vertical registration, and restores the true fundus curvature through spherical curvature correction, effectively compensating for the structural discontinuity and distortion at the splicing junction of the traditional segmented correction method; finally, seamlessly synthesizes the anterior segment and retinal mosaic images on the basis of unified physical scale and scanning axis positioning, achieving both high resolution and overall coherence, thereby obtaining a more complete structure and clearer details of the whole-eye image.

[0071] Figure 8 FIG. is a structural block diagram of an automatic whole-eye image stitching system based on swept-source OCT provided by an embodiment of the present application. The device at least includes the following modules: An image acquisition module, configured to collect anterior segment images and retinal images from different perspectives by combining manual guidance; An anterior segment image stitching module, configured to preprocess the anterior segment images from different perspectives to obtain anterior chamber angle information, perform rigid registration on the anterior segment images based on the anterior chamber angle information, and fuse the registered anterior segment images to obtain an anterior segment OCT mosaic image; A retinal image stitching module, configured to use the simultaneously acquired fundus laser images to perform horizontal and vertical registration on the retinal images from different perspectives, fuse and curvature correct the registered retinal images to obtain a retinal OCT mosaic image; A whole-eye image stitching module, configured to stitch the anterior segment OCT mosaic image and the retinal OCT mosaic image to obtain a whole-eye image.

[0072] For related details, refer to the above method embodiment.

[0073] Figure 9 FIG. is a block diagram of an electronic device provided by an embodiment of the present application. The device at least includes a processor 401 and a memory 402.

[0074] The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 401 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 401 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0075] The memory 402 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 402 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 401 to implement the method embodiment of the automatic stitching method of full-eye images based on swept-source OCT provided in this application.

[0076] In some embodiments, the electronic device may further optionally include: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402, and the peripheral device interface may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface through a bus, signal lines, or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply, etc.

[0077] Of course, the electronic device may also include fewer or more components, and this embodiment does not limit this.

[0078] Optionally, this application also provides a computer-readable storage medium, and a program is stored in the computer-readable storage medium, and the program is loaded and executed by the processor to implement the automatic stitching method of full-eye images based on swept-source OCT in the above method embodiment.

[0079] Optionally, the present application also provides a computer product, which includes a computer-readable storage medium. A program is stored in the computer-readable storage medium and is loaded and executed by a processor to implement the method for automatically stitching full-eye images based on swept-source OCT in the above method embodiments.

[0080] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0081] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. An automatic full-eye image stitching method based on swept-source OCT, characterized in that, The method includes: Combining manual guidance to collect anterior segment images and retinal images from different perspectives; the retinal images from different perspectives include corresponding fundus laser images; Preprocessing the anterior segment images from different perspectives to obtain anterior chamber angle information, rigidly registering the anterior segment images based on the anterior chamber angle information, and fusing the registered anterior segment images to obtain an anterior segment OCT mosaic image; Using the simultaneously acquired fundus laser images to perform horizontal and vertical registration on the retinal images from different perspectives, fusing and curvature correction on the registered retinal images to obtain a retinal OCT mosaic image; Stitching the anterior segment OCT mosaic image and the retinal OCT mosaic image to obtain a full-eye image.

2. The automatic full-eye image stitching method based on swept-source OCT according to claim 1, wherein The combining manual guidance to collect anterior segment images and retinal images from different perspectives includes: The anterior segment images from different perspectives include anterior segment OCT frontal images, anterior segment OCT left images, and anterior segment OCT right images; The retinal images from different perspectives include retinal OCT frontal images and corresponding fundus laser images, retinal OCT left images and corresponding fundus laser images, and retinal OCT right images and corresponding fundus laser images.

3. The automatic full-eye image stitching method based on swept-source OCT according to claim 2, wherein The preprocessing the anterior segment images from different perspectives to obtain anterior chamber angle information, rigidly registering the anterior segment images based on the anterior chamber angle information, and fusing the registered anterior segment images to obtain an anterior segment OCT mosaic image includes: Performing refractive correction on the anterior segment OCT frontal image in the anterior segment images; Preprocessing the anterior segment images to obtain a mask image of the effective region including the cornea and iris; After obtaining the mask image of each anterior segment image, accurately locating the upper and lower boundary point sets of the anterior chamber angle from the mask image, and applying the RANSAC algorithm to the two sets of points of the upper and lower boundaries respectively through random sampling and inlier consistency test to remove the outliers with large deviation from the overall trend; Use the remaining candidate points to perform linear fitting on the upper and lower boundaries respectively, obtaining two smooth lines representing both sides of the anterior chamber angle. After completing the line fitting, calculate the intersection point of these two boundary lines , the intersection point is the vertex coordinates of the anterior chamber angle. Obtain the inclination angle of any boundary line with the horizontal axis , the intersection point and the inclination angle together constitute the registration parameters required for rigid registration.

4. The automatic full-eye image stitching method based on swept-source OCT according to claim 3, wherein The preprocessing the anterior segment images from different perspectives to obtain anterior chamber angle information, rigidly registering the anterior segment images based on the anterior chamber angle information, and fusing the registered anterior segment images to obtain an anterior segment OCT mosaic image further includes: Taking the anterior segment OCT frontal image as the reference image, and the anterior segment OCT left image and the anterior segment OCT right image as the floating images, and performing rigid registration according to the obtained registration parameters; After completing the rigid registration, the floating image is finely adjusted longitudinally. The normalized cross-correlation matching is used to measure the matching degree between the corresponding regions of the template image and the floating image. The key region that coincides with the floating image is extracted from the reference image as the template image, and all the pixels in the template image are arranged in column order to form a row vector , that is, the feature vector of the template image; a vertical sliding window is selected in the floating image, and candidate regions are extracted by sliding row by row in its corresponding overlapping region, and are also converted into feature vectors , that is, the feature vector of the detection region; the similarity between regions is measured by calculating the cosine value of the angle between the two feature vectors, as shown below: ； Among them, and are the Euclidean norms of the feature vectors and respectively. After completing the entire sliding process, the cosine values of the angles corresponding to all candidate regions are statistically calculated, and the position corresponding to the largest cosine value of the angle is selected as the position with the optimal match between the template image and the floating image. The vertical offset of this position relative to the initial search starting point of the template image is used as the compensation translation amount, and the floating image is vertically translated using this offset amount; Using a method based on gray weight to fuse the registered anterior segment images to generate an anterior segment OCT mosaic image.

5. The automatic full-eye image stitching method based on swept-source OCT according to claim 2, wherein The using the simultaneously acquired fundus laser images to perform horizontal and vertical registration on the retinal images from different perspectives, fusing and curvature correction on the registered retinal images to obtain a retinal OCT mosaic image includes: Taking the retinal OCT frontal image as the reference image, and the left retinal OCT image and the right retinal OCT image as the floating images. For any one of the retinal OCT frontal image and the left or right retinal OCT image, each corresponds to a fundus laser image. Using the two fundus laser images as the registration input, the scale-invariant feature transform algorithm is used to extract the feature points of the fundus laser image, and the fast nearest neighbor search package is used to match the extracted feature points to find similar feature point pairs; Use the RANSAC algorithm to screen the matching points. Based on the filtered valid feature point pairs, calculate the mean of their horizontal distances to obtain the horizontal coincidence length of the fundus laser image , and calculate the horizontal coincidence length of the retinal image according to the horizontal coincidence length of the fundus laser image As follows: ； Among them, is the image width of the fundus laser image in the registration pair, is the image width of the corresponding retinal OCT image. According to the horizontal coincidence length of the retinal images , determine the alignment position of the reference image and the floating image in the horizontal direction, and adjust the position of the floating image in the horizontal direction; According to the calculated horizontal coincidence length, determine the column pixel positions corresponding to the reference image and the floating image at the splicing position. Based on this column of pixels, calculate the displacement difference between the reference image and the floating image in the vertical direction to obtain the vertical translation parameter. Perform a vertical translation transformation operation on the floating image according to the vertical translation parameter to achieve vertical position alignment.

6. The automatic full-eye image stitching method based on swept-source OCT according to claim 5, wherein The method of using the synchronously acquired fundus laser images to perform horizontal and vertical registration on retinal images with different perspectives, fusing the registered retinal images, and performing curvature correction to obtain the retinal OCT spliced image further includes: Adopt a method based on gray-weighting to fuse the registered retinal images to generate a retinal OCT spliced image; According to the retinal OCT imaging characteristics and the optical structure model, perform curvature correction on the retinal OCT spliced image to generate a retinal OCT spliced image close to the true fundus curvature; The curvature correction maps the spliced image to a fan-shaped ring image structure from the posterior end of the vitreous body to the fundus oculi, and the center of the circle where the fan-shaped ring is located is the scanning axis point at the pupil. According to the splicing width of the image and the optical parameters of OCT imaging, the opening angle corresponding to the complete spliced image is calculated as follows: ； Among them, is the scanning field of view angle of the retinal OCT image, is the width of the retinal OCT mosaic image, is the image width of the retinal OCT image; For any point in the frustum of a circular cone structure , its physical distance to the center of the circle is estimated based on the optical path and the refractive index of the medium, and its calculation formula is as follows: ​ ； Among them, and are respectively the optical path thicknesses of the lens and the vitreous at the point position, and are respectively the refractive indices of the lens and the vitreous; After the calculation is completed, each pixel in the retinal OCT mosaic image is combined with its scanning angle determined relative to the opening angle and the corresponding physical distance , transformed from the Cartesian coordinate system to the polar coordinate system with the scanning axis point as the pole to obtain its true spatial position distribution, and then through the inverse transformation from polar coordinates to Cartesian coordinates, the retinal OCT mosaic image is mapped back to a planar image to complete the correction of curvature distortion.

7. The automatic full-eye image stitching method based on swept-source OCT according to claim 1, characterized in that The method of splicing the anterior segment OCT spliced image and the retinal OCT spliced image to obtain the whole-eye image includes: Adjust the size of the retinal OCT spliced image to make its physical resolution consistent with that of the anterior segment OCT spliced image; When splicing the images, find the intersection point of the visual axis and the connecting line of the two anterior chamber angles in the anterior segment OCT spliced image, and determine the relative positions of the anterior segment OCT spliced image and the retinal OCT spliced image according to the coordinates of the intersection point; According to the obtained relative positions of the anterior segment OCT spliced image and the retinal OCT spliced image, move and splice the two to obtain the whole-eye spliced image.

8. An automatic full-eye image stitching system based on swept-source OCT, characterized in that, Including: An image acquisition module for collecting anterior segment images and retinal images with different perspectives in combination with manual guidance; An anterior segment image splicing module for preprocessing the anterior segment images with different perspectives to obtain anterior chamber angle information, performing rigid registration on the anterior segment images based on the anterior chamber angle information, and fusing the registered anterior segment images to obtain an anterior segment OCT spliced image; A retinal image splicing module for using the synchronously acquired fundus laser images to perform horizontal and vertical registration on retinal images with different perspectives, fusing the registered retinal images, and performing curvature correction to obtain a retinal OCT spliced image; A whole-eye image splicing module for splicing the anterior segment OCT spliced image and the retinal OCT spliced image to obtain a whole-eye image.

9. An electronic device, characterized in that, The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement a method for automatically splicing a whole-eye image based on swept-source OCT as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A program is stored in the storage medium, and when the program is executed by a processor, it is used to implement an automatic full-eye image stitching method based on swept-source OCT according to any one of claims 1 to 7.

Images