Fundus camera multi-image adaptive panoramic stitching method, device and storage medium

Abstract

This application discloses a method, device, and storage medium for adaptive panoramic stitching of multiple images from a fundus camera. The method includes: matching feature points of a reference image and the images to be stitched to obtain a transformation matrix; binarizing the reference image and the perspective-transformed images to be stitched to extract the maximum outer contour, performing a logical AND operation on the filled regions of the maximum outer contour to obtain the contour intersection; extracting connected regions from the contour intersection, using the geometric center of the connected regions as intersection points to delineate boundary lines in the overlapping areas of the reference image and the perspective-transformed images to be stitched to generate a fusion mask; fusing the reference image and the images to be stitched using the fusion mask across multiple frequency bands to obtain a stitched image; and repeatedly performing the image stitching operation on the stitched image as a new reference image until all images to be stitched are processed and the reference image is output. This application achieves the technical effect of providing anatomically complete and globally consistent panoramic fundus images through adaptive intersection point positioning of the fusion boundary, multi-image joint verification to eliminate accumulated errors, and roundness quality gating.

Description

Technical Field

[0001] This application relates to the field of fundus camera control technology, and in particular to a method, device and storage medium for adaptive panoramic stitching of multiple images from a fundus camera. Background Technology

[0002] Fundus cameras are widely used in clinical ophthalmology examinations for imaging and diagnosis of the retina, optic disc, and macular region. Due to the large curvature of the eyeball and the limited pupil diameter, the field of view of a single fundus image is typically limited to 30 to 50 degrees (45 degrees is the standard), making it impossible to simultaneously cover the entire macular region, optic disc, and peripheral retina. To achieve panoramic fundus imaging, clinicians need to capture multiple local fundus images from different angles, and then synthesize a wide-field panoramic image using image stitching technology. In recent years, the demand for panoramic fundus stitching technology has been increasing in scenarios such as diabetic retinopathy screening, optic disc assessment in glaucoma, and follow-up of retinopathy of prematurity, and related technical solutions have become a research hotspot in the field of ophthalmic imaging.

[0003] Traditional manual feature extraction methods use scale-invariant feature transformation and accelerated robust feature extraction local feature extractors to detect key points. Feature correspondence is established through approximate nearest neighbor search or brute force matching. Then, the homography matrix is ​​estimated using random sampling consistency. Finally, the stitching is completed through weighted averaging or multi-band fusion.

[0004] However, in terms of multi-image stitching, the chain registration strategy gradually amplifies the error as it propagates along the chain when stitching more than three images. Although some solutions use bundle adjustment for global optimization, the computational load is too large, making it difficult to achieve real-time stitching on embedded fundus cameras.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a method, device, and storage medium for adaptive panoramic stitching of multiple images from a fundus camera, aiming to solve the technical problem that existing fundus image stitching lacks systematic adaptive capabilities, resulting in the inability to stably output high-quality panoramic fundus images under complex clinical imaging conditions.

[0007] To achieve the above objectives, this application proposes a multi-image adaptive panoramic stitching method for fundus cameras, the method comprising: Feature points are extracted from the obtained baseline image and the image to be pieced together, respectively. Matching is performed based on the extracted feature points, and the transformation matrix of the image to be pieced together to the baseline image is estimated based on the matching results. The puzzle image is then subjected to a perspective transformation using the transformation matrix, and the perspective-transformed puzzle image is then... Figure 2 Value-based extraction to extract the maximum outer contour; The validity of the puzzle image after perspective transformation is verified based on the maximum outer contour. After successful verification, the reference image and the puzzle image after perspective transformation are binarized respectively, and the maximum outer contour after binarization is extracted respectively. The intersection of the contours is obtained by performing a logical AND operation on the filled area of ​​the maximum outer contour. The connected regions are extracted from the intersection of the contours. The geometric center of the connected regions is used as the intersection point. The intersection point is used to draw a boundary line in the overlapping area of ​​the reference image and the image to be pieced together after perspective transformation. A fusion mask is generated based on the boundary line. The reference image and the perspective-transformed image to be stitched together are multi-band fused according to the fusion mask to obtain a stitched image. The fusion mask divides the overlapping area into a reference image retention area and an image to be stitched together retention area. The stitched image is used as a new reference image. The next image to be stitched is obtained as a new image to be stitched, and the image stitching operation is repeated until all images to be stitched are processed. The reference image obtained from the last image stitching operation is output as a panoramic fundus image.

[0008] In one embodiment, when the number of stitched images exceeds a preset number, after the steps of binarizing the reference image and the perspective-transformed image to be stitched, extracting the maximum outer contour of the binarized image, and performing a logical AND operation on the filled area of ​​the maximum outer contour to obtain the contour intersection, the method further includes: Obtain the overall outline of the stitched image, calculate the global outline intersection between the maximum outer outline of the image to be stitched after perspective transformation and the overall outline of the stitched image, and extract the global intersection points from the global outline intersection. Each historical reference image used as the reference image in each image stitching operation is obtained. The intersection of the historical contour between the maximum outer contour of the image to be stitched after perspective transformation and the maximum outer contour of each historical reference image is calculated. Historical intersection points are extracted from each historical contour intersection. Each historical intersection point is matched with the global intersection point, and a matching relationship between each historical intersection point and the global intersection point is determined based on the coordinate distance threshold. When there are two or more matching relationships, calculate the intersection point of the line corresponding to each matching relationship, and add the line intersection point as a new intersection point.

[0009] In one embodiment, after the step of defining a boundary line in the overlapping area of ​​the reference image and the perspective-transformed image to be pieced together using the intersection point, and generating a fusion mask based on the boundary line, the method further includes: The number of intersections of the puzzle to be completed is obtained. When the number of intersections is less than a first preset threshold, the center distance between the center of the image of the puzzle to be completed and the center of the image of the reference image is obtained. When the center distance is less than a preset distance threshold, a weighted superposition and fusion is performed on the reference image and the perspective-transformed image to be stitched together to obtain a stitched image. When the center distance is greater than or equal to the preset distance threshold, the puzzle piece to be processed is abandoned and the next puzzle piece is obtained as the new puzzle piece; Based on the new image to be pieced together and the baseline image, the step of extracting feature points is repeated.

[0010] In one embodiment, after the step of obtaining the number of intersections of the puzzle piece, the method further includes: Count the number of intersections; When the number of intersections is greater than the second preset number threshold, it is determined that the overlapping area between the image to be stitched and the reference image is too large. Multi-band fusion is abandoned for the image to be stitched, and the reference image is used as the stitched image and the multi-band fusion operation is skipped. The second preset number threshold is greater than the first preset number threshold.

[0011] In one embodiment, before the step of extracting feature points from the acquired reference image and the image to be pieced together, the method further includes: For each input raw fundus image, perform circular region of interest segmentation, generate a circular mask with the center of the raw fundus image as the center and half the width of the raw fundus image as the radius, and remove the black corner areas of the raw fundus image through the circular mask; Contrast-limited adaptive histogram equalization enhancement is performed on the cropped image to improve the contrast between the vascular texture and the background region in the image. The enhanced image is scaled down proportionally to a preset scaling size to reduce the computational load of subsequent feature extraction and matrix operations; A preset number of black pixel borders are added to the top, bottom, left, and right sides of the reduced image to reserve pixel buffer space for perspective transformation, thus obtaining the reference image or the image to be pieced together.

[0012] In one embodiment, the steps of extracting feature points from the acquired reference image and the image to be pieced together, matching based on the extracted feature points, and estimating the transformation matrix of the image to be pieced together to the reference image based on the matching results include: Extract feature points from the baseline image and the image to be pieced together; An approximate nearest neighbor search algorithm is used to match the feature points of the baseline image with the feature points of the image to be pieced together, and the nearest neighbor feature point and the second nearest neighbor feature point are returned for each query point; Calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance for each query point, and retain matching pairs whose ratio is less than a first ratio threshold as initial screening matching pairs; When the number of initial screening matches is less than the preset number of matches, the first ratio threshold is relaxed to the second ratio threshold, and the matches are re-screened. The transformation matrix from the selected matching pairs to the base image is estimated using a random sampling consensus algorithm.

[0013] In one embodiment, before the step of validating the perspective-transformed puzzle image based on the maximum outer contour, the method further includes: The center point of the reference image is transformed to the coordinate system of the image to be pieced together using the transformation matrix to obtain the transformed center point; Calculate the displacement distance between the transformed center point and the center point of the reference image; When the displacement distance exceeds the preset displacement threshold, it is determined that the shooting position of the image to be stitched is too far away from the shooting position of the reference image, the image to be stitched is abandoned and the next image to be stitched is obtained as the new image to be stitched. Based on the new image to be pieced together and the baseline image, the step of extracting feature points is repeated.

[0014] In one embodiment, after the steps of using the stitched image as a new reference image, obtaining the next image to be stitched as a new image to be stitched, and repeating the image stitching operation until all images to be stitched are processed, and outputting the reference image obtained from the last image stitching operation as a panoramic fundus image, the method further includes: Count the number of images in the entire dataset that passed the validity verification and were successfully stitched into the panoramic fundus image; When the number of images is zero, output a stitching failure message and output the initially acquired reference image; When the number of images is greater than zero and less than the total number of images to be pieced together, a success message is output. The message includes a list of successfully pieced-together image names and the displacement vector of each image.

[0015] In addition, to achieve the above objectives, this application also proposes a fundus camera multi-image adaptive panoramic stitching device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the fundus camera multi-image adaptive panoramic stitching method as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the fundus camera multi-image adaptive panoramic stitching method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: Feature points are extracted from both the acquired baseline image and the image to be pieced together. Matching is performed based on the extracted feature points, and a transformation matrix is ​​estimated based on the matching results to transform the image to be pieced together to the baseline image. A perspective transformation is then performed on the image to be pieced together using the transformation matrix, and the perspective-transformed image to be pieced together is then... Figure 2 The image is binarized to extract the maximum outer contour. The validity of the perspective-transformed image to be stitched is verified based on the maximum outer contour. Upon successful verification, the reference image and the perspective-transformed image to be stitched are binarized, and the maximum outer contour of each binarized image is extracted. A logical AND operation is performed on the filled area of ​​the maximum outer contour to obtain the contour intersection. Connected regions are extracted from the contour intersection, and the geometric center of the connected region is used as the intersection point. A boundary line is drawn in the overlapping area of ​​the reference image and the perspective-transformed image to be stitched, and a fusion mask is generated based on the boundary line. Multi-band fusion is performed on the reference image and the perspective-transformed image to be stitched based on the fusion mask to obtain a stitched image. The fusion mask divides the overlapping area into a reference image retention area and an image to be stitched retention area. The stitched image is used as the new reference image, and the next image to be stitched is obtained as the new image to be stitched, and the image stitching operation is repeated until all images to be stitched are processed. The reference image obtained from the last image stitching operation is output as a panoramic fundus image.

[0018] The technical solution of this application achieves a comprehensive technological leap from fixed midline to adaptive anatomical structure for fusion boundary, from chain error accumulation to global consistency constraint for multi-image stitching, and from all-or-nothing to intelligent degradation of system response. It effectively solves the technical problem that existing fundus image stitching methods cannot stably output high-quality panoramic fundus images under complex clinical imaging conditions. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the fundus camera multi-image adaptive panoramic stitching method of this application; Figure 2 This is a detailed step diagram of step S10 in one embodiment; Figure 3 This is a flowchart illustrating the second embodiment of the fundus camera multi-image adaptive panoramic stitching method of the present invention; Figure 4 This is a flowchart illustrating the third embodiment of the fundus camera multi-image adaptive panoramic stitching method of the present invention; Figure 5 This is a flowchart illustrating the fourth embodiment of the fundus camera multi-image adaptive panoramic stitching method of the present invention; Figure 6 This is a flowchart illustrating the intersection detection method. Figure 7 Schematic diagram of the Laplacian pyramid fusion process guided by contour mask; Figure 8 A flowchart illustrating the cross-point verification of multiple spliced ​​graphs; Figure 9 This is a schematic diagram of the overall architecture of the fundus camera multi-image adaptive panoramic stitching of the present invention; Figure 10 This is a flowchart illustrating the process based on fully automated splicing units; Figure 11 This is a schematic diagram of the device structure of the hardware operating environment involved in the fundus camera multi-image adaptive panoramic stitching method in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] In the field of fundus image stitching, the mainstream technical approach mainly follows the scheme of fixing the midline after local feature matching and fusion, which has inherent defects: the fusion boundary is rigid and easily cuts off blood vessels and optic discs, multi-image chain registration leads to error accumulation, and there is a lack of effective filtering mechanism for fundus-specific artifacts.

[0026] This type of scheme typically involves a host computer extracting local feature points such as scale-invariant feature transforms or accelerated robust features, establishing correspondences through approximate nearest neighbor search or brute-force matching, estimating the homography matrix using random sampling consistency, and then completing the fusion using fixed weighting or a median-splitting method. In scenarios with rich image texture and moderate overlap, this scheme can achieve basic image stitching functionality. However, this scheme is essentially an open-loop processing flow: the fusion boundary is determined based on fixed rules and is not adjusted, failing to incorporate anatomical information into the fusion decision; during multi-image stitching, adjacent frames are registered sequentially, with errors propagating step-by-step and lacking global constraints; and it directly reports an error and exits after a matching failure, lacking intelligent degradation processing capabilities.

[0027] The core challenge faced by the above approaches lies in the fact that while the workflow architecture of fixed fusion boundaries, chained registration, and single-mode execution is intuitive, its fusion path lacks anatomical adaptive capabilities, global error constraints, and robust anti-interference quality gating and algorithm failure tolerance mechanisms. This contrasts sharply with the requirements of clinical panoramic fundus imaging for anatomical fidelity and multi-dimensionality. Figure 1 There is a fundamental contradiction between consistency and the inherent need for stability under complex conditions.

[0028] To address the aforementioned shortcomings, this application proposes a multi-image adaptive panoramic stitching method for fundus cameras. This method uses binary contour logic and operational geometry to locate the intersection points, determining the adaptive fusion boundary and allowing the fusion path to extend naturally along the fundus tissue contour, avoiding the severing of vascular structures. When stitching three or more images, multi-image stitching is achieved through joint matching of global intersection points and intersection points of each historical reference image. Figure 1 Consistency constraints are implemented to eliminate chain-like cumulative errors; a circularity quality-gated filtering abnormal registration is constructed, and a three-level degradation fusion strategy of weighted superposition or skipping is provided when there are insufficient cross points.

[0029] Through the above-mentioned technical means, this application has achieved a comprehensive leap from fixed midline to anatomical structure adaptation in fusion boundary, from chain error accumulation to global consistency constraint in multi-image stitching, and from all-or-nothing to intelligent degradation in system response. It effectively solves the technical problem that existing fundus image stitching methods cannot stably output high-quality panoramic fundus images under complex clinical imaging conditions.

[0030] Based on this, embodiments of this application provide a method for adaptive panoramic stitching of multiple images from a fundus camera, referring to... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the fundus camera multi-image adaptive panoramic stitching method of this application. In this embodiment, the fundus camera multi-image adaptive panoramic stitching method includes steps S10~S70: Step S10: Extract feature points from the obtained reference image and the image to be pieced together, perform matching based on the extracted feature points, and estimate the transformation matrix of the image to be pieced together to the reference image based on the matching results; In this embodiment, during the panoramic stitching of multiple fundus images from a fundus camera, the spatial transformation relationship between two adjacent fundus images is first established. To this end, one of the two adjacent fundus images is used as the reference image, and the other as the image to be stitched. Based on the reference image and the image to be stitched, scale-invariant feature transformation points are extracted from these two images respectively. This feature extraction process is performed in scale space. An extremum point is detected by constructing a Gaussian difference pyramid as candidate keypoints. These candidate keypoints are then precisely located, and low-contrast points and edge response points are eliminated. Finally, a principal direction is assigned to each keypoint, and a feature description vector is generated. This feature extraction method is invariant to image rotation, scale changes, and illumination changes, making it suitable for handling differences in fundus images caused by different shooting angles and lighting conditions.

[0031] After feature point extraction, an approximate nearest neighbor search algorithm is used to match the feature points of the baseline image with the feature points of the puzzle image. This matching method constructs multiple KD-trees to index the feature space. For each query point in the baseline image, it searches for the two feature points in the set of feature points to be pieced together that are closest to its feature description vector, namely the nearest neighbor and the second nearest neighbor. During the search, multiple KD-trees are traversed and a preset number of leaf nodes are checked to ensure the accuracy of the search results.

[0032] After obtaining the nearest and second nearest neighbors, the ratio of the nearest neighbor distance to the second nearest neighbor distance is calculated. If this ratio is less than a pre-set first ratio threshold, it indicates that the match between the nearest neighbor feature point and the query point is significantly better than that between the second nearest neighbor. In this case, the matching pair is determined to be a reliable initial screening matching pair and is retained. If the ratio is large, it indicates that the distinction between the nearest and second nearest neighbors is insufficient, and the matching pair has a high risk of mismatch, so it is discarded. This ratio test mechanism can effectively filter out a large number of erroneous matches.

[0033] After the ratio test screening, the number of initial matching pairs is counted. If this number is less than the preset number of matching pairs, it indicates that the texture information between the current image pairs is insufficient to support a reliable estimation of the subsequent transformation matrix. In this case, the first ratio threshold is automatically relaxed to the second ratio threshold, and the matching pair screening is re-executed to retain more matching pairs for subsequent calculations. This fallback strategy ensures that sufficient matching points can be obtained even in low-texture regions of the fundus image.

[0034] After obtaining a sufficient number of matching pairs, a random sampling consensus algorithm is used to estimate the transformation matrix from the plane to be pieced together to the reference image. This algorithm calculates candidate transformation matrices by randomly sampling matching pairs, then counts the number of interior points that conform to the mapping relationship of these matrices. After multiple iterations, the candidate matrix with the most interior points is selected as the final transformation matrix estimate. This transformation matrix is ​​a homography matrix in three-dimensional space, describing the projection transformation relationship between the plane to be pieced together and the reference image plane, providing a mathematical basis for subsequent image space transformations.

[0035] Step S20: Perform perspective transformation on the puzzle image using the transformation matrix, and then... Figure 2 Value-based extraction to extract the maximum outer contour; After obtaining the transformation matrix that transforms the image to be pieced together to the reference image, this transformation matrix is ​​applied to the image to be pieced together to achieve spatial alignment. The coordinates of each pixel in the image to be pieced together are projected according to the transformation matrix to obtain the corresponding position of that pixel in the coordinate system of the reference image. Since the transformation matrix describes the projection mapping relationship between planes, after this transformation, the image to be pieced together is mapped to the spatial coordinate system of the reference image, and the overlapping areas of the two images are aligned in spatial position.

[0036] After perspective transformation, the resulting image to be stitched together is binarized. A low grayscale threshold is set, assigning white pixels with grayscale values ​​greater than the threshold and black pixels with grayscale values ​​less than or equal to the threshold. In fundus images, the grayscale value of the effective fundus tissue area is usually higher than that of the background area. After this threshold segmentation, the effective fundus area appears as a white area, while the image background appears as a black area.

[0037] The maximum outer contour is extracted from the binarized image. The contour extraction algorithm scans from the boundary of the binarized image, tracing the boundary pixels between white and black areas to form a closed contour line. In fundus images, since the field of view of the fundus camera is circular, the effective area to be stitched together after perspective transformation should theoretically present an approximately circular shape. Therefore, the extracted maximum outer contour is the boundary contour of this circular effective area.

[0038] After extracting the maximum outer contour, the circularity of this contour is calculated. Circularity is calculated using the ratio of area to perimeter, specifically four times pi multiplied by the contour area and then divided by the square of the contour perimeter. This ratio is one when the contour shape is a perfect circle; the further the contour deviates from a circle, the smaller the ratio. Since the physical characteristics of the fundus camera's imaging field of view dictate that the effective fundus area should ideally be circular, the calculated circularity value reflects whether the image to be pieced together after perspective transformation maintains its normal geometric shape.

[0039] The calculated circularity value is compared with a preset circularity threshold. If the circularity is less than the threshold, it indicates severe geometric distortion in the image to be pieced together after perspective transformation. This distortion is usually caused by abnormal homography matrix estimation results, which may stem from feature point matching errors or poor image quality. In this case, the current image to be pieced together is abandoned, and the next image to be pieced together is obtained as a new image to be pieced together, and the processing flow is re-executed. If the circularity is greater than or equal to the threshold, it indicates that the image to be pieced together after perspective transformation maintains a normal circular geometry. The registration result passes the circularity verification and can proceed to the subsequent stitching process.

[0040] Step S30: Verify the validity of the perspective-transformed puzzle image based on the maximum outer contour. After successful verification, binarize the reference image and the perspective-transformed puzzle image respectively, and extract the maximum outer contour after binarization. Perform a logical AND operation on the filled area of ​​the maximum outer contour to obtain the contour intersection. In this embodiment, after the image to be stitched passes the circularity verification, it indicates that the image has maintained a normal geometric shape after perspective transformation and can be fused and stitched. At this time, the reference image and the perspective-transformed image to be stitched are binarized separately, and the two images are segmented using a grayscale threshold. The effective fundus area is marked as white, and the background area is marked as black.

[0041] After binarization, the maximum outer contour is extracted from both binarized images. The maximum outer contour of the reference image represents the boundary of the effective fundus region in the reference image, and the maximum outer contour of the image to be stitched together after perspective transformation represents the boundary of the effective fundus region in that image. Since the two images have been spatially aligned using the transformation matrix, these two maximum outer contours overlap in space. This overlapping area is the fundus tissue region jointly covered by the two fundus images.

[0042] After extracting the two largest outer contours, the areas enclosed by these two contours are drawn on a blank canvas with white fill, forming two contour-filled images. A logical AND operation is performed on these two contour-filled images, checking pixel-by-pixel whether corresponding pixel values ​​in both images are white. If both are white, the output for that location is white; otherwise, the output is black. The result of the logical AND operation is a new binary image, where the white area represents the spatial intersection of the effective fundus regions of the two images—that is, the fundus tissue area commonly covered by both images.

[0043] The geometric significance of this contour intersection region lies in its ability to delineate the actual overlapping area of ​​fundus tissue in the two fundus images. In traditional stitching methods, the fusion boundary of the overlapping region is typically determined using a fixed midline or a weighted method based on feature point density. These methods may result in the fusion boundary traversing important anatomical structures such as blood vessels and the optic disc. However, the overlapping region determined through contour intersection geometric operations naturally extends along the geometric intersection of the effective areas of the two images, providing a geometric basis for subsequent adaptive fusion boundary localization.

[0044] Step S40: Extract connected regions from the intersection of the contours, use the geometric center of the connected regions as the intersection point, and use the intersection point to draw a boundary line in the overlapping area of ​​the reference image and the perspective-transformed image to be pieced together, and generate a fusion mask based on the boundary line. After obtaining the contour intersection image, the intersection points used to determine the fusion boundary are extracted from the image. The contour intersection image may contain multiple connected regions, where larger connected regions represent the main overlapping areas, while smaller connected regions may correspond to minor bumps or noise-induced discrete points on the contour edges. Connected regions with areas smaller than a preset area threshold are extracted from the contour intersection. These small connected regions typically appear near the intersection of two circular contours, representing geometric feature points at the intersection of the circular contours.

[0045] For each extracted small-area connected region, calculate its minimum bounding rectangle. The minimum bounding rectangle is the smallest rectangle that can contain all the pixels of the connected region, and its direction is aligned with the principal axis of the connected region. Take the center point of the minimum bounding rectangle as the geometric center of the connected region, which is an intersection point.

[0046] The physical meaning of the intersection points is that they are the geometric points where the effective circular region outlines of the reference image and the effective circular region outlines of the image to be stitched together meet at the edge of the overlapping area. The circular outlines of the two fundus images intersect at the edge of the overlapping area, and the line connecting these intersection points is the natural boundary line of the effective regions of the two images. The number of intersection points reflects the intersection relationship between the two circular outlines. Typically, when the two images have moderate overlap, there are two main intersection points, located above and below the overlapping area, respectively.

[0047] After obtaining the intersection points, these intersection points are connected sequentially to draw a boundary line in the overlapping area of ​​the reference image and the perspective-transformed image to be stitched together. This boundary line extends along the geometric intersection of the circular contour of the fundus, dividing the overlapping area into two parts. A fusion mask is then generated based on this. This mask is a binary image of the same size as the original image, where the white portion corresponds to the area retained in the reference image, and the black portion corresponds to the area to be stitched together. The boundary of the mask is the boundary line connecting the intersection points, which extends along the natural circular contour of the fundus tissue, minimizing the risk of transverse anatomical structures such as blood vessels and the optic disc.

[0048] To eliminate potential noise and glitch near the boundary, a morphological opening operation is performed on the generated fusion mask. Small rectangular structuring elements are used to erode and then dilate the mask to remove isolated noise points near the boundary and smooth the edges. After the opening operation, the contours in the mask are re-detected, and the contour region containing the center point of the reference image is selected as the final fusion mask. This ensures that the center region of the reference image is always preserved within the reference image itself, avoiding any impact of the fusion operation on the critical diagnostic region in the image center. For more details on this implementation, please refer to [link to implementation details]. Figure 6 , Figure 6 This is a schematic diagram of the intersection detection method.

[0049] Based on the above limitations, after the step of defining a boundary line in the overlapping area of ​​the reference image and the perspective-transformed image to be pieced together using the intersection point, and generating a fusion mask based on the boundary line, the method further includes: The number of intersections of the puzzle to be completed is obtained. When the number of intersections is less than a first preset threshold, the center distance between the center of the image of the puzzle to be completed and the center of the image of the reference image is obtained. When the center distance is less than a preset distance threshold, a weighted superposition and fusion is performed on the reference image and the perspective-transformed image to be stitched together to obtain a stitched image. When the center distance is greater than or equal to the preset distance threshold, the puzzle piece to be processed is abandoned and the next puzzle piece is obtained as the new puzzle piece; Based on the new image to be pieced together and the baseline image, the step of extracting feature points is repeated.

[0050] In this embodiment, during panoramic image stitching, after perspective transformation and alignment between the reference image and the image to be stitched, the intersection points are extracted using binary contour logical AND operation. Due to individual differences in fundus images and the influence of imaging conditions, the intersection point detection results for some image pairs may deviate from the normal range. Therefore, the number of intersection points is an important indicator for measuring the degree of overlap of the effective areas of two images and the determinability of the fusion boundary. Under normal overlap conditions, two main intersection points can usually be detected, located above and below the overlapping area, respectively. However, in actual clinical imaging, factors such as slight eye movements due to poor fixation, decreased image contrast caused by refractive media turbidity such as cataracts, or local brightness abnormalities caused by tear film reflection can all lead to a decrease in the geometric quality of the binarized contour, thereby affecting the accuracy and completeness of intersection point detection.

[0051] Therefore, the number of intersection points is counted, and when the number of intersection points is less than a first preset threshold, it indicates that the contour intersection area between the current image to be stitched and the reference image has not yielded a sufficient number of geometric intersection points. At this point, a reliable fusion mask boundary cannot be constructed based on the intersection points, and the process enters a downgraded fusion process. This downgraded process first obtains the center distance between the image center of the image to be stitched and the image center of the reference image. This center distance reflects the degree of spatial deviation of the image to be stitched relative to the reference image and is an important basis for determining whether the two images have sufficient overlapping areas. During the acquisition of fundus images, the operator usually acquires fundus images from different locations sequentially according to a predetermined shooting path. The center distance between adjacent images is generally within a stable range. If the center distance is too large, it indicates that the two images may come from different regions of the fundus, resulting in insufficient overlap; if the center distance is within a reasonable range, it indicates that the two images have sufficient overlapping areas.

[0052] When the center distance is less than a preset distance threshold, it is determined that although the number of intersection points between the image to be stitched and the reference image is insufficient for precise mask-guided multi-band fusion, the two images have sufficient spatial overlap, allowing for stitching using a weighted overlay fusion method. Weighted overlay fusion is a simplified fusion method. The image to be stitched is overlaid onto the reference image according to its effective area mask. The pixel values ​​of the two images are then weighted and averaged in the overlapping area, with the weights determined based on the distance from each pixel to its respective effective area boundary. Compared to multi-band fusion, weighted overlay fusion does not rely on intersection points to define boundaries, making the calculation simpler. Even when intersection point detection fails but the image pair does have effective overlap, it can complete the stitching with lower fusion quality requirements, avoiding the discarding of an image that could have been stitched due to intersection point detection failure.

[0053] When the center distance is greater than or equal to a preset distance threshold, it indicates that the center point of the image to be stitched is too far from the center point of the reference image. The actual overlap area between the two images is insufficient to support meaningful fusion, and even weighted overlay fusion will introduce severe stitching misalignment or artifacts. In this case, the current image to be stitched is abandoned, and the next image to be stitched is obtained as the new image to be stitched. The step of extracting feature points is then repeated based on the current reference image. This judgment mechanism avoids stitching low-quality or invalid images into the panoramic image, preventing poor stitching results from polluting the quality of the constructed panoramic image.

[0054] The three-level degradation fusion strategy described above provides two fault-tolerant paths when there is insufficient intersection detection: one is to use weighted superposition fusion to complete the stitching when the two images have sufficient overlap, and the other is to skip the current image and continue processing subsequent images when the overlap between the two images is insufficient. This hierarchical response mechanism ensures that even in extreme scenarios where fully automatic feature matching fails or intersection detection is abnormal, the system can still retain usable stitching results to the greatest extent possible through simplified fusion methods or skipping strategies, rather than simply exiting with an error and causing the entire check to fail.

[0055] In addition, after the step of obtaining the number of intersections of the puzzle piece, the method further includes: Count the number of intersections; When the number of intersections is greater than the second preset number threshold, it is determined that the overlapping area between the image to be stitched and the reference image is too large. Multi-band fusion is abandoned for the image to be stitched, and the reference image is used as the stitched image and the multi-band fusion operation is skipped. The second preset number threshold is greater than the first preset number threshold.

[0056] In the previous degradation strategy, when the number of intersections is less than the first preset threshold, a degradation fusion process based on center distance is triggered. In contrast to the case of insufficient intersections, there is another extreme case in the actual splicing process, namely, an abnormally large number of intersections. This case also requires a dedicated degradation processing mechanism.

[0057] The number of intersection points is closely related to the geometric relationship between the effective region contours of the two images. When the baseline image and the circular effective region contour to be pieced together have a moderate overlap, the two circular contours form two geometric intersection points on the boundary of the overlapping area. Under normal circumstances, the number of detectable intersection points is usually two, located on the upper and lower sides of the overlapping area, respectively. As the overlap of the effective regions of the two images increases, the intersecting arc between the two circular contours lengthens, and the contour intersection points gradually separate to both sides, while the area of ​​the contour intersection continuously increases. When the overlap area of ​​the two images becomes large enough, the two circular contours almost completely overlap. At this point, the contour intersection points are no longer limited to two main intersection points, but rather multiple scattered geometric intersection features appear along the edge of the overlapping arc, leading to a significant increase in the number of extracted intersection points. In extreme cases, if the effective areas of the two images almost completely overlap, that is, the fundus area covered by the image to be pieced together is highly consistent with the area covered by the reference image, the two circular contours basically coincide. The shape of the contour intersection changes from two intersection points to multiple intersection features distributed along almost the entire circumference, and the number of intersection points may reach dozens or even more.

[0058] To address this extreme case, after obtaining the number of intersections to be stitched together, this number is counted and compared with a second preset threshold. The second preset threshold is greater than the first preset threshold, and its setting is based on statistical analysis of the normal overlap of fundus images. During the sequential acquisition of fundus images along a predetermined shooting path, the overlap between adjacent images is typically between 20% and 40%, at which point the number of intersections between the two circular outlines stabilizes at around two. When the number of intersections exceeds the second preset threshold, it indicates that the overlap between the two images is far greater than the reasonable range required for normal stitching, and the fundus areas covered by the two images highly overlap.

[0059] When the number of intersections exceeds a second preset threshold, the overlap between the image to be stitched and the baseline image is deemed too large. In such a highly overlapping situation, if multi-band fusion is still performed, the fundus tissue information in almost identical areas of the two images will be repeatedly fused, resulting in two nearly identical sets of vascular pathways and optic disc structures in the overlapping area of ​​the fused image. Although multi-band fusion can smooth the transition to some extent, the difference in the timing of the two images' capture means that even slight fixational movements by the patient can lead to sub-pixel-level deviations in the position of blood vessels in the same fundus region between the two images. Under highly overlapping conditions, this deviation manifests as blurred or ghosted blood vessels after fusion, thus reducing the clarity and diagnostic value of the panoramic image.

[0060] Therefore, when the number of intersections exceeds the second preset threshold, the multi-band fusion operation on the image to be stitched is abandoned, and the current reference image is directly used as the stitched image, skipping the fusion operation. The rationale for this approach is that since the fundus regions covered by the two images are highly consistent, the image to be stitched does not contribute new fundus tissue information to the panoramic view. Retaining the original reference image is sufficient to maintain the integrity of the panoramic image, without introducing highly redundant information from the image to be stitched.

[0061] This skip-fusion degradation strategy, in real-world clinical scenarios, avoids the possibility of operators mistakenly acquiring multiple images from similar locations during fundus image acquisition, or mistakenly recording images from the same location repeatedly in the acquisition sequence, due to improper operation or poor patient cooperation. If these highly overlapping images are stitched into a panoramic image, they will not only fail to expand the field of view but may also degrade image quality due to fusion artifacts. By statistically analyzing the number of intersection points, this abnormal overlap can be automatically identified and appropriate processing methods can be adopted, improving the automation and robustness of the stitching process while ensuring panoramic image quality.

[0062] Step S50: Perform multi-band fusion on the reference image and the perspective-transformed image to be stitched together according to the fusion mask to obtain a stitched image. The fusion mask divides the overlapping area into a reference image retention area and an image to be stitched together retention area. After generating the fusion mask, multi-band fusion of the reference image and the perspective-transformed image to be pieced together needs to be performed based on the mask. The pixel values ​​of the reference image and the perspective-transformed image to be pieced together are normalized from eight-bit unsigned integers to a 32-bit floating-point type in the range of zero to one to ensure the accuracy of subsequent pyramid construction and fusion operations.

[0063] A Gaussian pyramid is constructed for each of the two normalized images. The construction of the Gaussian pyramid begins with the original resolution image. The current layer image is smoothed using a Gaussian filter and then downsampled to half its original size to obtain the next layer image. This process is repeated until a predetermined number of layers, such as a four-layer Gaussian pyramid, is completed, where the bottom layer is the original resolution image and the top layer is the lowest resolution image.

[0064] The Laplacian pyramid is constructed based on the Gaussian pyramid. Each layer of the Laplacian pyramid is obtained by subtracting the upsampled and Gaussian-filtered image of the corresponding layer from the image of the Gaussian pyramid. The Laplacian pyramid effectively stores high-frequency detail information of the image at different scales, with the bottom layer containing the finest image details and the top layer containing the coarsest image contour information.

[0065] Simultaneously, a Gaussian pyramid with corresponding layers is constructed for the fused mask image. Each layer of the Gaussian pyramid represents the spatial distribution of the fusion weights at that scale. The masks of low-resolution layers are smoother, while the masks of high-resolution layers retain the boundary details of the original mask.

[0066] After the pyramid is constructed, a weighted fusion process is performed layer by layer. For each layer of the Laplacian pyramid, the Laplacian layer of the reference image and the Laplacian layer of the image to be pieced together are weighted and combined according to the weight values ​​of the masks in the same layer. That is, areas with mask values ​​close to one retain more information from the reference image, areas with mask values ​​close to zero retain more information from the image to be pieced together, and transitional areas with mask values ​​between zero and one are mixed with information from the two images according to the weight ratio.

[0067] After weighted fusion, the fused image is reconstructed level by level, starting from the top of the Laplacian pyramid. The fusion result of the current layer is upsampled to the size of the next layer and then superimposed with the fusion result of the next layer. This process is repeated until the original resolution is restored. After reconstruction, the output values ​​are cropped to the range of zero to one and then converted back to eight-bit unsigned integers to obtain the final stitched image. Through this multi-band fusion method, image information of different scales achieves a smooth transition at the fusion boundary, high-frequency details are naturally connected at the boundary, and low-frequency brightness gradually transitions at the boundary, thus eliminating visible stitching seams. The above content can also be viewed... Figure 7 , Figure 7 A schematic diagram of the Laplacian pyramid fusion process for the contour mask vagina.

[0068] Step S60: Use the stitched image as a new reference image, obtain the next image to be stitched as a new image to be stitched, and repeat the image stitching operation until all images to be stitched are processed. Output the reference image obtained from the last image stitching operation as a panoramic fundus image.

[0069] After a piece of the image to be stitched together is merged with the current reference image, the resulting stitched image is used as the new reference image. The new reference image contains information from all the stitched images, and its effective area expands as more images are added.

[0070] The next unprocessed fundus image is acquired as the new image to be stitched. Using the updated reference image and the new image to be stitched as the processing objects, the complete image stitching operation from steps S10 to S50 is re-executed. In the new stitching operation, the new reference image already contains panoramic information from the stitching of multiple images, and the new image to be stitched needs to be transformed into this ever-expanding panoramic coordinate system.

[0071] If a new image fails the circularity verification step—meaning the circularity of the image after perspective transformation is less than a preset circularity threshold—then the image is discarded and the next image to be acquired is obtained. This mechanism ensures that only high-quality registration results that pass geometric verification are stitched into the panoramic image, preventing inferior images from contaminating the already stitched panoramic result.

[0072] All images to be stitched are processed sequentially. Each successfully stitched image further expands the coverage of the panoramic image, while images that fail verification are skipped without affecting the existing content of the panoramic image. Once all images to be stitched have been processed, the reference image obtained from the final image stitching operation is a complete panoramic fundus image containing information from all images that can be stitched in.

[0073] The panoramic fundus image is used as the final output. Unlike traditional chain registration schemes, this method uses a reference image that already contains information from all previously stitched images when stitching a new image. The new image to be stitched is geometrically corrected through circularity verification, and the reasonableness of the fusion boundary is ensured through an adaptive fusion mask guided by intersection points. During the process of stitching multiple images into the panoramic image, each new image undergoes rigorous geometric verification and adaptive fusion processing. The final output panoramic fundus image exhibits excellent performance in terms of anatomical structure fidelity, geometric consistency, and visual continuity.

[0074] Furthermore, after the steps of using the stitched image as a new reference image, obtaining the next image to be stitched as a new image to be stitched, and repeating the image stitching operation until all images to be stitched are processed, and outputting the reference image obtained from the last image stitching operation as a panoramic fundus image, the method further includes: Count the number of images in the entire dataset that passed the validity verification and were successfully stitched into the panoramic fundus image; When the number of images is zero, output a stitching failure message and output the initially acquired reference image; When the number of images is greater than zero and less than the total number of images to be pieced together, a success message is output. The message includes a list of successfully pieced-together image names and the displacement vector of each image.

[0075] In the panoramic stitching process of multiple fundus images, after completing the iterative processing of all images to be stitched, the execution results of the entire stitching process are statistically analyzed to provide feedback to the operator on the completion status of the stitching operation. This statistical and feedback process constitutes the final stage of stitching, allowing the operator to understand the actual content of the panoramic image that was successfully or unsuccessfully stitched in, as well as the possible reasons for stitching failure. In clinical applications, this feedback mechanism provides the operator with objective evidence to determine whether it is necessary to re-capture fundus images of specific locations, effectively avoiding duplicate examinations and diagnostic delays caused by information asymmetry.

[0076] The statistical operation targets all images in the panoptic image that have passed circularity verification and been successfully stitched into the panoramic fundus image. During the iterative stitching process, each image undergoes a complete processing chain: feature extraction, transformation matrix estimation, perspective transformation, circularity verification, intersection detection, and fusion stitching. Only images that pass circularity verification and whose intersection detection results meet the fusion conditions are actually stitched into the panoramic image; images that fail circularity verification or are skipped from fusion due to abnormal intersection counts are not counted in the successful stitching count. The statistics iterate through all images in the panoptic image, recording whether each image was ultimately stitched into the panoramic image, and summing these entries to obtain the total number of successfully stitched images.

[0077] When the number of images obtained is zero, it indicates that none of the images in the entire dataset passed the circularity verification and were successfully stitched into the panoramic image. This result is usually caused by the following reasons: poor fixation in the patient's eyes, resulting in the image quality of all images failing to meet the registration requirements; small pupils or cloudy refractive media leading to excessively low image contrast, causing complete failure of feature extraction and matching; or a lack of effective overlap between the images to be stitched and the reference image during the acquisition process. In this case, the stitching operation fails to expand the field of view, and the panoramic image actually only contains single-image information from the initial reference image. The system outputs a stitching failure message, informing the operator that no additional images could be stitched in this attempt, and simultaneously outputs the initially acquired reference image as the result image for the operator's reference and to determine whether re-acquisition is necessary.

[0078] When the number of images obtained is greater than zero and less than the total number of images to be stitched, it indicates that the stitching operation has been partially successful. This means at least one image has been successfully stitched into the panoramic image, but several other images have been skipped due to failing verification. This partial success is common in clinical practice. For example, in stitching fundus images of cataract patients, the quality of some images may vary due to differences in lens opacity; higher-quality images may be successfully stitched in, while lower-quality images are automatically skipped. The system outputs a stitching success message, which includes a list of successfully stitched image names and the displacement vectors of each image. The image name list allows the operator to visually understand which locations have been successfully included in the panoramic image, and the displacement vectors record the translational offset of each successfully stitched image relative to the reference image, allowing the operator to assess the distribution of each image. If the operator finds that the image name corresponding to a key location is not in the success list, they can specifically re-acquire fundus images for that location and attempt stitching again, thus avoiding the inefficient operation of repeatedly acquiring images from all locations.

[0079] When the number of images obtained equals the total number of images to be stitched, it indicates that each image has passed the circularity verification and has been successfully stitched into the panoramic image, and the stitching operation is completely successful. Although this situation is not explicitly listed in the claims, as a boundary case where the number equals the total, it is logically considered a successful stitching in the output, and the output will include a stitching completion message along with a list of all image names and their displacement vectors.

[0080] Through the aforementioned statistical and feedback mechanisms, the results of the stitching operation are presented to the operator in a clear information format, creating a complete information loop from image input to output. Clinical operators no longer need to manually determine which images are stitched in and which are skipped; the system automatically completes the statistics and reporting, significantly improving the usability and operational efficiency of panoramic fundus stitching in clinical settings.

[0081] Further, see Figure 2 , Figure 2 This is a detailed schematic diagram of step S10 in one embodiment. The steps of extracting feature points from the acquired reference image and the image to be pieced together, matching based on the extracted feature points, and estimating the transformation matrix of the image to be pieced together to the reference image based on the matching results include S11~S12: Step S11: Extract feature points from the reference image and the image to be pieced together; Step S12: Use an approximate nearest neighbor search algorithm to match the feature points of the reference image with the feature points of the image to be pieced together, and return the nearest neighbor feature point and the second nearest neighbor feature point for each query point; Step S13: Calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance for each query point, and retain matching pairs whose ratio is less than the first ratio threshold as initial screening matching pairs; Step S14: When the number of initial screening matching pairs is less than the preset number of matching pairs, the first ratio threshold is relaxed to the second ratio threshold, and the matching pairs are re-screened to ensure that the number of screened matching pairs can be used for the estimation of the subsequent transformation matrix. Step S15: Use the random sampling consensus algorithm to estimate the transformation matrix from the selected matching pairs to the base image.

[0082] In this embodiment, after obtaining the baseline image and the image to be stitched together, scale-invariant feature transformation (SMT) feature points are extracted from both images. This feature extraction process is performed in scale space. An extremum point is detected using a difference-of-Gaussian pyramid as candidate keypoints. These candidate keypoints are then precisely located, and low-contrast points and edge response points are eliminated. Finally, a principal direction is assigned to each keypoint, and a feature description vector is generated. The feature description vector, in the form of a gradient histogram, describes the texture distribution features within the neighborhood of the keypoint. It is invariant to image rotation, scale changes, and illumination changes, making it suitable for handling differences in fundus images caused by varying shooting angles and lighting conditions.

[0083] After feature point extraction, an approximate nearest neighbor search algorithm is used to match the feature points of the baseline image with the feature points of the puzzle image. This matching method constructs multiple KD-trees to index the feature space. For each query point in the baseline image, it searches for the two feature points in the set of feature points to be pieced together that are closest to its feature description vector, namely the nearest neighbor and the second nearest neighbor. During the search, multiple KD-trees are traversed and a preset number of leaf nodes are checked to ensure the accuracy of the search results.

[0084] After obtaining the nearest and second nearest neighbors, the ratio of the nearest neighbor distance to the second nearest neighbor distance is calculated. Under ideal matching conditions, the distance between a correctly matched query point and its nearest neighbor feature point should be significantly smaller than the distance with its second nearest neighbor; therefore, this ratio should be small. Conversely, if the nearest and second nearest neighbors of a query point are close in distance, it indicates that there are multiple similar feature points near the query point, and the match is highly ambiguous, likely a false match. A first ratio threshold is pre-set. If the ratio of the nearest and second nearest neighbor distances is less than this threshold, it indicates that the match between the nearest neighbor feature point and the query point is significantly better than that between the second nearest neighbor, and the matching pair is considered a reliable initial screening match and retained. If the ratio is greater than or equal to the threshold, it indicates that the distinction between the nearest and second nearest neighbors is insufficient, and the matching pair has a high risk of false matching, and is therefore discarded. This ratio testing mechanism can effectively filter out a large number of false matches, significantly improving the overall accuracy of the matching pair set.

[0085] After the ratio test screening, the number of initially screened matching pairs is counted. This number is an important indicator of whether the current image pair has sufficient texture information to support stable registration. If the number of initially screened matching pairs is less than the preset number of matching pairs, it means that the reliable matching pairs retained after the ratio test between the current image pairs are insufficient to support the reliable estimation of the subsequent transformation matrix. This situation usually occurs in low-texture areas of fundus images, where the overlapping area is too small, or where the image quality is severely degraded due to factors such as cataracts. In this case, the first ratio threshold is automatically relaxed to the second ratio threshold, and the matching pair screening process is re-executed. The value of the second ratio threshold is greater than the first ratio threshold. The relaxed threshold allows more matching pairs that were rejected under strict ratio conditions to re-enter the retention set. As the retention conditions are relaxed, the number of initially screened matching pairs increases accordingly, thereby ensuring that enough matching points can still be obtained for subsequent calculations even under conditions of sparse image texture. The triggering and execution of this fallback strategy are fully automated, requiring no operator intervention, ensuring the continuity and stability of the stitching process under conditions of image quality fluctuations.

[0086] After obtaining a sufficient number of matching pairs, the random sampling consensus algorithm is used to estimate the transformation matrix from the plane to be pieced together to the reference image. This algorithm calculates candidate transformation matrices by randomly sampling matching pairs, then counts the number of inliers that conform to the mapping relationship of these matrices. After multiple iterations, the candidate matrix with the most inliers is selected as the final transformation matrix estimate. This transformation matrix is ​​a homography matrix in three-dimensional space, describing the projection transformation relationship between the plane to be pieced together and the plane of the reference image. Compared to directly using all matching pairs for least squares estimation, the random sampling consensus algorithm can effectively eliminate residual erroneous matching pairs, making the estimated transformation matrix more robust and accurate, providing a reliable mathematical basis for subsequent image space transformations.

[0087] Furthermore, you can also view Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the fundus camera multi-image adaptive panoramic stitching method of the present invention, based on... Figure 3 When the number of stitched images exceeds a preset number, after the steps of binarizing the reference image and the perspective-transformed image to be stitched, extracting the maximum outer contour of the binarized image, and performing a logical AND operation on the filled area of ​​the maximum outer contour to obtain the contour intersection, the method further includes steps S70~S100: Step S70: Obtain the overall outline of the stitched image, calculate the global outline intersection between the maximum outer outline of the image to be stitched after perspective transformation and the overall outline of the stitched image, and extract the global intersection point from the global outline intersection. Step S80: Obtain each historical reference image used as the reference image in each image stitching operation, calculate the historical contour intersection between the maximum outer contour of the image to be stitched after perspective transformation and the maximum outer contour of each historical reference image, and extract the historical intersection points from each historical contour intersection. Step S90: Match each of the historical intersection points with the global intersection point in terms of position, and determine whether there is a matching relationship between each of the historical intersection points and the global intersection point based on the coordinate distance threshold. Step S100: When there are two or more sets of matching relationships, calculate the intersection points of the lines corresponding to each set of matching relationships, and add the line intersection points as new intersection points to the intersection points.

[0088] In this embodiment, when the number of fundus images to be stitched reaches three or more, the contour shape of the stitched area becomes more complex as the number of image frames in the panoramic image increases. Each time a new image is stitched, the current reference image itself already contains panoramic information from multiple images, and its effective area is no longer the standard circle of a single original fundus image, but rather the union of the effective areas of multiple images, presenting an irregular geometric shape. In this case, calculating only the pairwise intersections between the image to be stitched and the current reference image can determine the fusion boundary between the image to be stitched and the existing panoramic image as a whole, but this boundary is only constrained by the globally accumulated image and fails to fully consider the local geometric relationship between the image to be stitched and each previously stitched historical reference image.

[0089] Therefore, when the number of stitched images reaches three or more, a multi-image joint intersection verification process is executed. This process first obtains the overall contour of the stitched images, that is, the union contour of the effective regions of all stitched images in the current reference image. The intersection of the maximum outer contour of the image to be stitched after perspective transformation and the overall contour of the stitched images is calculated, and global intersection points are extracted from this global contour intersection. These global intersection points represent the geometric intersection position between the image to be stitched and the overall contour of the existing panoramic image, reflecting the boundary relationship of the image to be stitched relative to the overall panoramic image.

[0090] Simultaneously, historical reference images used as reference images in each image stitching operation are acquired. The reference image used in the first round of stitching is the initial reference image; the reference image used in the second round of stitching is the stitched image after the first image to be stitched is incorporated, and so on. The updated reference image after each round of stitching is recorded as a historical reference image. For each historical reference image, the contour intersection between the maximum outer contour of the image to be stitched after perspective transformation and the maximum outer contour of the historical reference image is calculated, and historical intersection points are extracted from each historical contour intersection. Each set of historical intersection points represents the pairwise geometric intersection relationship between the image to be stitched and the corresponding historical reference image, reflecting the local overlap boundary between the image to be stitched and each already stitched image.

[0091] After obtaining the global intersection point and multiple sets of historical intersection points, each historical intersection point is matched with the global intersection point. The specific method for position matching is as follows: the Euclidean distance between the historical intersection point and the global intersection point is calculated. If this distance is less than a preset coordinate distance threshold, a matching relationship is determined between the historical intersection point and the global intersection point; if the distance is greater than or equal to the threshold, no matching relationship is determined. This coordinate distance threshold is set in image pixels, taking a value of tens of pixels at a typical fundus image resolution to accommodate minute positional shifts caused by registration errors.

[0092] When two or more sets of matching relationships exist, it indicates that at least two historical baseline images have corresponding intersection points with the global intersection point. In this case, matching point pairs are selected from each set of matching relationships. Each set of matching relationships contains one historical intersection point and one matching global intersection point. The intersection point of the lines connecting the matching point pairs is calculated. The geometric meaning of this line intersection point is that the fusion boundary between the image to be stitched and the two different historical baseline images intersects at the same point. This point represents the globally optimal position of the fusion boundary between the image to be stitched and the existing panoramic image in the context of multi-image stitching. This line intersection point is added as a new intersection point to the original set of intersection points, so that the final fusion boundary is simultaneously constrained by the global contour and multiple historical baseline images.

[0093] Through the aforementioned point-line fusion mechanism involving first-to-last detection and one-to-one comparison, the cumulative registration error of multi-image stitching is effectively suppressed. The fusion boundary of newly added images no longer depends on pairwise relationships with the current reference image, but rather on joint verification with each historical reference image, ensuring boundary consistency across different regions of the stitched panoramic image. This mechanism fundamentally solves the inherent problems of inconsistent boundaries and visible stitching seams that arise with the increase in the number of images in traditional chain registration methods. For the specific process of multi-image joint cross-validation described above, please refer to [link to documentation / documentation]. Figure 8 , Figure 8 This is a flowchart illustrating the verification of the intersection points of multiple spliced ​​graphs.

[0094] Furthermore, you can also view Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the fundus camera multi-image adaptive panoramic stitching method of the present invention, based on... Figure 4 Before the step of extracting feature points from the obtained reference image and the image to be pieced together, steps S110~S140 are included: Step S110: Perform circular region of interest segmentation on each input original fundus image, generate a circular mask with the center of the original fundus image as the center and half the width of the original fundus image as the radius, and remove the black corner areas of the original fundus image through the circular mask; Step S120: Perform contrast-limited adaptive histogram equalization enhancement on the cut image to improve the contrast between the blood vessel texture and the background area in the image. Step S130: The enhanced image is scaled down proportionally to a preset scaling size to reduce the computational load of subsequent feature extraction and matrix operations; Step S140: Add a preset number of black pixel borders to the top, bottom, left, and right sides of the reduced image to reserve pixel buffer space for perspective transformation, thereby obtaining the reference image or the image to be stitched together.

[0095] Before feature extraction and registration, the input raw fundus images undergo a series of preprocessing operations. The raw images acquired by the fundus camera are typically 3,000 by 3,000 pixels in size, with a circular fundus tissue imaging region at the center and invalid black areas at the four corners. These black corner areas contain no valid information about the fundus tissue; retaining them in subsequent processing would not only consume computational resources but also potentially interfere with the accuracy of feature extraction and contour detection. Therefore, a circular region of interest (ROI) segmentation operation is performed on each raw fundus image. This operation generates a circular mask with the same dimensions as the image, centered at the center point of the raw fundus image and with a radius equal to half the width of the raw fundus image. Pixels inside the circular region in the mask are marked as valid regions, while pixels outside the circular region are marked as invalid regions. By performing a bitwise AND operation between this mask and the raw image, the pixel values ​​inside the circular region remain unchanged, while the pixel values ​​outside the circular region are set to zero, effectively removing the invalid black areas at the four corners and retaining only the circular fundus tissue imaging region for subsequent processing.

[0096] After circular region of interest (ROI) segmentation, the contrast of retinal blood vessels in the image may vary significantly due to individual differences and shooting conditions. In some images, the blood vessels are clearly discernible, while in others, the grayscale difference between the blood vessels and the background is small, which can complicate subsequent feature extraction. To unify and improve the contrast of blood vessel texture, contrast-limited adaptive histogram equalization enhancement is performed on the segmented image. This enhancement method divides the image into multiple equally sized grid cells, calculates a grayscale histogram for each grid cell, and performs histogram equalization to make the grayscale distribution within each grid cell more uniform. To prevent over-enhancement in uniform or noisy regions, the method limits the height of the histogram; portions exceeding the limit are cropped and redistributed across the entire histogram range. After this enhancement process, the blood vessel regions, which were originally close to the background grayscale, are effectively stretched, making the blood vessel texture clearer and sharper, providing a higher-contrast input image for subsequent feature point detection.

[0097] After image enhancement, the enhanced image is proportionally scaled down to a preset size. Feature extraction and matrix operations on the original image at a resolution of 3000 x 3000 pixels are computationally intensive, especially when processing multiple images, where the computation time increases significantly. By scaling the image down to one-quarter of its original size while maintaining the aspect ratio, the total number of pixels is reduced to one-sixteenth, significantly reducing the computational load for subsequent feature point detection, descriptor generation, and matching operations. Although this scaling operation sacrifices some spatial resolution, the scaled-down image provides sufficient texture information for feature point detection and matching during the feature extraction and registration stages.

[0098] After scaling, a preset number of black pixel borders are added to the top, bottom, left, and right sides of the reduced image. Perspective transformation is an indispensable step in image stitching. After the stitched image undergoes perspective transformation and is mapped to the coordinate system of the reference image, the transformed image boundaries may exceed the original image size, and some image content may be cropped and lost after transformation. By adding black borders around the image in advance, the image size is effectively enlarged, reserving sufficient buffer space for the transformed image content, ensuring that the complete transformed image content can be preserved within the image frame. In fully automatic stitching mode, the number of borders added is adapted to the scaled image size, ensuring sufficient pixel buffer space without introducing too much blank area due to excessively large borders, which would affect the efficiency of subsequent processing. After the above complete preprocessing chain, the original fundus image is transformed into a preprocessed image with uniform size, enhanced contrast, centered effective area, and pixel buffer space. This image retains the key texture information used for feature extraction and registration, and has appropriate size and margins to support efficient subsequent processing.

[0099] Furthermore, you can also view Figure 5 , Figure 5 This is a flowchart illustrating the fourth embodiment of the fundus camera multi-image adaptive panoramic stitching method of the present invention, based on... Figure 5 Before the step of validating the puzzle image after perspective transformation based on the maximum outer contour, steps S150~S180 are further included: Step S150: Transform the center point of the reference image to the coordinate system of the image to be pieced together using the transformation matrix to obtain the transformed center point; Step S160: Calculate the displacement distance between the transformed center point and the center point of the reference image; Step S170: When the displacement distance exceeds the preset displacement threshold, it is determined that the shooting position of the image to be stitched is too far away from the shooting position of the reference image, the image to be stitched is abandoned and the next image to be stitched is obtained as a new image to be stitched. Step S180: Based on the new image to be pieced together and the reference image, the step of extracting feature points is performed again.

[0100] After perspective transformation, the image to be stitched together has been mapped to the spatial coordinate system of the reference image. At this point, a preliminary verification of the registration result is needed to ensure the basic spatial validity of the objects processed in subsequent circularity verification and fusion operations. This preliminary verification is conducted with the center point of the reference image as the reference object. The center point of the reference image is located at the geometric center of the reference image plane, representing the center position of the effective fundus region of the reference image. The center point of the reference image is spatially transformed using the transformation matrix estimated in step S10, mapping this center point to the coordinate system of the image to be stitched together, resulting in the transformed center point. This transformed center point represents the corresponding position of the central region of the reference image in the space of the image to be stitched together.

[0101] After obtaining the transformed center point, calculate the displacement distance between this transformed center point and the center point of the reference image. The geometric meaning of this displacement distance is the spatial deviation between the mapped position of the reference image's central region in the coordinate system of the puzzle and the center position of the puzzle itself. If there is a reasonable overlap between the puzzle and the reference image, the transformed center point of the reference image should fall within or near the effective area of ​​the puzzle, and the distance between the transformed center point and the center point of the puzzle is within a reasonable range. If the actual overlap between the puzzle and the reference image is very small or nonexistent, the transformed center point will fall far outside the effective area of ​​the puzzle, and the displacement distance between them will be significantly larger.

[0102] In actual clinical image acquisition, two fundus images from adjacent shooting positions typically have an overlap of 20% to 40%, corresponding to a displacement distance between the transformed center point and the center point of the image to be stitched, roughly distributed within a few hundred pixels. The upper and lower limits of this range are related to factors such as the shooting path planning, image resolution, and scaling ratio of the fundus images. A preset displacement threshold is set, and the calculated displacement distance is compared with this threshold. If the displacement distance does not exceed the preset threshold, it indicates a reasonable spatial overlap between the image to be stitched and the reference image, and the subsequent circularity verification and intersection detection processes can proceed. If the displacement distance exceeds the preset threshold, it indicates that the shooting position of the image to be stitched deviates too much from the shooting position of the reference image, there is no effective overlap between the two images, or the estimation result of the transformation matrix contains a serious error.

[0103] When the displacement distance exceeds a preset displacement threshold, the image to be stitched is determined to have deviated too much from the image position of the reference image. In this case, even if subsequent circularity verification and fusion operations are performed, a clinically meaningful stitching result cannot be obtained, and the panoramic image may even be contaminated due to incorrect registration. In this situation, the current image to be stitched is abandoned, and the next image to be stitched is obtained as the new image. Based on the current reference image and the new image to be stitched, the entire process from feature extraction to transformation matrix estimation is re-executed. This displacement distance verification mechanism builds the first quality gating defense before circularity verification, enabling the processing of the current image to be terminated early when there is a serious spatial deviation in the registration result. This avoids invalid computation blocking the subsequent image processing queue and improves the overall efficiency of the stitching process.

[0104] Furthermore, based on the technical content provided in the above embodiments, you can also view... Figure 9 , Figure 9 This is a schematic diagram of the overall architecture of the fundus camera multi-image adaptive panoramic stitching system of the present invention. In the system implemented by the fundus camera multi-image adaptive panoramic stitching architecture, the input is N original fundus images, each with a size of 3,000 x 3,000 pixels, using BGR color format. After receiving these N images, the system first sends them to the preprocessing module for processing. The preprocessing module performs circular region of interest segmentation, contrast-limited adaptive histogram equalization enhancement, proportional scaling, and black border expansion operations on each image in sequence. After preprocessing, the image size in fully automatic mode is adjusted to 1,750 x 1,750 pixels, and the image size in manual mode is adjusted to 3,150 x 3,150 pixels. Images in both modes are output in the form of preprocessed image vectors.

[0105] After preprocessing, the system enters the mode selection unit. This unit determines the subsequent execution path based on the user's operation instructions: if the user triggers a fully automatic stitching instruction, the system sends the preprocessed image to the fully automatic stitching unit; if the user triggers a manual stitching instruction, the system enters the manual stitching unit. In fully automatic mode, the system uses the first preprocessed image as the initial reference image and processes the remaining images in reverse order. Each image sequentially undergoes feature extraction and matching, transformation matrix estimation, perspective transformation, displacement distance verification, circularity verification, intersection detection, fusion mask generation, and multi-band fusion. After stitching is completed, the reference image is updated, and the above process is repeated until all images are processed. In manual mode, the system receives a translation vector specified by the user on the UI interface, performs spatial transformation on the preprocessed image according to the translation vector, and then executes intersection detection and a three-level degradation fusion strategy.

[0106] Finally, the output from the fully automatic or manual stitching unit is cropped and resized to restore the standard output size of 3,000 by 3,000 pixels, and then output as a panoramic fundus image via the data interface. The entire system drives the collaborative work of each module through an asynchronous message mechanism. When the user clicks the fully automatic stitching button, a fully automatic stitching message is triggered; when the user confirms the manual drag position, a manual stitching message is triggered. Data flow and status synchronization between modules are achieved through message passing.

[0107] Furthermore, based on the technical content provided in the above embodiments, you can also view... Figure 10 , Figure 10 This is a flowchart illustrating a process based on fully automated splicing units. Figure 10 In fully automatic stitching mode, the system uses the pre-processed first image as the initial reference image and traverses the remaining images to be stitched in reverse order, processing them sequentially from the last image forward and stitching them into the panoramic image one by one. For each image to be stitched, the system first extracts scale-invariant feature transformation (SMT) feature points from its contrast-limited adaptive histogram equalization enhanced image, setting a maximum number of feature points to ensure sufficient feature density in areas rich in vascular texture. After feature point extraction, multiple KD-tree indexes are constructed using an approximate nearest neighbor search library. For each feature point in the reference image, the system searches for nearest and second-nearest neighbor feature points in the feature point set of the images to be stitched, filtering high-quality matching pairs through a low-ratio test. If the number of filtered matching pairs is insufficient, the ratio threshold is automatically relaxed to ensure the number of matching points used for homography estimation. The random sampling consensus algorithm is used to estimate the homography matrix of the images to be stitched to the reference image from the filtered matching pairs.

[0108] After obtaining the transformation matrix, a perspective transformation is performed on the image to be pieced together, mapping it to the coordinate system of the reference image. Then, a dual quality verification process is initiated. The first verification is a displacement distance verification: the center point of the reference image is mapped to the coordinate system of the image to be pieced together using the transformation matrix, and the Euclidean distance between this point and the center point of the image to be pieced together is calculated. If this distance exceeds a preset displacement threshold, it indicates misalignment, and the image is directly discarded. The second verification is a roundness verification: the perspective-transformed image is binarized, and the maximum outer contour is extracted. The roundness of the contour is calculated. If the roundness is below a preset threshold, it indicates severe distortion of the transformed image, and the image is discarded. Images that pass both verifications proceed to the intersection detection process.

[0109] The intersection detection process first binarizes the baseline image and the perspective-transformed image to be stitched, extracting the maximum outer contour. A logical AND operation is performed on the two contour-filled regions to obtain the contour intersection. Connected regions with areas smaller than a preset threshold are extracted from the intersection, and the geometric center of each connected region is used as the intersection point. When the number of stitched images reaches three or more, multi-image joint verification is enabled. Global intersection points between the image to be stitched and the accumulated stitched images, as well as single-frame intersection points with each historical baseline image, are calculated. A point-line fusion mechanism using first-end detection and one-to-one comparison ensures that the fusion boundary is constrained by the multi-image joint verification. Subsequently, the intersection points are used to delineate boundaries in the overlapping area to generate a fusion mask. A four-layer Laplacian pyramid is constructed for multi-band fusion of the baseline image and the image to be stitched, resulting in a stitched image that is updated as a new baseline image. The process continues with the next image to be stitched until all images are processed.

[0110] This application provides a fundus camera multi-image adaptive panoramic stitching device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fundus camera multi-image adaptive panoramic stitching method in the first embodiment described above.

[0111] The following is for reference. Figure 11 This document illustrates a structural schematic diagram of a fundus camera multi-image adaptive panoramic stitching device suitable for implementing embodiments of this application. The fundus camera multi-image adaptive panoramic stitching device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 11 The illustrated fundus camera multi-image adaptive panoramic stitching device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 11 As shown, the fundus camera multi-image adaptive panoramic stitching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the fundus camera multi-image adaptive panoramic stitching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the fundus camera multi-image adaptive panoramic stitching device to wirelessly or wiredly communicate with other devices to exchange data. Although fundus camera multi-image adaptive panoramic stitching devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0113] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0114] The fundus camera multi-image adaptive panoramic stitching device provided in this application, employing the fundus camera multi-image adaptive panoramic stitching method in the above embodiments, can solve the technical problem that existing fundus image stitching lacks systematic adaptive capability, resulting in the inability to stably output high-quality panoramic fundus images under complex clinical imaging conditions. Compared with the prior art, the beneficial effects of the fundus camera multi-image adaptive panoramic stitching device provided in this application are the same as those of the fundus camera multi-image adaptive panoramic stitching method provided in the above embodiments, and other technical features in this fundus camera multi-image adaptive panoramic stitching device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0115] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0117] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the fundus camera multi-image adaptive panoramic stitching method in the above embodiments.

[0118] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable storage medium may be included in the fundus camera multi-image adaptive panoramic stitching device; or it may exist independently and not be assembled into the fundus camera multi-image adaptive panoramic stitching device.

[0120] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the fundus camera multi-image adaptive panoramic stitching device, enable the fundus camera multi-image adaptive panoramic stitching device to implement the technical content of the fundus camera multi-image adaptive panoramic stitching method embodiment shown above.

[0121] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0124] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described fundus camera multi-image adaptive panoramic stitching method. This solves the technical problem that existing fundus image stitching lacks systematic adaptive capabilities, resulting in the inability to stably output high-quality panoramic fundus images under complex clinical imaging conditions. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the fundus camera multi-image adaptive panoramic stitching method provided in the above embodiments, and will not be repeated here.

Claims

1. A method for adaptive panoramic stitching of multiple images from a fundus camera, characterized in that, Includes the following steps: Feature points are extracted from the obtained baseline image and the image to be pieced together, respectively. Matching is performed based on the extracted feature points, and the transformation matrix of the image to be pieced together to the baseline image is estimated based on the matching results. The transformation matrix is ​​used to perform a perspective transformation on the puzzle image, and the transformed puzzle image is binarized to extract the maximum outer contour. The validity of the puzzle image after perspective transformation is verified based on the maximum outer contour. After successful verification, the reference image and the puzzle image after perspective transformation are binarized respectively, and the maximum outer contour after binarization is extracted respectively. The intersection of the contours is obtained by performing a logical AND operation on the filled area of ​​the maximum outer contour. The connected regions are extracted from the intersection of the contours. The geometric center of the connected regions is used as the intersection point. The intersection point is used to draw a boundary line in the overlapping area of ​​the reference image and the image to be pieced together after perspective transformation. A fusion mask is generated based on the boundary line. The reference image and the perspective-transformed image to be stitched together are multi-band fused according to the fusion mask to obtain a stitched image. The fusion mask divides the overlapping area into a reference image retention area and an image to be stitched together retention area. The stitched image is used as a new reference image. The next image to be stitched is obtained as a new image to be stitched, and the image stitching operation is repeated until all images to be stitched are processed. The reference image obtained from the last image stitching operation is output as a panoramic fundus image.

2. The fundus camera multi-image adaptive panoramic stitching method as described in claim 1, characterized in that, When the number of stitched images exceeds a preset number, after the steps of binarizing the reference image and the perspective-transformed image to be stitched, extracting the maximum outer contour of the binarized image, and performing a logical AND operation on the filled area of ​​the maximum outer contour to obtain the contour intersection, the method further includes: Obtain the overall outline of the stitched image, calculate the global outline intersection between the maximum outer outline of the image to be stitched after perspective transformation and the overall outline of the stitched image, and extract the global intersection points from the global outline intersection. Each historical reference image used as the reference image in each image stitching operation is obtained. The intersection of the historical contour between the maximum outer contour of the image to be stitched after perspective transformation and the maximum outer contour of each historical reference image is calculated. Historical intersection points are extracted from each historical contour intersection. Each historical intersection point is matched with the global intersection point, and a matching relationship between each historical intersection point and the global intersection point is determined based on the coordinate distance threshold. When there are two or more matching relationships, calculate the intersection point of the line corresponding to each matching relationship, and add the line intersection point as a new intersection point.

3. The fundus camera multi-image adaptive panoramic stitching method as described in claim 1, characterized in that, After the step of defining a boundary line in the overlapping area of ​​the reference image and the perspective-transformed image to be pieced together using the intersection point, and generating a fusion mask based on the boundary line, the method further includes: The number of intersections of the puzzle to be completed is obtained. When the number of intersections is less than a first preset threshold, the center distance between the center of the image of the puzzle to be completed and the center of the image of the reference image is obtained. When the center distance is less than a preset distance threshold, a weighted superposition and fusion is performed on the reference image and the perspective-transformed image to be stitched together to obtain a stitched image. When the center distance is greater than or equal to the preset distance threshold, the puzzle piece to be processed is abandoned and the next puzzle piece is obtained as the new puzzle piece; Based on the new image to be pieced together and the baseline image, the step of extracting feature points is repeated.

4. The fundus camera multi-image adaptive panoramic stitching method as described in claim 3, characterized in that, After the step of obtaining the number of intersections of the puzzle piece, the method further includes: Count the number of intersections; When the number of intersections is greater than the second preset number threshold, it is determined that the overlapping area between the image to be stitched and the reference image is too large. Multi-band fusion is abandoned for the image to be stitched, and the reference image is used as the stitched image and the multi-band fusion operation is skipped. The second preset number threshold is greater than the first preset number threshold.

5. The fundus camera multi-image adaptive panoramic stitching method as described in claim 1, characterized in that, Before the step of extracting feature points from the obtained baseline image and the image to be pieced together, the method further includes: For each input raw fundus image, perform circular region of interest segmentation, generate a circular mask with the center of the raw fundus image as the center and half the width of the raw fundus image as the radius, and remove the black corner areas of the raw fundus image through the circular mask; Contrast-limited adaptive histogram equalization enhancement is performed on the cropped image to improve the contrast between the vascular texture and the background region in the image. The enhanced image is scaled down proportionally to a preset scaling size to reduce the computational load of subsequent feature extraction and matrix operations; A preset number of black pixel borders are added to the top, bottom, left, and right sides of the reduced image to reserve pixel buffer space for perspective transformation, thus obtaining the reference image or the image to be pieced together.

6. The fundus camera multi-image adaptive panoramic stitching method as described in claim 1, characterized in that, The steps of extracting feature points from the acquired reference image and the image to be pieced together, performing matching based on the extracted feature points, and estimating the transformation matrix of the image to be pieced together to the reference image based on the matching results include: Extract feature points from the baseline image and the image to be pieced together; An approximate nearest neighbor search algorithm is used to match the feature points of the baseline image with the feature points of the image to be pieced together, and the nearest neighbor feature point and the second nearest neighbor feature point are returned for each query point; Calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance for each query point, and retain matching pairs whose ratio is less than a first ratio threshold as initial screening matching pairs; When the number of initial screening matches is less than the preset number of matches, the first ratio threshold is relaxed to the second ratio threshold, and the matches are re-screened. The transformation matrix from the selected matching pairs to the base image is estimated using a random sampling consensus algorithm.

7. The fundus camera multi-image adaptive panoramic stitching method as described in claim 1, characterized in that, Before the step of validating the perspective-transformed puzzle image based on the maximum outer contour, the method further includes: The center point of the reference image is transformed to the coordinate system of the image to be pieced together using the transformation matrix to obtain the transformed center point; Calculate the displacement distance between the transformed center point and the center point of the reference image; When the displacement distance exceeds the preset displacement threshold, it is determined that the shooting position of the image to be stitched is too far away from the shooting position of the reference image, the image to be stitched is abandoned and the next image to be stitched is obtained as the new image to be stitched. Based on the new image to be pieced together and the baseline image, the step of extracting feature points is repeated.

8. The fundus camera multi-image adaptive panoramic stitching method as described in claim 1, characterized in that, After the steps of using the stitched image as a new reference image, obtaining the next image to be stitched as a new image to be stitched, and repeating the image stitching operation until all images to be stitched are processed, and outputting the reference image obtained from the last image stitching operation as a panoramic fundus image, the method further includes: Count the number of images in the entire dataset that passed the validity verification and were successfully stitched into the panoramic fundus image; When the number of images is zero, output a stitching failure message and output the initially acquired reference image; When the number of images is greater than zero and less than the total number of images to be pieced together, a success message is output. The message includes a list of successfully pieced-together image names and the displacement vector of each image.

9. A multi-image adaptive panoramic stitching device for fundus cameras, characterized in that, The fundus camera multi-image adaptive panoramic stitching device stores a computer program, which, when executed by a processor, implements the fundus camera multi-image adaptive panoramic stitching method according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the fundus camera multi-image adaptive panoramic stitching method according to any one of claims 1-8.

Images