Multi-view eye fundus image splicing method and system suitable for multiple modes

Abstract

The invention provides a multi-view eye fundus image splicing method suitable for multiple modalities, and the method comprises the steps: obtaining a plurality of to-be-spliced images of different modalities, carrying out the corresponding preprocessing according to the imaging characteristics of the modalities to which the images belong, and obtaining a preprocessed image group; selecting one of the preprocessed image groups as a reference image, and performing image registration on the other images and the reference image to obtain corresponding registration image groups; on the basis of a preset registration precision threshold value and the proportion of overlapping areas between the images, screening out the images meeting the requirements from the registration image group; and performing multi-scale fusion processing according to the spatial relationship between the screened images to obtain a spliced eye fundus image. According to the multi-view eye fundus image splicing method provided by the invention, through preprocessing, registration, screening and multi-scale fusion, images of different modes can be efficiently integrated, the splicing precision and the image quality are improved, and the method is suitable for accurate eye fundus analysis and medical diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-view fundus image stitching method and system applicable to multiple modalities. Background Art

[0002] Laser fundus cameras are commonly used in clinical ophthalmology examinations. They utilize confocal laser imaging at different wavelengths to capture the anatomical structures and pathological features of the fundus at all levels, including color imaging and angiography. These imaging modalities provide clinicians with a wealth of fundus information, enabling more accurate diagnosis and follow-up of fundus diseases. However, due to limitations in the structure of the eye and the angle of capture, a single fundus image has a limited field of view and cannot fully display all fundus regions. Therefore, image stitching technology is widely used to combine multiple images to expand the field of view and obtain a more comprehensive fundus image.

[0003] Existing fundus image stitching methods are mostly designed for single-modality or fixed-viewpoint images, making them difficult to adapt to the differences in imaging characteristics, noise, and perspective variations among multimodal images. This is particularly true when dealing with complex overlapping relationships between multiple images and registration and fusion at random perspectives, making both efficiency and accuracy difficult to guarantee. Furthermore, images of different modalities are often acquired using different devices, resulting in inconsistent imaging parameters between devices, further increasing the technical difficulty of multimodal image stitching. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-view fundus image stitching method and system applicable to multiple modalities, so as to solve the problems in the prior art that it is difficult to take into account the characteristics of multi-modal images, cannot effectively handle random view angles and complex overlapping relationships, and thus lead to insufficient stitching accuracy and applicability.

[0005] According to a first aspect of the present invention, a multi-view fundus image stitching method applicable to multiple modalities is provided, the method comprising:

[0006] Acquire multiple images to be stitched of different modalities, and perform corresponding preprocessing according to the imaging characteristics of the modality to which each image belongs, to obtain a preprocessed image group;

[0007] Selecting one image from the pre-processed image group as a reference image, and performing image registration on the remaining images respectively with the reference image to obtain a corresponding registered image group;

[0008] Based on a preset registration accuracy threshold and an overlapping area ratio between images, selecting images that meet the requirements from the registered image group;

[0009] Multi-scale fusion processing is performed according to the spatial relationship between the filtered images to obtain the spliced ​​fundus image.

[0010] Optionally, the specific steps of acquiring a plurality of images to be stitched of different modalities and performing corresponding preprocessing according to the imaging characteristics of the modality to which each image belongs to obtain a preprocessed image group include:

[0011] dividing the images to be stitched into MC modality images, IR / RF modality images, and FFA, ICGA, or AF modality images;

[0012] The MC modality image is sequentially subjected to non-local mean denoising, histogram equalization and Gaussian filtering processing;

[0013] performing non-local mean denoising, histogram equalization, Gaussian filtering, and weighted superposition processing on the IR / RF modality image in sequence;

[0014] The FFA, ICGA or AF modality images are sequentially subjected to Gaussian filtering and weighted superposition processing.

[0015] Optionally, the specific steps of selecting one image from the preprocessed image group as a reference image and performing image registration on the remaining images with the reference image to obtain a corresponding registered image group include:

[0016] Based on the local features of the reference image and the image to be registered, adjusting the position and angle of the image to be registered to achieve coarse registration;

[0017] Based on the structural information of the reference image and the image to be registered, a non-rigid transformation is performed on the image to be registered to achieve precise registration.

[0018] Optionally, the specific steps of adjusting the position and angle of the image to be registered based on local features of the reference image and the image to be registered to achieve coarse registration include:

[0019] The SIFT algorithm is used to extract feature points of the reference image and the image to be registered, and the feature points are matched based on the KNN algorithm;

[0020] Filter out valid key point pairs according to the matching results, and calculate the translation and rotation parameters of the image to be registered relative to the reference image;

[0021] A rigid transformation is performed on the image to be registered based on the translation and rotation parameters to obtain a coarse registration image.

[0022] Optionally, based on the structural information of the reference image and the image to be registered, performing a non-rigid transformation on the image to be registered to achieve precise registration includes:

[0023] extracting blood vessel structures from the reference image and the coarse registration image;

[0024] Calculating a pixel-level displacement field between the reference image and the coarse registration image based on an optical flow method;

[0025] A non-rigid transformation is performed on the coarse registration image according to the displacement field to obtain a fine registration image.

[0026] Optionally, the specific step of screening out images that meet the requirements from the registered image group based on a preset registration accuracy threshold and an overlapping area ratio between images includes:

[0027] Calculating a registration quality indicator of an overlapping area between the registered image and the reference image, the registration quality indicator comprising mutual information, peak signal-to-noise ratio, or perceptual hash similarity;

[0028] Screening out images whose registration quality index is lower than a preset threshold;

[0029] When the ratio of overlapping areas between images exceeds a preset ratio threshold, image deduplication is performed and only one image is retained.

[0030] Optionally, the specific steps of performing multi-scale fusion processing according to the spatial relationship between the filtered images to obtain a spliced ​​fundus image include:

[0031] Based on the intersection relationship between the filtered images, the spatial coverage of each image in the overlapping area is determined;

[0032] According to the intersection relationship, a fusion mask is generated for each image to indicate the fusion weight of the image in each overlapping area.

[0033] Optionally, the specific step of performing multi-scale fusion processing according to the spatial relationship between the filtered images to obtain a spliced ​​fundus image further includes:

[0034] Constructing a Gaussian pyramid and a Laplacian pyramid for the filtered image respectively, and constructing a Gaussian pyramid for the fusion mask corresponding to the filtered image;

[0035] At each resolution level, the image Laplacian pyramid is weighted fused based on the Gaussian pyramid of the fusion mask;

[0036] The fused image is reconstructed layer by layer in the order of low to high resolution to generate the final high-resolution stitched image.

[0037] Optionally, the reference image is selected based on the degree of centering of the macular area in each image to be stitched, and the image with the highest degree of centering of the macular area is selected as the reference image.

[0038] According to a second aspect of the present invention, a multi-view fundus image stitching system applicable to multiple modalities is provided, the system comprising:

[0039] An image acquisition module, used to acquire multiple images to be stitched in different modalities;

[0040] An image preprocessing module is used to perform corresponding preprocessing on the images to be stitched according to the imaging characteristics of the modality to which each image belongs, to obtain a preprocessed image group;

[0041] An image registration module is configured to select a reference image from the preprocessed image group, and perform image registration on the remaining images respectively with the reference image to obtain a registered image group;

[0042] An image screening module is used to screen out images that meet the requirements from the registered image group based on a preset registration accuracy threshold and an overlapping area ratio between images;

[0043] The image fusion module is used to perform multi-scale fusion processing based on the spatial relationship between the filtered images to obtain a spliced ​​fundus image.

[0044] According to the present invention, targeted preprocessing based on the imaging characteristics of images from different modalities improves image similarity and registration accuracy. Furthermore, through dual screening of baseline image registration, registration accuracy, and overlapping regions, the alignment quality of the images involved in the fusion is ensured. Multi-scale fusion, leveraging the spatial relationships between images, not only enhances the structural continuity and detail fidelity of the images but also enables unified stitching of multimodal, multi-view fundus images, effectively expanding the imaging field of view and enhancing the image support capabilities for clinical diagnosis.

[0045] Furthermore, the present invention effectively improves the accuracy and robustness of image alignment by designing differentiated preprocessing strategies for different modal images, combining local feature registration with structure-guided non-rigid registration methods. A registration quality screening mechanism can eliminate substandard images, preventing the stitching effect from being affected by low-quality or redundant images. The multi-scale fusion strategy achieves a natural transition while preserving image detail, ensuring the integrity and visual consistency of the final stitching result. Furthermore, the present invention further optimizes image layout based on the selection of a reference image centered in the macula, enhancing the clinical utility of the stitched images.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic flow chart of a multi-view fundus image stitching method applicable to multiple modalities according to an embodiment of the present invention is shown;

[0048] Figure 2 Shown Figure 1 A schematic flow chart of a method for obtaining a plurality of images to be stitched of different modalities in step S100 and performing corresponding preprocessing according to the imaging characteristics of the modality to which each image belongs to obtain a preprocessed image group;

[0049] Figure 3 Shown Figure 1 A schematic flow chart of a method for selecting one image from the pre-processed image group as a reference image and performing image registration on the remaining images with the reference image to obtain a corresponding registered image group in step S200;

[0050] Figure 4 Shown Figure 1 A schematic flow chart of a method for selecting images that meet the requirements from a group of registered images based on a preset registration accuracy threshold and an overlapping area ratio between images in step S300;

[0051] Figure 5 Shown Figure 1 A schematic flow chart of a method for screening out images that meet the requirements from a group of registered images based on a preset registration accuracy threshold and an overlapping area ratio between images in step S400;

[0052] Figure 6 FIG2 shows a stitching result diagram based on a color fundus camera (MC) image according to one embodiment of the present invention;

[0053] Figure 7 FIG2 shows a mosaic result diagram based on infrared imaging (IR) images according to one embodiment of the present invention;

[0054] Figure 8 FIG. 2 shows a stitching result diagram based on red-free (RF) images according to one embodiment of the present invention;

[0055] Figure 9 FIG2 shows a stitching result diagram based on fluorescein fundus angiography (FFA) images according to one embodiment of the present invention;

[0056] Figure 10 FIG2 shows a mosaic result diagram based on indocyanine green angiography (ICGA) images according to one embodiment of the present invention;

[0057] Figure 11 FIG. 4 shows a stitching result diagram based on autofluorescence (AF) images according to one embodiment of the present invention;

[0058] Figure 12 A structural block diagram of a multi-view fundus image stitching system applicable to multiple modalities according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It will be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] The terms "comprise," "comprising," and "having," and any variations thereof, as used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0061] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] Figure 1 FIG. 1 is a schematic flow chart of a multi-view fundus image stitching method applicable to multiple modalities according to an embodiment of the present invention. Figure 1 As shown, the multi-view fundus image stitching method includes the following steps:

[0063] Step S100 : obtaining a plurality of images to be stitched of different modalities, and performing corresponding preprocessing on each image according to the imaging characteristics of the modality to which the image belongs, to obtain a preprocessed image group.

[0064] Since images of different modalities have different imaging characteristics and image quality, targeted preprocessing helps to improve the consistency and comparability of images, thereby improving the accuracy and stability of subsequent image registration.

[0065] In step S200 , one image is selected from the pre-processed image group as a reference image, and the remaining images are registered with the reference image to obtain a corresponding registered image group.

[0066] By establishing a unified coordinate system transformation, geometric differences such as translation and rotation between the multi-view images are accurately corrected to generate a set of spatially aligned registered images. This step S200 achieves spatial consistency alignment of the multi-view images, laying the geometric foundation for subsequent image fusion and ensuring that the stitched wide-field image has accurate anatomical structure correspondence.

[0067] Step S300 : Based on a preset registration accuracy threshold and an overlapping area ratio between images, images meeting the requirements are screened out from the registered image group.

[0068] By evaluating the registration error and effective overlap range, images with poor registration or substandard overlap are eliminated, retaining a set of images that meet the fusion quality standards. This step S300 establishes a dual quality control mechanism, ensuring both reliable registration accuracy and an appropriate overlap between the images, providing data assurance for achieving optimal stitching results.

[0069] Step S400 : performing multi-scale fusion processing according to the spatial relationship between the filtered images to obtain a spliced ​​fundus image.

[0070] In this step, by analyzing the spatial overlap between images, image features are aligned and fused in a multi-scale spatial domain, ultimately generating a wide-field fundus image with an expanded field of view, complete details, and natural transitions. This step S400 effectively addresses the common issues of visible seams and uneven brightness seen in traditional stitching methods, maintaining the continuity of fundus structure and the integrity of details while expanding the field of view.

[0071] According to the above embodiment, by combining the imaging characteristics of different modal images for targeted preprocessing, the similarity between images and the accuracy of registration are improved. On this basis, through the dual screening of reference image registration and registration accuracy and overlapping areas, the images involved in the fusion are ensured to have good alignment quality. At the same time, the spatial relationship between images is utilized for multi-scale fusion, which not only enhances the structural continuity and detail fidelity of the images, but also realizes the unified splicing of multimodal and multi-view fundus images, effectively expanding the imaging field of view and improving the image support capabilities for clinical diagnosis.

[0072] In one embodiment, the reference image is selected based on the degree of centering of the macular area in each image to be stitched, and the image with the highest degree of centering of the macular area is selected as the reference image.

[0073] By selecting the image with the highest degree of centering in the macular area as the reference image, it can ensure that the subsequent alignment process uses the most important anatomical landmarks of the retina as the central reference. This not only significantly improves the spatial alignment accuracy of multi-view images, but also effectively avoids the distortion of peripheral retinal structures caused by improper selection of reference images. At the same time, due to the key position of the macular area in clinical diagnosis, this selection strategy can prioritize the image quality of the central field of view area, so that the final stitching result maintains optimal geometric accuracy and detail integrity in diagnostically sensitive areas.

[0074] Figure 2 Shown Figure 1 A schematic flow chart of the method of obtaining a plurality of images to be stitched of different modalities in step S100 and performing corresponding preprocessing according to the imaging characteristics of the modality to which each image belongs to obtain a preprocessed image group. Figure 2 As shown, step S100 includes:

[0075] In step S110 , the images to be stitched are divided into MC modality images, IR / RF modality images, and FFA, ICGA, or AF modality images.

[0076] Step S110 first performs modality identification and classification on the input fundus images to be stitched, accurately categorizing them into three modality categories: MC (color fundus photography), IR / RF (infrared / red-free imaging), and FFA / ICGA / AF (fluorescence angiography). This standardized modality classification system provides a clear path for subsequent targeted preprocessing, effectively resolving the issue of incompatible processing parameters caused by differences in imaging principles for multimodal images and ensuring that each modality receives the preprocessing flow best suited to its imaging characteristics.

[0077] In step S120 , the MC modality image is subjected to non-local mean denoising, histogram equalization, and Gaussian filtering in sequence.

[0078] For MC modality images, step S120 sequentially performs a three-stage process: non-local means denoising, histogram equalization, and Gaussian filtering. Non-local means denoising effectively suppresses speckle noise while preserving details such as vascular texture; histogram equalization enhances the contrast of retinal structures; and Gaussian filtering smoothes the processed image. This processing chain is particularly well-suited to the characteristics of color fundus photography, eliminating reflection artifacts while significantly improving the visibility of key structures such as the optic disc and macula, providing high-quality, feature-rich images for subsequent registration.

[0079] In step S130 , non-local mean denoising, histogram equalization, Gaussian filtering, and weighted superposition processing are sequentially performed on the IR / RF modality image.

[0080] For IR / RF modality images, step S130 performs the same denoising and enhancement as for MC modality, but additionally adds weighted overlay processing. Initial non-local means denoising and histogram equalization optimize the visualization of deep retinal vascular structures; large-scale Gaussian filtering captures background information; and the final overlay further enhances retinal vascular features through a weighted fusion of the two. This processing approach is particularly well-suited to the low-contrast and significant noise characteristics of infrared imaging, significantly improving the imaging quality of deep tissues such as the choroid and providing a stable source of features for multimodal registration.

[0081] Step S140 : performing Gaussian filtering and weighted superposition processing on the FFA, ICGA or AF modality images in sequence.

[0082] For fluorescence angiography (FFA / ICGA / AF) images, step S140 employs a streamlined processing flow combining Gaussian filtering and weighted superposition. Gaussian filtering directly applies to the original image, smoothing noise while preserving vascular fluorescence morphology; weighted superposition integrates the effective signal from multiple frames. This step takes into account the high dynamic range and large inter-frame variability of fluorescence images. While ensuring that pathological features such as vascular leakage are not blurred, it effectively suppresses background fluorescence noise, providing a reliable imaging foundation for accurate localization of pathological areas.

[0083] Figure 3 Shown Figure 1 A schematic flow chart of the method of selecting one image from the pre-processed image group as a reference image in step S200, and performing image registration on the remaining images with the reference image to obtain a corresponding registered image group. Figure 3 As shown, step S200 includes:

[0084] Step S210 : Based on the local features of the reference image and the image to be registered, the position and angle of the image to be registered are adjusted to achieve coarse registration.

[0085] This step S210 significantly improves the efficiency and stability of subsequent registration by preliminarily correcting the overall displacement and rotation differences between images, effectively avoiding the excessive deformation and distortion problems that may be caused by direct fine registration, and provides reliable initial transformation parameters for the precise alignment of multi-view fundus images.

[0086] Step S220 : Based on the structural information of the reference image and the image to be registered, a non-rigid transformation is performed on the image to be registered to achieve precise registration.

[0087] This step S220 further optimizes the local alignment accuracy based on the coarse registration, and can accurately compensate for subtle deformations caused by the curvature of the eyeball and the difference in imaging perspective, ensure the continuity of the vascular network and retinal layer structure, and ultimately achieve sub-pixel registration effect, laying a precise geometric foundation for the seamless fusion of multimodal images.

[0088] In one embodiment, the above step S210 includes the following specific steps:

[0089] Step S211 : extracting feature points of the reference image and the image to be registered using the SIFT algorithm, and matching the feature points based on the KNN algorithm.

[0090] In this step, local feature points in the image are extracted using the SIFT (Scale Invariant Feature Transform) algorithm. The advantage of SIFT is that it can handle the effects of scale, rotation, and illumination changes on image matching. During the feature extraction process, SIFT first detects potential points of interest in the image using the Gaussian difference function. These points are scale-invariant, ensuring that these feature points can still be effectively identified even if the image is scaled. Next, SIFT assigns a direction to each feature point and describes it based on its local gradient direction to ensure robustness to rotation and illumination changes.

[0091] After feature point extraction, the K-nearest neighbor (KNN) algorithm is used to match feature points in the reference image and the image to be registered. The KNN algorithm calculates the Euclidean distance between each feature descriptor and the other set of descriptors and selects the top K pairs of feature points with the smallest distance as candidate matches. This matching step allows the algorithm to efficiently find key point pairs between the two images, providing a foundation for subsequent registration.

[0092] Step S212 , filtering out valid key point pairs according to the matching results, and calculating the translation and rotation parameters of the image to be registered relative to the reference image.

[0093] In this step, by analyzing the matched key point pairs, valid point pairs are selected and used to calculate the translation and rotation parameters between the images. Valid point pairs ensure the accuracy of the registration, and the calculated translation and rotation parameters provide the necessary data for subsequent image transformations. This step S212 ensures high precision in the image registration process. By properly calculating translation and rotation, displacement and angular differences between the images can be eliminated, laying the foundation for precise registration.

[0094] Step S213 , performing a rigid transformation on the image to be registered based on the translation and rotation parameters to obtain a coarse registration image.

[0095] In this step, a rigid transformation is performed on the image to be registered, based on the translation and rotation parameters calculated in the previous step. This transformation translates and rotates the image to align it with the reference image. This rigid transformation ensures that the overall structure and proportions of the image remain unchanged, ensuring that there is no additional deformation or distortion during the registration process. The benefit of this step S213 is that it performs preliminary image registration, providing a coarse registration result that is closer to the reference image for subsequent fine registration, thereby improving registration efficiency and accuracy.

[0096] In one embodiment, the above step S220 includes the following specific steps:

[0097] Step S221 : extracting blood vessel structures from the reference image and the coarse registration image.

[0098] Vascular structures are prominent and relatively fixed features in fundus images, serving as a crucial reference during image registration. Extracting vascular structures ensures that key anatomical features in the image are preserved during subsequent registration. This step uses image processing algorithms to identify and enhance vascular textures in the image, providing reliable reference information that helps accurately align key anatomical structures within the image and lays a solid foundation for further registration.

[0099] Step S222 : Calculate the pixel-level displacement field between the reference image and the coarse registration image based on the optical flow method.

[0100] Optical flow is a motion estimation method based on local image brightness changes. It estimates the pixel-level displacement field by tracking the changes in adjacent pixels between two images. Optical flow can accurately calculate the movement of each pixel in space and is particularly suitable for processing images with rich details and complex structures. In fundus image registration, the precise pixel-level displacement field provides an accurate basis for subsequent non-rigid transformations, ensuring precise alignment of image details.

[0101] Step S223 , performing a non-rigid transformation on the coarse registration image according to the displacement field to obtain a fine registration image.

[0102] In this step, a non-rigid transformation is performed on the coarsely registered image based on the displacement field calculated in the previous step. This non-rigid transformation allows for more subtle image deformation during the registration process, not just translation and rotation. Instead, it adjusts local regions of the image based on pixel-by-pixel changes in the displacement field, thereby more accurately matching the structure of the reference image. This transformation is crucial for processing complex vascular structures and other details in fundus images, effectively improving registration accuracy and ensuring efficient alignment and accurate reconstruction of images across different regions, ultimately generating a finely registered image.

[0103] Figure 4 Shown Figure 1 Schematic flow chart of the method for screening out images that meet the requirements from the registered image group based on the preset registration accuracy threshold and the overlap area ratio between images in step S300. Figure 4 As shown, step S300 includes:

[0104] Step S310 , calculating a registration quality index of an overlapping area between the registered image and the reference image, where the registration quality index includes mutual information, peak signal-to-noise ratio, or perceptual hash similarity.

[0105] In this step, the similarity between the overlapping areas of the registered image and the reference image is first calculated. The matching accuracy of each pair of images is then evaluated using various registration quality metrics. For images in the MC, IR, and RF modalities, mutual information and peak signal-to-noise ratio (PSNR) are used to measure image registration. Mutual information quantifies the amount of information shared between two images and effectively evaluates image similarity. PSNR measures the degree of image distortion; higher values ​​indicate better registration accuracy. For images in the FFA, ICGA, and AF modalities, mutual information and perceptual hashing similarity are used to evaluate image matching. Perceptual hashing extracts features from the image and generates a fingerprint (i.e., a perceptual number sequence) that effectively describes the perceptual characteristics of the image and provides good comparability, thereby accurately assessing the similarity between images. The combined calculation of these quality metrics allows for an objective assessment of the accuracy of image registration, providing a basis for subsequent image screening.

[0106] Step S320: Screen out images whose registration quality index is lower than a preset threshold.

[0107] In this step, images with registration quality indicators below a preset threshold are filtered out. Registration quality is an important metric for measuring image registration effectiveness. A low quality indicator for an image indicates poor matching accuracy with the reference image, potentially leading to mismatches or inaccurate overlap. By setting a reasonable threshold, poor-quality images can be automatically removed, preventing them from negatively impacting the stitching result and ensuring the accuracy and consistency of the final stitched image.

[0108] Step S330 : When the ratio of overlapping areas between images exceeds a preset ratio threshold, image deduplication is performed, and only one image is retained.

[0109] In actual photography, multiple images may overlap in the same or similar areas. Excessive duplicate images not only increases the computational burden but can also lead to image redundancy and unnatural transitions during the stitching process. In step S330, by setting a reasonable overlap ratio threshold, duplicate images can be removed and the best matching image retained, effectively reducing redundant information and optimizing the stitching effect, resulting in a more concise and accurate final stitched image.

[0110] In one embodiment, the preset ratio threshold is 95%, that is, when the overlapping area between each pair of registered images accounts for more than 95% of the entire image, the two images are determined to have highly duplicated content, and only one of the images is retained.

[0111] Figure 5 Shown Figure 1 Schematic flow chart of the method for screening out images that meet the requirements from the registered image group based on the preset registration accuracy threshold and the overlap area ratio between images in step S400. Figure 5 As shown, step S400 includes:

[0112] Step S410 : determining the spatial coverage of each image in the overlapping area based on the intersection relationship between the filtered images.

[0113] In this step, the overlapping seams between the selected images are analyzed to identify the spatial locations covered by each image in the multi-image mosaic scene and the areas of overlap with adjacent images. This process helps to clearly define the spatial boundaries of each image in the target mosaic image, providing an accurate basis for region demarcation in subsequent fusion operations, thereby avoiding occlusion conflicts and fusion errors between images.

[0114] Step S420 : generating a fusion mask for each image based on the intersection relationship, to indicate the fusion weight of the image in each overlapping area.

[0115] After completing the spatial coverage analysis, a blending mask is generated for each image based on the position and overlap of the intersection lines between the images. This mask represents the fusion weight of the images in different overlapping areas in units of pixels, ensuring that the images in the final stitching can smoothly transition based on their spatial position and the degree of overlap between adjacent images, thereby improving the naturalness of the blended area and overall visual consistency.

[0116] In step S430 , a Gaussian pyramid and a Laplacian pyramid are constructed for the filtered image, and a Gaussian pyramid is constructed for the fusion mask corresponding to the filtered image.

[0117] In this step, a corresponding Gaussian pyramid and Laplacian pyramid are constructed for each image to be stitched, supporting subsequent multi-resolution image fusion operations. A corresponding Gaussian pyramid is also constructed for each fusion mask, ensuring a smooth and consistent weight distribution at all levels of resolution. By processing image details and contours at different scales, the edge preservation and layering of the fused image are significantly enhanced.

[0118] Step S440 : performing weighted fusion on the Laplacian pyramid of the image at each resolution level based on the Gaussian pyramid of the fusion mask.

[0119] In this step, the Laplacian pyramid images of all corresponding image levels are fused, using the Gaussian pyramid of the fusion mask as a weighting factor. This method strikes a good balance between image detail and overall structure, ensuring that the fused image exhibits smooth transitions and sharp edges at all scales, avoiding common issues such as noticeable stitching seams and uneven brightness.

[0120] Step S450 : reconstructing the fused image layer by layer in order of resolution from low to high to generate a final high-resolution stitched image.

[0121] Finally, the system gradually reconstructs the fused image from lower to higher resolution layers, stacking the pyramidal images layer by layer to produce a high-resolution, seamlessly transitioned, and fully detailed fundus image. This process ensures consistent and high-quality fusion across the full resolution range, meeting the image clarity and integrity requirements for clinical or scientific research.

[0122] Figure 6 FIG. 4 shows a stitching result diagram based on a color fundus camera (MC) image according to an embodiment of the present invention. Figure 7 A diagram showing a stitching result based on infrared imaging (IR) images according to an embodiment of the present invention is shown. Figure 8 FIG. 4 shows a stitching result diagram based on red-free (RF) images according to an embodiment of the present invention. Figure 9 A diagram showing a stitching result based on fluorescein fundus angiography (FFA) images according to an embodiment of the present invention is shown. Figure 10 A diagram showing a stitching result based on indocyanine green angiography (ICGA) images according to one embodiment of the present invention is shown. Figure 11 FIG. 4 shows a stitching result diagram based on autofluorescence (AF) images according to an embodiment of the present invention.

[0123] These images demonstrate the effectiveness of the present method in stitching images from different modalities, achieving high-quality stitching across various fundus imaging techniques. Comparing the stitching results from different modalities clearly demonstrates the high precision and stability of the present method, resolving issues such as ghosting, distortion, and inaccurate alignment that plague existing stitching techniques. Therefore, the present method holds significant promise for application in multimodal fundus image analysis, particularly in medical imaging diagnosis and treatment.

[0124] The embodiment of the present invention also provides a multi-view fundus image stitching system applicable to multiple modalities, such as Figure 12 As shown, the multi-view fundus image stitching system includes an image acquisition module 101 , an image preprocessing module 102 , an image registration module 103 , an image screening module 104 and an image fusion module 105 .

[0125] The image acquisition module 101 is used to acquire multiple images to be stitched of different modalities. The image preprocessing module 102 is used to perform corresponding preprocessing on the images to be stitched according to the imaging characteristics of the modality to which each image belongs, to obtain a preprocessed image group. The image registration module 103 is used to select a reference image from the preprocessed image group, and perform image registration on the remaining images with the reference image to obtain a registered image group. The image screening module 104 is used to screen out images that meet the requirements from the registered image group based on a preset registration accuracy threshold and the ratio of overlapping areas between images. The image fusion module 105 is used to perform multi-scale fusion processing based on the spatial relationship between the screened images to obtain a stitched fundus image.

[0126] In the multi-view fundus image stitching system applicable to multiple modalities, the specific implementation method of obtaining the stitched image refers to the relevant content of the embodiment of the above method, and will not be elaborated here.

[0127] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A multi-view fundus image stitching method applicable to multiple modalities, characterized in that: The method comprises: Acquire multiple images to be stitched of different modalities, and perform corresponding preprocessing according to the imaging characteristics of the modality to which each image belongs, to obtain a preprocessed image group; Selecting one image from the pre-processed image group as a reference image, and performing image registration on the remaining images respectively with the reference image to obtain a corresponding registered image group; Based on a preset registration accuracy threshold and an overlapping area ratio between images, selecting images that meet the requirements from the registered image group; Multi-scale fusion processing is performed according to the spatial relationship between the filtered images to obtain the spliced ​​fundus image.

2. The multi-view fundus image stitching method according to claim 1, characterized in that: The specific steps of obtaining a plurality of images to be stitched of different modalities and performing corresponding preprocessing according to the imaging characteristics of the modality to which each image belongs to obtain a preprocessed image group include: dividing the images to be stitched into MC modality images, IR / RF modality images, and FFA, ICGA, or AF modality images; The MC modality image is sequentially subjected to non-local mean denoising, histogram equalization and Gaussian filtering processing; performing non-local mean denoising, histogram equalization, Gaussian filtering, and weighted superposition processing on the IR / RF modality image in sequence; The FFA, ICGA or AF modality images are sequentially subjected to Gaussian filtering and weighted superposition processing.

3. The multi-view fundus image stitching method according to claim 1, wherein: The specific steps of selecting one image from the pre-processed image group as a reference image and performing image registration on the remaining images with the reference image to obtain a corresponding registered image group include: Based on the local features of the reference image and the image to be registered, adjusting the position and angle of the image to be registered to achieve coarse registration; Based on the structural information of the reference image and the image to be registered, a non-rigid transformation is performed on the image to be registered to achieve precise registration.

4. The multi-view fundus image stitching method according to claim 3, characterized in that: The specific steps of adjusting the position and angle of the image to be registered based on the local features of the reference image and the image to be registered to achieve coarse registration include: The SIFT algorithm is used to extract feature points of the reference image and the image to be registered, and the feature points are matched based on the KNN algorithm; Filter out valid key point pairs according to the matching results, and calculate the translation and rotation parameters of the image to be registered relative to the reference image; A rigid transformation is performed on the image to be registered based on the translation and rotation parameters to obtain a coarse registration image.

5. The multi-view fundus image stitching method according to claim 4, characterized in that: Based on the structural information of the reference image and the image to be registered, the specific steps of performing non-rigid transformation on the image to be registered to achieve precise registration include: extracting blood vessel structures from the reference image and the coarse registration image; Calculating a pixel-level displacement field between the reference image and the coarse registration image based on an optical flow method; A non-rigid transformation is performed on the coarse registration image according to the displacement field to obtain a fine registration image.

6. The multi-view fundus image stitching method according to claim 5, characterized in that: The specific steps of screening out images that meet the requirements from the registered image group based on a preset registration accuracy threshold and an overlapping area ratio between images include: Calculating a registration quality indicator of an overlapping area between the registered image and the reference image, the registration quality indicator comprising mutual information, peak signal-to-noise ratio, or perceptual hash similarity; Screening out images whose registration quality index is lower than a preset threshold; When the ratio of overlapping areas between images exceeds a preset ratio threshold, image deduplication is performed and only one image is retained.

7. The multi-view fundus image stitching method according to claim 6, characterized in that: The specific steps of performing multi-scale fusion processing according to the spatial relationship between the filtered images to obtain the spliced ​​fundus image include: Based on the intersection relationship between the filtered images, the spatial coverage of each image in the overlapping area is determined; According to the intersection relationship, a fusion mask is generated for each image to indicate the fusion weight of the image in each overlapping area.

8. The multi-view fundus image stitching method according to claim 7, characterized in that: The specific step of performing multi-scale fusion processing according to the spatial relationship between the filtered images to obtain the spliced ​​fundus image also includes: Constructing a Gaussian pyramid and a Laplacian pyramid for the filtered image respectively, and constructing a Gaussian pyramid for the fusion mask corresponding to the filtered image; At each resolution level, the image Laplacian pyramid is weighted fused based on the Gaussian pyramid of the fusion mask; The fused image is reconstructed layer by layer in the order of low to high resolution to generate the final high-resolution stitched image.

9. The method according to claim 1, characterized in that The selection of the reference image is based on the degree of centering of the macular area in each image to be stitched, and the image with the highest degree of centering of the macular area is selected as the reference image.

10. A multi-view fundus image stitching system applicable to multiple modalities, characterized in that: The system comprises: An image acquisition module, used to acquire multiple images to be stitched in different modalities; An image preprocessing module is used to perform corresponding preprocessing on the images to be stitched according to the imaging characteristics of the modality to which each image belongs, to obtain a preprocessed image group; An image registration module is configured to select a reference image from the preprocessed image group, and perform image registration on the remaining images respectively with the reference image to obtain a registered image group; An image screening module is used to screen out images that meet the requirements from the registered image group based on a preset registration accuracy threshold and an overlapping area ratio between images; The image fusion module is used to perform multi-scale fusion processing based on the spatial relationship between the filtered images to obtain a spliced ​​fundus image.

Images