An image detection-based eye surgery positioning method and system

Abstract

This invention relates to the field of medical image processing technology, and discloses a method and system for ocular surgical localization based on image detection. The method includes: acquiring intraoperative fundus color images, microscopic images, and optical coherence tomography (OCT) data; performing multimodal fine registration; generating enhanced fused images; performing retinal key structure detection and localization on the enhanced fused images; and generating intraoperative navigation overlay display information based on three-dimensional pose information using a residual-gated dynamic tracking mechanism. Compared to methods that rely solely on single-modal images for assisted manual localization in teaching demonstrations and simulation training systems, especially under conditions of intraoperative tissue occlusion and significant grayscale differences in multimodal imaging, this invention addresses the technical problem of failing to achieve stable and continuous tissue localization under complex intraoperative environments through unified calibration of multimodal images and a residual-gated dynamic tracking mechanism.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for locating eye surgery based on image detection. Background Technology

[0002] Currently, in ophthalmological procedures, especially delicate retinal surgeries, surgeons primarily rely on intraoperative microscopic observation, intraoperative OCT-assisted scanning, and preoperative fundus images for manual comparison and localization. These methods have significant shortcomings in practical application. For example, intraoperative microscopic images only provide two-dimensional surface visualization information, making it difficult to reflect the interlayer structure and depth relationships of retinal tissues; while OCT can provide high-precision tomographic information, its field of view is limited, and there is a lack of precise spatial correspondence between OCT and microscopic images, requiring surgeons to manually compare and rely on experience across different display interfaces; and preoperative fundus images, due to differences in shooting angle, lighting conditions, and eye posture, are often difficult to directly correlate with the current visual field during surgery.

[0003] Furthermore, during retinal surgery, the following complex situations often arise: surgical instruments frequently enter the field of vision, causing temporary occlusion of critical areas; minor deformations of retinal tissue or slight eye movements occur during surgery; significant differences in grayscale expression between different imaging modalities make it impossible to establish an effective comparison through simple image overlay; and low-contrast lesion boundaries or micro-tears are difficult to reliably identify in a single modality. In these situations, existing technologies typically rely on frame-by-frame manual judgment or simple algorithm-assisted localization based on single-modality images, which cannot fully meet the intraoperative requirements for continuous, stable, and high-precision three-dimensional localization. Once image occlusion, decreased contrast, or temporary loss of the target occurs, the localization results are prone to jumps or even interruptions, affecting the safety and accuracy of the surgical procedure.

[0004] Therefore, there is an urgent need for an ocular surgical positioning method that can still achieve precise alignment of multi-source images, information enhancement and fusion, and continuous and stable positioning and tracking even when there are significant differences in grayscale in multimodal imaging, the target is easily obscured during surgery, and there are subtle dynamic changes in retinal tissue, so as to improve the navigation accuracy, stability and safety during retinal surgery. Summary of the Invention

[0005] To address the aforementioned technical shortcomings, the purpose of this invention is to propose an image detection-based method for localization in ocular surgery. This method aims to solve the technical problem that relying solely on a single modal image for manual localization in teaching demonstrations and simulation training systems, especially under conditions of intraoperative tissue occlusion and significant grayscale differences in multimodal imaging, makes it impossible to achieve stable and continuous tissue localization.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides an image detection-based method for eye surgery localization.

[0007] The image detection-based ocular surgical localization method includes: Step S10: Acquire multimodal intraoperative image data, and construct initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism; wherein, the multimodal intraoperative image data includes fundus color images, microscopic images and OCT scan data; Step S20: Based on the initial alignment results, a joint optimization registration mechanism of feature point geometric constraints and mutual information is used to perform a fine registration task of multimodal images, and output a set of registration transformation parameters; Step S30: Based on the registration transformation parameter set, an information contribution-based optimal fusion mechanism is used to perform the fusion enhancement preprocessing task and output the enhanced fused image; Step S40: Based on the enhanced fused image, the edge-preserving structural saliency detection and template consistency verification mechanism is used to perform the retinal target structure detection and localization task, and output the three-dimensional pose information of the retinal target structure; Step S50: Based on the output three-dimensional pose information, a residual gating tracking mechanism is used to continuously track the retinal target structure and generate navigation overlay display information.

[0008] Preferably, step S10, which involves acquiring multimodal intraoperative image data and constructing initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism, specifically includes: Step S101: Acquire multimodal intraoperative image data, which includes intraoperative fundus color images, microscopic images and OCT scan data, and generate a unified timestamp for each frame of the multimodal intraoperative image data to form a multimodal synchronous image sequence. Step S102: Read the intrinsic parameters, extrinsic parameters and lens distortion parameters of each imaging device corresponding to the multimodal synchronous image sequence, and establish a distortion correction model and an initial projection model for mapping each modal image to a unified reference coordinate system. Step S103: Using the pixel coordinate system or retinal reference plane coordinate system corresponding to the fundus color image as a unified reference coordinate system, the microscopic image and OCT scan data are projected and aligned by combining the distortion correction model and the initial projection model, and the initial alignment result is output.

[0009] Preferably, step S20, which involves performing a multimodal image fine registration task based on the initial alignment results using a joint optimization registration mechanism of feature point geometric constraints and mutual information, and outputting a set of registration transformation parameters, specifically includes: Step S201: Cross-modal stable structural feature extraction stage: Extract stable structural features from fundus color images and microscopic images. Stable structural features include vascular bifurcation points, main vessel orientation edges, and optic disc edge structural features. Construct OCT projection results corresponding to stable structural features in OCT scan data. Step S202: Mutual information-driven joint optimization stage of registration parameters: Using the OCT projection result as the initial constraint, the optimization strategy of maximizing mutual information is adopted to jointly solve the multimodal registration transformation parameters, so that the microscopic image and the OCT projection result achieve the maximum statistical correlation under a unified reference coordinate system, and the registration result is output. Step S203: Registration quality assessment and transformation parameter set output stage: Perform residual statistics and consistency assessment on the registration results, screen out abnormal matching points and update the registration parameters, and output the final registration transformation parameter set.

[0010] Preferably, in step S202, the joint optimization stage of registration parameters driven by mutual information includes: using the OCT projection result as the initial constraint, adopting the optimization strategy of maximizing mutual information to jointly solve the multimodal registration transformation parameters. In each round of solution, the microscopic image and the OCT projection result are sampled to a unified reference coordinate system to construct a joint gray-level distribution, and the mutual information of the joint gray-level distribution is used as the objective function. At the same time, the geometric consistency constraint of feature points is introduced as a regularization term to suppress local extrema, so that the process of each round of solution can still converge to the correct registration solution even when the gray-level mapping relationship of different modes is nonlinear.

[0011] Preferably, step S30, which involves performing a fusion enhancement preprocessing task based on an information contribution-based optimal fusion mechanism using the registration transformation parameter set, and outputting an enhanced fused image, specifically includes: Step S301: Call the output set of registration transformation parameters to map the microscopic image and OCT projection result to the reference coordinate system of the fundus color image to form a pixel-level correspondence; Step S302: Perform multi-resolution decomposition on the multimodal intraoperative image data based on pixel-level correspondence to generate low-frequency background components and high-frequency detail components, and calculate the edge intensity, texture energy and structural saliency of each modality in the local region as information contribution indicators. Step S303: Perform weighted combination of high-frequency detail components based on edge intensity, texture energy and structural saliency of local regions, and perform consistent balance fusion on low-frequency background components to finally output enhanced fused image.

[0012] Preferably, step S40, which involves performing the detection and localization of the retinal target structure based on the enhanced fused image using an edge-preserving structural saliency detection and template consistency verification mechanism, and outputting the three-dimensional pose information of the retinal target structure, specifically includes: Step S401: Candidate region generation based on structural saliency constraints: Construct a structural saliency map on the enhanced fused image, generate a set of candidate regions for retinal target structures, and perform connected component filtering on the candidate region set to remove noise artifacts; Step S402: Edge-preserving segmentation and key point localization: Perform edge-preserving segmentation on the candidate region after connected component screening, and extract the set of vessel bifurcation points, optic disc edge points and lesion boundary points; Step S403: Template consistency check and 3D pose calculation: The set of vascular bifurcation points, optic disc edge points, and lesion boundary points are matched and verified with the pre-set preoperative planning template structure to eliminate inconsistencies; and the two-dimensional positioning results are mapped into three-dimensional pose information by combining the depth and slice thickness information in the OCT scan data, and the three-dimensional pose information of the retinal target structure is output.

[0013] Preferably, step S50, which involves continuously tracking the retinal target structure using a residual-gated tracking mechanism based on the output three-dimensional pose information and generating navigation overlay display information, specifically includes: Step S501: Construct a three-dimensional pose state vector based on the output three-dimensional pose information, and perform state prediction using Kalman filtering based on the three-dimensional pose state vector, and output the predicted state vector. Step S502: Obtain the observed state vector, calculate the state residual based on the predicted state vector and the observed state vector, and if the state residual exceeds the preset gating threshold, reduce the observation weight of the corresponding mode or trigger relocation, and output the updated target state. Step S503: Generate navigation overlay markers, trajectory information, and abnormal prompt information based on the updated target state, and output navigation overlay display information.

[0014] The present invention also provides an image detection-based eye surgery positioning system comprising: The multimodal acquisition and calibration module is used to acquire multimodal intraoperative image data and construct initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism; the multimodal intraoperative image data includes fundus color images, microscopic images and OCT scan data; The fine registration module is used to perform fine registration of multimodal images based on the initial alignment results, using a joint optimization registration mechanism of feature point geometric constraints and mutual information, and outputs a set of registration transformation parameters. The fusion enhancement preprocessing module is used to perform fusion enhancement preprocessing tasks based on the registration transformation parameter set and adopt an information contribution-based optimal fusion mechanism to output enhanced fused images. The key structure detection and localization module is used to perform the detection and localization of retinal target structures based on enhanced fusion images using an edge-preserving structural saliency detection and template consistency verification mechanism, and outputs the three-dimensional pose information of the retinal target structures. The dynamic tracking and navigation output module is used to continuously track the retinal target structure and generate navigation overlay display information based on the output three-dimensional pose information using a residual gating tracking mechanism.

[0015] The present invention also provides an image detection-based eye surgery positioning device, comprising: a memory, a processor, and an image detection-based eye surgery positioning program stored in the memory and executable on the processor. When the image detection-based eye surgery positioning program is executed by the processor, it implements an image detection-based eye surgery positioning method.

[0016] The present invention also provides a computer program product, including an image detection-based eye surgery localization program, which, when executed by a processor, implements the image detection-based eye surgery localization method.

[0017] The beneficial effects of this invention are as follows: By introducing a multimodal image unified calibration and mutual information-driven fine registration mechanism, this invention achieves high-precision alignment of intraoperative fundus images, microscopic images and OCT images in the same reference coordinate system. Compared with the traditional single-modal or simple superposition method, it effectively reduces the positioning error caused by multi-source image misalignment and significantly improves the spatial positioning accuracy of key retinal structures.

[0018] This invention employs an image enhancement mechanism that utilizes multi-resolution information contribution optimization fusion to enhance the presentation of structural details in different modal images that are most discriminative for surgical localization. This avoids the problems of detail blurring and contrast reduction caused by traditional average fusion, enabling low-contrast, small retinal lesions and boundaries to still be clearly identified in the intraoperative environment, thereby improving the integrity and reliability of target detection.

[0019] This invention constructs a residual-gated dynamic tracking and navigation overlay mechanism based on three-dimensional pose. Even in the event of field-of-view occlusion, imaging fluctuations, or changes in spatial attitude, it can still maintain the continuity and stability of target structure positioning and navigation display, thereby improving the real-time performance, stability, and reliability of multimodal images in teaching demonstrations and simulation training systems. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of an image detection-based eye surgery localization method according to the present invention.

[0022] Figure 2 This is a schematic diagram of an equipment for an eye surgery localization method based on image detection according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of the image detection-based eye surgery localization method of the present invention, which presents the first embodiment of the image detection-based eye surgery localization method of the present invention.

[0025] In the first embodiment, the image detection-based eye surgery localization method includes: Step S10: Acquire multimodal intraoperative image data, and construct initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism; wherein, the multimodal intraoperative image data includes fundus color images, microscopic images and OCT scan data; It should be noted that the "deep perspective and distortion joint calibration mechanism" in this step is not a traditional calibration method that relies solely on planar mapping based on two-dimensional pixel coordinate relationships. Instead, it simultaneously considers: the effects of radial and tangential distortions introduced by the microscope's optical system; the influence of the actual tissue depth information provided by OCT on the projection relationship; and the differences in perspective relationships between fundus images and microscopic images due to differences in shooting angles. Therefore, when establishing the initial alignment, pixel-level alignment is not simply achieved through translation, scaling, or rotation. Instead, an imaging mapping relationship that includes depth and distortion terms is established, enabling the OCT three-dimensional information to be accurately projected onto the two-dimensional fundus reference coordinate system, providing a physically meaningful initial alignment basis for subsequent fine registration.

[0026] Understandably, this joint calibration mechanism can achieve relatively uniform spatial alignment accuracy across the entire retinal field of view, especially in the peripheral regions, areas with significant optical distortion, and areas with large viewing angle differences, while maintaining good correspondence between multimodal images. This provides a reliable spatial basis for subsequent fine registration, multimodal fusion, and tissue localization, eliminating the need for significant corrections to initial geometric errors in subsequent processing, thereby improving the stability and accuracy of the overall localization process. For example, in actual intraoperative images, a vascular bifurcation point located in the periretinal area showed a pixel deviation of more than ten pixels between the microscopic image and the fundus image before this mechanism was adopted. OCT projection onto this area even resulted in significant misalignment, causing the vascular structure to appear in multiple non-overlapping positions in different modal images. However, after adopting the depth fluoroscopy and distortion joint calibration mechanism, the positional error of this vascular bifurcation point in the three images can be controlled within a very small range, and the interlayer structure reflected by OCT can be accurately projected onto the corresponding vascular location, providing a reliable basis for subsequent fusion enhancement and precise localization.

[0027] Step S20: Based on the initial alignment results, a joint optimization registration mechanism of feature point geometric constraints and mutual information is used to perform a fine registration task of multimodal images, and output a set of registration transformation parameters; It should be noted that the "joint optimization registration mechanism of feature point geometric constraints and mutual information" in this step refers to the following: based on the initial alignment with spatial significance obtained in step S10, instead of directly relying on grayscale similarity or simple pixel correlation for alignment, it first extracts structural feature points with stable geometric significance in different modalities, such as blood vessel bifurcation points, optic disc edge points, and main blood vessel direction edge points, to construct cross-modal geometric correspondences. Based on this geometric constraint, mutual information is introduced as a statistical correlation metric to jointly optimize the registration transformation parameters, enabling high-precision registration even when there are significant differences in grayscale representation between images of different modalities. This mechanism utilizes both geometric structural consistency and statistical distribution consistency information simultaneously, avoiding the limitations of a single registration strategy.

[0028] Understandably, by first establishing an initial geometric correspondence using stable structural features and then performing global optimization through mutual information, the robustness and accuracy of multimodal registration can be significantly improved. Even if the same structure appears as a bright region, a shadow region, or a texture boundary in different modalities, this joint mechanism can still correctly identify its spatial correspondence, avoiding mismatches caused by traditional grayscale matching methods under grayscale inversion or contrast differences, thus ensuring that subsequent fusion and localization are based on accurate alignment.

[0029] It should be understood that, compared to traditional techniques that rely solely on cross-correlation, mean square error, or gray-level difference for registration, this step introduces geometric feature point constraints, providing a clear structural reference for the registration process and avoiding the local extremum problem that may occur when relying solely on gray-level statistics. Furthermore, compared to methods that only use feature point matching, this step further introduces mutual information optimization, enabling the overall region to achieve statistically optimal alignment. The combination of these two approaches gives the registration both structural stability and global consistency, making it particularly suitable for multimodal scenarios with vastly different gray-level representation mechanisms, such as OCT, fundus images, and microscopic images.

[0030] For example, in actual intraoperative images, a vascular bifurcation structure appears as a dark red stripe in a color fundus image, a bright edge structure in a microscopic image, and an area of ​​interlayer reflection difference in an OCT projection image. If a traditional grayscale correlation method is used, this area is easily mismatched to a neighboring texture area due to the difference in grayscale modes, resulting in an alignment error of 5 to 10 pixels. If only feature point matching is used, matching is prone to failure in cases of vascular occlusion or local texture loss. However, through the joint optimization mechanism in this step, an initial correspondence is first established using the geometric position of the vascular bifurcation point, and then mutual information is used for iterative optimization. Ultimately, the registration error of this area in the three modalities can be controlled within a very small range, laying a precise foundation for subsequent fusion enhancement and localization.

[0031] Step S30: Based on the registration transformation parameter set, an information contribution-based optimal fusion mechanism is used to perform the fusion enhancement preprocessing task and output the enhanced fused image; It should be noted that the "information contribution-optimized fusion mechanism" in this step refers to the following: after completing multimodal fine registration and obtaining pixel-level correspondence in step S20, instead of using a simple weighted average or image overlay method for fusion, the structural information contribution of each modality image at the same pixel position is evaluated, and the modal information most beneficial to target recognition is selected for combination and reconstruction at multiple scale levels. The information contribution includes, but is not limited to, indicators such as local edge gradient intensity, texture energy distribution, structural saliency response, and local contrast difference. By comprehensively evaluating these indicators, it is determined which modality best represents the structure within the current region, thus prioritizing the preservation of detailed information from that modality during the fusion process.

[0032] Understandably, fundus images, microscopic images, and OCT projection images have different imaging advantages in different regions. For example, vascular textures are clearer in fundus images, interstitial tissue structures are more obvious in OCT, and real-time illumination details are more prominent in microscopic images. Therefore, simple averaging would lead to a weakening of details. In contrast, information contribution optimization fusion can automatically select the "optimal source of expression" in each local region, so that the fusion result maintains the maximum clarity of local details while achieving overall brightness balance, thereby significantly enhancing the recognizability of key structures.

[0033] It should be understood that, compared to traditional fusion methods that use fixed weights or simple superposition, this step introduces a dynamic optimization mechanism based on the contribution of structural information, giving the fusion result a significant "structural enhancement" characteristic. Traditional average fusion often results in blurred edges and reduced contrast in detailed areas, while this step prioritizes the retention of the most discriminative modal information at the high-frequency detail level and maintains overall brightness consistency at the low-frequency background level, making the fused image both clear and stable, and more suitable for subsequent target detection and localization. For example, in actual intraoperative images, a tiny retinal tear has low contrast in fundus color images, slight edge contrast in microscopic images, and obvious inter-slice separation signals in OCT projection images. If traditional weighted average fusion is used, the contrast of this tear area is only slightly improved, and it is still difficult for detection algorithms to identify; however, after using information contribution-optimized fusion, this area prioritizes the retention of high-frequency structural information from OCT, while combining it with edge information from the microscopic image, so that the tear boundary shows a significant enhancement effect in the fused image. Subsequent detection algorithms can more stably identify this area and avoid missed detection.

[0034] Step S40: Based on the enhanced fused image, the edge-preserving structural saliency detection and template consistency verification mechanism is used to perform the retinal target structure detection and localization task, and output the three-dimensional pose information of the retinal target structure; It should be noted that the "edge-preserving structural saliency detection and template consistency verification mechanism" in this step refers to the following: Based on the enhanced fusion image obtained in step S30, instead of directly using ordinary threshold segmentation or general object detection algorithms for identification, a structural saliency map reflecting the degree of local structural abrupt changes is first constructed to highlight geometrically stable regions such as vessel boundaries, optic disc contours, and lesion edges. Based on this, an edge-preserving segmentation strategy is used to extract candidate structural regions. Subsequently, pre-operative or preset structural templates (such as vessel orientation templates, optic disc morphology templates, and lesion morphology templates) are introduced for consistency verification, thereby filtering out noise artifacts and falsely detected regions. Finally, the verified two-dimensional localization results are combined with the depth information provided by OCT to map the corresponding three-dimensional pose information.

[0035] Understandably, the enhanced fusion image already strengthens the edges and contrast of key structures in step S30. This step prioritizes these geometrically significant regions through structural saliency detection, avoiding sensitivity to background texture or lighting changes. Furthermore, template consistency verification ensures that the detection results are not only saliency in the current frame but also conform to the expected anatomical features in terms of structural morphology, thus significantly improving the reliability of the localization results. Combined with OCT depth data, the two-dimensional detection results can be accurately transformed into spatially meaningful three-dimensional pose information. For example, in actual intraoperative images, a segment of blood vessel may appear as a bright spot in the microscopic image due to reflection, easily misidentified as a structural edge. If a traditional edge detection algorithm is used, this area may be mis-segmented as the target structure. In this step, the actual blood vessel boundary is first located through structural saliency detection, and then compared with the preoperative blood vessel orientation template. The bright spot region is found to be inconsistent with the continuous orientation characteristics of the blood vessel, and thus it is removed. Simultaneously, the actual blood vessel boundary is significantly enhanced in the enhanced fusion image, preserved after template verification, and its accurate position in three-dimensional space is obtained by combining OCT depth information for subsequent navigation and tracking.

[0036] Step S50: Based on the output three-dimensional pose information, a residual gating tracking mechanism is used to continuously track the retinal target structure and generate navigation overlay display information.

[0037] It should be noted that the "residual-gated tracking mechanism" in this step refers to the following: after obtaining the 3D pose information of the retinal target structure output in step S40, the current frame detection result is not directly used as the sole basis for localization. Instead, a 3D pose state model of the target is constructed to predict the target's historical motion states. Simultaneously, real-time observation information from microscopic images and OCT data is acquired. The reliability of the observation data is determined by calculating the residual difference between the predicted state and the observed state. When the residual exceeds a preset gating threshold, the weight of that observation mode is automatically reduced or a relocalization process is triggered, thereby preventing abnormal observation data from interfering with the localization results.

[0038] Understandably, due to the complex intraoperative environment, retinal tissue may undergo slight displacement due to eye movements, and surgical instruments may temporarily obscure the target area. If localization updates are based solely on the detection results of the current frame, it can easily lead to localization jumps or even loss. This step introduces a state prediction and residual judgment mechanism to make the localization process continuous and anti-interference. Even if the observation is incomplete or contains errors at a certain moment, stable predictions can still be made based on historical states, and smooth corrections can be performed after the observation returns to normal.

[0039] It should be understood that, compared to the traditional frame-by-frame detection and update-by-frame localization method, this step transforms target localization from an "instantaneous detection problem" to a "continuous state estimation problem." Traditional methods immediately interrupt navigation once target recognition fails in a frame; however, through residual gating, observational anomalies can be identified and their impact actively suppressed, ensuring a continuous and stable localization trajectory. Furthermore, by dynamically adjusting the observation weights for different modalities, tracking can still be maintained using reliable information sources even under occlusion or noise conditions.

[0040] For example, during actual surgery, when instruments enter the field of view and obscure part of the vascular structure, the detection result of the current frame of the microscopic image may show a significant shift, resulting in a large difference between the observed position and the predicted position. If a traditional method is used, this erroneous observation value would be directly used to update the positioning, causing the navigation marker to jump instantaneously. However, in this step, by calculating the residual and finding that the deviation exceeds the gating threshold, the weight of the observation in that frame of the microscopic image is automatically reduced, and the target position display continues to be maintained based on the predicted state from the previous moment. When the instruments are removed and the observation returns to normal, a smooth update is performed, ensuring that the navigation trajectory remains continuous and stable, without any jumps or interruptions.

[0041] Example 2: Furthermore, the present invention provides an image detection-based eye surgery localization system, which employs an image detection-based eye surgery localization method from the above embodiments, and can solve the technical problem of image detection-based eye surgery localization. The beneficial effects of the image detection-based eye surgery localization system provided by the present invention are the same as those of the image detection-based eye surgery localization method provided in the above embodiments, and other technical features of the image detection-based eye surgery localization system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0042] Example 3: This invention provides an image detection-based eye surgery positioning device. Please refer to... Figure 2An image detection-based eye surgery positioning device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform an image detection-based eye surgery positioning method as described in Embodiment 1 above. An image detection-based eye surgery positioning device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. An image detection-based eye surgery positioning device is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this invention. An image detection-based eye surgery positioning device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of an image detection-based eye surgery positioning device. Processing device 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. I / O interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows an image detection-based eye surgery positioning device to communicate wirelessly or wiredly with other devices to exchange data. Although an image detection-based eye surgery positioning device with various systems is shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0043] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image detection-based eye surgery localization method described above. The computer program product provided by this invention can solve the technical problem of image detection-based eye surgery localization. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the image detection-based eye surgery localization method provided in the above embodiments, and will not be repeated here.

[0044] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0045] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0046] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for localizing eye surgery based on image detection, characterized in that, The methods include: Step S10: Acquire multimodal intraoperative image data, and construct initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism; wherein, the multimodal intraoperative image data includes fundus color images, microscopic images and OCT scan data; Step S20: Based on the initial alignment results, a joint optimization registration mechanism of feature point geometric constraints and mutual information is used to perform a fine registration task of multimodal images, and output a set of registration transformation parameters; Step S30: Based on the registration transformation parameter set, an information contribution-based optimal fusion mechanism is used to perform the fusion enhancement preprocessing task and output the enhanced fused image; Step S40: Based on the enhanced fused image, the edge-preserving structural saliency detection and template consistency verification mechanism is used to perform the retinal target structure detection and localization task, and output the three-dimensional pose information of the retinal target structure; Step S50: Based on the output three-dimensional pose information, a residual gating tracking mechanism is used to continuously track the retinal target structure and generate navigation overlay display information.

2. The method for localizing eye surgery based on image detection as described in claim 1, characterized in that, Step S10, which involves acquiring multimodal intraoperative image data and constructing initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism, specifically includes: Step S101: Acquire multimodal intraoperative image data, which includes intraoperative fundus color images, microscopic images and OCT scan data, and generate a unified timestamp for each frame of the multimodal intraoperative image data to form a multimodal synchronous image sequence. Step S102: Read the intrinsic parameters, extrinsic parameters and lens distortion parameters of each imaging device corresponding to the multimodal synchronous image sequence, and establish a distortion correction model and an initial projection model for mapping each modal image to a unified reference coordinate system. Step S103: Using the pixel coordinate system or retinal reference plane coordinate system corresponding to the fundus color image as a unified reference coordinate system, the microscopic image and OCT scan data are projected and aligned by combining the distortion correction model and the initial projection model, and the initial alignment result is output.

3. The method for localizing eye surgery based on image detection as described in claim 2, characterized in that, Step S20, which involves performing a multimodal image fine registration task based on the initial alignment results using a joint optimization registration mechanism combining feature point geometric constraints and mutual information, and outputting a set of registration transformation parameters, specifically includes: Step S201: Cross-modal stable structural feature extraction stage: Extract stable structural features from fundus color images and microscopic images. Stable structural features include vascular bifurcation points, main vessel orientation edges, and optic disc edge structural features. Construct OCT projection results corresponding to stable structural features in OCT scan data. Step S202: Mutual information-driven joint optimization stage of registration parameters: Using the OCT projection result as the initial constraint, the optimization strategy of maximizing mutual information is adopted to jointly solve the multimodal registration transformation parameters, so that the microscopic image and the OCT projection result achieve the maximum statistical correlation under a unified reference coordinate system, and the registration result is output. Step S203: Registration quality assessment and transformation parameter set output stage: Perform residual statistics and consistency assessment on the registration results, screen out abnormal matching points and update the registration parameters, and output the final registration transformation parameter set.

4. The method for localizing eye surgery based on image detection as described in claim 3, characterized in that, In step S202, the mutual information-driven joint optimization stage of registration parameters includes: using the OCT projection result as the initial constraint, adopting the optimization strategy of maximizing mutual information to jointly solve the multimodal registration transformation parameters. In each round of solution, the microscopic image and the OCT projection result are sampled to a unified reference coordinate system to construct a joint gray-level distribution, and the mutual information of the joint gray-level distribution is used as the objective function. At the same time, the geometric consistency constraint of feature points is introduced as a regularization term to suppress local extrema, so that the process of each round of solution can still converge to the correct registration solution even when the gray-level mapping relationship of different modes is nonlinear.

5. The method for localizing eye surgery based on image detection as described in claim 1, characterized in that, Step S30, which involves performing a fusion enhancement preprocessing task based on the registration transformation parameter set using an information contribution-based optimal fusion mechanism to output an enhanced fused image, specifically includes: Step S301: Call the output set of registration transformation parameters to map the microscopic image and OCT projection result to the reference coordinate system of the fundus color image to form a pixel-level correspondence; Step S302: Perform multi-resolution decomposition on the multimodal intraoperative image data based on pixel-level correspondence to generate low-frequency background components and high-frequency detail components, and calculate the edge intensity, texture energy and structural saliency of each modality in the local region as information contribution indicators. Step S303: Perform weighted combination of high-frequency detail components based on edge intensity, texture energy and structural saliency of local regions, and perform consistent balance fusion on low-frequency background components to finally output enhanced fused image.

6. The method for localizing eye surgery based on image detection as described in claim 1, characterized in that, Step S40, which involves performing the detection and localization of retinal target structures based on edge-preserving structural saliency detection and template consistency verification mechanisms using enhanced fused images, and outputting the three-dimensional pose information of the retinal target structures, specifically includes: Step S401: Candidate region generation based on structural saliency constraints: Construct a structural saliency map on the enhanced fused image, generate a set of candidate regions for retinal target structures, and perform connected component filtering on the candidate region set to remove noise artifacts; Step S402: Edge-preserving segmentation and key point localization: Perform edge-preserving segmentation on the candidate region after connected component screening, and extract the set of vessel bifurcation points, optic disc edge points and lesion boundary points; Step S403: Template consistency check and 3D pose calculation: The set of vascular bifurcation points, optic disc edge points, and lesion boundary points are matched and verified with the pre-set preoperative planning template structure to eliminate inconsistencies; and the two-dimensional positioning results are mapped into three-dimensional pose information by combining the depth and slice thickness information in the OCT scan data, and the three-dimensional pose information of the retinal target structure is output.

7. The method for localizing eye surgery based on image detection as described in claim 1, characterized in that, Step S50, which involves continuously tracking the retinal target structure using a residual-gated tracking mechanism based on the output three-dimensional pose information and generating navigation overlay display information, specifically includes: Step S501: Construct a three-dimensional pose state vector based on the output three-dimensional pose information, and perform state prediction using Kalman filtering based on the three-dimensional pose state vector, and output the predicted state vector. Step S502: Obtain the observed state vector, calculate the state residual based on the predicted state vector and the observed state vector, and if the state residual exceeds the preset gating threshold, reduce the observation weight of the corresponding mode or trigger relocation, and output the updated target state. Step S503: Generate navigation overlay markers, trajectory information, and abnormal prompt information based on the updated target state, and output navigation overlay display information.

8. An image detection-based eye surgery localization system, applied to the image detection-based eye surgery localization method according to any one of claims 1 to 7, characterized in that, The image detection-based ocular surgery localization system includes: The multimodal acquisition and calibration module is used to acquire multimodal intraoperative image data and construct initial alignment results based on the multimodal intraoperative image data using a combined depth fluoroscopy and distortion calibration mechanism; the multimodal intraoperative image data includes fundus color images, microscopic images and OCT scan data; The fine registration module is used to perform fine registration of multimodal images based on the initial alignment results, using a joint optimization registration mechanism of feature point geometric constraints and mutual information, and outputs a set of registration transformation parameters. The fusion enhancement preprocessing module is used to perform fusion enhancement preprocessing tasks based on the registration transformation parameter set and adopt an information contribution-based optimal fusion mechanism to output enhanced fused images. The key structure detection and localization module is used to perform the detection and localization of retinal target structures based on enhanced fusion images using an edge-preserving structural saliency detection and template consistency verification mechanism, and outputs the three-dimensional pose information of the retinal target structures. The dynamic tracking and navigation output module is used to continuously track the retinal target structure and generate navigation overlay display information based on the output three-dimensional pose information using a residual gating tracking mechanism.

9. An image detection-based ocular surgical positioning device, characterized in that, The image detection-based eye surgery localization device includes: a memory, a processor, and an image detection-based eye surgery localization program stored in the memory and executable on the processor. When the image detection-based eye surgery localization program is executed by the processor, it implements an image detection-based eye surgery localization method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes an image detection-based eye surgery localization program, which, when executed by a processor, implements an image detection-based eye surgery localization method according to any one of claims 1 to 7.

Images