A method for real-time navigation of ophthalmic surgery fusing multi-modal data

Abstract

The application discloses a kind of ophthalmic surgery real-time navigation methods of fusion multimodal data, it is related to medical operation navigation technical field, it solves the technical problem of navigation deviation caused by single data source when ophthalmic surgery navigation;The present application extracts standard navigation data by collecting real-time action data and eyeball data, and the data is denoised and calibrated and state prediction by Kalman filtering algorithm, the eyeball vector feature and lesion data are extracted by PnP algorithm and micro-motion trajectory parameter fitting, the lesion is accurately segmented and three-dimensional coordinates are updated in real time by encoding-decoding network and attention mechanism, and the operation path data is generated by weighting fusion of multimodal data based on the confidence calculation formula of variance;Effective matching group is screened by Euclidean distance and matching threshold, and least square method is used to calculate registration transformation matrix to adaptively select re-registration or path constraint strategy by double-layer residual threshold judgment mechanism, and the fitting of operation path is completed.

Description

Technical Field

[0001] This invention belongs to the field of medical surgical navigation technology, specifically a real-time navigation method for ophthalmic surgery that integrates multimodal data. Background Technology

[0002] In the field of ophthalmic surgical navigation, path planning and guidance typically combine preoperative imaging data with intraoperative instrument tracking. However, ophthalmic surgery involves extremely limited operating space and demands extremely high precision. During the procedure, the patient's eyeball often undergoes uncontrollable micro-movements, while tissue deformation and instrument manipulation deviations can cause deviations between the actual surgical path and the preoperatively planned path. Most existing navigation solutions rely on single-modal data or simple multimodal overlay, lacking a dynamic deep fusion mechanism for motion and eyeball data, and thus failing to effectively eliminate sensor noise and biases. Furthermore, in the spatial registration stage, current technologies struggle to accurately compensate for deviations caused by eyeball micro-movements and tissue deformations, lacking adaptive residual threshold judgment and local re-registration mechanisms. This results in insufficient navigation path fitting accuracy, poor real-time navigation reliability, and a high risk of surgical complications. Therefore, there is an urgent need for an ophthalmic surgical method that can effectively fuse multimodal data, accurately compensate for deviations, and achieve high-precision real-time navigation.

[0003] This invention proposes a real-time navigation method for ophthalmic surgery that integrates multimodal data. By collecting real-time multimodal data and combining it with standard navigation data for path fitting, the method achieves dynamic generation and precise alignment of the surgical path, overcoming the navigation deviation problem caused by a single data source and solving the aforementioned technical issues. Summary of the Invention

[0004] The present invention aims to at least solve one of the technical problems existing in the prior art; to this end, the present invention proposes a real-time navigation method for ophthalmic surgery that integrates multimodal data, in order to solve the technical problem of navigation deviation caused by a single data source in ophthalmic surgery navigation in the prior art.

[0005] To achieve the above objectives, a first aspect of the present invention provides a real-time navigation method for ophthalmic surgery that integrates multimodal data, comprising: Real-time motion data is collected through a built-in force sensor, eye data is obtained by acquiring two-dimensional images of the eye, and standard navigation data is extracted from the database. Surgical path data is obtained based on real-time motion data and eye data; By combining surgical path data with standard navigation data, the surgical path can be fitted and accurately navigated. Real-time motion data and lesion feature data are dynamically fused to generate surgical path data, including: Modal data is obtained by fusing real-time motion data with lesion feature data; The confidence level is calculated using the following formula: ; The calculated number based on data variance is: Confidence of modal data ;in, For the first The variance of the modal data The total number of modal data participating in the fusion; The sub-data in the modal data are weighted and fused by combining confidence scores, and then combined with the RCM remote motion center kinematic model to obtain surgical path data.

[0006] Preferably, the step of obtaining surgical path data based on real-time motion data and eye data includes: Data preprocessing is performed on the collected real-time motion data and eye data; Feature extraction is performed on eyeball data to obtain eyeball view features and lesion data; lesion data is then localized and segmented to obtain lesion feature data. Real-time motion data and lesion feature data are dynamically fused to generate surgical path data; The dynamic fusion is achieved by calculating the confidence level of the modal data after fusing real-time motion data and lesion feature data, and then combining the confidence level with the RCM remote motion center kinematic model to perform weighted fusion of the sub-data in the modal data.

[0007] Preferably, the data preprocessing of the collected real-time motion data and eye data includes: The real-time motion data and eye data are denoised and calibrated, and the Kalman filter algorithm is used to eliminate noise in the real-time motion data and correct sensor bias. Through the formula: ; Complete the prediction of eye movement trends; in, express The system state vector at any given time. Represents the state transition matrix. Represents the control input matrix. This represents the control input vector at time k-1; Through the formula: ; Complete the prediction of covariance; among which, Represents the covariance matrix. Represents the process noise covariance. Let the covariance matrix at the previous time step be denoted as . This represents the state transition matrix of the Kalman filter. Representation matrix The transpose of the matrix; Calculate the Kalman gain and perform the state update, then perform the covariance update.

[0008] Preferably, the step of extracting features from the eyeball data to obtain eyeball view features and lesion data includes: Using the bottom left corner of the obtained image as the initial point, construct a pixel coordinate system and set the corresponding pixel coordinate point in the coordinate system for each pixel. Using the PnP algorithm: ; Complete the 3D to 2D conversion; among which, The coordinates of the point in the pixel coordinate system Let these be the coordinates of the point in the camera coordinate system. Let these be the coordinates of the point in the world coordinate system. For the depth of the point, This is the intrinsic parameter matrix of the camera. For a point in the world coordinate system matrix, For a point in the world coordinate system Translation vector; The continuous frame coordinate changes of infrared markers and iris feature points on the surface of the eyeball are extracted from the eyeball data. Micro-motion trajectory parameters are fitted to obtain the micro-motion trajectory parameters. The infrared markers and iris feature points are adjusted by the micro-motion trajectory parameters to obtain the eyeball vector features, and the lesion pixels and lesion range in the image are located.

[0009] Preferably, the step of locating and segmenting the lesion data to obtain lesion feature data includes: The grayscale and edge features of the lesion area are extracted. An encoder-decoder network structure is adopted. The encoder extracts deep features of the eye image step by step through convolutional and pooling layers to capture the contour and texture information of the intraocular structure. The decoder performs feature map upsampling through deconvolutional layers and combines the shallow and deep features of the encoder to segment the key area containing the lesion. At the same time, by adding an attention mechanism, the feature weights of key structures are strengthened. Combined with the preoperative lesion coordinate benchmark, the three-dimensional coordinates of the lesion are updated in real time.

[0010] Preferably, the step of combining surgical path data with standard navigation data to achieve surgical path fitting and precise navigation includes: Spatial registration is performed between surgical path data and standard navigation data, aligning real-time path coordinates with standard path templates and three-dimensional anatomical models of the eyeball to compensate for path deviations caused by micro-movements of the eyeball, tissue deformation, and instrument operation deviations. Simultaneously, the path deviation value between the real-time path and the standard path is calculated, and the subsequent surgical path is corrected based on the path deviation value to obtain the navigation surgical path.

[0011] Preferably, the spatial registration of surgical path data with standard navigation data includes: The feature points in the surgical path data that correspond to the coordinates of standard feature points in the standard navigation data are marked as real-time feature points. The standard feature points and real-time feature points are combined to obtain a matching feature point group. The similarity of matching feature point groups is calculated using Euclidean distance, according to the Euclidean distance formula: ; Calculate the feature point similarity between matching feature point groups ;in, For real-time feature descriptors Dimensional value, For the standard feature descriptor Dimensional value; Set a matching threshold; determine whether the similarity of feature points in a matching feature point group is greater than the matching threshold; if yes, mark the corresponding matching feature point group as an incorrect matching group; if no, mark the corresponding matching feature point group as a valid matching group. The registration transformation matrix of the effective matching group is calculated using the least squares method. The specific formula is as follows: ; in, For the first The registration transformation matrix of each real-time feature point For the first Real-time feature point coordinates, For the first Coordinates of a standard feature point To effectively match the number of pairs, Represents the real-time feature point set Rotation registration matrix, Represents the real-time feature point set Translation vector; This matrix is ​​then used to complete spatial registration of real-time path data.

[0012] Preferably, the step of correcting the subsequent surgical path based on the path deviation value to obtain the navigation surgical path includes: S1: By formula: ; Calculate the registration residuals between the real-time path and the standard path. ;in, The coordinates of the standard feature points after transformation. For the first Real-time feature point coordinates, To ensure an effective number of matching pairs; S2: Set residual thresholds, including a first residual threshold and a second residual threshold; determine whether the registration residual is greater than the first residual threshold; if yes, re-execute the complete registration process; otherwise, jump to S3; S3: Determine whether the registration residual is greater than the second residual threshold; if yes, trigger local re-registration; if no, determine that the registration is qualified, and complete the path correction by using the least squares method to obtain the navigation surgical path.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a fundamental input source for multimodal data by collecting real-time motion data and eye data, and extracting standard navigation data, overcoming the deficiency of incomplete information representation in ophthalmic surgery due to a single data source. Through Kalman filtering, the data is denoised, calibrated, and state predicted, effectively eliminating sensor noise and bias, accurately predicting eye movement trends, and providing highly stable data support for subsequent path generation. By using the PnP algorithm and fitting micro-motion trajectory parameters, eye vector features and lesion data are extracted, achieving precise mapping between three-dimensional space and two-dimensional images, and dynamic capture of eye micro-movements. Finally, through an encoder-decoder network and attention mechanism, lesions are accurately segmented and three-dimensional coordinates are updated in real time. The system improves the accuracy of lesion feature extraction by using a variance-based confidence calculation formula to perform weighted fusion of multimodal data, achieving objective allocation and dynamic adaptive fusion of data weights, and generating high-precision surgical path data. It also improves the accuracy of spatial registration by filtering effective matching groups using Euclidean distance and matching thresholds, and calculating the registration transformation matrix using the least squares method. Finally, it adaptively selects re-registration or path constraint strategies through a two-layer residual threshold judgment mechanism, optimizing computational efficiency while ensuring registration accuracy, effectively compensating for path deviations caused by eyeball micro-movements, tissue deformation, and instrument operation deviations, ultimately achieving high-precision real-time navigation and path fitting in ophthalmic surgery. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram illustrating the working principle of the present invention.

[0016] Figure 2 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] A first aspect of the present invention provides a real-time navigation method for ophthalmic surgery that integrates multimodal data, comprising: Real-time motion data is collected through a built-in force sensor, eye data is obtained by acquiring two-dimensional images of the eyeball, and standard navigation data is extracted from the database. Surgical path data is obtained based on real-time motion data and eye data; By combining surgical path data with standard navigation data, the surgical path can be fitted and accurately navigated. Real-time motion data and lesion feature data are dynamically fused to generate surgical path data, including: Modal data is obtained by fusing real-time motion data with lesion feature data; The confidence level is calculated using the following formula: ; The calculated number based on data variance is: Confidence of modal data ;in, For the first The variance of the modal data The total number of modal data participating in the fusion; The sub-data in the modal data are weighted and fused by combining confidence scores, and then combined with the RCM remote motion center kinematic model to obtain surgical path data.

[0019] like Figure 1 As shown, this embodiment provides a real-time navigation method for ophthalmic surgery that integrates multimodal data. This method constructs a complete logical closed loop from multi-source data acquisition to dynamic path generation and then to standard template fitting, aiming to solve the pain point of navigation path deviation caused by micro-movements of the eyeball and tissue deformation during ophthalmic surgery.

[0020] Step S100: Collect real-time motion data and eye data, and extract standard navigation data from the database.

[0021] The real-time motion data in this application can be acquired through an electromagnetic tracker, an optical locator, or sensors built into a robotic arm; eye data can be acquired through an intraoperative microscope, an OCT scanner, or an infrared camera.

[0022] Meanwhile, the "standard navigation data" extracted from the database refers to the ideal surgical path template and the three-dimensional anatomical model of the eyeball based on preoperative image planning.

[0023] Step S200: Obtain surgical path data based on real-time motion data and eye data.

[0024] In this embodiment, to address the high-frequency electromagnetic interference and low-frequency drift of the sensors during surgery, the system employs a Kalman filter algorithm to eliminate noise in the real-time motion data and correct sensor biases. It predicts eye movement trends and covariance using formulas, and calculates the Kalman gain status and updates the covariance accordingly.

[0025] Through the formula: ; Complete the prediction of eye movement trends; in, express The system state vector at any given time. Represents the state transition matrix. Represents the control input matrix. This represents the control input vector at time k-1; The system state vector typically includes the position coordinates, velocity components, and sensor bias estimates of the surgical instruments in three-dimensional space. Its physical significance lies in comprehensively depicting the true dynamic state of the action data and eye data at the current moment. The state transition matrix F describes the natural evolution of the system from the previous moment to the current moment.

[0026] Through the formula: ; Complete the prediction of covariance; among which, Let Q represent the covariance matrix, and let Q represent the process noise covariance. Let the covariance matrix at the previous time step be denoted as . This represents the state transition matrix of the Kalman filter. Representation matrix The transpose of the matrix; The covariance matrix characterizes the correlation and spread range of errors in each dimension of the state vector, and its physical meaning is to measure the confidence interval of the prediction result; the process noise covariance Q reflects the inherent defect of the physical model itself in that it cannot perfectly characterize the real motion. For example, the elastic deformation of the eye tissue or the tiny frictional force change when the instrument comes into contact can be regarded as process noise.

[0027] It should be understood that the assignment logic of Q needs to be adaptively adjusted according to the actual intensity of the disturbance during the operation. If the surgical environment is stable, the value of Q is smaller to enhance the confidence of the model. If the eyeball moves violently, Q is appropriately increased to accommodate the model error. This dynamic assignment mechanism gives the algorithm a strong environmental adaptability.

[0028] Calculate the Kalman gain and perform the state update, then perform the covariance update; Kalman gain acts as a bridge between the prediction phase and the observation update phase. When the observation noise is high, the gain automatically decreases, making the system more reliant on prior predictions; when the process noise is high, the gain automatically increases, making the system more reliant on real-time observations.

[0029] By using the state update formula, the system superimposes the residuals of prior predictions and actual observations according to the gain ratio. This not only eliminates random noise from the sensor but, more importantly, dynamically corrects sensor biases. Subsequent covariance updates then narrow the error confidence interval, signifying a reduction in system uncertainty. This collaborative iterative mechanism of prediction and update ensures an optimal balance between noise suppression and real-time response in the output data.

[0030] The acquired eye images were converted into coordinates, establishing a two-dimensional Cartesian coordinate system with the lower left corner of the eye image as the origin, the horizontal rightward direction as the u-axis, and the vertical upward direction as the v-axis. A corresponding pixel coordinate point was assigned for each pixel; and a camera coordinate system and a world coordinate system were constructed with the camera position as the origin. The 3D to 2D conversion is achieved using the PnP algorithm. The specific formula is as follows: ; in, The coordinates of the point in the pixel coordinate system Let these be the coordinates of the point in the camera coordinate system. Let these be the coordinates of the point in the world coordinate system. For the depth of the point, This is the intrinsic parameter matrix of the camera. For a point in the world coordinate system matrix, For a point in the world coordinate system Translation vector; The PnP algorithm is used to transform the eye image from a 3D coordinate system in the world coordinate system to a 2D coordinate system for mapping. The world coordinate system coordinates represent the absolute position of the eye or lesion in the real 3D physical space. The pose transformation matrix describes the camera's rotation and translation in 3D space. The intrinsic parameter matrix K contains inherent optical parameters of the camera, such as focal length and principal point coordinates, and describes the geometric relationship of light rays projected from the camera coordinate system onto the imaging plane. Depth... This reflects the distance information of a spatial point along the camera's optical axis, used to correct the projection scale; finally, the pixel coordinates are calculated. This enables precise positioning from the macroscopic three-dimensional physical space to the microscopic two-dimensional image plane. This mapping mechanism ensures that the two-dimensional image features captured in real time during surgery can be accurately back-projected into the three-dimensional navigation space, providing a distortion-free spatial alignment basis for subsequent dynamic fusion and spatial registration.

[0031] Infrared markers are miniature infrared reflective spheres attached to the conjunctival region of the eye surface before surgery. They present highly contrasting bright spots under the infrared light source of the microscope during surgery, facilitating rapid algorithm localization. Iris feature points are inherent feature points in the iris texture. This embodiment specifically explains why infrared markers and iris feature points must be chosen as tracking targets, rather than traditional limbal vessel feature points. Limbal vessels are prone to congestion and deformation during surgery due to instrument contact or changes in intraocular pressure, leading to feature point drift and instability. Infrared markers, on the other hand, have physical rigidity, while iris feature points, located deep within the eye, are minimally affected by surface deformation. The combination of these two forms a stable and reliable tracking target combination, preventing infringers from using easily deformable surface vessel feature points as a substitute to circumvent the micro-motion tracking mechanism of this invention.

[0032] By extracting the pixel coordinate changes of the target in consecutive frame images, the algorithm can fit the micro-motion trajectory parameters. The micro-motion trajectory parameters referred to in this application are a set of dynamic vector data describing the amplitude, frequency and phase of eye micro-movements. Subsequently, the original coordinates of the infrared marker points and iris feature points are adjusted and compensated in reverse using the micro-motion trajectory parameters to eliminate the coordinate jitter caused by micro-movements, thereby extracting stable eye vector features.

[0033] It should be understood that although this embodiment elaborates on the mapping relationship of each parameter in the PnP algorithm formula and the micro-motion capture logic of infrared markers and iris feature points, the specific number of targets selected and the specific functional form of the fitting parameters should be adjusted according to different surgical types and the physiological characteristics of the patient's eyeball. The description in this embodiment is only illustrative and not restrictive. Any target substitution or fitting function modification that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this invention should be included within the protection scope of this invention.

[0034] The lesion data is located and segmented to obtain lesion feature data; This step first extracts the grayscale and edge features of the lesion area as preliminary clues, and then introduces an encoder-decoder network structure for in-depth feature mining and spatial reconstruction.

[0035] First, the eye image is progressively reduced in dimensionality and abstracted through multiple convolutional and pooling layers. Shallow convolutional kernels primarily capture local grayscale features such as edges and textures, while pooling operations continuously shrink the feature map size and expand the receptive field. Deep convolutional kernels, on the other hand, can extract global semantic information such as the overall topological contours of the retina and choroid. This progressive extraction mechanism from local to global allows the network to infer the true anatomical attributes of a region even when local pixels are occluded by shadows, relying on the global topological structure.

[0036] Simultaneously, by adding an attention mechanism, the feature weights of key structures are strengthened, and the three-dimensional coordinates of the lesion are updated in real time by combining the preoperative lesion coordinate benchmark.

[0037] It's important to note that the attention mechanism doesn't passively accept all fused features. Instead, it actively recalibrates the spatial channels and locations in the feature map using learned weight parameters. It can automatically identify and enhance the feature weights of key structures such as the retina, choroid, and lesions, while significantly suppressing the interference weights of background areas, shadow areas, and reflective artifacts.

[0038] The method involves dynamically fusing real-time motion data with lesion feature data to generate surgical path data. First, these two heterogeneous data sets are initially concatenated and aligned in the feature space to form a modal data set containing multidimensional information. The confidence score is then calculated using the following formula: ; The confidence level of modal data numbered i was calculated based on the data variance. ;in, Let be the variance of the i-th modality data. This represents the total number of modal data participating in the fusion.

[0039] The variance of each modal data is obtained by calculating the dispersion of the data in real time using a workstation; then, the reciprocal of the variance of each modal data is calculated according to the formula. The larger the reciprocal value, the smaller the dispersion of the corresponding modal data and the more stable the data; subsequently, the calculation... The sum of the reciprocals of the variances of the modal data Finally, the confidence level of each modality is obtained by dividing the inverse of its variance by the sum above. The confidence level ranges from (0,1), and The sum of the confidence levels of the various modalities is 1. The higher the confidence level, the greater the influence weight of the modality on the generation of the surgical path.

[0040] Sub-data within each modality (such as the instrument end-effector coordinates in the motion modality and the target 3D coordinates in the lesion modality) are categorized according to their corresponding confidence levels. By weighted superposition and combining it with a kinematic model, the system can not only determine the ideal target position of the instrument, but also generate a feasible motion trajectory based on physical constraints, and finally generate surgical path data containing path coordinate sequence, direction vector, and real-time distance between the instrument and the target.

[0041] Step S300: Combine surgical path data with standard navigation data to complete the path fitting and precise navigation of the surgery.

[0042] The feature points in the surgical path data that correspond to the coordinates of standard feature points in the standard navigation data are marked as real-time feature points. The standard feature points and real-time feature points are combined to obtain a matching feature point group. The similarity of matching feature point groups is calculated using Euclidean distance, according to the Euclidean distance formula: ; Calculate the feature point similarity between matching feature point groups ;in, For real-time feature descriptors Dimensional value, For the standard feature descriptor Dimensional value.

[0043] Understandably, Euclidean distance The smaller the value of d, the smaller the numerical difference between the two vectors in each dimension, meaning that the local texture patterns they describe are more similar, and the higher the reliability of the matching. Conversely, the larger the value of d, the more significant the texture difference, and the more likely the matching is wrong. By determining the matching feature points, the path can be better standardized and fitted.

[0044] In this embodiment, the matching threshold is an empirically or adaptively calculated distance threshold, and its setting logic needs to comprehensively consider the intraoperative image noise level and the stability of feature extraction.

[0045] The system determines whether the similarity of feature points in a matching feature point group is greater than a matching threshold. If yes, the corresponding matching feature point group is marked as an incorrect matching group; otherwise, it is marked as a valid matching group. This binary classification screening mechanism based on hard thresholds effectively purifies the dataset and ensures that the observation data input to the least squares method are all high-quality interior points.

[0046] After obtaining a clean and effective matching group, the registration transformation matrix of the effective matching group is calculated using the least squares method. The specific formula is as follows: ; in, For the first The registration transformation matrix of each real-time feature point For the first Real-time feature point coordinates, For the first Coordinates of a standard feature point To effectively match the number of pairs, Represents the real-time feature point set Rotation registration matrix, Represents the real-time feature point set Translation vector.

[0047] The optimization goal of the least squares method is to find an optimal set of M' and T' such that the sum of squared errors between the coordinates of all valid matching pairs after transformation by this matrix and the standard coordinates is minimized globally. The essence of this optimization mechanism is to ensure that the transformation matrix best reflects the distribution trend of all valid observation points as a whole, rather than being influenced by individual residuals. After solving for the registration transformation matrix, the system applies it to every coordinate point in the real-time path data, forcibly mapping the intraoperative real-time path to the preoperative standard coordinate system. This achieves alignment of the real-time path with the standard path template and the three-dimensional anatomical model of the eyeball at a macroscopic level, initially compensating for path offsets caused by micro-movements of the eyeball and instrument operation deviations.

[0048] Through the formula: ; Calculate the registration residual e between the real-time path and the standard path; where, The registration residuals are the coordinates of the standard feature points after transformation. Essentially, it represents the average distance error remaining after registration for all valid matching pairs. It accurately reflects the degree of local fit in the current global registration. The smaller the value of e, the closer the registered path fits the standard template. The larger the value, the more it indicates that the local deformation or transient disturbance exceeds the compensation capability of the global registration, requiring the triggering of a higher-level correction mechanism.

[0049] S301: Set residual thresholds, which include a first residual threshold and a second residual threshold; determine whether the registration residual is greater than the first residual threshold; if yes, re-execute the complete registration process; otherwise, jump to S302. S302: Determine whether the registration residual is greater than the second residual threshold; if yes, trigger local re-registration; if no, determine that the registration is qualified, and complete the path correction by using the least squares method to obtain the navigation surgical path.

[0050] The system adopts a compromise local re-registration strategy: only the feature points of the deformed local areas are re-extracted and matched, while the main framework of the global attitude matrix remains unchanged. This significantly reduces the amount of computation while correcting the accuracy, achieving a dynamic balance between accuracy and latency.

[0051] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0052] Working principle of the invention: This invention constructs a basic input source for multimodal data by collecting real-time motion data and eye data, and extracting standard navigation data. It uses a Kalman filter algorithm for data denoising, calibration, and state prediction, and employs the PnP algorithm and micro-motion trajectory parameter fitting to extract eye vector features and lesion data. A encoder-decoder network and attention mechanism are used to accurately segment lesions and update their three-dimensional coordinates in real time. A variance-based confidence calculation formula is used to weight and fuse the multimodal data, generating high-precision surgical path data. Effective matching groups are selected using Euclidean distance and a matching threshold, and the registration transformation matrix is ​​calculated using the least squares method. Finally, a two-layer residual threshold judgment mechanism adaptively selects between re-registration and path constraint strategies, ultimately achieving high-precision real-time navigation and path fitting for ophthalmic surgery.

[0053] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A real-time navigation method for ophthalmic surgery that integrates multimodal data, characterized in that, include: Real-time motion data is collected through a built-in force sensor, eye data is obtained by acquiring two-dimensional images of the eyeball, and standard navigation data is extracted from the database. Surgical path data is obtained based on real-time motion data and eye data; By combining surgical path data with standard navigation data, the surgical path can be fitted and accurately navigated. Real-time motion data and lesion feature data are dynamically fused to generate surgical path data, including: Modal data is obtained by fusing real-time motion data with lesion feature data; The confidence level is calculated using the following formula: ； The calculated number based on data variance is: Confidence of modal data ;in, For the first The variance of the modal data The total number of modal data participating in the fusion; The sub-data in the modal data are weighted and fused by combining confidence scores, and then combined with the RCM remote motion center kinematic model to obtain surgical path data; The process of obtaining surgical path data based on real-time motion data and eye data includes: Data preprocessing is performed on the collected real-time motion data and eye data; Feature extraction is performed on eyeball data to obtain eyeball view features and lesion data; lesion data is then localized and segmented to obtain lesion feature data. Real-time motion data and lesion feature data are dynamically fused to generate surgical path data; The dynamic fusion is achieved by calculating the confidence level of the modal data after fusing real-time motion data and lesion feature data, and then combining the confidence level with the RCM remote motion center kinematic model to perform weighted fusion of the sub-data in the modal data. The data preprocessing of the collected real-time motion data and eye data includes: The real-time motion data and eye data are denoised and calibrated, and the Kalman filter algorithm is used to eliminate noise in the real-time motion data and correct sensor bias. Through the formula: ； Complete the prediction of eye movement trends; in, express The system state vector at any given time. The state transition matrix is ​​represented by [value 1], and the control input matrix is ​​represented by [value 2]. This represents the control input vector at time k-1; Through the formula: ； Complete the prediction of covariance; among which, Represents the covariance matrix. Represents the process noise covariance. Let the covariance matrix at the previous time step be denoted as . This represents the state transition matrix of the Kalman filter. Representation matrix The transpose of the matrix; Calculate the Kalman gain and perform the state update, then perform the covariance update; The step of extracting features from eyeball data to obtain eyeball view features and lesion data includes: Using the bottom left corner of the obtained image as the initial point, construct a pixel coordinate system and set the corresponding pixel coordinate point in the coordinate system for each pixel. Using the PnP algorithm: ； Complete the 3D to 2D conversion; among which, The coordinates of the point in the pixel coordinate system Let these be the coordinates of the point in the camera coordinate system. Let these be the coordinates of the point in the world coordinate system. For the depth of the point, This is the intrinsic parameter matrix of the camera. For a point in the world coordinate system matrix, For a point in the world coordinate system Translation vector; The continuous frame coordinate changes of infrared markers and iris feature points on the surface of the eyeball are extracted from the eyeball data. Micro-motion trajectory parameters are fitted to obtain the micro-motion trajectory parameters. The infrared markers and iris feature points are adjusted by the micro-motion trajectory parameters to obtain the eyeball vector features, and the lesion pixels and lesion range in the image are located.

2. The real-time navigation method for ophthalmic surgery fusion based on multimodal data according to claim 1, characterized in that, The step of locating and segmenting lesion data to obtain lesion feature data includes: The grayscale and edge features of the lesion area are extracted. An encoder-decoder network structure is adopted. The encoder extracts deep features of the eye image step by step through convolutional and pooling layers to capture the contour and texture information of the intraocular structure. The decoder performs feature map upsampling through deconvolutional layers and combines the shallow and deep features of the encoder to segment the key area containing the lesion. At the same time, by adding an attention mechanism, the feature weights of key structures are strengthened. Combined with the preoperative lesion coordinate benchmark, the three-dimensional coordinates of the lesion are updated in real time.

3. The real-time navigation method for ophthalmic surgery fusion based on multimodal data according to claim 1, characterized in that, The process of combining surgical path data with standard navigation data to achieve path fitting and precise navigation for the surgery includes: Spatial registration is performed between surgical path data and standard navigation data, aligning real-time path coordinates with standard path templates and three-dimensional anatomical models of the eyeball to compensate for path deviations caused by micro-movements of the eyeball, tissue deformation, and instrument operation deviations. Simultaneously, the path deviation value between the real-time path and the standard path is calculated, and the subsequent surgical path is corrected based on the path deviation value to obtain the navigation surgical path.

4. The real-time navigation method for ophthalmic surgery fusion using multimodal data according to claim 3, characterized in that, The spatial registration of surgical path data with standard navigation data includes: The feature points in the surgical path data that correspond to the coordinates of standard feature points in the standard navigation data are marked as real-time feature points. The standard feature points and real-time feature points are combined to obtain a matching feature point group. The similarity of matching feature point groups is calculated using Euclidean distance, according to the Euclidean distance formula: ； Calculate the feature point similarity between matching feature point groups ;in, For real-time feature descriptors Dimensional value, For the standard feature descriptor Dimensional value; Set a matching threshold; determine whether the similarity of feature points in a matching feature point group is greater than the matching threshold; if yes, mark the corresponding matching feature point group as an incorrect matching group; if no, mark the corresponding matching feature point group as a valid matching group. The registration transformation matrix of the effective matching group is calculated using the least squares method. The specific formula is as follows: ； in, For the first The registration transformation matrix of each real-time feature point For the first Real-time feature point coordinates, For the first Coordinates of a standard feature point To effectively match the number of pairs, Represents the real-time feature point set Rotation registration matrix, Represents the real-time feature point set Translation vector; This matrix is ​​then used to complete spatial registration of real-time path data.

5. The real-time navigation method for ophthalmic surgery fusion based on multimodal data according to claim 3, characterized in that, The process of correcting subsequent surgical paths based on path deviation values ​​to obtain a navigation surgical path includes: S1: By formula: ； Calculate the registration residuals between the real-time path and the standard path. ;in, The coordinates of the standard feature points after transformation. For the first Real-time feature point coordinates, To ensure an effective number of matching pairs; S2: Set residual thresholds, including a first residual threshold and a second residual threshold; determine whether the registration residual is greater than the first residual threshold; if yes, re-execute the complete registration process; otherwise, jump to S3; S3: Determine whether the registration residual is greater than the second residual threshold; if yes, trigger local re-registration; if no, determine that the registration is qualified, and complete the path correction by using the least squares method to obtain the navigation surgical path.

Images