An intraoperative endoscopic imaging three-dimensional reconstruction navigation method

Abstract

The present application relates to a kind of intraoperative endoscope imaging three-dimensional reconstruction navigation method, belong to intraoperative imaging reconstruction and navigation technical field.The present application installs multiple electromagnetic positioners in surgical operation space, and establishes nonlinear electromagnetic positioning equation set according to magnetic field model and electromagnetic positioner, the conversion relationship of electromagnetic positioner and surgical endoscope base point is calculated, electromagnetic positioning endoscope pose is obtained;RGBD point cloud image is obtained using the camera of binocular endoscope and is mapped and positioned, and visual endoscope pose is obtained;Adaptive kalman filter is used to electromagnetic positioning and visual positioning result, and then further three-dimensional mapping work is carried out using the fused pose information.Based on the given pose information and mapping result, intraoperative navigation system is carried out.The present application improves the precision and robustness of intraoperative endoscope navigation by multi-sensor fusion of electromagnetic positioning and visual odometry, and can realize real-time, accurate three-dimensional positioning in surgical environment.

Description

Technical Field

[0001] This invention relates to the field of intraoperative imaging reconstruction and navigation technology, specifically to an intraoperative laparoscopic imaging three-dimensional reconstruction navigation method. Background Technology

[0002] Laparoscopy is one of the most commonly used medical imaging instruments in clinical surgery. It enters the body through natural orifices or minimally invasive incisions, playing a crucial role in the examination and surgical treatment of human organs and tissues by observing and imaging the surgical area. Laparoscopy is widely used in gastroscopy, colonoscopy, respiratory disease diagnosis, laparoscopic surgery, and oral examinations. In clinical practice, laparoscopy can be divided into monocular and binocular laparoscopy based on the number of camera lenses. Monocular endoscopes are suitable for routine diagnostic examinations such as gastroscopy and colonoscopy; traditional monocular laparoscopy typically only generates two-dimensional images and lacks depth perception capabilities, limiting its application in three-dimensional tissue reconstruction, surgical navigation, and precise lesion localization. Binocular laparoscopy, on the other hand, acquires depth information directly by acquiring images from two different perspectives, making it more suitable for complex surgical procedures requiring high spatial positioning and operational precision, such as endoscopic submucosal dissection and transoral endoscopic myotomy. This invention primarily focuses on binocular laparoscopy. Binocular laparoscopic 3D navigation integrates real-time intraoperative endoscopic images with spatial positioning technology. It is used to estimate the endoscopic pose as the endoscope moves within the cavity, and simultaneously reconstructs a 3D map of the cavity in real time. This allows the endoscope to accurately reach the target lesion in the narrow cavity, avoiding dangerous structures such as blood vessels and nerves, thereby improving the accuracy, safety, and efficiency of the surgery.

[0003] In recent years, the continuous advancement of deep learning technology has enabled computer vision-based 3D reconstruction techniques to recover depth information from images under binocular endoscopy, effectively improving the accuracy and real-time performance of reconstruction results in surgical environments. Therefore, these technologies have been widely applied in minimally invasive surgery. Algorithms such as endoGSLAM and endoGaussian employ a uniform motion model as their foundation, assuming the endoscope system is in uniform motion, providing pose changes under this uniform motion, and then performing secondary correction based on visual information. However, this motion model inherently does not meet the demands of real-time surgery. Operators cannot maintain constant movement during surgery and frequently need to stop for careful observation; even when movement is required, uniform speed cannot be guaranteed. This makes it impossible to directly replicate traditional SLAM algorithms used in large-scale scenarios, such as autonomous driving and building surveying, in surgical settings. These algorithms exhibit significant jumps and distortion regions when facing complex situations. Visual-only navigation faces several challenges: 1. Poor accuracy: Unlike large-scale scenarios (such as autonomous driving and building surveying), single movements in surgical settings are often only a few centimeters, and the overall distance is generally no more than one meter. In such confined spaces, the observation errors of conventional LiDAR and inertial navigation systems are too large to be directly applied in intraoperative navigation. Using high-precision inertial navigation equipment would incur significant costs. 2. Defects in image information: In the intraoperative environment, image information faces several challenges, including insufficient lighting, surgical instruments and personnel easily obstructing the lens, and insufficient visual feature texture and contrast. These unfavorable factors can lead to errors in image feature matching and reconstruction, reducing the overall accuracy of the algorithm and failing to meet the requirements of real-time intraoperative navigation. 3. Insufficient real-time performance: Due to the numerous shortcomings of visual-only navigation, various methods are employed to improve its effectiveness, such as lengthening observation sequences, increasing computational complexity, using large models for secondary rendering, and expanding observation constraints. However, all these methods lead to a significant increase in computation time. Therefore, pure vision navigation often requires several minutes of comprehensive observation of the surgical area, followed by tens of minutes or even hours of global reconstruction, and then the reconstruction results are back-projected onto the surgical environment for use. This results in a severe lack of real-time performance, thus greatly reducing its application scenarios and effectiveness.

[0004] However, due to the unique imaging conditions of endoscopic scenarios, various factors contribute to blurred or noisy images in endoscopic imaging, making it difficult for single deep learning methods to effectively extract depth information. These factors include: confined and enclosed spatial environments, excessively close proximity of the target object, imaging distortion, and dynamic changes caused by the patient's physiological activities (such as respiration, blood flow, and organ peristalsis).

[0005] Based on this, the present invention provides an intraoperative laparoscopic imaging three-dimensional reconstruction navigation method, which can provide real-time three-dimensional images and positioning feedback in surgical navigation, bringing surgeons a better user experience and more valuable output results. Summary of the Invention

[0006] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, and in view of the above-mentioned shortcomings, the present invention provides an intraoperative laparoscopic imaging three-dimensional reconstruction navigation method.

[0007] In a first aspect, the present invention provides a three-dimensional reconstruction navigation method for intraoperative laparoscopic imaging, comprising: installing multiple external electromagnetic positioning base stations in the surgical operating space, configuring a magnetic source on the laparoscopic probe, obtaining the spatial position of each electromagnetic positioning base station, establishing a nonlinear electromagnetic positioning equation set based on the spatial positions of the electromagnetic positioning base stations and the magnetic source, calculating the conversion relationship between the electromagnetic positioner and the surgical laparoscopic base point, and obtaining the electromagnetic positioning laparoscopic pose. The camera of the binocular endoscope is used to acquire RGBD point cloud images and perform mapping and localization to obtain the visual localization endoscope pose based on image information. The normalized innovation square is obtained by using the observation residuals and innovation covariance matrix of arbitrary modes, and compared with the chi-square distribution quantile to determine whether the current motion of the endoscope is in quasi-static mode, slow motion mode or fast motion mode, and the corresponding motion model and process noise covariance matrix are selected. The fused cavity pose is obtained through adaptive fusion; in the Kalman filtering process, the selected motion model and process noise covariance matrix are used to perform state prediction to obtain the prior state and prior covariance; the Mahalanobis distance between the electromagnetic estimate and the visual estimate is calculated to test statistical consistency; if statistically consistent, Kalman filtering is performed to update, otherwise covariance cross-fusion is performed to update. Three-dimensional images are obtained by fusion of laparoscopic poses for three-dimensional mapping. Intraoperative laparoscopic 3D navigation is performed using the reconstructed 3D images and the 3D spatial coordinates, pitch angle, roll angle, and yaw angle of the endoscope.

[0008] Furthermore, the nonlinear electromagnetic positioning solution equations are as follows: ; ; ; in, Magnetic intensity The components of the magnetic field in three directions are measured for the electromagnetic positioning base station; the magnetic source is located at the base point of the endoscope probe, and the coordinates of the base point are... The direction of the magnetic dipole moment is consistent with the axis of the endoscope probe, and the unit direction vector is... There are i electromagnetic positioning base stations deployed in the surgical operating space. The coordinates of the spatial location of each electromagnetic positioning base station are obtained as follows: Base stations are evenly distributed around the surgical operating space to avoid being collinear or coplanar.

[0009] Furthermore, the weighted nonlinear least squares method is used to solve the problem: ; Constraints: ; in, The vector form of the nonlinear electromagnetic positioning solution equations: ; Defined by the nonlinear electromagnetic positioning solution equations; The weight of the observation at the i-th base station is determined based on the geometric distribution of the base stations and the signal-to-noise ratio. The solution process employs a two-stage search strategy. First, let... Using the reference vector, the equations are simplified to obtain an initial estimate of the position. Then, using the estimated position as the initial value, the Levenberg-Marquardt method is used iteratively to obtain the unit direction vector. The coordinates of the base point are ; Solve for the angle between the unit direction vector and the reference vector, determine the unit rotation axis from the reference vector to the unit direction vector, and obtain the rotation matrix R of the endoscope probe by combining the Rodriguez rotation formula.

[0010] Furthermore, a hierarchical decoupling strategy is adopted to solve the pose of the electromagnetic positioning laparoscope. The process includes: Attitude-assisted initialization is performed based on magnetic field characteristics, where the partial derivative matrix of the magnetic field vector at the electromagnetic positioning base station with respect to spatial coordinates is used. The eigenvector corresponding to the largest eigenvalue approximately points in the direction of the magnetic source, thus obtaining an initial estimate of the unit direction vector; by utilizing the characteristic that the magnetic field amplitude decays with the cube of the distance, the initial distance and position are estimated by the amplitude ratio of at least two electromagnetic positioning base stations; Utilizing the property that magnetic field amplitude is independent of direction, based on the amplitude of all electromagnetic positioning base stations and magnetic field amplitude model To construct an optimization objective function containing only position variables: ; in, To determine the weights set based on the positioning scores of the electromagnetic positioning base stations, at each sampling time, the Jacobian matrix of the observation equations of each electromagnetic positioning base station with respect to the state variables is calculated based on the current position of the endoscope. The comprehensive positioning score of the electromagnetic positioning base station is obtained using the Jacobian matrix: ; It is the reciprocal of the product of the comprehensive positioning score of the electromagnetic positioning base station and the variance of the observation noise; The Levenberg-Marquardt algorithm is used to solve the 3D optimization problem. During the localization process, the solution of the previous frame and the initial position estimate are used as dual initial values. The position in the pose of the electromagnetic positioning cavity mirror can be solved with a small number of iterations. The solved position is used to solve for the direction in a set of nonlinear electromagnetic positioning equations.

[0011] Furthermore, RGBD point cloud images are acquired using the binocular camera built into the endoscope, feature points are extracted using the SuperPoint network, feature point matching is performed using the LightGlue network, and the feature point matching relationship is used to solve the visual positioning of the endoscope pose.

[0012] Furthermore, by utilizing the correspondence between RGB images and aligned depth map points, the feature point pixels of the RGB image are back-projected onto the left eye camera coordinate system to obtain the corresponding 3D points. ; The pose of the endoscope in visual localization is solved using feature points and a visual pose observation model. The visual pose observation model is as follows:

[0013] in The scale factor of the endoscope is calculated through baseline correction; The intrinsic parameters of the camera in the laparoscope are based on the intrinsic parameters of the left camera of the binocular laparoscope; ui and vi represent the pixel coordinates of feature point i, and Xi, Yi, and Zi represent the three-dimensional coordinates of feature point i in the world coordinate system. The pose of the endoscope camera relative to the world coordinate system is used for visual positioning of the endoscope pose. By using the correspondence of feature points within several consecutive frames, the PnP algorithm combined with RANSAC robust estimation is employed to solve the pose of the current endoscope camera relative to the world coordinate system.

[0014] Furthermore, the normalized innovation squared is compared with the chi-square distribution quantile with 6 degrees of freedom: if the normalized innovation squared is less than or equal to The mode is determined to be quasi-static, and a static motion model is adopted. The prior state at time t is directly taken as the posterior state at time t-1. The process noise covariance matrix is... The noise covariance matrix is ​​observed using an electromagnetic positioning system; if the normalized innovation squared is greater than... and less than or equal to The motion pattern is determined to be slow, so a uniform motion model is adopted. The prior state at time t is the posterior state at time t-1 plus the sampling time and the velocity estimation vector at time t-1. Process noise covariance matrix In addition to the process noise covariance matrix of the quasi-static mode, the product of the velocity estimation covariance and the square of the sampling time interval is introduced; if the normalized innovation square is greater than The motion is classified as a fast motion mode, and a uniform acceleration motion model is adopted. The prior state at time t is the posterior state at time t-1 plus the sampling time and the velocity estimation vector at time t-1. Then, add half the acceleration estimation vector and the square of the sampling time interval to the process noise covariance matrix. In addition to the process noise covariance matrix of the slow motion mode, the product of the fourth power of the acceleration error covariance sampling time interval and then divided by four is introduced; and the mode switch is only performed after the new mode has been continuously for K frames, where K is the preset number of lag frames.

[0015] Furthermore, the Kalman filter update includes: Using the prior covariance matrix Calculate the Kalman gain matrix: Where H is the observation matrix, The visual observation noise covariance matrix is... The prior covariance matrix; Using the Kalman gain matrix The posterior state estimate is calculated by comparing the poses of the electromagnetic positioning endoscope and the visual positioning endoscope. The posterior state estimate is the fusion laparoscope pose; Using the Kalman gain matrix and prior covariance matrix Calculate the posterior covariance matrix: I is a 6×6 identity matrix. This is the posterior covariance matrix after fusion.

[0016] Furthermore, the covariance cross-fusion update includes: The inverse of the posterior covariance matrix The inverse matrix of the electromagnetic positioning covariance matrix Visual positioning covariance matrix Weighted combination of the inverse matrices: The weights ω are determined by minimizing the determinant or trace of the fusion covariance matrix, and are solved using a one-dimensional search with a set step size or the Brent method. Posterior state estimation is .

[0017] Secondly, the present invention provides an intraoperative laparoscopic imaging three-dimensional reconstruction navigation device, comprising: at least one processing unit, wherein the processing unit is connected to a storage unit, a binocular endoscope, a magnetic source and an electromagnetic locator via a bus unit, the storage unit stores a computer program, and the processing unit implements the intraoperative laparoscopic imaging three-dimensional reconstruction navigation method by running the computer program stored in the storage unit.

[0018] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art: This invention combines an electromagnetic locator with a laparoscopic system to obtain the real-time pose of the endoscope. This pose complements the pose of the computer vision navigation system, providing complementary information for intraoperative 3D reconstruction and navigation.

[0019] This invention improves positioning accuracy by using a motion pattern adaptive detection mechanism based on normalized innovation squares and combining it with chi-square distribution test switching, enabling the Kalman filter to automatically match various motion states during surgery, such as stillness, slow operation, and rapid switching of field of view.

[0020] The adaptive Kalman filter of this invention adopts a covariance cross-fusion strategy. It uses Mahalanobis distance to detect the statistical consistency between electromagnetic and visual positioning in real time. When there is inconsistency, it automatically switches the covariance cross-fusion strategy to avoid overconfidence and divergence of the filter. This reduces positioning error in degenerate scenarios such as electromagnetic interference or visual occlusion.

[0021] This invention decomposes a 6-dimensional nonlinear problem into two 3-dimensional subproblems through a hierarchical decoupling optimization strategy, and uses magnetic gradient tensor to assist in initial value estimation, thereby reducing the number of iterations and significantly improving convergence reliability.

[0022] This application uses dynamic electromagnetic positioning base station positioning score to weight the contribution of electromagnetic positioning base stations in order to improve positioning accuracy. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a three-dimensional reconstruction navigation method for intraoperative laparoscopic imaging provided in an embodiment of the present invention; Figure 2A flowchart illustrating the electromagnetic positioning laparoscope pose calculation using a hierarchical decoupling strategy provided in this embodiment of the invention; Figure 3 This is a flowchart of obtaining the fused laparoscope pose through adaptive fusion, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the intraoperative laparoscopic imaging three-dimensional reconstruction navigation device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0028] Example 1 like Figure 1 As shown, the present invention provides a three-dimensional reconstruction navigation method for intraoperative laparoscopic imaging, comprising: The S100 is equipped with multiple external electromagnetic positioning base stations installed in the surgical operating space. The endoscope probe is equipped with a magnetic source to obtain the spatial position of each electromagnetic positioning base station.

[0029] The magnetic source is located at the base point of the endoscope probe, and the coordinates of the base point are: The direction of the magnetic dipole moment is consistent with the axis of the endoscope probe, and the unit direction vector is... There are i electromagnetic positioning base stations deployed in the surgical operating space. The spatial coordinates of each electromagnetic positioning base station are obtained as follows: Base stations are evenly distributed around the surgical operating space to avoid being collinear or coplanar.

[0030] Furthermore, based on the spatial positions of the electromagnetic positioning base station and the magnetic source, a set of nonlinear electromagnetic positioning equations is established to calculate the conversion relationship between the electromagnetic locator and the surgical endoscope base point, thereby obtaining the electromagnetic positioning endoscope pose. Specifically, the conversion relationship between the electromagnetic locator and the surgical endoscope base point is obtained by solving the above nonlinear electromagnetic positioning equations using the weighted least squares method, thus obtaining the electromagnetic positioning endoscope pose.

[0031] The specific set of nonlinear electromagnetic positioning equations is as follows: ; ; ; in, Magnetic intensity Magnetic field measured for electromagnetic positioning base station i Components in three directions, Let be the distance to the electromagnetic positioning base station i measured by the magnetic field. The equation has six unknowns: a, b, c, m, n, and p. i must be greater than or equal to 2 to solve the equation. In practical applications, four base stations are selected for positioning.

[0032] In practice, magnetic interference is unavoidable in the surgical operating space. To avoid magnetic interference and improve the accuracy of electromagnetic positioning, no-load calibration is performed before the surgery begins to measure the ambient magnetic field of the surgical operating space; and the known interference magnetic field of the surgical equipment is pre-measured or modeled. After the electromagnetic positioning base station collects the magnetic signal, it compensates for the ambient magnetic field and the interference magnetic field of the equipment to obtain the magnetic field signal of the magnetic source.

[0033] Solving the nonlinear electromagnetic positioning equations yields the coordinates of the endoscope probe. and unit direction vector .

[0034] One solution method: Based on the magnetic intensity and the three directional components of the magnetic field measured by each electromagnetic positioning base station, substitute them into the equation system to solve for the coordinates of the endoscope probe. and unit direction vector .

[0035] Specifically, the system of equations can be written in vector form:

[0036] in, Defined by the nonlinear electromagnetic positioning equations, the weighted nonlinear least squares method is used to solve them: ; Constraints:

[0037] in, This represents the weight of the observation from the i-th base station, determined based on the geometric distribution of the base stations and the signal-to-noise ratio. For example, the solution process employs a two-stage search strategy, first letting... Using the reference vector, the equations are simplified to obtain an initial estimate of the position. Then, using the estimated position as the initial value, the Levenberg-Marquardt method is used iteratively to obtain the unit direction vector. The coordinates of the base point are Reference vector This is a reference direction manually specified in the surgical environment, used as the original direction of the surgical coordinate system. If no reference vector is specified, the unit vector is used. As a reference vector.

[0038] The above solution method solves for six unknowns (a,b,c) and (m,n,p) simultaneously, resulting in a high-dimensional search space and slow iteration convergence.

[0039] Furthermore, a hierarchical decoupling strategy is adopted to solve the pose of the electromagnetic positioning cavity mirror. A magnetic field gradient tensor matrix is ​​constructed using magnetic field measurements from multiple electromagnetic positioning base stations, and eigenvalue decomposition is performed. The eigenvector corresponding to the largest eigenvalue is used as the initial estimate of the magnetic source direction vector. The hierarchical decoupling strategy involves two stages: the first stage utilizes the property that magnetic field amplitude is independent of direction to solve for the spatial position of the magnetic source; the second stage fixes the position and then solves for the direction vector. For example... Figure 2 As shown, the process includes: First, attitude-assisted initialization is performed based on the magnetic field characteristics. Specifically, the partial derivative matrix of the magnetic field vector generated by the magnetic dipole at the i-th electromagnetic positioning base station with respect to spatial coordinates is calculated. , Given a 3×3 symmetric unscented matrix G, by performing eigenvalue decomposition on G, the eigenvector corresponding to the largest eigenvalue approximately points in the direction of the magnetic source, thus obtaining an initial estimate of the unit direction vector. Simultaneously, by utilizing the characteristic that the magnetic field amplitude attenuates with the cube of the distance, the initial distance and position are estimated by the amplitude ratio of at least two electromagnetic positioning base stations. .

[0040] Utilizing the property that the magnitude of the magnetic field is independent of its direction, an equation is constructed to describe the relationship between the angle between the magnetic moment direction and the line connecting the source station, the distance from the source station, and the square of the measured magnetic field magnitude: ; in It is the angle between the direction of the magnetic moment and the line connecting the source station. , The unit vector pointing from the magnetic source to the i-th electromagnetic positioning base station contains direction information; The direction-related terms in the equation are reparameterized as follows: ; Rewrite the equation as a quadratic form of u after reparameterization: ; The magnetic field amplitude model is obtained: ; in, It only depends on A symmetric matrix; given a trial location, the distances R of all base stations relative to the magnetic source. i and Given that the amplitude model, which originally depended on direction, now becomes a family of known functions with u as the variable, at the current trial location, the amplitudes of all measured electromagnetic positioning base stations are used. By working backwards to deduce the best-matching u, we arrive at the following linear / quadratic optimization problem: ; in, To determine the weights set based on the positioning scores of the electromagnetic positioning base stations, at each sampling time, the Jacobian matrix of the observation equations of each electromagnetic positioning base station with respect to the state variables is calculated based on the current position of the endoscope. The comprehensive positioning score of the electromagnetic positioning base station is obtained using the Jacobian matrix: ; This is the reciprocal of the product of the comprehensive positioning score and the observation noise variance of the electromagnetic positioning base station. Weights based on the comprehensive positioning score are used to restrict poorly positioned electromagnetic positioning base stations, thus preventing solution contamination.

[0041] After obtaining u, the optimal model amplitude at the trial positions (a, b, c) is then calculated: ; Although the value is generated by the direction parameter, the direction parameter itself is uniquely determined by the position. Therefore, It is a function that depends only on position. Therefore, utilizing and Construct an optimization objective function containing only position variables: ; The Levenberg-Marquardt algorithm is used to solve the 3D optimization problem. During the localization process, the solution of the previous frame and the initial position estimate are used as dual initial values. The position in the electromagnetic positioning cavity mirror pose can be solved with a small number of iterations, and convergence is usually achieved in 3-5 iterations.

[0042] The solved position is substituted into the nonlinear electromagnetic positioning equations to solve for the direction, which is much more efficient than solving a six-dimensional variable optimization problem.

[0043] Solve for the unit direction vector Then, based on the unit direction vector With reference vector The formula for determining the angle between the two directions is: ; Unit rotation axis from reference vector to direction vector The plane perpendicular to the direction vector and the reference vector is determined by the cross product as follows: ; According to Rodriguez's rotation formula, the rotation about the unit axis is obtained. Rotation angle The rotation matrix is ​​used as the rotation matrix of the current endoscopic probe. : ; Where K is the unit axis of rotation The corresponding antisymmetric matrix is ​​represented as: ; Unit rotation axis The amount.

[0044] The rotation matrix of the current endoscopic probe obtained by electromagnetic positioning is in the form of: .

[0045] The S200 uses a binocular endoscope camera to acquire RGBD point cloud images and perform mapping and localization to obtain the visual positioning endoscope pose based on image information.

[0046] The RGBD point cloud image is acquired using the binocular camera built into the cavity. The RGBD point cloud image contains an RGB image and an aligned depth map, which includes the color and depth of each pixel. The depth map of the RGBD point cloud image is aligned with the left eye camera, avoiding the dependence of triangulation on motion parallax in pure vision methods. It is especially suitable for cavity environments with weak texture and large changes in lighting.

[0047] For RGB images in RGBD point cloud images, the SuperPoint network is used to extract feature points. The SuperPoint network simultaneously detects key points such as corners and spots on the image and calculates their descriptors. It is robust to changes in illumination and viewpoint and achieves sub-pixel accuracy. The number of output feature points is adjustable. Then, the LightGlue network is used for feature point matching. The LightGlue network is a graph neural network based on Transformer. It takes the descriptor sets of two frames of RGBD point cloud images and outputs the matching relationship. It has strong robustness to repeated textures and viewpoint changes and can effectively eliminate false matches.

[0048] By utilizing the correspondence between RGB images and aligned depth map points, the feature point pixels of the RGB image are back-projected onto the coordinate system of the left-eye camera to obtain the corresponding 3D points. .

[0049] The pose of the endoscope in visual localization is solved using feature points and a visual pose observation model. Specifically, the visual pose observation model is as follows:

[0050] in The scale factor of the endoscope is calculated through baseline correction; is the camera intrinsic parameter of the endoscope, with the intrinsic parameter of the left camera of the binocular endoscope as the reference; ui and vi represent the pixel coordinates of feature point i in the pixel coordinate system, and Xi, Yi, and Zi represent the three-dimensional coordinates of feature point i in the world coordinate system. The pose of the endoscope camera relative to the world coordinate system is used for visual positioning of the endoscope pose.

[0051] By using the correspondence of feature points within several consecutive frames, the PnP algorithm combined with RANSAC robust estimation is employed to solve the pose of the current endoscope camera relative to the world coordinate system.

[0052] in, Let be the translation vector for visually locating the laparoscope pose. The rotation matrix is ​​used for visual positioning of the endoscope pose.

[0053] The poses of the electromagnetic positioning cavity mirror and the visual positioning cavity mirror are unified into the Euler pose.

[0054] The electromagnetic positioning cavity mirror pose observations at time t are as follows: , The observed pitch angle of the electromagnetic positioning cavity mirror at time t is: ; The observed roll angle of the electromagnetic positioning cavity mirror at time t is: ; The observed yaw angle of the electromagnetic positioning cavity mirror at time t is: .

[0055] The base position of the electromagnetic positioning cavity mirror pose at time t .

[0056] Similarly, the visual positioning cavity mirror pose observations determined from the RGBD point cloud image are as follows:

[0057] Among them, the observed pitch angle of the visual positioning endoscope pose at time t is: ; The observed roll angle of the visual positioning laparoscope pose at time t is: ; The observed yaw angle of the visual positioning endoscope pose at time t is: .

[0058] The base position of the visual positioning laparoscope pose at time t .

[0059] The pointed symbol above the letters indicates observation, the 'm' at the top left of the letters indicates electromagnetic, the 'v' indicates visual, and the 'r' indicates visual. 11 r 21 r 31 r 32 r 33 This represents the elements of the corresponding row and column in the rotation matrix obtained by electromagnetic or visual means.

[0060] S300 uses the observation residuals and innovation covariance matrix of any mode to obtain the normalized innovation square, and compares it with the chi-square distribution quantile to determine whether the current motion of the endoscope is quasi-static mode, slow motion mode or fast motion mode, and selects the corresponding motion model and noise covariance matrix.

[0061] Specifically, taking visual positioning as an example, the normalized squared innovation value of the current frame t is calculated: ; in: For visual positioning observation residual vector, For visual positioning of the endoscope. This is the prior state estimate for frame t, predicted from the state of the previous frame, where H is a 6×6 observation matrix. The new information covariance matrix, Let be the prior state covariance matrix. The observation noise covariance matrix of the vision system.

[0062] The normalized squared innovation follows a chi-square distribution with 6 degrees of freedom under the Kalman filter consistency assumption. Set a chi-square quantile threshold with 6 degrees of freedom and a confidence level of 0.95. And the chi-square quantile threshold with 6 degrees of freedom and a confidence level of 0.99. The judgment rule is as follows: If the normalized square of the new information is less than or equal to The mode is determined to be quasi-static, and a static motion model is adopted. The prior state at time t is directly taken as the posterior state at time t-1. The process noise covariance matrix is... The noise covariance matrix is ​​observed using an electromagnetic positioning system; if the normalized innovation squared is greater than... and less than or equal to The motion pattern is determined to be slow, so a uniform motion model is adopted. The prior state at time t is the posterior state at time t-1 plus the sampling time and the velocity estimation vector at time t-1. Process noise covariance matrix In addition to the process noise covariance matrix of the quasi-static mode, the product of the velocity estimation covariance and the square of the sampling time interval is introduced; if the normalized innovation square is greater than The motion is classified as a fast motion mode, and a uniform acceleration motion model is adopted. The prior state at time t is the posterior state at time t-1 plus the sampling time and the velocity estimation vector at time t-1. Then, add half the acceleration estimation vector and the square of the sampling time interval to the process noise covariance matrix. In addition to the process noise covariance matrix of the slow motion mode, the product of the fourth power of the acceleration error covariance sampling time interval and then divided by four is introduced; and the mode switch is only performed after the new mode has been continuously for K frames, where K is the preset number of lag frames.

[0063] This enables the subsequent adaptive Kalman filter to automatically match various motion states and process noises during surgery, such as stillness, slow operation, and rapid switching of field of view, with tracking accuracy superior to schemes with fixed motion models and noise.

[0064] S400 obtains the fused cavity mirror pose through adaptive fusion; it calculates the Mahalanobis distance between the electromagnetic and visual estimates to test statistical consistency; if statistically consistent, it performs standard Kalman filtering; otherwise, it performs covariance cross-fusion update. During the Kalman filtering process, it uses the selected motion model to perform state prediction, obtaining the prior state and prior covariance.

[0065] S401, Calculate the prior laparoscopic pose state estimate based on the selected motion mode. Using the selected process noise covariance matrix The posterior covariance matrix of the previous time step Calculate the prior covariance matrix , where F is the state transition matrix (6×6 matrix) for electromagnetic positioning. Let be the posterior covariance matrix of the previous time step. The selected process noise covariance matrix.

[0066] S402, using the covariance matrix estimated by electromagnetic positioning The covariance matrix of visual localization estimation Electromagnetic positioning cavity mirror pose Visual positioning of the endoscopic position Calculate the pose of the electromagnetic positioning laparoscope Visual positioning of the endoscopic position The Mahalanobis distance between them is used to perform a statistical consistency test, where the Mahalanobis distance is: .

[0067] If the Mahalanobis distance is less than or equal to the chi-square quantile with 6 degrees of freedom and a confidence level of 0.95, the pose of the electromagnetic positioning endoscope and the pose of the visual positioning endoscope are statistically consistent, and S403A is executed; otherwise, the two estimated statistics are inconsistent, such as the electromagnetic system being affected by metal instruments causing magnetic field distortion, or the visual system being obstructed by smoke / bleed causing feature point loss, and S403B is executed.

[0068] S403A, perform Kalman filter update: Using the prior covariance matrix Calculate the Kalman gain matrix: Where H is the observation matrix, The visual observation noise covariance matrix is... The prior covariance matrix; Using the Kalman gain matrix The posterior state estimate is calculated by comparing the poses of the electromagnetic positioning endoscope and the visual positioning endoscope. The posterior state estimate is the fusion laparoscope pose; Using the Kalman gain matrix and prior covariance matrix Calculate the posterior covariance matrix: I is a 6×6 identity matrix with 1s on the diagonal and 0s on the rest. This is the posterior covariance matrix after fusion.

[0069] Update on S403B covariance cross-fusion: The inverse of the posterior covariance matrix The inverse matrix of the electromagnetic positioning covariance matrix Visual positioning covariance matrix Weighted combination of the inverse matrices: The weights ω are determined by minimizing the determinant or trace of the fusion covariance matrix, and are solved using a one-dimensional search method with a set step size. Covariance crossover ensures that the fusion covariance is not less than the covariance of any subsystem, avoiding the overconfidence problem of standard Kalman filtering when information is inconsistent.

[0070] Using the inverse of the posterior covariance matrix We combine weighted visual and electromagnetic localization to calculate the posterior state estimate: .

[0071] S404, using an exponentially weighted moving average to update the estimated value of the visual positioning observation noise covariance matrix online: , in: α and t-1 are the online estimates of the visual positioning observation noise covariance matrix at time t and t-1, respectively; α∈(0,1) is the forgetting factor (dimensionless, typically α=0.9). The closer the forgetting factor α is to 1, the more historical information is retained, the smoother the estimate is, but the slower the response to new changes. The closer α is to 0, the more sensitive the estimate is, but the greater the noise. This is the current observation residual vector.

[0072] Iterate until the set number of iterations is reached or the error meets the set threshold.

[0073] The S500 uses fusion laparoscopic pose to create a 3D map and obtain a 3D image; the reconstructed 3D image and the 3D spatial coordinates, pitch angle, roll angle and yaw angle of the laparoscopy are used for intraoperative laparoscopic 3D navigation.

[0074] Use fusion endoscopic posture Obtain the corrected pose of the endoscope camera relative to the world coordinate system. Using the corrected pose of the endoscopic camera relative to the world coordinate system Perform visual 3D reconstruction: .

[0075] The S600 uses the reconstructed 3D image and the 3D spatial coordinates, pitch angle, roll angle, and yaw angle of the endoscope for intraoperative 3D endoscopic navigation.

[0076] Example 2 like Figure 4 As shown, this embodiment of the invention provides an intraoperative laparoscopic imaging three-dimensional reconstruction navigation device, comprising: at least one processing unit, the processing unit being connected to a storage unit, a binocular endoscope, a magnetic source, and an electromagnetic locator via a bus unit; the storage unit, as a computer-readable storage medium, being used to store software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to the intraoperative laparoscopic imaging three-dimensional reconstruction navigation method in this embodiment of the invention. The processing unit implements the aforementioned intraoperative laparoscopic imaging three-dimensional reconstruction navigation method by running the software program, computer-executable program, and module stored in the storage unit.

[0077] Of course, the computer program stored in the storage unit of the intraoperative laparoscopic imaging three-dimensional reconstruction navigation device provided in the embodiments of the present invention is not limited to the method operation described above, but can also execute related operations in the intraoperative laparoscopic imaging three-dimensional reconstruction navigation method provided in any embodiment of the present invention.

[0078] Example 3 This invention provides a computer-readable storage medium storing a computer program that, when executed, implements the intraoperative laparoscopic imaging three-dimensional reconstruction navigation method.

[0079] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for intraoperative laparoscopic imaging three-dimensional reconstruction navigation, characterized in that, include: Multiple external electromagnetic positioning base stations are installed in the surgical operating space. The endoscope probe is equipped with a magnetic source to obtain the spatial position of each electromagnetic positioning base station. Based on the spatial positions of the electromagnetic positioning base stations and the magnetic source, a set of nonlinear electromagnetic positioning equations is established to calculate the conversion relationship between the electromagnetic positioner and the surgical endoscope base point, thereby obtaining the electromagnetic positioning endoscope pose. The camera of the binocular endoscope is used to acquire RGBD point cloud images and perform mapping and localization to obtain the visual localization endoscope pose based on image information. The normalized innovation square is obtained by using the observation residuals and innovation covariance matrix of arbitrary modes, and compared with the chi-square distribution quantile to determine whether the current motion of the endoscope is quasi-static mode, slow motion mode or fast motion mode, and the corresponding motion model and noise covariance matrix are selected. The fused cavity mirror pose is obtained through adaptive fusion. In the Kalman filtering process, the selected motion model and noise covariance matrix are used to perform state prediction to obtain the prior state and prior covariance. The Mahalanobis distance between the electromagnetic estimate and the visual estimate is calculated to test statistical consistency. If the statistics are consistent, Kalman filtering is performed to update; otherwise, covariance cross-fusion is performed to update. Three-dimensional images are obtained by fusion of laparoscopic poses for three-dimensional mapping. Intraoperative laparoscopic 3D navigation is performed using the reconstructed 3D images and the 3D spatial coordinates, pitch angle, roll angle, and yaw angle of the endoscope.

2. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 1, characterized in that, The nonlinear electromagnetic positioning solution equation set: ； ； ； in, Magnetic intensity The components of the magnetic field in three directions are measured for the electromagnetic positioning base station; the magnetic source is located at the base point of the endoscope probe, and the coordinates of the base point are... The direction of the magnetic dipole moment is consistent with the axis of the endoscope probe, and the unit direction vector is... There are i electromagnetic positioning base stations deployed in the surgical operating space. The coordinates of the spatial location of each electromagnetic positioning base station are obtained as follows: Base stations are evenly distributed around the surgical operating space to avoid being collinear or coplanar.

3. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 2, characterized in that, Solve using the weighted nonlinear least squares method: ； Constraints: ; in, The vector form of the nonlinear electromagnetic positioning solution equations: ; Defined by the nonlinear electromagnetic positioning solution equations; The weight of the observation at the i-th base station is determined based on the geometric distribution of the base stations and the signal-to-noise ratio. The solution process employs a two-stage search strategy. First, let... Using the reference vector, the equations are simplified to obtain an initial estimate of the position. Then, using the estimated position as the initial value, the Levenberg-Marquardt method is used iteratively to obtain the unit direction reference vector. The coordinates of the base point are ; Solve for the angle between the unit direction vector and the reference vector to determine the unit rotation axis from the reference vector to the unit direction vector, and obtain the rotation matrix R of the endoscope probe by combining the Rodriguez rotation formula.

4. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 3, characterized in that, A hierarchical decoupling strategy is employed to solve the pose of the electromagnetic positioning laparoscope. The process includes: Attitude-assisted initialization is performed based on magnetic field characteristics, where the partial derivative matrix of the magnetic field vector at the electromagnetic positioning base station with respect to spatial coordinates is used. The eigenvector corresponding to the largest eigenvalue approximately points in the direction of the magnetic source, thus obtaining an initial estimate of the unit direction vector; by utilizing the characteristic that the magnetic field amplitude decays with the cube of the distance, the initial distance and position are estimated by the amplitude ratio of at least two electromagnetic positioning base stations; Utilizing the property that magnetic field amplitude is independent of direction, based on the amplitude of all electromagnetic positioning base stations and magnetic field amplitude model To construct an optimization objective function containing only position variables: ； in, To determine the weights set based on the positioning scores of the electromagnetic positioning base stations, at each sampling time, the Jacobian matrix of the observation equations of each electromagnetic positioning base station with respect to the state variables is calculated based on the current position of the endoscope. The comprehensive positioning score of the electromagnetic positioning base station is obtained using the Jacobian matrix: ； It is the reciprocal of the product of the comprehensive positioning score of the electromagnetic positioning base station and the variance of the observation noise; The Levenberg-Marquardt algorithm is used to solve the 3D optimization problem. During the localization process, the solution of the previous frame and the initial position estimate are used as dual initial values. The position in the pose of the electromagnetic positioning cavity mirror can be solved with a small number of iterations. The solved position is used to solve for the direction in a set of nonlinear electromagnetic positioning equations.

5. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 1, characterized in that, RGBD point cloud images are acquired using the binocular camera built into the endoscope. Feature points are extracted using the SuperPoint network, and feature point matching is performed using the LightGlue network. The feature point matching relationship is then used to solve the visual positioning of the endoscope pose.

6. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 5, characterized in that, By utilizing the correspondence between RGB images and aligned depth map points, feature point pixels of the RGB image are back-projected onto the left-eye camera coordinate system to obtain the corresponding 3D points. ; The pose of the endoscope in visual localization is solved using feature points and a visual pose observation model. The visual pose observation model is as follows: in The scale factor of the endoscope is calculated through baseline correction; The camera intrinsic reference of the endoscope is based on the intrinsic reference of the left camera of the binocular endoscope; u i v i X represents the pixel coordinates of feature point i. i Y i Z i Represents the three-dimensional coordinates of feature point i in the world coordinate system. The pose of the endoscope camera relative to the world coordinate system is used for visual positioning of the endoscope pose. By using the correspondence of feature points within several consecutive frames, the PnP algorithm combined with RANSAC robust estimation is employed to solve the pose of the current endoscope camera relative to the world coordinate system.

7. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 1, characterized in that, Compare the normalized innovation square with the quantile of the chi-square distribution with 6 degrees of freedom: if the normalized innovation square is less than or equal to The mode is determined to be quasi-static, and a static motion model is adopted. The prior state at time t is directly taken as the posterior state at time t-1. The process noise covariance matrix is... The noise covariance matrix is ​​observed using an electromagnetic positioning system; if the normalized squared innovation is greater than... and less than or equal to The motion pattern is determined to be slow, so a uniform motion model is adopted. The prior state at time t is the posterior state at time t-1 plus the sampling time and the velocity estimation vector at time t-1. Process noise covariance matrix In addition to the process noise covariance matrix of the quasi-static mode, the product of the velocity estimation covariance and the square of the sampling time interval is introduced; if the normalized innovation square is greater than The motion is classified as a fast motion mode, and a uniform acceleration motion model is adopted. The prior state at time t is the posterior state at time t-1 plus the sampling time and the velocity estimation vector at time t-1. Then, add half the acceleration estimation vector and the square of the sampling time interval to the process noise covariance matrix. In addition to the process noise covariance matrix of the slow motion mode, the product of the fourth power of the acceleration error covariance sampling time interval and then divided by four is introduced; and the mode switch is only performed after the new mode has been continuously for K frames, where K is the preset number of lag frames.

8. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 1, characterized in that, Kalman filter updates include: Using the prior covariance matrix Calculate the Kalman gain matrix: Where H is the observation matrix, The visual observation noise covariance matrix; Using the Kalman gain matrix The posterior state estimate is calculated by comparing the poses of the electromagnetic positioning endoscope and the visual positioning endoscope. The posterior state estimate is the fusion laparoscope pose; Using the Kalman gain matrix and prior covariance matrix Calculate the posterior covariance matrix: I is a 6×6 identity matrix with 1s on the diagonal and 0s on the rest. This is the posterior covariance matrix after fusion.

9. The intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to claim 1, characterized in that, Covariance cross-fusion update includes: The inverse of the posterior covariance matrix The inverse matrix of the electromagnetic positioning covariance matrix Visual positioning covariance matrix Weighted combination of the inverse matrices: The weights ω are determined by minimizing the determinant or trace of the fusion covariance matrix, and are solved using a one-dimensional search with a set step size or the Brent method. Posterior state estimation is .

10. A three-dimensional reconstruction navigation device for intraoperative laparoscopic imaging, comprising: At least one processing unit is provided, wherein the processing unit is connected to a storage unit, a binocular endoscope, a magnetic source, and an electromagnetic positioner via a bus unit, wherein the storage unit stores a computer program, and the processing unit implements the intraoperative laparoscopic imaging three-dimensional reconstruction navigation method according to any one of claims 1-9 by running the computer program stored in the storage unit.

Images