An endoscopic surgery navigation method and device based on video image augmented reality

Abstract

This invention relates to an endoscopic surgical navigation method and device based on video image augmented reality. The method includes: segmenting the key surgical areas of interest and selecting anatomical landmarks from preoperative CT images; installing passive tracking tools on the patient's surgical site and the endoscope; registering the real patient with a 3D model using calibrated probes and anatomical landmarks; calibrating the endoscope parameters under the navigation system; determining the transformation matrix of the passive tracking tool coordinate system relative to the endoscope coordinate system; projecting and rendering the 3D model of the preoperatively segmented key surgical areas of interest using a pinhole model simulation and processing it with distortion parameters to obtain a virtual image; and fusing the virtual image with images acquired in real time by the endoscope and displaying it. Compared with existing technologies, this invention fuses virtual images with real endoscopic images and presents them on the same screen, reducing the surgeon's intraoperative judgment time and lowering the difficulty of hand-eye coordination.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of augmented reality technology, and in particular to an endoscopic surgery navigation method and device based on video image augmented reality. BACKGROUND

[0002] With the development of computer technology and the continuous progress of navigation technology, computer-aided surgery navigation technology is increasingly valued in various surgical operations. Under the trend of surgical precision, doctors need to rely on surgical navigation to assist in performing fine operations. However, in some surgical scenarios, the target area of the surgery is often blocked. For example, under the view of various endoscopes such as laparoscopes and sinus scopes, the doctor cannot directly observe the lesion site, or there are important nerves and blood vessels near the lesion site, and the surgery needs to avoid damaging these tissues. In order to obtain these positions, it is necessary to combine navigation software and endoscopic image information.

[0003] At present, the surgical navigation method in endoscopic surgery mainly is to register the preoperative CT of the patient with the real patient, so as to mark the position of the surgical instrument in the CT space. However, most of the schemes cannot fuse the endoscopic image and the CT image information, and the doctor needs to judge and operate comprehensively the two kinds of information, which tests the doctor's hand-eye coordination. At the same time, during the surgery, the instrument needs to be used to open the channel to the surgical lesion, which destroys the original cavity structure, so that the preoperative CT image cannot accurately correspond to the patient's condition, and the doctor needs to make a judgment according to experience.

[0004] In summary, the existing endoscopic surgery navigation method has the defects of poor precision, incomplete information fusion, testing the doctor's hand-eye coordination, and mismatching the navigation information with the actual image during the surgery. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide an endoscopic surgery navigation method and device based on video image augmented reality. The augmented reality method fuses the intraoperative endoscopic information and the navigation CT information to assist the doctor in surgery.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] An endoscopic surgery navigation method based on video image augmented reality, comprising the following steps:

[0008] S1, obtaining the preoperative CT image of the patient, segmenting the key parts of the surgery, generating the corresponding three-dimensional model, and selecting the anatomical landmark points in the preoperative CT image;

[0009] S2, installing a passive tracking tool on the patient's surgical site and the endoscope, respectively, setting up a navigation system, and the conversion matrix relationship chain under the navigation system is as follows:

[0010]

[0011] wherein I is an identity matrix, is a transformation matrix of the endoscope coordinate system relative to the endoscope passive tracking tool coordinate system, is a transformation matrix of the endoscope passive tracking tool coordinate system relative to the navigation system world coordinate system, is a transformation matrix of the navigation system world coordinate system relative to the patient surgical site passive tracking tool coordinate system, is a transformation matrix of the patient surgical site passive tracking tool coordinate system relative to the endoscope coordinate system;

[0012] S3, under the navigation system, using the probe which has been calibrated under the navigation system, using the selected anatomical landmark points to register the real patient and the three-dimensional model, obtaining the registration matrix T R ;

[0013] S4, calibrating the parameters of the endoscope under the navigation system, including camera intrinsic parameters, camera extrinsic parameters and distortion parameters, the camera extrinsic parameters being a transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system

[0014] S5, according to the transformation matrix relationship chain under the navigation system, determining the transformation matrix of the patient surgical site passive tracking tool coordinate system relative to the endoscope coordinate system, using pinhole model simulation imaging to project and render the three-dimensional model of the preoperative segmented key part of the surgical focus, using the distortion parameters of the endoscope calibration to process the rendering result, obtaining a virtual image, and fusing and displaying the virtual image with the image collected by the endoscope in real time.

[0015] Further, when registering the CT space and the patient space using the selected anatomical landmark points, the registration target is to minimize the distance between the two point clouds by continuously iterating the rotation and translation matrix, and finally obtaining the registration matrix T R .

[0016] Further, the calibration of the camera extrinsic parameters is specifically:

[0017] Setting a marker board, determining the relative position of the marker board and the navigation system;

[0018] Under different motion states of the endoscope, the navigation system tracks the motion of the endoscope passive tracking tool, and determines the relative attitude between the navigation system and the endoscope passive tracking tool;

[0019] Using the endoscope to acquire the image of the marker board when the endoscope moves;

[0020] The relative attitude between the signboard and the endoscope is calculated based on the image of the signboard acquired by the endoscope. The relative attitude between the navigation system and the endoscope is calculated based on the relative position of the signboard and the navigation system. The camera extrinsic parameters are then calculated based on the relative attitudes between the navigation system and the passive tracking tool of the endoscope, and between the navigation system and the endoscope.

[0021] Furthermore, the pinhole imaging model is as follows:

[0022]

[0023] Where Z represents the scaling factor, u and v represent the pixel coordinates in the image coordinate system, and d x d y The dimensions of the photosensitive element in the x and y directions are represented by u0 and v0, respectively, which represent the offset between the actual imaging center and the theoretical center. f represents the focal length. This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. T is the transformation matrix between the navigation system's world coordinate system and the passive tracking tool's coordinate system at the patient's surgical site. R Let X represent the registration matrix. w Y w Z w These are the three-dimensional coordinates of a point in space.

[0024] Furthermore, the process for determining the real-time position of the passive tracking tool of the endoscope under the navigation system is as follows:

[0025] The navigation system acquires the pose of the passive tracking tool of the endoscope in real time, the endoscope acquires surgical images in real time, and the motion of the endoscope is calculated by solid geometry based on the surgical images. The fusion result of the motion of the endoscope and the pose of the passive tracking tool of the endoscope is recorded as the measurement value of the system state.

[0026] The motion process of the passive endoscopic tracking tool recorded by the navigation system is obtained, and the motion parameters of the passive endoscopic tracking tool, including velocity and acceleration, are calculated based on the motion process of the passive endoscopic tracking tool.

[0027] Using the pose and motion parameters of the passive endoscope tracking tool at the previous moment, the position of the passive endoscope tracking tool at this moment is calculated, and this position is recorded as the predicted value of the system state.

[0028] The real-time position of the passive endoscopic tracking tool is determined based on the measured and predicted values.

[0029] An endoscopic surgical navigation device based on video image augmented reality includes:

[0030] The preoperative CT module is used to acquire the patient's preoperative CT images, segment out the key areas of concern in the surgery, generate corresponding three-dimensional models, and select anatomical landmarks in the preoperative CT images.

[0031] A passive tracking tool for endoscopes, mounted on the endoscope;

[0032] A passive tracking device for the patient's surgical site is installed at the patient's surgical site.

[0033] The navigation system is used for navigating and locating passive tracking tools for endoscopes and passive tracking tools for patient surgical sites. The transformation matrix relationship chain under the navigation system is as follows:

[0034]

[0035] Where I is the identity matrix, This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. This is the transformation matrix between the navigation system's world coordinate system and the coordinate system of the passive tracking tool at the patient's surgical site. The transformation matrix for the passive tracking tool coordinate system relative to the endoscope coordinate system at the patient's surgical site;

[0036] The registration module, under the navigation system, uses probes already calibrated within the navigation system and selected anatomical landmarks to register the real patient with the 3D model, obtaining the registration matrix T. R ;

[0037] The calibration module is used to calibrate the parameters of the endoscope under the navigation system, including camera intrinsic parameters, camera extrinsic parameters, and distortion parameters. The camera extrinsic parameters are the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system.

[0038] The display imaging module is used to determine the transformation matrix of the passive tracking tool coordinate system relative to the endoscope coordinate system based on the transformation matrix relationship chain under the navigation system. It uses a pinhole model to simulate imaging and project and render the 3D model of the key surgical area segmented before surgery. The rendering result is processed using the distortion parameters calibrated by the endoscope to obtain a virtual image. The virtual image is then fused with the image acquired in real time by the endoscope and displayed.

[0039] Furthermore, when registering the CT space and patient space using selected anatomical landmarks, the registration objective is to minimize the distance between the two point clouds by iteratively calculating the rotation and translation matrices, ultimately obtaining the registration matrix T. R .

[0040] Furthermore, the calibration of camera extrinsic parameters is specifically as follows:

[0041] Set up a signboard and determine the relative position of the signboard to the navigation system;

[0042] Under different motion states of the endoscope, the navigation system tracks the movement of the passive tracking tool of the endoscope and determines the relative attitude between the navigation system and the passive tracking tool of the endoscope.

[0043] The endoscope is used to acquire images of the marker plate during its movement.

[0044] The relative attitude between the signboard and the endoscope is calculated based on the image of the signboard acquired by the endoscope. The relative attitude between the navigation system and the endoscope is calculated based on the relative position of the signboard and the navigation system. The camera extrinsic parameters are then calculated based on the relative attitudes between the navigation system and the passive tracking tool of the endoscope, and between the navigation system and the endoscope.

[0045] Furthermore, the pinhole imaging model is as follows:

[0046]

[0047] Where Z represents the scaling factor, u and v represent the pixel coordinates in the image coordinate system, and d x d y The dimensions of the photosensitive element in the x and y directions are represented by u0 and v0, respectively, which represent the offset between the actual imaging center and the theoretical center. f represents the focal length. This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. T is the transformation matrix between the navigation system's world coordinate system and the passive tracking tool's coordinate system at the patient's surgical site. R Let X represent the registration matrix. w Y w Z w These are the three-dimensional coordinates of a point in space.

[0048] Furthermore, the process for determining the real-time position of the passive tracking tool of the endoscope under the navigation system is as follows:

[0049] The navigation system acquires the pose of the passive tracking tool of the endoscope in real time, the endoscope acquires surgical images in real time, and the motion of the endoscope is calculated by solid geometry based on the surgical images. The fusion result of the motion of the endoscope and the pose of the passive tracking tool of the endoscope is recorded as the measurement value of the system state.

[0050] The motion process of the passive endoscopic tracking tool recorded by the navigation system is obtained, and the motion parameters of the passive endoscopic tracking tool, including velocity and acceleration, are calculated based on the motion process of the passive endoscopic tracking tool.

[0051] Using the pose and motion parameters of the passive endoscope tracking tool at the previous moment, the position of the passive endoscope tracking tool at this moment is calculated, and this position is recorded as the predicted value of the system state.

[0052] The real-time position of the passive endoscopic tracking tool is determined based on the measured and predicted values.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) Install a passive tracking tool on the endoscope, record the position and attitude of the endoscope under the navigation system, use the navigation system to calibrate the internal and external parameters of the endoscope, and calculate the transformation matrix from the three-dimensional model space to the endoscope image space according to the transformation matrix relationship chain under the navigation system. This solves the transformation relationship between the endoscope image space and the three-dimensional model space and reduces the amount of computation.

[0055] (2) Virtual images are obtained by rendering pinhole molding models, and the virtual images are fused with the real images of the endoscope and presented on the same screen, which reduces the doctor's judgment time during the operation, reduces the difficulty of hand-eye coordination, and reduces the operation time.

[0056] (3) It can intuitively and accurately display important tissues and organs that are covered by opaque tissues, which helps doctors make judgments and avoids damage to important tissues. It solves the problem of the impact of the inconsistency between the real-time images during the operation and the preoperative CT images caused by the doctor's necessary operation damaging the cavity or tissue structure.

[0057] (4) The use of filtering methods to eliminate the positioning error of the navigation system tracking the passive tracking tool improves the positioning accuracy, thereby improving the accuracy of calibration and registration. Attached Figure Description

[0058] Figure 1 A flowchart of an endoscopic surgical navigation method based on video image augmented reality;

[0059] Figure 2 A schematic diagram illustrating the implementation scenario of an endoscopic surgical navigation method based on video image augmented reality;

[0060] Attached reference numerals: 1. Navigation system; 2. Endoscope; 3. Display imaging module; 4. Patient surgical site; 5. Passive tracking tool. Detailed Implementation

[0061] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, providing detailed implementation methods and specific operating procedures. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them, and the scope of protection of the present invention is not limited to the following embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0062] As used herein, "an embodiment" or "embodiment" refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0063] This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment), or the execution order of steps without timing constraints can be adjusted.

[0064] Example 1:

[0065] This application proposes a video image augmented reality-based endoscopic surgical navigation method. Taking endoscopic ophthalmology surgery as an example, the surgical site is the nasal cavity of the patient's head. Figure 1 and Figure 2 As shown, it includes the following steps:

[0066] S1. Before the operation, take CT images of the patient to obtain the patient's preoperative CT images, segment the key areas of concern for the operation, namely the patient's optic nerve canal segment (and surrounding important blood vessels), generate the corresponding three-dimensional model, and select anatomical landmarks in the preoperative CT images; In this application, the lesion area is directly manually segmented during preoperative preparation, making preoperative preparation simpler.

[0067] S2. Install passive tracking tools at the patient's surgical site and on the endoscope, respectively, and set up a navigation system. The transformation matrix relationship chain under the navigation system is as follows:

[0068]

[0069] Where I is the identity matrix, This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. This is the transformation matrix between the world coordinate system of the navigation system and the coordinate system of the passive head tracking tool for the patient. The transformation matrix is ​​the coordinate system of the passive head tracking tool relative to the coordinate system of the endoscope.

[0070] S3. Under the navigation system, using probes already calibrated within the system, the selected anatomical landmarks are used to register the real patient with the 3D model. The registration objective is to minimize the distance between the two point clouds by iteratively calculating the rotation and translation matrices, ultimately obtaining the registration matrix T. R ;

[0071] S4. Calibrate the endoscope parameters under the navigation system, including camera intrinsic parameters, camera extrinsic parameters, and distortion parameters. The camera extrinsic parameters are the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system.

[0072] The calibration of camera extrinsic parameters is specifically as follows:

[0073] 1. Set up signboards and determine their relative positions to the navigation system;

[0074] 2. Under different motion states of the endoscope, the navigation system tracks the movement of the passive tracking tool of the endoscope and determines the relative attitude between the navigation system and the passive tracking tool of the endoscope;

[0075] 3. Acquire images of the marker plate using the endoscope during its movement;

[0076] 4. Calculate the relative attitude between the signboard and the endoscope based on the signboard image acquired by the endoscope. Calculate the relative attitude between the navigation system and the endoscope based on the relative position of the signboard and the navigation system. Calculate the camera extrinsic parameters based on the relative attitudes between the navigation system and the passive tracking tool of the endoscope, and between the navigation system and the endoscope.

[0077] In addition, camera intrinsic parameters and distortion parameters can be provided by the manufacturer or determined by self-calibration. The marker board images collected during the external parameter calibration process can be used to calculate and calibrate camera intrinsic parameters and distortion parameters.

[0078] S5. Based on the transformation matrix relationship chain under the navigation system, determine the transformation matrix between the passive tracking tool coordinate system and the endoscope coordinate system at the patient's surgical site. in, For the calibrated camera extrinsic parameters, It can be determined directly based on the navigation system tracking the movement of the passive tracking tool of the endoscope. It can be determined directly based on the movement of the head-tracking passive tracking tool tracked by the navigation system. This transformation relationship can be obtained.

[0079] Based on the registration results in step S3 and the calibration results in step S4 This can be converted to a transformation matrix from CT space (3D model) to the endoscopic coordinate system. Using a pinhole model to simulate imaging, the preoperatively segmented intravascular optic nerve model is projected and rendered. The pinhole imaging model is as follows:

[0080]

[0081] Where Z represents the scaling factor, u and v represent the pixel coordinates in the image coordinate system, and d x d y The dimensions of the photosensitive element in the x and y directions are represented by u0 and v0, respectively, which represent the offset between the actual imaging center and the theoretical center. f represents the focal length. This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. T is the transformation matrix between the navigation system's world coordinate system and the passive tracking tool's coordinate system at the patient's surgical site. R Let X represent the registration matrix. w Y w Z w These are the three-dimensional coordinates of a point in space.

[0082] The rendering results are processed using distortion parameters calibrated by the endoscope to obtain a virtual image. The virtual image is then fused with the image acquired in real time by the endoscope and displayed to complete the augmented reality display of the optic nerve on the endoscopic image.

[0083] During the surgery, based on the transformation matrix, a virtual image of the surgical site is rendered in real time using a pinhole model to simulate imaging, and then fused with the real endoscopic image to achieve augmented reality display. This application addresses the registration problem between the endoscopic image and the 3D model to be projected by fixing a passive tracking tool on the endoscope. By recording the position and orientation of the endoscope under the navigation system, the navigation system calibrates the endoscope's internal and external parameters, calculating the transformation matrix from the 3D model space to the endoscopic image space, thus avoiding the computational burden of point cloud registration.

[0084] The accuracy of the data depends on the navigation system's positioning of the passive tracking tool. This application employs a filtering method to control positioning errors, thereby reducing them. The process of determining the real-time position of the endoscope's passive tracking tool under the navigation system during registration and calibration is as follows:

[0085] 1. The navigation system acquires the pose of the passive tracking tool of the endoscope in real time. At the same time, the endoscope can acquire surgical images in real time. At any time, it can acquire two frames of images from adjacent time points. Based on the image features of the surgical images, the motion of the endoscope is calculated by solid geometry. The fusion result of the motion of the endoscope and the pose of the passive tracking tool of the endoscope is recorded as the measured value of the system state.

[0086] 2. Obtain the motion process of the endoscope passive tracking tool recorded by the navigation system, and calculate the motion parameters of the endoscope passive tracking tool, including velocity and acceleration, based on the motion process of the endoscope passive tracking tool;

[0087] 3. Using the pose and motion parameters of the passive endoscopic tracking tool at the previous moment, calculate the position of the passive endoscopic tracking tool at this moment based on kinematics. This position is recorded as the predicted value of the system state.

[0088] 4. Based on the measured and predicted values ​​of the passive tracking tool, the real-time position of the passive tracking tool is determined, thereby achieving position optimization of the passive tracking tool.

[0089] The above steps are repeated iteratively to achieve real-time optimization and noise reduction of the passive tracking tool's position.

[0090] Compared with the prior art, the advantages of the present invention are as follows:

[0091] 1) This invention enables navigation to guide doctors to determine the exact location of lesions and assist in completing the surgery.

[0092] 2) This invention can visually and accurately display important tissues and organs that are covered by opaque tissue during surgery, which helps doctors make judgments and avoids damage to important tissues.

[0093] 3) This invention solves the problem of the inconsistency between intraoperative real-time images and preoperative CT images caused by necessary doctor operations damaging cavities or tissue structures, thus improving the safety of the operation.

[0094] 4) By fusing endoscopic images with CT images and presenting them on the same screen, doctors can reduce the time required for intraoperative judgment, lower the difficulty of hand-eye coordination, and shorten the operation time.

[0095] This application also provides an endoscopic surgical navigation device based on video image augmented reality, comprising:

[0096] The preoperative CT module is used to acquire the patient's preoperative CT images, segment out the key areas of concern in the surgery, generate corresponding three-dimensional models, and select anatomical landmarks in the preoperative CT images.

[0097] The passive tracking tool for the endoscope and the passive tracking tool for the patient's surgical site 5 are respectively installed on the endoscope 2 and the patient's surgical site 4;

[0098] Navigation system 1 is used for navigating and locating passive tracking tools for endoscopes and passive tracking tools for patient surgical sites. The transformation matrix relationship chain under the navigation system is as follows:

[0099]

[0100] Where I is the identity matrix, This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. This is the transformation matrix between the navigation system's world coordinate system and the coordinate system of the passive tracking tool at the patient's surgical site. The transformation matrix for the passive tracking tool coordinate system relative to the endoscope coordinate system at the patient's surgical site;

[0101] The registration module, under the navigation system, uses probes already calibrated within the navigation system and selected anatomical landmarks to register the real patient with the 3D model, obtaining the registration matrix T. R ;

[0102] The calibration module is used to calibrate the parameters of the endoscope under the navigation system, including camera intrinsic parameters, camera extrinsic parameters, and distortion parameters. The camera extrinsic parameters are the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system.

[0103] The display imaging module 3 is used to determine the transformation matrix of the passive tracking tool coordinate system relative to the endoscope coordinate system based on the transformation matrix relationship chain under the navigation system. It uses a pinhole model to simulate imaging and project and render the three-dimensional model of the key surgical area segmented before surgery. The rendering result is processed using the distortion parameters calibrated by the endoscope to obtain a virtual image. The virtual image is then fused with the image acquired in real time by the endoscope and displayed.

[0104] The method of surgical navigation using the aforementioned video image augmented reality endoscopic surgical navigation device is the same as the video image augmented reality-based endoscopic surgical navigation method described above, and will not be repeated here.

[0105] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An endoscopic surgical navigation device based on video image augmented reality, characterized in that, include: The preoperative CT module is used to acquire the patient's preoperative CT images, segment out the key areas of concern in the surgery, generate corresponding three-dimensional models, and select anatomical landmarks in the preoperative CT images. A passive tracking tool for endoscopes, mounted on the endoscope; A passive tracking device for the patient's surgical site is installed at the patient's surgical site. The navigation system is used for navigating and locating passive tracking tools for endoscopes and passive tracking tools for patient surgical sites. The transformation matrix relationship chain under the navigation system is as follows: in, It is the identity matrix. This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. This is the transformation matrix between the navigation system's world coordinate system and the coordinate system of the passive tracking tool at the patient's surgical site. The transformation matrix for the passive tracking tool coordinate system relative to the endoscope coordinate system at the patient's surgical site; The registration module, under the navigation system, uses probes already calibrated within the navigation system and selected anatomical landmarks to register the real patient with the 3D model, obtaining a registration matrix. ; The calibration module is used to calibrate the parameters of the endoscope under the navigation system, including camera intrinsic parameters, camera extrinsic parameters, and distortion parameters. The camera extrinsic parameters are the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. ; The display imaging module is used to determine the transformation matrix of the passive tracking tool coordinate system relative to the endoscope coordinate system based on the transformation matrix relationship chain under the navigation system. It uses a pinhole model to simulate imaging and project and render the 3D model of the key surgical area segmented before surgery. The rendering result is processed using the distortion parameters calibrated by the endoscope to obtain a virtual image. The virtual image is then fused with the image acquired in real time by the endoscope and displayed.

2. The endoscopic surgical navigation device based on video image augmented reality according to claim 1, characterized in that, When registering CT space and patient space using selected anatomical landmarks, the registration objective is to minimize the distance between the two point clouds by iteratively calculating the rotation and translation matrices, ultimately obtaining the registration matrix. .

3. The endoscopic surgical navigation device based on video image augmented reality according to claim 1, characterized in that, The calibration of camera extrinsic parameters is specifically as follows: Set up a signboard and determine the relative position of the signboard to the navigation system; Under different motion states of the endoscope, the navigation system tracks the movement of the passive tracking tool of the endoscope and determines the relative attitude between the navigation system and the passive tracking tool of the endoscope. The endoscope is used to acquire images of the marker plate during its movement. The relative attitude between the signboard and the endoscope is calculated based on the image of the signboard acquired by the endoscope. The relative attitude between the navigation system and the endoscope is calculated based on the relative position of the signboard and the navigation system. The camera extrinsic parameters are then calculated based on the relative attitudes between the navigation system and the passive tracking tool of the endoscope, and between the navigation system and the endoscope. .

4. The endoscopic surgical navigation device based on video image augmented reality according to claim 1, characterized in that, The pinhole imaging model is as follows: in, This represents the proportionality coefficient. , This represents the coordinates of a pixel in the image coordinate system. , Characterized by the dimensions of the photosensitive element in the x and y directions. , Characterizing the offset between the actual imaging center and the theoretical center, Indicates focal length. This is the transformation matrix between the endoscope coordinate system and the endoscope passive tracking tool coordinate system. This is the transformation matrix between the coordinate system of the passive tracking tool for the endoscope and the world coordinate system of the navigation system. This is the transformation matrix between the navigation system's world coordinate system and the coordinate system of the passive tracking tool at the patient's surgical site. Represents the registration matrix. , , These are the three-dimensional coordinates of a point in space.

5. The endoscopic surgical navigation device based on video image augmented reality according to claim 1, characterized in that, The process of determining the real-time position of the passive tracking tool of the endoscope under the navigation system is as follows: The navigation system acquires the pose of the passive tracking tool of the endoscope in real time, the endoscope acquires surgical images in real time, and the motion of the endoscope is calculated by solid geometry based on the surgical images. The fusion result of the motion of the endoscope and the pose of the passive tracking tool of the endoscope is recorded as the measurement value of the system state. The motion process of the passive endoscopic tracking tool recorded by the navigation system is obtained, and the motion parameters of the passive endoscopic tracking tool, including velocity and acceleration, are calculated based on the motion process of the passive endoscopic tracking tool. Using the pose and motion parameters of the passive endoscope tracking tool at the previous moment, the position of the passive endoscope tracking tool at this moment is calculated, and this position is recorded as the predicted value of the system state. The real-time position of the passive endoscopic tracking tool is determined based on the measured and predicted values.

Images