Surgical instrument three-dimensional positioning method and device, terminal and medium
By combining a neural network model based on the Transformer architecture and an extended Kalman filter model, high-precision and robust three-dimensional positioning of surgical instruments in robot-assisted surgery is achieved. This solves the problems of adjustment delay and detection instability in existing endoscopic vision systems, and improves the automation and safety of surgery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTHERN UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2026-02-06
- Publication Date
- 2026-05-12
AI Technical Summary
In existing robot-assisted surgery, endoscopic vision systems rely on manual camera adjustments, which result in heavy cognitive burden, adjustment delays, and poor operational continuity. Automatic adjustment technologies suffer from problems such as complex hardware for stereo vision solutions that are susceptible to interference, increased sterilization risks and easy detachment of markers, single-frame deep learning that only obtains two-dimensional positions and is not robust to occlusion, and the lack of a temporal fusion mechanism that leads to detection jumps, affecting the stability of camera control.
A neural network model based on the Transformer architecture is used to detect the three-dimensional pose of surgical instruments in images acquired by an endoscopic camera. By using the two-dimensional key point coordinates of multiple frames and historical images, the three-dimensional key point coordinates of multiple frames and historical images are used to solve the technical problems of the two-dimensional key points of multiple frames and historical images, and the three-dimensional key point observation coordinates of multiple frames are used to perform temporal fusion using an extended Kalman filter model.
In scenarios with complex abdominal lighting, frequent obstruction, and rapid instrument movement, high-precision and robust 3D positioning can be achieved, improving the automation and safety of minimally invasive surgery, without the need for stereo cameras or markers.
Smart Images
Figure CN122023531A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical robot vision and intelligent control technology, and in particular to a method, device, terminal and medium for three-dimensional positioning of surgical instruments. Background Technology
[0002] With the development of robot-assisted surgery, endoscopic vision systems have become the sole visual source for doctors to observe tissue structures, locate surgical instruments, and perform delicate procedures. Current technologies typically rely on manual foot pedals or hand-operated devices to adjust the position of the endoscopic camera to maintain the field of view within the appropriate area. However, this approach has significant limitations: doctors must frequently switch between manipulating surgical instruments and controlling the endoscope, creating an additional cognitive burden; camera adjustments may be delayed or inaccurate, potentially reducing surgical efficiency; and the surgical rhythm is disrupted by human actions, affecting the continuity of the procedure.
[0003] However, while existing automatic adjustment technologies that combine automatic camera adjustment with intelligent camera tracking can solve the problems associated with manually adjusting the position of the endoscope camera using foot pedals or hand-operated devices, these technologies still have shortcomings. For example, stereo vision solutions have complex hardware and are susceptible to interference; marker methods increase the risk of sterilization and are prone to falling off; single-frame deep learning only obtains two-dimensional positions and is not robust to occlusion; and the lack of a temporal fusion mechanism leads to detection jumps, affecting the stability of camera control. Summary of the Invention
[0004] The purpose of this invention is to provide a method, device, terminal and medium for three-dimensional positioning of surgical instruments, which can maintain high precision and robustness in three-dimensional positioning in scenarios with complex abdominal lighting, frequent occlusion and rapid instrument movement, without the need for stereo cameras or markers.
[0005] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention discloses a three-dimensional positioning method for surgical instruments, wherein the method includes: The system acquires a current frame image and historical multi-frame images containing surgical instruments captured by an endoscope camera, and inputs the current frame image and the historical multi-frame images into a pre-trained instrument keypoint detection network to detect the two-dimensional keypoint coordinates of the surgical instruments in the current frame image and the two-dimensional keypoint coordinates of the surgical instruments in the historical multi-frame images; the instrument keypoint detection network is a neural network model based on the Transformer architecture. Based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system are calculated using the multi-frame triangulation method. The observed coordinates of the three-dimensional key points are input into a pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimation of the surgical instrument; the state variables of the extended Kalman filter model contain the three-dimensional pose of the surgical instrument.
[0006] Optionally, the instrument keypoint detection network is a YOLOX-STrans network, and the YOLOX-STrans network includes a convolution-based backbone network, a feature extraction module embedding a Swing Transformer, a feature pyramid structure, and a detection head for predicting two-dimensional keypoint coordinates.
[0007] Optionally, the step of calculating the three-dimensional keypoint observation coordinates of the surgical instrument in the camera coordinate system based on the two-dimensional keypoint coordinates of the surgical instrument in the current frame image and the two-dimensional keypoint coordinates of the surgical instrument in the historical multi-frame images, and using multi-frame triangulation, includes: Based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, the camera pose transformation between adjacent image frames is calculated. A projection matrix for triangulation is constructed based on the camera intrinsic parameter matrix and the camera pose transformation; the camera intrinsic parameter matrix is a matrix determined when calibrating the intrinsic parameters of the endoscopic camera. Based on the projection matrix, multi-frame triangulation is performed on the two-dimensional key point coordinates of the surgical instrument in a series of consecutive images consisting of the current frame image and the historical multi-frame images to obtain the three-dimensional key point observation coordinates of the two-dimensional key point coordinates of the surgical instrument in the camera coordinate system.
[0008] Optionally, linear algebra can be used to solve for the three-dimensional homogeneous coordinates during multi-frame triangulation operations.
[0009] Optionally, the extended Kalman filter model uses an approximate uniform motion model or a uniformly accelerated motion model as the state prediction model.
[0010] Optionally, after performing temporal fusion of the observed coordinates of the three-dimensional key points using the extended Kalman filter model to obtain the three-dimensional pose estimate of the surgical instrument, the method further includes: Based on the three-dimensional pose estimation of the surgical instrument and the preset view planning strategy, the desired pose of the endoscope camera is calculated; wherein, the preset view planning strategy is a strategy that keeps the end of the surgical instrument located in the central region of the camera imaging plane and limits the angle between the camera optical axis and the surgical instrument axis within a preset range. Based on the desired pose, joint control commands are generated and sent to the surgical robot controller, so that the surgical robot controller can drive the endoscopic robotic arm to move the endoscopic camera to the desired pose according to the joint control commands.
[0011] Optionally, the joint control commands include joint position control commands or joint speed control commands.
[0012] Secondly, the present invention also discloses a three-dimensional positioning device for surgical instruments, wherein the device comprises: A two-dimensional key point detection module is used to acquire the current frame image and historical multi-frame images containing surgical instruments captured by the endoscope camera, and input the current frame image and the historical multi-frame images into a pre-trained instrument key point detection network to detect the two-dimensional key point coordinates of the surgical instruments in the current frame image and the two-dimensional key point coordinates of the surgical instruments in the historical multi-frame images; the instrument key point detection network is a neural network model based on the Transformer architecture; The three-dimensional key point calculation module is used to calculate the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, and using the multi-frame triangulation method. The three-dimensional pose determination module is used to input the observed coordinates of the three-dimensional key points into a pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimate of the surgical instrument; the state variables of the extended Kalman filter model contain the three-dimensional pose of the surgical instrument.
[0013] Thirdly, the present invention discloses a terminal, comprising: a memory, a processor, and a surgical instrument three-dimensional positioning program stored in the memory and executable on the processor, wherein the surgical instrument three-dimensional positioning program, when executed by the processor, implements the steps of the surgical instrument three-dimensional positioning method as described above.
[0014] Fourthly, the present invention discloses a computer-readable storage medium storing a computer program that can be executed to implement the steps of the surgical instrument three-dimensional positioning method as described above.
[0015] This invention provides a method, device, terminal, and medium for three-dimensional positioning of surgical instruments. The method includes: acquiring a current frame image and historical multi-frame images containing surgical instruments captured by an endoscopic camera; inputting the current frame image and the historical multi-frame images into a pre-trained instrument keypoint detection network to detect the two-dimensional keypoint coordinates of the surgical instrument in the current frame image and the two-dimensional keypoint coordinates of the surgical instrument in the historical multi-frame images; the instrument keypoint detection network is a neural network model based on the Transformer architecture; calculating the three-dimensional keypoint observation coordinates of the surgical instrument in the camera coordinate system based on the two-dimensional keypoint coordinates of the surgical instrument in the current frame image and the two-dimensional keypoint coordinates of the surgical instrument in the historical multi-frame images, and using multi-frame triangulation; inputting the three-dimensional keypoint observation coordinates into a pre-constructed extended Kalman filter model for temporal fusion to obtain a three-dimensional pose estimate of the surgical instrument; the state variables of the extended Kalman filter model contain the three-dimensional pose of the surgical instrument. Therefore, this invention detects the 2D key point coordinates of surgical instruments in the current frame image containing the instruments using a neural network model based on the Transformer architecture. This enables robust identification of instrument key points in laparoscopic surgery scenarios. Multi-frame triangulation is then used to recover the 3D key point observation coordinates of the surgical instruments, achieving 3D key point reconstruction based on triangulation. Finally, an extended Kalman filter model is used to temporally fuse the 3D key point observation coordinates to obtain continuous and stable 3D pose estimation of the surgical instruments, achieving temporally robust pose fusion. In other words, the technical solution of this application can maintain high-precision and robust 3D positioning in scenarios with complex lighting, frequent occlusion, and rapid instrument movement in the laparoscopic cavity, without requiring a stereo camera or markers, significantly improving the automation and safety of minimally invasive surgery. Attached Figure Description
[0016] Figure 1 This is a flowchart of a preferred embodiment of the three-dimensional positioning method for surgical instruments in this invention; Figure 2 This is a schematic diagram of a YOLOX-STrans network in which a Swing Transformer module is embedded in the backbone network, as disclosed in this invention. Figure 3 This is a schematic diagram of a specific feature extraction module embedded with a Swing Transformer disclosed in this invention; Figure 4 This is a schematic diagram of a specific extended Kalman filter disclosed in this invention; Figure 5 This is a logical block diagram of a preferred embodiment of the three-dimensional positioning method for surgical instruments in this invention; Figure 6 This is a schematic block diagram of a specific surgical instrument positioning method disclosed in this invention. Figure 7 This is a functional principle block diagram of a preferred embodiment of the three-dimensional positioning device for surgical instruments in this invention; Figure 8 This is a functional principle block diagram of a preferred embodiment of the terminal in this invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0018] With the development of robot-assisted surgery, endoscopic vision systems have become the sole visual source for doctors to observe tissue structures, locate surgical instruments, and perform delicate procedures. Current technologies typically rely on manual foot pedals or hand-operated devices to adjust the position of the endoscopic camera to maintain the field of view within the appropriate area. However, this approach has significant limitations: doctors must frequently switch between manipulating surgical instruments and controlling the endoscope, creating an additional cognitive burden; camera adjustments may be delayed or inaccurate, potentially reducing surgical efficiency; and the surgical rhythm is disrupted by human actions, affecting the continuity of the procedure.
[0019] However, while existing automatic camera adjustment technologies combining automatic camera adjustment and intelligent camera following can solve problems such as heavy cognitive burden, adjustment delay, and poor operational consistency caused by reliance on manual camera adjustments, these technologies still have shortcomings. For example, stereo vision-based solutions require dual optical channels or dual sensors, resulting in complex hardware structures and poor stability. Smoke, reflections, and surgical instrument interference in surgical scenarios can cause stereo matching failures, and deployment costs are high, making widespread clinical application difficult. Marker-based methods locate surgical instruments by attaching optical / infrared markers, but this increases the risk of sterilization, and the markers are prone to falling off or being occluded, failing to meet clinical device standards. Single-frame deep learning detection methods such as YOLO, Mask R-CNN, and Transformer-based models have improved instrument detection performance, but they only provide two-dimensional positions, are not robust to occlusion, and produce discontinuous outputs, causing jitter when directly driving the robot. The lack of a stable temporal fusion mechanism is also a concern. Surgical instruments are affected by changes in force, tissue occlusion, and smoke, leading to significant jumps in detection results. Without filters to perform temporal fusion of the results, the stability of automatic camera control will be severely affected.
[0020] In summary, in robot-assisted surgery, the endoscopic vision system relies on manual camera adjustment, which leads to problems such as heavy cognitive burden, adjustment delay, and poor operational consistency. Existing automatic adjustment technologies have shortcomings: stereo vision solutions have complex hardware and are susceptible to interference; marker methods increase the risk of sterilization and are prone to detachment; single-frame deep learning only obtains two-dimensional position and is not robust to occlusion; the lack of a temporal fusion mechanism leads to detection jumps, affecting the stability of camera control.
[0021] Therefore, this application provides a three-dimensional positioning scheme for surgical instruments, which can maintain high precision and robust three-dimensional positioning in scenarios with complex abdominal lighting, frequent occlusion, and rapid instrument movement, without the need for stereo cameras or markers, and can significantly improve the automation and safety of minimally invasive surgery.
[0022] Please see Figure 1 , Figure 1 This is a flowchart of the three-dimensional positioning method for surgical instruments in this invention. For example... Figure 1 As shown, the three-dimensional positioning method for surgical instruments according to an embodiment of the present invention includes: Step S11: Acquire the current frame image and historical multi-frame images containing surgical instruments captured by the endoscope camera, and input the current frame image and the historical multi-frame images into a pre-trained instrument key point detection network to detect the two-dimensional key point coordinates of the surgical instruments in the current frame image and the two-dimensional key point coordinates of the surgical instruments in the historical multi-frame images; the instrument key point detection network is a neural network model based on the Transformer architecture.
[0023] In this embodiment, an endoscope camera acquires real-time images of the abdominal cavity scene, i.e., images containing surgical instruments. The image resolution can be set to 640×480 or higher. The system collects current frame images and historical multi-frame images containing surgical instruments from the endoscope camera, where the historical multi-frame images are consecutive, and the current frame image and historical multi-frame images constitute a continuous multi-frame image. A high-precision instrument keypoint detection network based on deep learning is used to determine the two-dimensional keypoints of the surgical instruments in the current frame image and the two-dimensional keypoints of the surgical instruments in the historical multi-frame images. Specifically, the current frame image and the historical multi-frame images are input into a pre-trained instrument keypoint detection network for keypoint detection to detect the coordinates of the two-dimensional keypoints of the surgical instruments in the current frame image and the two-dimensional keypoints of the surgical instruments in the historical multi-frame images. The endoscope camera can be a monocular endoscope camera.
[0024] It should be noted that the instrument key point detection network is a YOLOX-STrans network based on the YOLOX framework with a SwinTransformer module embedded in the backbone network. See [link to relevant documentation]. Figure 2As shown, the YOLOX-STrans network includes a convolutional backbone network, a feature extraction module embedding a Swing Transformer, a feature pyramid structure, and a detection head for predicting 2D keypoint coordinates. It can input the current frame image into the pre-trained YOLOX-STrans network for keypoint detection, thus enabling 2D keypoint detection of surgical instruments based on the YOLOX-STrans network. This detection network can further include a global attention guidance module to enhance the feature response of image regions containing surgical instruments, improving the robustness of instrument detection in scenes with partial occlusion, reflection, and motion blur. The feature extraction module embedding a Swing Transformer is described in [reference needed]. Figure 3 As shown, it is used to capture local and global context, exhibiting high robustness. Its global modeling capability enhances the stability of occluded scenes. In other words, the Swin Transformer, through window attention and cross-window transfer mechanisms, enables the instrument keypoint detection network to maintain high detection accuracy even under conditions of partial instrument occlusion, illumination reflection, and motion blur. Feature pyramid structures, such as FPN+PAN, are used to fuse multi-scale information.
[0025] Furthermore, the output of the instrument key point detection network includes not only the two-dimensional pixel coordinates of multiple key points of the surgical instrument, but also... Furthermore, there is the confidence level corresponding to the key point. Therefore, for key points with a confidence level lower than the preset confidence threshold, they can be directly ignored or completed through neighborhood interpolation.
[0026] In this embodiment, before acquiring the current frame image containing surgical instruments captured by the endoscopic camera, the process may further include: calibrating the intrinsic parameters of the endoscopic camera and establishing the extrinsic parameter transformation relationship between the endoscopic camera coordinate system and the surgical robot base coordinate system. Specifically, the Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the endoscopic camera to obtain the camera intrinsic parameter matrix, radial distortion coefficient, and tangential distortion coefficient. The extrinsic parameter transformation relationship between the endoscopic camera coordinate system and the surgical robot base coordinate system is also known as the rigid body transformation matrix. This rigid body transformation matrix can correctly map the robot end-effector pose to the camera coordinate system, thereby providing an accurate projection model for subsequent multi-frame triangulation.
[0027] Step S12: Based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system are calculated using the multi-frame triangulation method.
[0028] In this embodiment, 3D keypoint reconstruction based on triangulation can be achieved through a monocular, markerless 3D reconstruction mechanism, that is, multi-frame triangulation is used to calculate the observed coordinates of the surgical instruments in the camera coordinate system. Monocular 3D localization relies on only a single camera to recover the coordinates of stable 3D keypoints.
[0029] Specifically, based on the 2D keypoint coordinates of the surgical instruments in the current frame image and the 2D keypoint coordinates of the surgical instruments in historical multi-frame images, the camera pose transformation between adjacent image frames is calculated. A projection matrix for triangulation is constructed based on the camera intrinsic parameter matrix and the camera pose transformation. The camera intrinsic parameter matrix is a matrix determined during the intrinsic parameter calibration of the endoscopic camera. Based on the projection matrix, multi-frame triangulation is performed on the 2D keypoint coordinates of the surgical instruments in a series of consecutive frames consisting of the current frame image and historical multi-frame images, obtaining the 3D keypoint observation coordinates of the surgical instruments in the camera coordinate system. In the multi-frame triangulation operation, linear algebra can be used to solve for the 3D homogeneous coordinates, and least squares optimization or multi-frame joint optimization can be used to reduce 3D reconstruction errors. Furthermore, at least two frames are used to construct the projection matrix during the triangulation operation, and the projection matrix is jointly determined by the camera intrinsic parameter matrix and the camera pose transformation between adjacent time points.
[0030] It should be noted that multi-frame camera pose calculation can be performed based on robot instance kinematics. The surgical robot provides encoder readings for each joint angle, and the spatial pose of the camera at any given time can be calculated using the robot's kinematic model. In other words, the camera poses provided by the surgical robot at adjacent time points are calculated from the robot encoder readings and the robot's kinematic model, and are used together with the camera's extrinsic parameter registration relationship to construct the projection matrix for triangulation.
[0031] For example, based on the joint angles of the surgical robot at adjacent time points and the robot's kinematic model, the pose transformation of the endoscopic camera at those adjacent time points is calculated. Then, using the two-dimensional key points of the corresponding surgical instruments in multiple consecutive frames, a monocular multi-frame triangulation operation is performed to obtain the three-dimensional observation coordinates of the surgical instrument key points in the camera coordinate system. Specifically, for time points... and time The two-dimensional keypoint coordinates of all detected keypoints are denoted as follows: and Then, using the robot's kinematics model, the camera pose transformation between adjacent frames can be obtained, i.e.: ; Then, based on the camera intrinsic parameter matrix and camera pose transformation, a projection matrix for triangulation is constructed, namely: ; ; Furthermore, during the multi-frame triangulation operation, the three-dimensional homogeneous coordinates are solved using linear algebra methods. ,Right now: ; Among them, matrix By projection constraints It unfolds into shape.
[0032] The minimum eigenvector of A is obtained using Singular Value Decomposition (SVD), and is used as... The solution.
[0033] Considering the significant imaging noise in the abdominal cavity, multi-frame joint triangulation was employed, selecting the nearest frame. Each frame contains an image with matching key points. The 3D coordinates that best meet the projection constraints of all frames are solved by least squares optimization. The reconstruction error of triangulation can be controlled within a small range.
[0034] Step S13: Input the observed coordinates of the three-dimensional key points into the pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimation of the surgical instrument; the state variables of the extended Kalman filter model include the three-dimensional pose of the surgical instrument.
[0035] In this embodiment, in order to overcome the problems of large jitter in single-frame triangulation and occasional loss due to occlusion, the extended Kalman filter (EKF) model is used to perform temporal fusion of the observed coordinates of 3D key points to achieve temporally robust pose fusion. The extended Kalman filter model can reduce jump error, and the observation noise covariance of the extended Kalman filter is dynamically adjusted according to the uncertainty of multi-frame triangulation.
[0036] The pre-constructed Extended Kalman Filter (EKF) model incorporates position, velocity, and orientation parameters. This EKF model utilizes an approximate uniform motion model or a uniformly accelerated motion model as the state prediction model. Furthermore, the temporally continuous EKF enables smooth detection of transitions, improving visual servoing stability and achieving temporal fusion based on EKF. Specifically, a state transition equation is established based on the surgical instrument motion model. An observation equation is established based on the obtained 3D observation coordinates of the surgical instrument keypoints in the camera coordinate system. Temporal fusion of these 3D observation coordinates yields a continuous and smooth 3D pose estimate of the surgical instrument.
[0037] Furthermore, the state vector can be: ; in, This refers to the three-dimensional position of the surgical instrument tip. This refers to the movement speed of the surgical instrument tip.
[0038] When using an approximate uniform motion model as the state prediction model, the state prediction model is as follows: ; The observation model is: ; in, This is for triangulation noise.
[0039] It should be noted that when the detection network fails to output keypoints in a certain frame, the Extended Kalman Filter (EKF) model can still maintain the continuity of 3D pose by relying on the prediction step, achieving robust tracking under short-term occlusion conditions. For the specific structure of the EKF model, please refer to [link to EKF model documentation]. Figure 4 As shown.
[0040] As can be seen, in this embodiment of the invention, by detecting the two-dimensional key point coordinates of surgical instruments in the current frame image containing surgical instruments using a neural network model based on the Transformer architecture, robust identification of instrument key points in abdominal surgery scenarios can be achieved. Multi-frame triangulation is then used to recover the three-dimensional key point observation coordinates of the surgical instruments, achieving three-dimensional key point reconstruction of the surgical instruments based on triangulation. Then, an extended Kalman filter model is used to temporally fuse the three-dimensional key point observation coordinates to obtain continuous and stable three-dimensional pose estimation of the surgical instruments, achieving temporally robust pose fusion. In other words, the technical solution of this application can maintain high-precision and high-robustness three-dimensional positioning in scenarios with complex abdominal lighting, frequent occlusion, and rapid instrument movement, without requiring a stereo camera or markers, significantly improving the automation and safety of minimally invasive surgery. Furthermore, by fusing deep learning visual perception and probabilistic filtering estimation, the three-dimensional positioning of monocular surgical instruments becomes more stable, thereby ensuring the robustness and real-time performance of automatic camera adjustment.
[0041] See Figure 5 As shown, this embodiment of the invention discloses a specific three-dimensional positioning method for surgical instruments. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.
[0042] Step S21: Acquire the current frame image and historical multi-frame images containing surgical instruments captured by the endoscope camera, and input the current frame image and the historical multi-frame images into a pre-trained instrument key point detection network to detect the two-dimensional key point coordinates of the surgical instruments in the current frame image and the two-dimensional key point coordinates of the surgical instruments in the historical multi-frame images; the instrument key point detection network is a neural network model based on the Transformer architecture.
[0043] Step S22: Based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system are calculated using the multi-frame triangulation method.
[0044] Step S23: Input the observed coordinates of the three-dimensional key points into the pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimation of the surgical instrument; the state variables of the extended Kalman filter model include the three-dimensional pose of the surgical instrument.
[0045] Step S24: Calculate the desired pose of the endoscope camera based on the three-dimensional pose estimation of the surgical instrument and the preset view planning strategy; wherein, the preset view planning strategy is a strategy that keeps the end of the surgical instrument located in the central region of the camera imaging plane and limits the angle between the camera optical axis and the surgical instrument axis within a preset range.
[0046] It should be noted that the preset viewpoint planning strategy aims to keep the surgical instrument tip in the center of the camera's imaging plane and limit the angle between the camera's optical axis and the surgical instrument's axis within a preset range. In other words, the goal of the endoscope's automatic adjustment is to keep the surgical instrument tip in the center of the image and maintain the preset observation distance and optical axis angle. Furthermore, the desired pose of the endoscope is calculated by aligning the optical axis with the surgical instrument tip, and then the desired pose can be transformed to the surgical robot's base coordinate system using a rigid body transformation matrix.
[0047] For example, based on the three-dimensional pose estimation of surgical instruments and preset viewing distances, camera optical axis constraints, and other viewpoint planning strategies, the target observation position and orientation of the endoscopic camera are calculated. Furthermore, when the three-dimensional position of the surgical instrument tip is... The expected observation distance is The camera optical axis vector is denoted as Then the target camera position can be: ; in, It is the direction vector determined by the camera orientation strategy, such as the direction of the instrument axis or a fixed observation direction.
[0048] Step S25: Generate joint control commands based on the desired pose and send the joint control commands to the surgical robot controller so that the surgical robot controller drives the endoscopic robotic arm to move the endoscopic camera to the desired pose according to the joint control commands.
[0049] In this embodiment, joint control commands are generated based on the desired pose. These commands can include joint position control commands or joint velocity control commands. Specifically, the desired pose can be input into an inverse kinematics solver to generate the joint control commands. This involves converting the target observation position and orientation into the desired pose of the surgical robot's end effector. The joint control commands are then obtained through inverse kinematics solving. The target observation position is obtained by translating the three-dimensional coordinates of the surgical instrument's end effector along a preset distance. The preset distance is determined based on the safe range of the surgical operation. Furthermore, the inverse kinematics solver can employ numerical iteration, damped least squares, or analytical methods to improve convergence and numerical stability. The joint control commands are then sent to the surgical robot controller. The controller drives the endoscopic robotic arm according to the joint commands, causing the camera to gradually approach the target viewpoint, i.e., controlling the endoscopic camera to move to the desired pose to achieve automatic camera adjustment.
[0050] After a new image frame arrives, detection, triangulation, filtering, planning, and control can be repeatedly performed to form a complete vision-control closed loop. This allows the endoscope camera to continuously follow the three-dimensional movement of surgical instruments and operate stably at 30 to 60 FPS, i.e., at a frequency of no less than 30 frames per second. This enables real-time automatic adjustment of the endoscope camera during surgery, with an automatic camera adjustment response time of less than 50 ms. It achieves real-time automatic camera tracking in highly dynamic scenes and can maintain stable tracking under conditions of smoke, reflection, and obstruction. It can cover complex instrument trajectories such as continuous movement, arc movement, and puncture.
[0051] It is understandable that by integrating deep learning detection, monocular multi-frame triangulation, and extended Kalman filtering to calculate the 3D positioning of surgical instruments, and then using this positioning result to achieve automatic adjustment of the endoscope camera, camera view planning based on instrument pose, and automatic adjustment of the endoscope robotic arm.
[0052] For details regarding steps S21 to S23, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.
[0053] As can be seen, in this embodiment of the invention, by detecting the two-dimensional key point coordinates of surgical instruments in the current frame image containing surgical instruments using a neural network model based on the Transformer architecture, robust identification of instrument key points in laparoscopic surgery scenarios can be achieved. Multi-frame triangulation is then used to recover the three-dimensional key point observation coordinates of the surgical instruments, achieving three-dimensional key point reconstruction of the surgical instruments based on triangulation. Then, an extended Kalman filter model is used to temporally fuse the three-dimensional key point observation coordinates to obtain continuous and stable three-dimensional pose estimation of the surgical instruments, achieving temporally robust pose fusion. In other words, the technical solution of this application can maintain high-precision and high-robustness three-dimensional positioning in scenarios with complex abdominal lighting, frequent occlusion, and rapid instrument movement, without requiring a stereo camera or markers, significantly improving the automation and safety of minimally invasive surgery. Furthermore, by fusing deep learning visual perception and probabilistic filtering estimation, the three-dimensional positioning of monocular surgical instruments becomes more stable, thereby ensuring the robustness and real-time performance of the endoscopic camera's automatic adjustment.
[0054] It should be noted that the technical solution of this application can be applied to the automatic adjustment scene of the camera in robot vision servoing, and has strong real-time performance, achieving real-time processing of 30 to 60 FPS.
[0055] For example, see Figure 6 As shown, the current frame image containing the surgical instruments is input into a pre-trained YOLOX-STrans network for keypoint detection to determine the coordinates of the two-dimensional keypoints of the surgical instruments. Then, using the two-dimensional key points of the corresponding surgical instruments in multiple consecutive frames, a monocular multi-frame triangulation operation is performed to obtain the three-dimensional observation coordinates of the surgical instrument key points in the camera coordinate system. Then, the three-dimensional observation coordinates of the key points of the surgical instrument in the camera coordinate system are input into the pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimation of the surgical instrument.
[0056] In one embodiment, such as Figure 7 As shown, based on the above-described three-dimensional positioning method for surgical instruments, the present invention also provides a three-dimensional positioning device for surgical instruments, comprising: The two-dimensional key point detection module 11 is used to acquire the current frame image and historical multi-frame images containing surgical instruments captured by the endoscope camera, and input the current frame image and the historical multi-frame images into a pre-trained instrument key point detection network to detect the two-dimensional key point coordinates of the surgical instruments in the current frame image and the two-dimensional key point coordinates of the surgical instruments in the historical multi-frame images; the instrument key point detection network is a neural network model based on the Transformer architecture; The three-dimensional key point calculation module 12 is used to calculate the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, and using the multi-frame triangulation method. The three-dimensional pose determination module 13 is used to input the observed coordinates of the three-dimensional key points into a pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimation of the surgical instrument; the state variables of the extended Kalman filter model include the three-dimensional pose of the surgical instrument.
[0057] In some specific embodiments, the instrument keypoint detection network is a YOLOX-STrans network, and the YOLOX-STrans network includes a convolution-based backbone network, a feature extraction module embedded with a Swing Transformer, a feature pyramid structure, and a detection head for predicting two-dimensional keypoint coordinates.
[0058] As can be seen, in this embodiment of the invention, by detecting the two-dimensional key point coordinates of surgical instruments in the current frame image containing surgical instruments using a neural network model based on the Transformer architecture, robust identification of instrument key points in laparoscopic surgery scenarios can be achieved. Multi-frame triangulation is then used to recover the three-dimensional key point observation coordinates of the surgical instruments, achieving three-dimensional key point reconstruction of the surgical instruments based on triangulation. Then, an extended Kalman filter model is used to temporally fuse the three-dimensional key point observation coordinates to obtain continuous and stable three-dimensional pose estimation of the surgical instruments, achieving temporally robust pose fusion. In other words, the technical solution of this application can maintain high-precision and high-robustness three-dimensional positioning in scenarios with complex abdominal lighting, frequent occlusion, and rapid instrument movement, without requiring a stereo camera or markers, significantly improving the automation and safety of minimally invasive surgery. Furthermore, by fusing deep learning visual perception and probabilistic filtering estimation, the three-dimensional positioning of monocular surgical instruments becomes more stable, thereby ensuring the robustness and real-time performance of the endoscopic camera's automatic adjustment.
[0059] In some specific embodiments, the three-dimensional key point calculation module 12 may specifically include: The pose transformation calculation unit is used to calculate the camera pose transformation between adjacent image frames based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images. The projection matrix construction unit is used to construct a projection matrix for triangulation based on the camera intrinsic parameter matrix and the camera pose transformation; the camera intrinsic parameter matrix is a matrix determined when calibrating the intrinsic parameters of the endoscope camera. The triangulation unit is used to perform multi-frame triangulation on the coordinates of the two-dimensional key points of the surgical instrument in a series of images consisting of the current frame image and the historical multi-frame images, based on the projection matrix, so as to obtain the three-dimensional key point observation coordinates of the two-dimensional key point coordinates of the surgical instrument in the camera coordinate system.
[0060] In some specific embodiments, linear algebra is used to solve for the three-dimensional homogeneous coordinates during multi-frame triangulation operations.
[0061] In some specific embodiments, the extended Kalman filter model uses an approximate uniform motion model or a uniformly accelerated motion model as the state prediction model.
[0062] In some specific embodiments, the surgical instrument three-dimensional positioning device may further include: The camera pose calculation module is used to calculate the desired pose of the endoscope camera based on the three-dimensional pose estimation of the surgical instrument and a preset view planning strategy; wherein, the preset view planning strategy is a strategy that keeps the end of the surgical instrument located in the central region of the camera imaging plane and limits the angle between the camera optical axis and the surgical instrument axis within a preset range. A control command generation module is used to generate joint control commands based on the desired pose and send the joint control commands to the surgical robot controller, so that the surgical robot controller drives the endoscopic robotic arm to move the endoscopic camera to the desired pose according to the joint control commands.
[0063] In some specific embodiments, the joint control commands include joint position control commands or joint speed control commands.
[0064] Furthermore, it is worth noting that the working process of the surgical instrument three-dimensional positioning device provided in this embodiment is the same as the working process of the surgical instrument three-dimensional positioning method described above, so it will not be repeated here. For details, please refer to the working process of the surgical instrument three-dimensional positioning method described above.
[0065] It should be noted that the aforementioned three-dimensional positioning device for surgical instruments has high compatibility and can be directly deployed with existing medical robot systems.
[0066] Figure 8 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0067] When the processor 502 executes the program, it implements the three-dimensional positioning method for surgical instruments provided in the above embodiments.
[0068] Furthermore, the terminal also includes: Communication interface 503 is used for communication between memory 501 and processor 502.
[0069] The memory 501 is used to store computer programs that can run on the processor 502.
[0070] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0071] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.
[0072] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0073] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0074] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional positioning method for surgical instruments.
[0075] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.
[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0077] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can read and execute instructions from and from an instruction execution system, apparatus or device).
[0078] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0079] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A three-dimensional positioning method for surgical instruments, characterized in that, The method includes: The system acquires a current frame image and historical multi-frame images containing surgical instruments captured by an endoscope camera, and inputs the current frame image and the historical multi-frame images into a pre-trained instrument key point detection network to detect the two-dimensional key point coordinates of the surgical instruments in the current frame image and the two-dimensional key point coordinates of the surgical instruments in the historical multi-frame images; the instrument key point detection network is a neural network model based on the Transformer architecture. Based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system are calculated using the multi-frame triangulation method. The observed coordinates of the three-dimensional key points are input into a pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimation of the surgical instrument; the state variables of the extended Kalman filter model contain the three-dimensional pose of the surgical instrument.
2. The three-dimensional positioning method for surgical instruments according to claim 1, characterized in that, The instrument key point detection network is a YOLOX-STrans network, which includes a convolution-based backbone network, a feature extraction module embedded with SwinTransformer, a feature pyramid structure, and a detection head for predicting two-dimensional key point coordinates.
3. The three-dimensional positioning method for surgical instruments according to claim 1, characterized in that, The step of calculating the three-dimensional keypoint observation coordinates of the surgical instrument in the camera coordinate system based on the two-dimensional keypoint coordinates of the surgical instrument in the current frame image and the two-dimensional keypoint coordinates of the surgical instrument in the historical multi-frame images, and using multi-frame triangulation, includes: Based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, the camera pose transformation between adjacent image frames is calculated. A projection matrix for triangulation is constructed based on the camera intrinsic parameter matrix and the camera pose transformation; the camera intrinsic parameter matrix is a matrix determined when calibrating the intrinsic parameters of the endoscopic camera. Based on the projection matrix, multi-frame triangulation is performed on the two-dimensional key point coordinates of the surgical instrument in a series of consecutive images consisting of the current frame image and the historical multi-frame images to obtain the three-dimensional key point observation coordinates of the two-dimensional key point coordinates of the surgical instrument in the camera coordinate system.
4. The three-dimensional positioning method for surgical instruments according to claim 1, characterized in that, In performing multi-frame triangulation operations, linear algebra is used to solve for the three-dimensional homogeneous coordinates.
5. The three-dimensional positioning method for surgical instruments according to claim 1, characterized in that, The extended Kalman filter model uses an approximate uniform motion model or a uniformly accelerated motion model as the state prediction model.
6. The three-dimensional positioning method for surgical instruments according to any one of claims 1 to 5, characterized in that, After using the extended Kalman filter model to perform temporal fusion of the observed coordinates of the three-dimensional key points to obtain the three-dimensional pose estimate of the surgical instrument, the method further includes: Based on the three-dimensional pose estimation of the surgical instrument and the preset view planning strategy, the desired pose of the endoscope camera is calculated; wherein, the preset view planning strategy is a strategy that keeps the end of the surgical instrument located in the central region of the camera imaging plane and limits the angle between the camera optical axis and the surgical instrument axis within a preset range. Based on the desired pose, joint control commands are generated and sent to the surgical robot controller, so that the surgical robot controller can drive the endoscopic robotic arm to move the endoscopic camera to the desired pose according to the joint control commands.
7. The three-dimensional positioning method for surgical instruments according to claim 6, characterized in that, The joint control commands include joint position control commands or joint speed control commands.
8. A three-dimensional positioning device for surgical instruments, characterized in that, The device includes: A two-dimensional key point detection module is used to acquire the current frame image and historical multi-frame images containing surgical instruments captured by the endoscope camera, and input the current frame image and the historical multi-frame images into a pre-trained instrument key point detection network to detect the two-dimensional key point coordinates of the surgical instruments in the current frame image and the two-dimensional key point coordinates of the surgical instruments in the historical multi-frame images; the instrument key point detection network is a neural network model based on the Transformer architecture; The three-dimensional key point calculation module is used to calculate the three-dimensional key point observation coordinates of the surgical instrument in the camera coordinate system based on the two-dimensional key point coordinates of the surgical instrument in the current frame image and the two-dimensional key point coordinates of the surgical instrument in the historical multi-frame images, and using the multi-frame triangulation method. The three-dimensional pose determination module is used to input the observed coordinates of the three-dimensional key points into a pre-constructed extended Kalman filter model for temporal fusion to obtain the three-dimensional pose estimate of the surgical instrument; the state variables of the extended Kalman filter model contain the three-dimensional pose of the surgical instrument.
9. A terminal, characterized in that, include: The device includes a memory, a processor, and a surgical instrument three-dimensional positioning program stored in the memory and executable on the processor, wherein the surgical instrument three-dimensional positioning program, when executed by the processor, implements the steps of the surgical instrument three-dimensional positioning method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed to implement the steps of the surgical instrument three-dimensional positioning method as described in any one of claims 1 to 7.