Surgical instrument pose estimation method and device and surgical navigation method and system
By extracting and fusing the projected image block feature map of surgical instruments, iteratively solves the position information of surgical instruments, and solving the problems of high equipment costs and inaccurate position estimation in the prior art, and achieving accurate position prediction and accuracy of surgical instruments Navigation improves the safety and efficiency of surgery.
Patent Information
- Application Number
- CN202410945899.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-12-06
AI Technical Summary
The existing surgical instrument position estimation method has high cost to invest in equipment, which can easily increase the burden on medical device usage scenarios, and in complex scenarios, resulting in inaccurate pose estimation results, affecting the accuracy and safety of the surgery.
By obtaining the video frame and three-dimensional model of the doctor operating the surgical instrument, the outline feature map, foreground probability map and boundary probability map of the projected image block are extracted, and the posture information of the surgical instrument is iteratively solved, and the posture information of the fusion probability map is used to improve the posture The accuracy of the estimate reduces the cost of equipment and ensures the range of movement of surgical instruments.
Accurate prediction and precise navigation of surgical instrument position is achieved, equipment costs are reduced, additional auxiliary equipment is avoided, and the accuracy and safety of the surgery is ensured.
Smart Images

Figure CN119090955A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a surgical instrument posture estimation method, device, surgical navigation method and system. Background Art
[0002] Surgical robots and intelligent assisted surgery are important development directions in today's smart medical field. How to achieve robust, accurate and minimally dependent on auxiliary equipment minimally invasive surgical tool and instrument posture estimation is an important key basic technology for surgical robots and intelligent assisted surgery.
[0003] Related technologies: Common methods for realizing surgical instrument pose estimation are mainly divided into two categories. The first category usually requires adding other auxiliary tools in addition to the monocular endoscope required for general surgery itself to locate the surgical instrument. For example, at the tail end of the surgical instrument, a marker such as a chessboard is fixed, and the pose of the marker is estimated by collecting the image of the marker, and the hand-eye calibration technology is used to indirectly estimate the pose of the surgical instrument. The second widely used method is to add additional color or depth cameras, match them by binocular or color and depth cameras, and then obtain the three-dimensional information of the features on the surgical instrument through the epipolar geometry relationship, thereby estimating the pose of the surgical instrument.
[0004] However, the two types of posture estimation methods in the related art have high equipment costs and are likely to add additional burdens to the use scenarios of medical devices. In complex scenarios, they will also affect the range of movement of surgical instruments, resulting in inaccurate posture estimation results of medical devices. Summary of the invention
[0005] The present invention provides a surgical instrument posture estimation method, device, surgical navigation method and system, which are used to solve the defects existing in the related technology.
[0006] The present invention provides a method for estimating the position and posture of a surgical instrument, comprising: Acquire a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument; Based on the historical posture of the surgical instrument in the previous video frame, projecting the three-dimensional model into the current video frame to obtain a projected image block in the current video frame; Extracting a contour feature map, a foreground probability map, and a boundary probability map of the projected image block, and fusing the contour feature map, the foreground probability map, and the boundary probability map to obtain a fused probability map; Based on the fusion probability map, the position and posture information of the surgical instrument is iteratively solved.
[0007] According to a method for estimating the posture of a surgical instrument provided by the present invention, extracting a contour feature map of the projection image block comprises: Extracting a multi-scale feature map of the projected image block; Based on the corresponding line model of the surgical instrument in the current video frame, interpolation sampling is performed on the multi-scale feature map to obtain the contour feature map.
[0008] According to a method for estimating the posture of a surgical instrument provided by the present invention, the posture information of the surgical instrument is iteratively solved based on the fusion probability map, including: In the current iteration process, based on the posture obtained in the previous iteration process, the three-dimensional contour points of the three-dimensional model are projected onto the fusion probability map to obtain two-dimensional projection points, and the reprojection error is calculated based on the two-dimensional projection points and the adjacent two-dimensional points on the fusion probability map; Determining a likelihood function based on the reprojection error and the fused probability map; For corresponding point pairs of the three-dimensional contour points and the two-dimensional projection points, use the likelihood function to calculate likelihood estimates, and based on the likelihood estimates, update the pose obtained in the previous iteration process to obtain the pose obtained in the current iteration process; When the iteration end condition is met, the posture obtained in the current iteration process is used as the posture information of the surgical instrument; otherwise, the current iteration process is updated to the next iteration process, and the current iteration process is continued.
[0009] The present invention also provides a surgical navigation method, comprising: Based on the above-mentioned surgical instrument posture estimation method, determine the posture information of the surgical instrument; Converting the position information to a patient coordinate system to obtain target position information of the surgical instrument in the patient coordinate system; The target posture information is registered with the patient's medical image to obtain a registered image, and the registered image is displayed.
[0010] A surgical navigation method provided by the present invention further includes: Meshing the three-dimensional model to determine a target three-dimensional mesh of the surgical instrument; The target three-dimensional mesh is rendered to the current video frame, and the current video frame is displayed.
[0011] According to a surgical navigation method provided by the present invention, the three-dimensional model is gridded to determine a target three-dimensional grid of the surgical instrument, including: Meshing the three-dimensional model to obtain an initial three-dimensional mesh; The initial three-dimensional mesh is subjected to vertex sparse processing to obtain the target three-dimensional mesh.
[0012] The present invention also provides a surgical instrument posture estimation device, comprising: An acquisition module, used to acquire a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument; A projection module, configured to project the three-dimensional model into the current video frame based on the historical posture of the surgical instrument in the previous video frame to obtain a projected image block in the current video frame; A fusion module, used to extract a contour feature map, a foreground probability map and a boundary probability map of the projected image block, and fuse the contour feature map, the foreground probability map and the boundary probability map to obtain a fused probability map; A solution module is used to iteratively solve the posture information of the surgical instrument based on the fusion probability map.
[0013] The present invention also provides a surgical navigation system, comprising: A posture estimation module, used to determine the posture information of the surgical instrument based on the above-mentioned surgical instrument posture estimation method; A navigation registration module, used for converting the posture information into a patient coordinate system to obtain target posture information of the surgical instrument in the patient coordinate system; The visualization module is used to register the target posture information with the patient's medical image to obtain a registered image and display the registered image.
[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a surgical instrument posture estimation method or a surgical navigation method as described above.
[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a surgical instrument posture estimation method or a surgical navigation method as described above.
[0016] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for estimating the posture of a surgical instrument or a surgical navigation method.
[0017] The present invention provides a method, device, surgical navigation method and system for estimating the posture of a surgical instrument. The method for estimating the posture of a surgical instrument applies a contour feature map, a foreground probability map and a boundary probability map of a projected image block, and combines them to improve the prediction accuracy of the boundary of the surgical instrument. The fused probability map can then be used to obtain more accurate posture information of the surgical instrument, which is convenient for subsequent more accurate surgery. Moreover, the method does not require the introduction of additional auxiliary equipment, which can reduce the equipment cost and ensure the range of motion of the surgical instrument. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flow chart of the surgical instrument posture estimation method provided by the present invention.
[0020] Figure 2 It is a flowchart of the surgical navigation method provided by the present invention.
[0021] Figure 3 It is a schematic diagram of the application flow of the surgical navigation method provided by the present invention.
[0022] Figure 4 It is a structural schematic diagram of the surgical instrument posture estimation device provided by the present invention.
[0023] Figure 5 It is a structural schematic diagram of the surgical navigation system provided by the present invention.
[0024] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] Since the equipment cost of the pose estimation method in the related technology is high and it is easy to add additional burden to the use scenarios of medical devices, it will also affect the range of motion of surgical instruments in complex scenarios, resulting in inaccurate pose estimation results of medical devices. Moreover, the operation of the surgical navigation system depends on specific surgical instruments, and doctors need additional training and time to learn and master the functions of the system. This may affect the doctor's workflow and operation time, and may introduce additional risks. In addition, although there are some learning-based methods that can complete the 6DoF pose estimation of instruments, the limited computational efficiency makes it impossible to complete real-time tracking and application.
[0027] Based on this, a method for estimating the posture of a surgical instrument is provided in an embodiment of the present invention.
[0028] Figure 1 FIG. 1 is a flow chart of a method for estimating the posture of a surgical instrument provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S11, obtaining a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument; S12, projecting the three-dimensional model into the current video frame based on the historical posture of the surgical instrument in the previous video frame to obtain a projected image block in the current video frame; S13, extracting a contour feature map, a foreground probability map, and a boundary probability map of the projected image block, and fusing the contour feature map, the foreground probability map, and the boundary probability map to obtain a fused probability map; S14, iteratively solving the position and posture information of the surgical instrument based on the fusion probability map.
[0029] Specifically, the surgical instrument posture estimation method provided in the embodiment of the present invention is performed by a surgical instrument posture estimation device, which can be configured in a surgical navigation system, and the surgical navigation system can be installed in a computer, which can be a local computer, such as a computer, a tablet, etc., which is not specifically limited here. The surgical navigation system is a technical tool that combines medical imaging, computer vision and navigation technology, and is designed to provide real-time navigation and visualization support during surgery to help doctors locate and operate more accurately.
[0030] First, step S11 is performed to obtain a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument. The doctor may be a doctor who needs to use a surgical instrument to perform surgery, and the surgical instrument may be a handheld surgical instrument, such as a super-dexterous handheld bendable minimally invasive surgical instrument.
[0031] The current video frame may be any video frame in a video of a doctor operating a surgical instrument acquired by an RGB camera, and the current video frame may be an RGB color image.
[0032] The three-dimensional model of surgical instruments can be drawn through computer-aided design (CAD).
[0033] Then, step S12 is executed, and the three-dimensional model is projected into the current video frame using the historical posture of the surgical instrument in the previous video frame to obtain the projected image block in the current video frame. Here, the posture of the surgical instrument in each video frame is iteratively estimated according to the order of the video frames in the video. Therefore, when solving the posture information of the surgical instrument in the current video frame, the historical posture of the surgical instrument in the previous video frame is known, and is also estimated by the surgical instrument posture estimation method provided in the embodiment of the present invention. The historical posture can be six-degree-of-freedom (6DoF) posture information, including three-dimensional position information and three-dimensional posture information.
[0034] The historical pose can be represented by a rotation matrix and a translation vector, and the three-dimensional model can be projected into the current video frame using the historical pose. Here, the three-dimensional model coordinates can be converted into two-dimensional image coordinates using a perspective projection matrix based on the historical pose of the surgical instrument in the previous video frame and the relative position relationship between the camera and the three-dimensional model, and the projection position of the three-dimensional model in the current video frame can be calculated. Based on the projection position, the area outside the edge of the current video frame can be removed, and the projected image block containing the edge features can be cropped. It can be understood that the historical pose of the surgical instrument in the first video frame can be a preset unit matrix.
[0035] Thereafter, step S13 is performed to extract the contour feature map, foreground probability map and boundary probability map of the projected image block. The projected image block can be input into a contour feature extraction model, and the contour feature extraction model outputs a contour feature map. The contour feature extraction model can be a neural network model and can be trained by image samples with contour feature labels.
[0036] When extracting the foreground probability map, the foreground and background pixels that have been segmented in the previous video frame can be used to establish the distribution models of the foreground color and background color respectively by statistically analyzing the color histograms of the foreground and background pixels or using the kernel density estimation algorithm (KDE). Then, using the Bayesian theorem, the probability of each pixel in the projected image block of the current video frame belonging to the foreground is calculated based on the distribution model.
[0037] When extracting the boundary probability map, the gradient strength and direction of each pixel in the projected image block can be calculated based on the gradient edge detection algorithm, and the edge information in the projected image block can be highlighted by using morphological operations such as dilation and erosion. Thereafter, the obtained gradient strength can be converted into the boundary probability of each pixel in the projected image block to obtain the boundary probability map.
[0038] When extracting the boundary probability map, a lightweight convolutional neural network (CNN) can also be used to implement it. The projected image block is input into the lightweight CNN, and the boundary probability map is obtained through the lightweight CNN.
[0039] Thereafter, the contour feature map, the foreground probability map, and the boundary probability map may be fused to obtain a fused probability map to improve the prediction accuracy of the surgical instrument boundary.
[0040] Finally, step S14 is executed, and the posture information of the surgical instrument is solved by using the fusion probability map and an iterative optimization method. Here, an objective function can be defined first, which can reflect the difference between the current posture estimate and the true posture. The difference can be quantified by comparing the fusion probability map with the predicted image or data generated by the current posture estimate. At the same time, an iteration rule can be formulated, that is, how to update the posture estimate according to the gradient or other information of the objective function in each iteration. In addition, it is necessary to set an end condition, such as the change in the objective function is less than a certain set threshold, or the number of iterations reaches a preset value. Based on this, the objective function is calculated according to the current posture estimate. According to the iteration rule, the current posture estimate of the surgical instrument is updated. If the end condition is met, the iteration is stopped; otherwise, the next iteration process continues.
[0041] After obtaining the final pose estimate, its accuracy is verified by actual testing or comparison with data from other high-precision devices. If the error is large, the parameters in the iterative optimization algorithm can be adjusted or the data fusion method can be improved to improve the accuracy of the pose estimate.
[0042] A method for estimating the posture of a surgical instrument provided in an embodiment of the present invention first obtains the current video frame of a doctor operating a surgical instrument and the three-dimensional model of the surgical instrument; then, based on the historical posture of the surgical instrument in the previous video frame, the three-dimensional model is projected into the current video frame to obtain a projection image block in the current video frame; thereafter, the contour feature map, foreground probability map and boundary probability map of the projection image block are extracted, and the contour feature map, foreground probability map and boundary probability map are fused to obtain a fusion probability map; finally, based on the fusion probability map, the posture information of the surgical instrument is iteratively solved. This method applies the contour feature map, foreground probability map and boundary probability map of the projection image block and combines them to improve the prediction accuracy of the surgical instrument boundary, and then uses the fusion probability map to solve more accurate posture information of the surgical instrument, which is convenient for subsequent more accurate surgery. Moreover, this method does not require the introduction of additional auxiliary equipment, which can reduce the equipment cost invested and ensure the range of motion of the surgical instrument.
[0043] On the basis of the above embodiment, extracting the contour feature map of the projection image block includes: Extracting a multi-scale feature map of the projected image block; Based on the corresponding line model of the surgical instrument in the current video frame, interpolation sampling is performed on the multi-scale feature map to obtain the contour feature map.
[0044] Specifically, when extracting the contour feature map of the projected image block, a lightweight network, such as an FPN-Lite network, can be used to extract a multi-scale feature map. The multi-scale feature map can include a series of feature information from coarse to fine.
[0045] Then, the corresponding line model of the surgical instrument can be obtained from the current video frame, and then interpolation sampling is performed on the multi-scale feature map according to the corresponding line model to obtain the contour feature map.
[0046] On the basis of the above embodiment, the iteratively solving the posture information of the surgical instrument based on the fusion probability map includes: In the current iteration process, based on the posture obtained in the previous iteration process, the three-dimensional contour points of the three-dimensional model are projected onto the fusion probability map to obtain two-dimensional projection points, and the reprojection error is calculated based on the two-dimensional projection points and the adjacent two-dimensional points on the fusion probability map; Determining a likelihood function based on the reprojection error and the fused probability map; For corresponding point pairs of the three-dimensional contour points and the two-dimensional projection points, use the likelihood function to calculate likelihood estimates, and based on the likelihood estimates, update the pose obtained in the previous iteration process to obtain the pose obtained in the current iteration process; When the iteration end condition is met, the posture obtained in the current iteration process is used as the posture information of the surgical instrument; otherwise, the current iteration process is updated to the next iteration process, and the current iteration process is continued.
[0047] Specifically, in the iterative solution process, for each iteration process, taking the current iteration process as an example, in the current iteration process, the posture obtained in the previous iteration process is used, combined with the camera intrinsic parameter matrix and the posture matrix, to project the three-dimensional contour points of the three-dimensional model to the fusion probability map, and obtain the two-dimensional projection points on the fusion probability map.
[0048] The reprojection error is calculated using the two-dimensional projection point and one or more adjacent two-dimensional points on the fusion probability map that are close to the two-dimensional projection point. The adjacent two-dimensional points can be obtained by interpolation or nearest neighbor search. On the two-dimensional plane, for each two-dimensional projection point, the Euclidean distance or weighted distance can be used to calculate the reprojection error between each two-dimensional projection point and its adjacent two-dimensional points. The reprojection error can reflect the accuracy of the current pose estimation.
[0049] Afterwards, using the reprojection error and the fused probability map, a likelihood function can be defined, which can be used to describe the probability of observing the boundary probability map under the current pose estimation.
[0050] After that, for each corresponding point pair of the 3D contour point and the 2D projection point, the likelihood value is calculated using the likelihood function, and the boundary probability can be considered as a weight in the process. The likelihood of all corresponding point pairs is simply summarized, such as taking the logarithm of the product or sum, to obtain the overall likelihood estimate.
[0051] Using the likelihood estimate, an optimization algorithm such as gradient descent, Levenberg-Marquardt algorithm or PnP algorithm is applied to update the pose obtained in the previous iteration to obtain the pose obtained in the current iteration. It can be understood that the optimization goal of the optimization algorithm is to find the pose that maximizes the likelihood estimate.
[0052] When the iteration end condition is met, the posture obtained in the current iteration process is used as the posture information of the surgical instrument; otherwise, the current iteration process is updated to the next iteration process and the current iteration process is continued.
[0053] like Figure 2 As shown, based on the above embodiment, a surgical navigation method is also provided in an embodiment of the present invention, including: S21, determining the posture information of the surgical instrument based on the surgical instrument posture estimation method provided in the above embodiments; S22, converting the posture information into a patient coordinate system to obtain target posture information of the surgical instrument in the patient coordinate system; S23, registering the target posture information with the patient's medical image to obtain a registered image, and displaying the registered image.
[0054] Specifically, the surgical navigation method provided in the embodiment of the present invention is executed by a surgical navigation system, which may have a built-in surgical instrument posture estimation device. The surgical navigation system may be installed in a computer, which may be a local computer, such as a computer, a tablet, etc., which is not specifically limited here.
[0055] First, execute step S21 to determine the posture information of the surgical instrument using the surgical instrument posture estimation method provided in the above embodiments.
[0056] Then execute step S22. Since the posture information of the surgical instrument obtained in step S21 is the posture information in the camera coordinate system, it is necessary to align the camera coordinate system with the patient coordinate system to complete the navigation registration, that is, convert the posture information to the patient coordinate system to obtain the target posture information of the surgical instrument in the patient coordinate system.
[0057] Finally, step S23 is performed to register the target posture information with the patient's medical image to obtain a registered image, and the registered image is displayed. The patient's medical image may be an image of an in vivo lesion obtained during minimally invasive surgery using a surgical instrument.
[0058] Here, the registration process can be understood as converting the target pose information of the surgical instrument into the medical image coordinate system, and calculating the deviation between the predicted position and the actual position of the surgical instrument in the patient's medical image by comparing the predicted position and the actual position of the surgical instrument in the patient's medical image. The target pose information of the surgical instrument is adjusted according to the deviation until the actual position of the surgical instrument in the patient's medical image is consistent with the predicted position.
[0059] The specific implementation methods of registration can include feature-based registration methods and mutual information-based registration methods. Feature-based registration methods use features such as anatomical landmarks, edges, and textures in the patient's medical images for registration. For example, image segmentation technology can be used to extract the target structure and calculate its centroid or center line as the registration feature.
[0060] Mutual information is a concept in information theory that is used to measure the statistical correlation between two random variables. In patient medical image registration, the patient's medical image and the target pose information of the surgical instrument can be regarded as two random variables, and the registration is achieved by maximizing the mutual information between them.
[0061] Finally, the registered images are displayed on the monitor in real time to provide doctors with intuitive visual feedback. Doctors can then adjust surgical strategies or instrument operations based on the feedback to ensure the accuracy and safety of the surgery.
[0062] The surgical navigation method provided in the embodiment of the present invention first uses the surgical instrument posture estimation method provided in the above embodiments to determine the posture information of the surgical instrument; then the posture information is converted to the patient coordinate system to obtain the target posture information of the surgical instrument in the patient coordinate system; finally, the target posture information is registered with the patient's medical image to obtain a registered image, and the registered image is displayed. This method can make the obtained posture information more accurate by improving the estimation method of the surgical instrument's posture information, and thus make the obtained registered image more accurate, providing accurate navigation information for doctors. Moreover, the display of the registered image in this method can enable doctors to observe the registered image more intuitively, which is convenient for doctors to quickly perform surgical navigation work.
[0063] On the basis of the above embodiments, the surgical navigation method provided in the embodiments of the present invention further includes: Meshing the three-dimensional model to determine the target three-dimensional mesh of the surgical instrument; Render the target 3D mesh to the current video frame and display the current video frame.
[0064] Specifically, the 3D model of the surgical instrument can be meshed to determine the target 3D mesh of the surgical instrument, and then the target 3D mesh can be rendered to the current video frame according to the predicted homography matrix, and the current video frame can be displayed. The homography matrix represents the rotation and offset of the surgical instrument relative to the starting position, which can be obtained by learning and predicting the fusion probability map.
[0065] This method displays the current video frame, allowing the doctor to more intuitively observe the posture information of the surgical instrument from the current video frame, making it easier to operate the surgical instrument.
[0066] Based on the above embodiment, the step of gridding the three-dimensional model to determine the target three-dimensional grid of the surgical instrument includes: Meshing the three-dimensional model to obtain an initial three-dimensional mesh; The initial three-dimensional mesh is subjected to vertex sparse processing to obtain the target three-dimensional mesh.
[0067] Specifically, when determining the target three-dimensional mesh of the surgical instrument, the mesh texture of the surgical instrument may be generated by a CAD tool to obtain an initial three-dimensional mesh.
[0068] Then, the initial three-dimensional mesh can be subjected to vertex sparse processing, and vertices and lines can be extracted to generate a target three-dimensional mesh, which can reduce the number and complexity of vertices of the target three-dimensional mesh and reduce the difficulty of generating the target three-dimensional mesh.
[0069] like Figure 3 FIG. 1 is an application flow chart of a surgical navigation method provided in an embodiment of the present invention, the method comprising: At the beginning of the operation, data acquisition is performed, including RGB images of the doctor operating the surgical instrument and the three-dimensional model of the surgical instrument.
[0070] Then, the projected image block is intercepted in the RGB image, and the multi-scale feature map in the projected image block is extracted.
[0071] Afterwards, according to the corresponding line model of the surgical instrument in the RGB image, interpolation sampling is performed on the multi-scale feature map to obtain the contour feature map.
[0072] Thereafter, the contour feature map is fused with the foreground probability map and the boundary probability map of the projected image block to obtain a fused probability map.
[0073] After that, the posture information of the surgical instrument is obtained by iteratively solving the fusion probability map.
[0074] After that, the camera coordinate system is aligned with the patient coordinate system, the posture information is converted to the patient coordinate system, the target posture information of the surgical instrument in the patient coordinate system is obtained, and the navigation registration is completed. At the same time, the three-dimensional model of the surgical instrument can be meshed to obtain the target three-dimensional mesh of the surgical instrument, and the target three-dimensional mesh is rendered to an RGB image, and the RGB image is displayed.
[0075] Finally, the target posture information is registered with the patient's medical image to obtain a registered image, which is then displayed to enable subsequent surgical navigation.
[0076] like Figure 4 As shown, based on the above embodiment, an embodiment of the present invention provides a surgical instrument posture estimation device, including: An acquisition module 41 is used to acquire a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument; A projection module 42, configured to project the three-dimensional model into the current video frame based on the historical posture of the surgical instrument in the previous video frame to obtain a projection image block in the current video frame; A fusion module 43 is used to extract a contour feature map, a foreground probability map and a boundary probability map of the projected image block, and fuse the contour feature map, the foreground probability map and the boundary probability map to obtain a fused probability map; The solving module 44 is used to iteratively solve the posture information of the surgical instrument based on the fusion probability map.
[0077] On the basis of the above-mentioned embodiment, in the surgical instrument posture estimation device provided in the embodiment of the present invention, the fusion module is specifically used for: Extracting a multi-scale feature map of the projected image block; Based on the corresponding line model of the surgical instrument in the current video frame, interpolation sampling is performed on the multi-scale feature map to obtain the contour feature map.
[0078] On the basis of the above-mentioned embodiment, in the surgical instrument posture estimation device provided in the embodiment of the present invention, the solution module is specifically used for: In the current iteration process, based on the posture obtained in the previous iteration process, the three-dimensional contour points of the three-dimensional model are projected onto the fusion probability map to obtain two-dimensional projection points, and the reprojection error is calculated based on the two-dimensional projection points and the adjacent two-dimensional points on the fusion probability map; Determining a likelihood function based on the reprojection error and the fused probability map; For corresponding point pairs of the three-dimensional contour points and the two-dimensional projection points, use the likelihood function to calculate likelihood estimates, and based on the likelihood estimates, update the pose obtained in the previous iteration process to obtain the pose obtained in the current iteration process; When the iteration end condition is met, the posture obtained in the current iteration process is used as the posture information of the surgical instrument; otherwise, the current iteration process is updated to the next iteration process, and the current iteration process is continued.
[0079] Specifically, the functions of each module in the surgical instrument posture estimation device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.
[0080] like Figure 5 As shown, based on the above embodiments, a surgical navigation system is provided in an embodiment of the present invention, including: A posture estimation module 51, used to determine the posture information of the surgical instrument based on the surgical instrument posture estimation method provided in the above embodiments; A navigation registration module 52, used for converting the posture information into a patient coordinate system to obtain target posture information of the surgical instrument in the patient coordinate system; The visualization module 53 is used to register the target posture information with the patient's medical image to obtain a registered image and display the registered image.
[0081] Based on the above embodiment, the surgical navigation system provided in the embodiment of the present invention further includes a gridding module for: Meshing the three-dimensional model to determine a target three-dimensional mesh of the surgical instrument; The target three-dimensional mesh is rendered to the current video frame, and the current video frame is displayed.
[0082] On the basis of the above-mentioned embodiment, in the surgical navigation system provided in the embodiment of the present invention, the gridding module is specifically used for: Meshing the three-dimensional model to obtain an initial three-dimensional mesh; The initial three-dimensional mesh is subjected to vertex sparse processing to obtain the target three-dimensional mesh.
[0083] Specifically, the functions of each module in the surgical navigation system provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.
[0084] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the surgical instrument posture estimation method or the surgical navigation method provided in the above-mentioned embodiments.
[0085] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical instrument posture estimation method or surgical navigation method provided in the above-mentioned embodiments.
[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the surgical instrument posture estimation method or surgical navigation method provided in the above-mentioned embodiments.
[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for estimating the posture of a surgical instrument, characterized in that: include: Acquire a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument; Based on the historical posture of the surgical instrument in the previous video frame, projecting the three-dimensional model into the current video frame to obtain a projected image block in the current video frame; Extracting a contour feature map, a foreground probability map, and a boundary probability map of the projected image block, and fusing the contour feature map, the foreground probability map, and the boundary probability map to obtain a fused probability map; Based on the fusion probability map, the position and posture information of the surgical instrument is iteratively solved.
2. The method for estimating the position and posture of a surgical instrument according to claim 1, characterized in that: The step of extracting a contour feature map of the projection image block comprises: Extracting a multi-scale feature map of the projected image block; Based on the corresponding line model of the surgical instrument in the current video frame, interpolation sampling is performed on the multi-scale feature map to obtain the contour feature map.
3. The method for estimating the position and posture of a surgical instrument according to claim 1, characterized in that: The iteratively solving the posture information of the surgical instrument based on the fusion probability map includes: In the current iteration process, based on the posture obtained in the previous iteration process, the three-dimensional contour points of the three-dimensional model are projected onto the fusion probability map to obtain two-dimensional projection points, and the reprojection error is calculated based on the two-dimensional projection points and the adjacent two-dimensional points on the fusion probability map; Determining a likelihood function based on the reprojection error and the fused probability map; For corresponding point pairs between the three-dimensional contour points and the two-dimensional projection points, use the likelihood function to calculate likelihood estimates, and based on the likelihood estimates, update the pose obtained in the previous iteration process to obtain the pose obtained in the current iteration process; When the iteration end condition is met, the posture obtained in the current iteration process is used as the posture information of the surgical instrument; otherwise, the current iteration process is updated to the next iteration process, and the current iteration process is continued.
4. A surgical navigation method, characterized in that: include: Determine the posture information of the surgical instrument based on the surgical instrument posture estimation method according to any one of claims 1 to 3; Converting the position information to a patient coordinate system to obtain target position information of the surgical instrument in the patient coordinate system; The target posture information is registered with the patient's medical image to obtain a registered image, and the registered image is displayed.
5. The surgical navigation method according to claim 4, characterized in that: Also includes: Meshing the three-dimensional model to determine a target three-dimensional mesh of the surgical instrument; The target three-dimensional mesh is rendered to the current video frame, and the current video frame is displayed.
6. The surgical navigation method according to claim 5, characterized in that: The step of meshing the three-dimensional model to determine a target three-dimensional mesh of the surgical instrument includes: Meshing the three-dimensional model to obtain an initial three-dimensional mesh; The initial three-dimensional mesh is subjected to vertex sparse processing to obtain the target three-dimensional mesh.
7. A surgical instrument posture estimation device, characterized in that: include: An acquisition module, used to acquire a current video frame of a doctor operating a surgical instrument and a three-dimensional model of the surgical instrument; A projection module, configured to project the three-dimensional model into the current video frame based on the historical posture of the surgical instrument in the previous video frame to obtain a projected image block in the current video frame; A fusion module, used to extract a contour feature map, a foreground probability map and a boundary probability map of the projected image block, and fuse the contour feature map, the foreground probability map and the boundary probability map to obtain a fused probability map; A solution module is used to iteratively solve the posture information of the surgical instrument based on the fusion probability map.
8. A surgical navigation system, characterized in that: include: A posture estimation module, used to determine the posture information of the surgical instrument based on the surgical instrument posture estimation method according to any one of claims 1 to 3; A navigation registration module, used for converting the posture information into a patient coordinate system to obtain target posture information of the surgical instrument in the patient coordinate system; The visualization module is used to register the target posture information with the patient's medical image to obtain a registered image and display the registered image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the surgical instrument posture estimation method as described in any one of claims 1-3, or the surgical navigation method as described in any one of claims 4-6.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for estimating the posture of a surgical instrument as described in any one of claims 1 to 3 or the method for surgical navigation as described in any one of claims 4 to 6 is implemented.
Citation Information
Patent Citations
3D guide wire tracking method and device based on blood vessel 3D / 2D matching
CN111798451A
Surgical navigation system and method based on three-dimensional model, equipment and medium
CN112155727A
Augmented reality surgical navigation method, device, equipment and medium
CN118319490A
Cited By
Intelligent pose estimation method for maintainability test component based on CAD model
CN120931726A
