Surgical instrument pose dataset production method and device, electronic equipment and storage medium

By fixing reference markers on a rotating positioning plate and using depth cameras and 3D reconstruction technology, a surgical instrument pose dataset is constructed, which solves the problem of insufficient 6DoF pose datasets for surgical instruments in the existing technology, realizes high-precision pose information acquisition, and supports surgical navigation and robot-assisted surgery.

CN119090954BActive Publication Date: 2026-11-10Artificial Intelligence and Robotics Innovation Center of Hong Kong Institute of Innovation, Chinese Academy of Sciences
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410945315.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2026-11-10
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The lack of high-precision six-degree-of-freedom (6DoF) pose datasets for surgical instruments in the current technology, especially the lack of methods for creating datasets for reflective metal tools in the surgical field, makes pose estimation difficult in surgical navigation and robot-assisted surgery.

Method used

By fixing reference markers on a rotating positioning plate, a depth camera is used to acquire scene point cloud videos, perform point cloud matching and surface reconstruction, and combine 3D model data and target pose map to construct a pose dataset for surgical instruments. Simultaneous localization and mapping algorithms and the Levenberg-Marquardt method are used to optimize the pose map and acquire high-precision pose information.

Benefits of technology

It enables the rapid and accurate construction of surgical instrument pose datasets, supports high-precision pose estimation, and is suitable for surgical navigation and robot-assisted surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090954B_ABST
    Figure CN119090954B_ABST
Patent Text Reader

Abstract

The application provides a surgical instrument pose dataset production method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method can construct a target pose graph of a depth camera by fixing a reference marker on a rotating positioning plate, and then, by combining the target pose graph and three-dimensional model data of a surgical instrument, the accurate acquisition of pose information of the surgical instrument can be automatically realized, and a surgical instrument pose dataset can be quickly and accurately constructed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, electronic device, and storage medium for creating surgical instrument pose datasets. Background Technology

[0002] With the increasing use of mixed reality in surgical scenarios, surgical navigation systems, surgeon skill assessment, and robot-assisted surgery are receiving more and more attention. Researchers are increasingly demanding six-degree-of-freedom (6DoF) pose estimation for surgical instruments. For example, tracking the 6DoF pose of surgical instruments is used for navigation registration and medical image registration in surgical navigation, while surgical robots can grasp instruments by estimating their pose in a single frame. All of these require high-precision 6DoF pose datasets of surgical instruments for algorithm training; therefore, creating 6DoF datasets of surgical instruments is particularly important. Summary of the Invention

[0003] This invention provides a method, apparatus, electronic device, and storage medium for creating surgical instrument pose datasets, in order to overcome the deficiencies existing in related technologies.

[0004] This invention provides a method for creating a surgical instrument pose dataset, comprising: Based on a depth camera, a scene point cloud video of a rotating positioning plate is acquired, wherein surgical instruments and reference markers are fixed on the positioning plate. The point cloud images of each frame in the scene point cloud video, except for the starting frame point cloud image, are matched with the starting frame point cloud image to obtain the matching results. Based on the matching results, the target pose map of the depth camera is constructed. For any frame of point cloud image in the scene point cloud video, calculate the normal vector of each point in the point cloud image, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument. Based on the 3D model data and the target pose map, the vertex and pose information of the surgical instrument are determined. Based on the vertex and pose information of the surgical instrument in each frame of the scene point cloud video, and combined with the segmentation mask of the surgical instrument in each frame of the point cloud video, the surgical instrument pose dataset is constructed.

[0005] According to a method for creating a surgical instrument pose dataset provided by the present invention, the reference markers include a plurality of reference markers, which are uniformly fixed on the positioning plate; the positioning plate rotates at a constant speed, and each frame of the scene point cloud video contains at least three reference markers.

[0006] According to a method for creating a surgical instrument pose dataset provided by the present invention, the step of constructing a target pose map of the depth camera based on the matching result includes: Based on the matching results, a simultaneous localization and mapping (SLAM) algorithm is applied to construct the initial pose map of the depth camera. Based on a 3D data processing library, the initial pose map is optimized to obtain the target pose map.

[0007] According to a method for creating a surgical instrument pose dataset provided by the present invention, the step of optimizing the initial pose map based on a 3D data processing library to obtain the target pose map includes: Based on the aforementioned 3D data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial pose map, thereby obtaining the target pose map.

[0008] According to a method for creating a surgical instrument pose dataset provided by the present invention, the step of performing surface reconstruction based on the normal vectors of each point to obtain the three-dimensional model data of the surgical instrument includes: Based on the normal vectors of each point, a three-dimensional geometric processing system is applied to perform surface reconstruction and obtain the initial reconstruction result; Delete the point cloud contained in any frame point cloud image in the initial reconstruction result to obtain the mesh data of the surgical instrument; The mesh data is flattened and hole-filling operations are performed to obtain the three-dimensional model data.

[0009] According to a method for creating a surgical instrument pose dataset provided by the present invention, the step of calculating the normal vector of each point in any frame point cloud image includes: Based on a three-dimensional geometric processing system, the foreground point cloud and background point cloud in any frame point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud. Calculate the normal vector of each point in the foreground cloud.

[0010] The present invention also provides a device for generating surgical instrument pose datasets, comprising: The video acquisition module is used to acquire scene point cloud video of a rotating positioning plate based on a depth camera. Surgical instruments and reference markers are fixed on the positioning plate. The matching module is used to match each frame point cloud image in the scene point cloud video except for the starting frame point cloud image with the starting frame point cloud image to obtain the matching result, and to construct the target pose map of the depth camera based on the matching result. The reconstruction module is used to calculate the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument. The construction module is used to determine the vertex and pose information of the surgical instrument based on the 3D model data and the target pose map, and to construct the surgical instrument pose dataset based on the vertex and pose information of the surgical instrument in each frame of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud video.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the surgical instrument pose dataset creation method as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the surgical instrument pose dataset creation method as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the surgical instrument pose dataset creation method as described above.

[0014] The present invention provides a method, apparatus, electronic device and storage medium for creating surgical instrument pose datasets. The method can construct a target pose map of a depth camera by fixing a reference marker on a rotating positioning plate. Then, by combining the target pose map with the three-dimensional model data of the surgical instruments, the pose information of the surgical instruments can be accurately acquired automatically, and thus a surgical instrument pose dataset can be constructed quickly and accurately. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts illustrating the method for creating a surgical instrument pose dataset provided by the present invention.

[0017] Figure 2 This is the second flowchart illustrating the method for creating a surgical instrument pose dataset provided by this invention.

[0018] Figure 3 This is a schematic diagram of the surgical instrument pose dataset generation device provided by the present invention.

[0019] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Since there is no existing method for creating a 6DoF dataset for surgical instruments, and methods such as LineMod, YCB-Video, and T-Less are designed for everyday objects and are not suitable for surgical instruments, especially reflective metal tools used in surgery, it is difficult to directly obtain their 6DoF datasets using these methods. Therefore, this invention provides a method for creating a surgical instrument pose dataset.

[0022] Figure 1 This is a flowchart illustrating a method for creating a surgical instrument pose dataset according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes: S1, Based on a depth camera, acquire scene point cloud video of a rotating positioning plate, on which surgical instruments and reference markers are fixed; S2, match the point cloud images of each frame in the scene point cloud video except for the starting frame point cloud image with the starting frame point cloud image to obtain the matching result, and construct the target pose map of the depth camera based on the matching result. S3. For any frame of point cloud image in the scene point cloud video, calculate the normal vector of each point in the frame of point cloud image, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument. S4. Based on the 3D model data and the target pose map, determine the vertex and pose information of the surgical instrument, and construct the surgical instrument pose dataset based on the vertex and pose information of the surgical instrument in each frame of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud video.

[0023] Specifically, the surgical instrument pose dataset creation method provided in this embodiment of the invention is executed by a surgical instrument pose dataset creation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0024] First, perform step S1 to acquire scene point cloud video of the rotating positioning plate using a depth camera. This depth camera can be a RealSense camera. The positioning plate can be a wooden board, which can be placed on a rotating table or turntable.

[0025] The positioning plate is fixed with surgical instruments and reference markers. The surgical instruments can be reflective metal tools used in the surgical field, and they can have a slender structure, such as ultra-dexterous handheld flexible minimally invasive surgical instruments. The surgical instruments are grayish-white in color, while the positioning plate can be a different color from the surgical instruments.

[0026] Because surgical instruments are flat, small objects such as transparent tape can be used for support under the surgical instruments in order to better separate the surgical instruments and positioning plates when processing point clouds later.

[0027] The reference marker can be a black and white image of ArUco tags, which can be affixed around the positioning plate to ensure that the reference marker does not shift when the positioning plate rotates.

[0028] Scene point cloud video acquired by a depth camera can include multiple frames of point cloud images. Each frame can contain point clouds of surgical instruments and reference markers. Each frame can be an RGB color image.

[0029] Then, step S2 is executed, matching each frame of the scene point cloud video (excluding the starting frame point cloud image) with the starting frame point cloud image to obtain matching results. Here, the rotation angle and translation transformation of the positioning plate in different frame point cloud images can be determined by the texture features on the reference marker. Texture feature matching or point cloud matching can be used to match each frame of the point cloud image (excluding the starting frame point cloud image) with the starting frame point cloud image to obtain matching results. The matching results can include matching point pairs with a similarity distance less than or equal to a preset distance, as well as the relative pose information of the depth camera determined by each matching point pair. The preset distance can be set as needed and is not specifically limited here. The relative pose information of the depth camera can include the rotation angle and translation transformation of the depth camera.

[0030] Using the obtained matching results, a target pose map of the depth camera can be constructed. The nodes in this target pose map represent the pose of the depth camera at different times and locations, and the edges represent the relative transformations and confidence levels between adjacent nodes.

[0031] Next, step S3 is executed. For any frame of point cloud image in the scene point cloud video, the normal vector of each point in the frame is calculated, and the surface is reconstructed using the normal vectors of each point to obtain the three-dimensional model data of the surgical instrument. This three-dimensional model data can be the mesh data of the surgical instrument.

[0032] Finally, step S4 is executed, using the 3D model data of the surgical instrument and the target pose map to determine the vertices and pose information of the surgical instrument. The vertex coordinates of the surgical instrument can be calculated from the 3D model data, and the pose and vertex registration of each frame of the point cloud image can be completed using the target pose map, thus determining the pose information of the surgical instrument. Here, the pose information of the surgical instrument can be 6DoF pose information.

[0033] Furthermore, by utilizing the vertex and pose information of the surgical instruments in each frame of the scene point cloud video, and combining this with the segmentation mask of the surgical instruments in each frame of the point cloud video, a surgical instrument pose dataset can be constructed. This segmentation mask can be obtained by segmenting each frame of the point cloud image.

[0034] This surgical instrument pose dataset contains the vertex and pose information of the surgical instruments in each frame of the scene point cloud video, as well as the segmentation mask of the surgical instruments.

[0035] Afterwards, the surgical instrument pose dataset can be stored in a specified path to facilitate subsequent model training and testing on the surgical instrument pose dataset.

[0036] The method for creating a surgical instrument pose dataset provided in this embodiment of the invention first acquires a scene point cloud video of a rotating positioning plate using a depth camera. The positioning plate is fixed with surgical instruments and reference markers. Then, each frame of the scene point cloud video, except for the initial frame, is matched with the initial frame point cloud image to obtain the matching result. Based on the matching result, a target pose map of the depth camera is constructed. Subsequently, for any frame of the scene point cloud video, the normal vector of each point in that frame is calculated, and surface reconstruction is performed based on the normal vectors of each point to obtain the three-dimensional model data of the surgical instrument. Finally, based on the three-dimensional model data and the target pose map, the vertex and pose information of the surgical instrument are determined. Based on the vertex and pose information of the surgical instrument in each frame of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame, a surgical instrument pose dataset is constructed. This method can construct a target pose map of the depth camera by fixing a reference marker on a rotating positioning plate. Then, by combining the target pose map with the 3D model data of the surgical instruments, the pose information of the surgical instruments can be accurately acquired automatically, and a surgical instrument pose dataset can be constructed quickly and accurately.

[0037] Based on the above embodiments, the reference markers may include multiple ones, for example, up to 10, and each reference marker may be evenly fixed around the positioning plate.

[0038] The positioning plate can rotate at a constant speed, and each frame of the scene point cloud video can contain at least three reference markers, which can improve the accuracy of subsequent matching and reduce the difficulty of matching.

[0039] Based on the above embodiments, constructing the target pose map of the depth camera according to the matching result includes: Based on the matching results, a simultaneous localization and mapping (SLAM) algorithm is applied to construct the initial pose map of the depth camera. Based on a 3D data processing library, the initial pose map is optimized to obtain the target pose map.

[0040] Specifically, when constructing the target pose map of the depth camera, the initial pose map of the depth camera can be constructed by first using the matching results and applying the Simultaneous Localization and Mapping (SLAM) algorithm.

[0041] Subsequently, the initial pose map can be optimized using a 3D data processing library (open3d) to obtain the target pose map, which can improve the accuracy, stability and consistency of the target pose map.

[0042] In this embodiment of the invention, by processing the initial attitude map, the accuracy and robustness of the target attitude map can be effectively improved. Even in an environment that stores a large amount of noise and uncertain information, a target attitude map with high accuracy and high robustness can still be obtained.

[0043] Based on the above embodiments, the step of optimizing the initial pose map using a 3D data processing library to obtain the target pose map includes: Based on the aforementioned 3D data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial pose map, thereby obtaining the target pose map.

[0044] Specifically, when optimizing the initial attitude map, a 3D data processing library can be used to apply the Levenberg-Marquardt algorithm to optimize the initial attitude map and obtain the target attitude map.

[0045] Here, an optimizer can be constructed first, specifying the preset distance for matching, the edge pruning threshold, and the reference point. The preset distance can be set to 1.5 times the size of the point cloud voxels. The edge pruning threshold can be set to 0.25, and the reference point is the initial frame point cloud image.

[0046] The initial pose graph is then optimized twice. The first optimization considers all edge cases to optimize the pose of the initial pose graph and tries to distinguish misalignments between uncertain edges. Among the edges considered are those with high uncertainty, which helps to initially correct most of the deviations in the initial pose graph and prunes uncertain or erroneous edges as much as possible.

[0047] The second optimization will produce a tighter global alignment without any uncertain or erroneous edges. This second optimization focuses on edges with higher weights, aiming to refine pose estimation and improve the accuracy of global alignment.

[0048] Based on the above embodiments, calculating the normal vector of each point in any frame point cloud image includes: Based on a three-dimensional geometric processing system, the foreground point cloud and background point cloud in any frame point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud. Calculate the normal vector of each point in the foreground cloud.

[0049] Based on the above embodiments, the step of performing surface reconstruction based on the normal vectors of each point to obtain the three-dimensional model data of the surgical instrument includes: Based on the normal vectors of each point, a three-dimensional geometric processing system is applied to perform surface reconstruction and obtain the initial reconstruction result; Delete the point cloud contained in any frame point cloud image in the initial reconstruction result to obtain the mesh data of the surgical instrument; The mesh data is flattened and hole-filling operations are performed to obtain the three-dimensional model data.

[0050] Specifically, when determining the 3D model data, a 3D geometry processing system (Meshlab) can be used to segment the foreground and background point clouds in each frame of the point cloud image, and the background point cloud can be cropped to obtain the foreground point cloud, which is the point cloud of the surgical instruments. By cropping the background point cloud, the subsequent computational load can be reduced, and only the normal vectors of each point in the foreground point cloud need to be calculated.

[0051] Subsequently, the surface can be reconstructed using the normal vectors of each point and a three-dimensional geometric processing system to obtain the initial reconstruction result.

[0052] Subsequently, the point clouds contained in each frame of the initial reconstruction result can be deleted to obtain the mesh data of the surgical instruments, thereby reducing the useless point clouds in the initial reconstruction result and further reducing the amount of subsequent computation.

[0053] Finally, the mesh data of the surgical instruments can be flattened and filled with holes to obtain the complete three-dimensional model data of the surgical instruments, which facilitates the subsequent definition of the size of the surgical instruments and the labeling of vertex coordinates.

[0054] In summary, such as Figure 2 As shown, the method for creating a surgical instrument pose dataset provided in this embodiment of the invention includes: The first step is to make a positioning plate and fix the surgical instruments and reference markers on the positioning plate; The second step is to use a depth camera to acquire scene point cloud video of the rotating positioning plate; The third step is to construct the initial pose map of the depth camera; The fourth step is to optimize the initial attitude map to obtain the target attitude map; The fifth step involves using a 3D geometry processing system to process each frame of point cloud image to obtain 3D model data, and combining this data with the target pose map to determine the vertices and pose information of the surgical instruments. The sixth step involves constructing a surgical instrument pose dataset based on the vertex and pose information of the surgical instruments, combined with the segmentation mask of the surgical instruments in each frame of point cloud image.

[0055] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides a surgical instrument pose dataset creation device, comprising: Video acquisition module 31 is used to acquire scene point cloud video of a rotating positioning plate based on a depth camera. Surgical instruments and reference markers are fixed on the positioning plate. The matching module 32 is used to match each frame point cloud image in the scene point cloud video except for the starting frame point cloud image with the starting frame point cloud image to obtain a matching result, and to construct the target pose map of the depth camera based on the matching result. The reconstruction module 33 is used to calculate the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument. The construction module 34 is used to determine the vertex and pose information of the surgical instrument based on the three-dimensional model data and the target pose map, and to construct the surgical instrument pose dataset based on the vertex and pose information of the surgical instrument in each frame of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud video.

[0056] Based on the above embodiments, the surgical instrument pose dataset generation device provided in this embodiment of the invention includes multiple reference markers, which are uniformly fixed on the positioning plate; the positioning plate rotates at a constant speed, and each frame of the scene point cloud video contains at least three reference markers.

[0057] Based on the above embodiments, the surgical instrument pose dataset generation device provided in this embodiment of the invention, wherein the matching module is specifically used for: Based on the matching results, a simultaneous localization and mapping (SLAM) algorithm is applied to construct the initial pose map of the depth camera. Based on a 3D data processing library, the initial pose map is optimized to obtain the target pose map.

[0058] Based on the above embodiments, the surgical instrument pose dataset generation device provided in this embodiment of the invention, wherein the matching module is further specifically used for: Based on the aforementioned 3D data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial pose map, thereby obtaining the target pose map.

[0059] Based on the above embodiments, the surgical instrument pose dataset creation device provided in this embodiment of the invention, wherein the reconstruction module is specifically used for: Based on the normal vectors of each point, a three-dimensional geometric processing system is applied to perform surface reconstruction and obtain the initial reconstruction result; Delete the point cloud contained in any frame point cloud image in the initial reconstruction result to obtain the mesh data of the surgical instrument; The mesh data is flattened and hole-filling operations are performed to obtain the three-dimensional model data.

[0060] Based on the above embodiments, the surgical instrument pose dataset generation device provided in this embodiment of the invention, wherein the reconstruction module is further specifically used for: Based on a three-dimensional geometric processing system, the foreground point cloud and background point cloud in any frame point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud. Calculate the normal vector of each point in the foreground cloud.

[0061] Specifically, the functions of each module in the surgical instrument pose dataset creation device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0062] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the surgical instrument pose dataset creation method provided in the above embodiments.

[0063] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the surgical instrument pose dataset creation method provided in the above embodiments.

[0065] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the surgical instrument pose dataset creation method provided in the above embodiments.

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0067] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for creating a surgical instrument pose dataset, characterized in that, include: Based on a depth camera, a scene point cloud video of a rotating positioning plate is acquired, wherein surgical instruments and reference markers are fixed on the positioning plate. The point cloud images of each frame in the scene point cloud video, except for the starting frame point cloud image, are matched with the starting frame point cloud image to obtain the matching results. Based on the matching results, the target pose map of the depth camera is constructed. For any frame of point cloud image in the scene point cloud video, calculate the normal vector of each point in the point cloud image, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument. Based on the 3D model data and the target pose map, the vertex and pose information of the surgical instrument are determined. Based on the vertex and pose information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, and combined with the segmentation mask of the surgical instrument in each frame of the point cloud image, the surgical instrument pose dataset is constructed. The reference markers include multiple ones, which are uniformly fixed on the positioning plate; the positioning plate rotates at a constant speed, and each frame of the scene point cloud video contains at least three of the reference markers. The step of constructing the target pose map of the depth camera based on the matching result includes: Based on the matching results, a simultaneous localization and mapping (SLAM) algorithm is applied to construct the initial pose map of the depth camera. Based on a 3D data processing library, the initial pose map is optimized to obtain the target pose map; The surface reconstruction based on the normal vectors of each point yields the three-dimensional model data of the surgical instrument, including: Based on the normal vectors of each point, a three-dimensional geometric processing system is applied to perform surface reconstruction and obtain the initial reconstruction result; Delete the point cloud contained in any frame point cloud image in the initial reconstruction result to obtain the mesh data of the surgical instrument; The mesh data is flattened and hole-filling is performed to obtain the three-dimensional model data; The calculation of the normal vector of each point in any frame point cloud image includes: Based on a three-dimensional geometric processing system, the foreground point cloud and background point cloud in any frame point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud. Calculate the normal vector of each point in the foreground cloud.

2. The method for creating a surgical instrument pose dataset according to claim 1, characterized in that, The optimization of the initial pose map based on a 3D data processing library to obtain the target pose map includes: Based on the aforementioned 3D data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial pose map, thereby obtaining the target pose map.

3. A device for generating a surgical instrument pose dataset, characterized in that, include: The video acquisition module is used to acquire scene point cloud video of a rotating positioning plate based on a depth camera. Surgical instruments and reference markers are fixed on the positioning plate. The matching module is used to match each frame point cloud image in the scene point cloud video except for the starting frame point cloud image with the starting frame point cloud image to obtain the matching result, and to construct the target pose map of the depth camera based on the matching result. The reconstruction module is used to calculate the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument. The construction module is used to determine the vertex and pose information of the surgical instrument based on the 3D model data and the target pose map, and to construct the surgical instrument pose dataset based on the vertex and pose information of the surgical instrument in each frame of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud video. The reference markers include multiple ones, which are uniformly fixed on the positioning plate; the positioning plate rotates at a constant speed, and each frame of the scene point cloud video contains at least three of the reference markers. The step of constructing the target pose map of the depth camera based on the matching result includes: Based on the matching results, a simultaneous localization and mapping (SLAM) algorithm is applied to construct the initial pose map of the depth camera. Based on a 3D data processing library, the initial pose map is optimized to obtain the target pose map; The surface reconstruction based on the normal vectors of each point yields the three-dimensional model data of the surgical instrument, including: Based on the normal vectors of each point, a three-dimensional geometric processing system is applied to perform surface reconstruction and obtain the initial reconstruction result; Delete the point cloud contained in any frame point cloud image in the initial reconstruction result to obtain the mesh data of the surgical instrument; The mesh data is flattened and hole-filling is performed to obtain the three-dimensional model data; The calculation of the normal vector of each point in any frame point cloud image includes: Based on a three-dimensional geometric processing system, the foreground point cloud and background point cloud in any frame point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud. Calculate the normal vector of each point in the foreground cloud.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for creating surgical instrument pose datasets as described in any one of claims 1-2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for creating surgical instrument pose datasets as described in any one of claims 1-2.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for creating surgical instrument pose datasets as described in any one of claims 1-2.

Citation Information

Patent Citations

  • Binocular vision-based reference mark pose estimation method

    CN116091625A

  • Real-time surgical instrument pose estimation method and system based on deep learning and stereoscopic vision

    CN117011381A