Surgical instrument pose data set manufacturing method and device, electronic equipment and storage medium

Through the cooperation of depth cameras and benchmark markers, synchronous positioning and map algorithms and Levinberg-Marquardt method optimization, the problem of difficult to produce 6DoF position data sets in the existing technology is solved, and the operation is realized Accurate acquisition and rapid construction of instrument position information.

CN119090954AActive Publication Date: 2024-12-06Artificial Intelligence and Robotics Innovation Center of Hong Kong Institute of Innovation, Chinese Academy of Sciences
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410945315.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-06
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The prior art lacks effective methods to produce high-precision six-degree-of-freedom (6DoF) position data sets for surgical instruments, especially for reflective metal tools in the field of surgery. It is difficult to directly obtain their 6DoF data sets using the above data set production methods.

Method used

By obtaining scene point cloud videos of rotating positioning plates based on the depth camera, using benchmark markers for matching and surface reconstruction, the target attitude map and three-dimensional model data are constructed, combining synchronous positioning with map algorithm and Levinberg-Marquardt method Optimize, determine the vertex and posture information of the surgical instrument, and construct the posture data set of the surgical instrument.

Benefits of technology

It realizes the accurate acquisition of surgical instrument position information, and quickly and accurately constructs the surgical instrument position data set, suitable for surgical navigation and robot-assisted surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090954A_ABST
    Figure CN119090954A_ABST
Patent Text Reader

Abstract

The invention provides a surgical instrument pose data set manufacturing method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision. According to the method, a target pose graph of a depth camera can be constructed by fixing a reference marker on a rotating positioning plate; and in combination with the target pose map and the three-dimensional model data of the surgical instrument, the pose information of the surgical instrument can be automatically and accurately acquired, so that the pose data set of the surgical instrument can be quickly and accurately constructed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, electronic device and storage medium for preparing a surgical instrument posture data set. Background Art

[0002] With the use of mixed reality in surgical scenarios, surgical navigation systems, doctor surgical skill assessment, and robot-assisted surgery have received more and more attention, and R&D personnel have more and more demands for six-degree-of-freedom (6DoF) pose estimation of surgical instruments. For example, navigation registration and medical image registration in surgical navigation can be completed by tracking the 6DoF pose of surgical instruments, and surgical robots can grasp surgical instruments by estimating the pose of single-frame instruments. All of these require a high-precision 6DoF pose dataset of surgical instruments to complete algorithm training, so how to create a 6DoF dataset for surgical instruments is particularly important. Summary of the invention

[0003] The present invention provides a method, device, electronic device and storage medium for preparing a surgical instrument posture data set, so as to solve the defects existing in the related art.

[0004] The present invention provides a method for preparing a surgical instrument posture data set, comprising: Based on the depth camera, a scene point cloud video of a rotating positioning plate is obtained, wherein the positioning plate is fixed with surgical instruments and reference markers; Matching each frame point cloud image except the starting frame point cloud image in the scene point cloud video with the starting frame point cloud image to obtain a matching result, and constructing a target posture graph of the depth camera according to the matching result; For any frame of point cloud image in the scene point cloud video, calculate the normal vector of each point in the frame of point cloud image, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument; Based on the three-dimensional model data and the target posture graph, the vertices and posture information of the surgical instrument are determined, and based on the vertices and posture information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image, the surgical instrument posture dataset is constructed.

[0005] According to a method for producing a surgical instrument posture data set provided by the present invention, the reference marker includes multiple reference markers, and the multiple reference markers are evenly fixed on the positioning plate; the positioning plate rotates at a uniform speed, and each frame of the point cloud image of the scene point cloud video contains at least three of the reference markers.

[0006] According to a method for preparing a surgical instrument posture data set provided by the present invention, constructing a target posture graph of the depth camera according to the matching result includes: Based on the matching results, applying a synchronous positioning and mapping algorithm to construct an initial posture graph of the depth camera; Based on a three-dimensional data processing library, the initial posture graph is optimized to obtain the target posture graph.

[0007] According to a method for preparing a surgical instrument posture data set provided by the present invention, the initial posture graph is optimized based on a three-dimensional data processing library to obtain the target posture graph, including: Based on the three-dimensional data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial posture graph to obtain the target posture graph.

[0008] According to a method for preparing a surgical instrument posture data set provided by the present invention, the surface reconstruction is performed based on the normal vectors of the points to obtain the three-dimensional model data of the surgical instrument, including: Based on the normal vectors of the points, a three-dimensional geometric processing system is applied to perform surface reconstruction to obtain an initial reconstruction result; Deleting the point cloud included in any frame of the point cloud image in the initial reconstruction result to obtain grid data of the surgical instrument; The mesh data is flattened and hole-filled to obtain the three-dimensional model data.

[0009] According to a method for preparing a surgical instrument posture data set provided by the present invention, the calculation of the normal vector of each point in any frame of the point cloud image includes: Based on a three-dimensional geometric processing system, the foreground point cloud and the background point cloud in any frame of the point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud; The normal vector of each point in the foreground point cloud is calculated.

[0010] The present invention also provides a surgical instrument posture data set preparation device, comprising: A video acquisition module, used to acquire a scene point cloud video of a rotating positioning plate based on a depth camera, wherein surgical instruments and reference markers are fixed on the positioning plate; A matching module, used to match each frame point cloud image in the scene point cloud video except the starting frame point cloud image with the starting frame point cloud image to obtain a matching result, and construct a target posture graph of the depth camera according to the matching result; A reconstruction module, for calculating the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and performing surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument; A construction module is used to determine the vertices and posture information of the surgical instrument based on the three-dimensional model data and the target posture graph, and to construct the surgical instrument posture data set based on the vertices and posture information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for producing a surgical instrument posture data set as described in any one of the above is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for producing a surgical instrument posture data set as described in any one of the above.

[0013] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the methods for producing a surgical instrument posture data set as described above.

[0014] The present invention provides a method, device, electronic device and storage medium for producing a surgical instrument posture data set. The method can construct a target posture map of a depth camera by fixing a reference marker on a rotating positioning plate, and then combine the target posture map with the three-dimensional model data of the surgical instrument to automatically and accurately obtain the posture information of the surgical instrument, thereby quickly and accurately constructing a surgical instrument posture data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is one of the flow charts of the method for preparing a surgical instrument posture data set provided by the present invention.

[0017] Figure 2 This is the second flow chart of the method for producing a surgical instrument posture data set provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of a surgical instrument posture data set production device provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Since there is no related method for making a 6DoF dataset for surgical instruments in the prior art, and the dataset making methods such as LineMod, YCB-Video, and T-Less are all for objects in life and are not suitable for surgical instruments, especially for reflective metal tools in the surgical field, it is difficult to directly obtain their 6DoF datasets using the above dataset making methods. Based on this, a method for making a surgical instrument pose dataset is provided in an embodiment of the present invention.

[0022] Figure 1 FIG. 1 is a flow chart of a method for preparing a surgical instrument posture data set provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S1, based on a depth camera, obtaining a scene point cloud video of a rotating positioning plate, wherein surgical instruments and reference markers are fixed on the positioning plate; S2, matching each frame point cloud image in the scene point cloud video except the starting frame point cloud image with the starting frame point cloud image to obtain a matching result, and constructing a target posture graph of the depth camera according to the matching result; S3, for any frame of point cloud image in the scene point cloud video, calculating the normal vector of each point in the any frame of point cloud image, and performing surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument; S4, based on the three-dimensional model data and the target posture graph, determine the vertices and posture information of the surgical instrument, and based on the vertices and posture information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image, construct the surgical instrument posture data set.

[0023] Specifically, the surgical instrument posture dataset production method provided in the embodiment of the present invention is executed by a surgical instrument posture dataset production device, which can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0024] First, step S1 is performed to obtain a scene point cloud video of a rotating positioning plate using a depth camera. The depth camera may be a RealSense. The positioning plate may be a wooden board and may be placed on a rotating table or turntable.

[0025] The positioning plate is fixed with surgical instruments and reference markers. The surgical instruments may be reflective metal tools in the surgical field, which may have a slender structure, such as ultra-dexterous handheld bendable minimally invasive surgical instruments. The color of the surgical instruments is off-white, and the positioning plate and the surgical instruments may be different in color.

[0026] Since the surgical instrument is flat, in order to better separate the surgical instrument and the positioning plate during subsequent point cloud processing, small objects such as transparent tape can be used as support under the surgical instrument.

[0027] The fiducial markers can be black and white images of ArUco tags, which can be attached around the positioning plate to ensure that the fiducial markers do not shift when the positioning plate is rotated.

[0028] The scene point cloud video acquired by the depth camera may include multiple frames of point cloud images, each of which may include the point cloud of the surgical instrument and the point cloud of the reference marker. Each frame of the point cloud image may be an RGB color image.

[0029] Then, step S2 is executed to match each frame of point cloud image in the scene point cloud video except the starting frame point cloud image with the starting frame point cloud image to obtain a matching result. Here, the rotation angle and translation transformation of the positioning plate in different frame point cloud images can be determined by the texture features on the benchmark marker. Texture feature matching or point cloud matching can be used to match each frame of point cloud image except the starting frame point cloud image with the starting frame point cloud image to obtain a matching result. The matching result may include each matching point pair whose similarity distance is less than or equal to the preset distance, and the relative posture information of the depth camera determined by each matching point pair. The preset distance can be set as needed and is not specifically limited here. The relative posture information of the depth camera may include the rotation angle and translation transformation of the depth camera.

[0030] Using the matching results, we can construct a target posture graph of the depth camera. The nodes in the target posture graph represent the postures of the depth camera at different times and locations, and the edges represent the relative transformation and confidence between adjacent nodes.

[0031] Then, step S3 is executed to calculate the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and use the normal vector of each point to perform surface reconstruction, so as to obtain the three-dimensional model data of the surgical instrument. The three-dimensional model data can be the mesh data of the surgical instrument.

[0032] Finally, step S4 is performed to determine the vertices and posture information of the surgical instrument using the three-dimensional model data of the surgical instrument and the target posture graph. The vertex coordinates of the surgical instrument can be calculated using the three-dimensional model data of the surgical instrument, and the posture and vertex registration of each frame of the point cloud image can be completed through the target posture graph to determine the posture information of the surgical instrument. Here, the posture information of the surgical instrument can be 6DoF posture information.

[0033] Furthermore, the surgical instrument pose data set can be constructed by using the vertices and pose information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image. The segmentation mask can be obtained by segmenting each frame of the point cloud image.

[0034] The surgical instrument pose data set contains the vertices, pose information and segmentation masks of the surgical instruments in each frame of the point cloud image of the scene point cloud video.

[0035] Thereafter, the surgical instrument posture dataset can be stored in a specified path to facilitate subsequent model training and testing on the surgical instrument posture dataset.

[0036] The method for producing a surgical instrument pose data set provided in an embodiment of the present invention first obtains a scene point cloud video of a rotating positioning plate based on a depth camera, and the positioning plate is fixed with surgical instruments and reference markers; then, each frame of point cloud image except the starting frame point cloud image in the scene point cloud video is matched with the starting frame point cloud image to obtain a matching result, and a target pose graph of the depth camera is constructed based on the matching result; thereafter, for any frame of point cloud image in the scene point cloud video, the normal vector of each point in any frame of point cloud image is calculated, and based on the normal vector of each point, surface reconstruction is performed to obtain three-dimensional model data of the surgical instrument; finally, based on the three-dimensional model data and the target pose graph, the vertices and pose information of the surgical instrument are determined, and based on the vertices and pose information of the surgical instrument in each frame of point cloud image in the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of point cloud image, a surgical instrument pose data set is constructed. This method can construct a target posture map of the depth camera by fixing the reference marker on a rotating positioning plate, and then combine the target posture map with the three-dimensional model data of the surgical instrument to automatically and accurately obtain the posture information of the surgical instrument, thereby quickly and accurately constructing a surgical instrument posture dataset.

[0037] On the basis of the above-mentioned embodiment, the reference marker may include a plurality, for example, 10, and each reference marker may be evenly fixed around the positioning plate.

[0038] The positioning plate can rotate at a constant speed, and each frame of the point cloud image of the scene point cloud video can contain at least three benchmark markers, which can improve the accuracy of subsequent matching and reduce the difficulty of matching.

[0039] On the basis of the above embodiment, constructing a target posture graph of the depth camera according to the matching result includes: Based on the matching results, applying a synchronous positioning and mapping algorithm to construct an initial posture graph of the depth camera; Based on a three-dimensional data processing library, the initial posture graph is optimized to obtain the target posture graph.

[0040] Specifically, when constructing a target posture graph of a depth camera, the matching result can be firstly used to apply a Simultaneous Localization and Mapping (SLAM) algorithm to construct an initial posture graph of the depth camera.

[0041] Afterwards, using a three-dimensional data processing library (open3d), the initial pose graph can be optimized to obtain a target pose graph, which can improve the accuracy, stability, and consistency of the target pose graph.

[0042] In the embodiment of the present invention, the accuracy and robustness of the target posture graph can be effectively improved by processing the initial posture graph, and a target posture graph with high accuracy and high robustness can still be obtained even in an environment storing a large amount of noise and uncertain information.

[0043] On the basis of the above embodiment, the initial posture graph is optimized based on the three-dimensional data processing library to obtain the target posture graph, including: Based on the three-dimensional data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial posture graph to obtain the target posture graph.

[0044] Specifically, when optimizing the initial posture graph, the three-dimensional data processing library can be used to apply the Levenberg-Marquardt algorithm to optimize the initial posture graph to obtain the target posture graph.

[0045] Here, you can first construct an optimizer and specify the preset distance, edge pruning threshold, reference point, etc. during matching. The preset distance can be set to 1.5 times the size of the point cloud voxel. The edge pruning threshold can be set to 0.25, and the reference point is the initial frame point cloud image.

[0046] Then the initial pose graph is optimized twice. The first optimization will optimize the pose of the initial pose graph by considering all edges and try to distinguish the wrong alignment between uncertain edges. Among them, all edges considered include those with high uncertainty, which helps to preliminarily correct most of the deviations in the initial pose graph and prune uncertain or wrong edges as much as possible.

[0047] The second optimization will produce a tighter global alignment without uncertain or erroneous edges. The second optimization focuses on edges with higher weights to refine the pose estimate and improve the accuracy of the global alignment.

[0048] On the basis of the above embodiment, the step of calculating the normal vector of each point in any frame of the point cloud image includes: Based on a three-dimensional geometric processing system, the foreground point cloud and the background point cloud in any frame of the point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud; The normal vector of each point in the foreground point cloud is calculated.

[0049] On the basis of the above embodiment, the surface reconstruction is performed based on the normal vectors of the points to obtain the three-dimensional model data of the surgical instrument, including: Based on the normal vectors of the points, a three-dimensional geometric processing system is applied to perform surface reconstruction to obtain an initial reconstruction result; Deleting the point cloud included in any frame of the point cloud image in the initial reconstruction result to obtain grid data of the surgical instrument; The mesh data is flattened and hole-filled to obtain the three-dimensional model data.

[0050] Specifically, when determining the 3D model data, the foreground point cloud and background point cloud in each frame of the point cloud image can be segmented based on the 3D geometry processing system (Meshlab), and the background point cloud can be cropped to obtain the foreground point cloud, which is the point cloud of the surgical instrument. By cropping the background point cloud, the subsequent calculation amount can be reduced, and only the normal vector of each point in the foreground point cloud needs to be calculated.

[0051] Afterwards, the normal vector of each point can be used to apply a three-dimensional geometry processing system to perform surface reconstruction and obtain an initial reconstruction result.

[0052] Thereafter, the point cloud contained in each frame of the point cloud image in the initial reconstruction result can be deleted to obtain the mesh data of the surgical instrument, so as to reduce the useless point cloud of the initial reconstruction result, which can further reduce the subsequent calculation amount.

[0053] Finally, the mesh data of the surgical instrument can be flattened and the holes can be filled to obtain the complete three-dimensional model data of the surgical instrument, which is convenient for the subsequent definition of the size of the surgical instrument and the marking of vertex coordinates.

[0054] In summary, if Figure 2 As shown, the method for preparing a surgical instrument posture data set provided in an embodiment of the present invention includes: The first step is to make a positioning plate and fix the surgical instruments and reference markers on the positioning plate; The second step is to obtain the scene point cloud video of the rotating positioning plate through the depth camera; The third step is to construct the initial posture graph of the depth camera; The fourth step is to optimize the initial posture graph to obtain the target posture graph; The fifth step is to use a 3D geometry processing system to process each frame of the point cloud image to obtain 3D model data, and combine it with the target posture graph to determine the vertices and posture information of the surgical instrument; The sixth step is to construct a surgical instrument pose dataset based on the vertices and pose information of the surgical instrument and the segmentation mask of the surgical instrument in each frame of the point cloud image.

[0055] like Figure 3 As shown, based on the above embodiment, an embodiment of the present invention provides a surgical instrument posture data set production device, including: A video acquisition module 31 is used to acquire a scene point cloud video of a rotating positioning plate based on a depth camera, wherein surgical instruments and reference markers are fixed on the positioning plate; A matching module 32 is used to match each frame point cloud image in the scene point cloud video except the starting frame point cloud image with the starting frame point cloud image to obtain a matching result, and construct a target posture graph of the depth camera according to the matching result; A reconstruction module 33 is used to calculate the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument; The construction module 34 is used to determine the vertices and posture information of the surgical instrument based on the three-dimensional model data and the target posture graph, and to construct the surgical instrument posture data set based on the vertices and posture information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image.

[0056] On the basis of the above-mentioned embodiments, the surgical instrument posture data set production device provided in the embodiments of the present invention comprises a plurality of reference markers, and the plurality of reference markers are evenly fixed on the positioning plate; the positioning plate rotates at a uniform speed, and each frame of the point cloud image of the scene point cloud video contains at least three of the reference markers.

[0057] On the basis of the above-mentioned embodiment, in the apparatus for preparing a surgical instrument posture data set provided in the embodiment of the present invention, the matching module is specifically used for: Based on the matching results, applying a synchronous positioning and mapping algorithm to construct an initial posture graph of the depth camera; Based on a three-dimensional data processing library, the initial posture graph is optimized to obtain the target posture graph.

[0058] On the basis of the above-mentioned embodiment, in the apparatus for preparing a surgical instrument posture data set provided in the embodiment of the present invention, the matching module is further specifically used for: Based on the three-dimensional data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial posture graph to obtain the target posture graph.

[0059] On the basis of the above-mentioned embodiment, in the apparatus for preparing a surgical instrument posture data set provided in the embodiment of the present invention, the reconstruction module is specifically used for: Based on the normal vectors of the points, a three-dimensional geometric processing system is applied to perform surface reconstruction to obtain an initial reconstruction result; Deleting the point cloud included in any frame of the point cloud image in the initial reconstruction result to obtain grid data of the surgical instrument; The mesh data is flattened and hole-filled to obtain the three-dimensional model data.

[0060] On the basis of the above-mentioned embodiment, in the apparatus for preparing a surgical instrument posture data set provided in the embodiment of the present invention, the reconstruction module is further specifically used for: Based on a three-dimensional geometric processing system, the foreground point cloud and the background point cloud in any frame of the point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud; The normal vector of each point in the foreground point cloud is calculated.

[0061] Specifically, the functions of each module in the surgical instrument posture data set production device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0062] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the surgical instrument posture data set preparation method provided in the above embodiments.

[0063] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0064] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical instrument posture data set production method provided in the above-mentioned embodiments.

[0065] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the surgical instrument posture data set production method provided in the above-mentioned embodiments.

[0066] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for producing a surgical instrument pose dataset. It is characterized in that include: Based on the depth camera, a scene point cloud video of a rotating positioning plate is obtained, wherein the positioning plate is fixed with surgical instruments and reference markers; Matching each frame point cloud image except the starting frame point cloud image in the scene point cloud video with the starting frame point cloud image to obtain a matching result, and constructing a target posture graph of the depth camera according to the matching result; For any frame of point cloud image in the scene point cloud video, calculate the normal vector of each point in the frame of point cloud image, and perform surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument; Based on the three-dimensional model data and the target posture graph, the vertices and posture information of the surgical instrument are determined, and based on the vertices and posture information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image, the surgical instrument posture dataset is constructed.

2. The method for preparing a surgical instrument posture data set according to claim 1, It is characterized in that The reference markers include a plurality of reference markers, and the plurality of reference markers are evenly fixed on the positioning plate; the positioning plate rotates at a uniform speed, and each frame of the point cloud image of the scene point cloud video contains at least three of the reference markers.

3. The method for preparing a surgical instrument posture data set according to claim 1, It is characterized in that The step of constructing a target posture graph of the depth camera according to the matching result includes: Based on the matching results, applying a synchronous positioning and mapping algorithm to construct an initial posture graph of the depth camera; Based on a three-dimensional data processing library, the initial posture graph is optimized to obtain the target posture graph.

4. The method for preparing a surgical instrument posture data set according to claim 3, It is characterized in that The step of optimizing the initial posture graph based on the three-dimensional data processing library to obtain the target posture graph includes: Based on the three-dimensional data processing library, the Levenberg-Marquardt algorithm is applied to optimize the initial posture graph to obtain the target posture graph.

5. The method for preparing a surgical instrument posture dataset according to any one of claims 1 to 4, It is characterized in that The surface reconstruction is performed based on the normal vectors of each point to obtain the three-dimensional model data of the surgical instrument, including: Based on the normal vectors of the points, a three-dimensional geometric processing system is applied to perform surface reconstruction to obtain an initial reconstruction result; Deleting the point cloud included in any frame of the point cloud image in the initial reconstruction result to obtain grid data of the surgical instrument; The mesh data is flattened and hole-filled to obtain the three-dimensional model data.

6. The method for preparing a surgical instrument posture dataset according to any one of claims 1 to 4, It is characterized in that The calculating the normal vector of each point in any frame of the point cloud image comprises: Based on a three-dimensional geometric processing system, the foreground point cloud and the background point cloud in any frame of the point cloud image are segmented, and the background point cloud is cropped to obtain the foreground point cloud; The normal vector of each point in the foreground point cloud is calculated.

7. A surgical instrument posture data set production device, It is characterized in that include: A video acquisition module, used to acquire a scene point cloud video of a rotating positioning plate based on a depth camera, wherein surgical instruments and reference markers are fixed on the positioning plate; A matching module, used to match each frame point cloud image in the scene point cloud video except the starting frame point cloud image with the starting frame point cloud image to obtain a matching result, and construct a target posture graph of the depth camera according to the matching result; A reconstruction module, for calculating the normal vector of each point in any frame of the point cloud image in the scene point cloud video, and performing surface reconstruction based on the normal vector of each point to obtain the three-dimensional model data of the surgical instrument; A construction module is used to determine the vertices and posture information of the surgical instrument based on the three-dimensional model data and the target posture graph, and to construct the surgical instrument posture data set based on the vertices and posture information of the surgical instrument in each frame of the point cloud image of the scene point cloud video, combined with the segmentation mask of the surgical instrument in each frame of the point cloud image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method for producing a surgical instrument posture data set according to any one of claims 1-6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, it implements the method for producing a surgical instrument posture data set as described in any one of claims 1-6.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, it implements the method for producing a surgical instrument posture data set as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Binocular vision-based reference mark pose estimation method

    CN116091625A

  • Surgical instrument positioning method and system based on ARUCO mark

    CN116473675A

  • Real-time surgical instrument pose estimation method and system based on deep learning and stereoscopic vision

    CN117011381A

  • Marker and method of estimating surgical instrument pose using the same

    US20140316252A1

  • Systems and methods for position and orientation tracking of anatomy and surgical instruments

    US20190090955A1