Surgical instrument three-dimensional reconstruction and calibration method based on pixel engraving

Abstract

The invention relates to a surgical instrument three-dimensional reconstruction and calibration method based on pixel engraving, and provides a real-time calibration system, which comprises a position sensing pattern arranged on a surgical instrument and used for assisting reconstruction and positioning of the surgical instrument; the visual positioning system is used for detecting a position sensing pattern on the surgical instrument and extracting the outline of the surgical instrument; the control system is used for detecting and recognizing position sensing pattern feature information, performing three-dimensional reconstruction on the surgical instrument, and acquiring and positioning the tip of the surgical instrument and the normal vector of the tip in real time; the real-time calibration system is used for three-dimensional reconstruction and registration of a surgical instrument in a preoperative preparation stage, and comprises incremental reconstruction of a non-planar three-dimensional model of a position sensing pattern, three-dimensional reconstruction of the surgical instrument, and calibration of a tip of the surgical instrument and a normal vector of the tip. According to the method, higher-precision and more reliable real-time navigation positioning can be provided for the surgical instrument.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer-assisted surgical treatment, and in particular relates to a three-dimensional reconstruction and calibration method of surgical instruments based on pixel engraving. Background Art

[0002] Minimally invasive hip replacement surgery and dental implant surgery are challenging due to their small surgical field and poor visibility. Currently, these procedures require specialized equipment such as endoscopes, electromagnetic sensors, and optical ball launchers to assist doctors. Improving the accuracy, robustness, and visibility of surgical positioning will help surgeons, especially those with limited experience, perform high-precision implants and minimally invasive procedures. This will reduce surgical errors and risks, and improve surgical efficiency and treatment outcomes.

[0003] At present, most computer-assisted surgeries use electromagnetic positioning, optical marker positioning or traditional pattern marker positioning to assist the navigation system in completing the calibration of the surgical instrument tip posture. Most of the existing optical or visual positioning markers are essentially positioning feature points with a single structure and are difficult to distinguish. They are difficult to deal with some scenes of occlusion and obstacles, which will lead to emergencies such as incorrect positioning information or even loss. On the other hand, the currently commonly used surgical instrument tip posture calibration method only obtains the relative position between the tip and the marker, and locates the tip point by identifying the marker. However, the normal vector of the surgical instrument tip point cannot be calibrated, especially for instruments aligned in non-central axes, and cannot provide doctors with more reliable positioning information in the direction of instrument implantation.

[0004] Considering the above-mentioned problems such as poor positioning, weak anti-occlusion ability and incomplete calibration of surgical instruments, it is necessary to develop a three-dimensional reconstruction and calibration method for surgical instruments based on pixel engraving to provide more accurate and reliable real-time navigation positioning for surgical instruments. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional reconstruction and calibration method for surgical instruments based on pixel engraving, which can provide more accurate and reliable real-time navigation positioning for surgical instruments.

[0006] To achieve the above objectives, the present invention adopts a technical solution: a method for 3D reconstruction and calibration of surgical instruments based on pixel engraving, providing a real-time calibration system, comprising:

[0007] Position sensing patterns, provided on surgical instruments, to assist in the reconstruction and positioning of surgical instruments;

[0008] A visual positioning system is used to detect the position sensing pattern on the surgical instrument and extract the contour of the surgical instrument;

[0009] Control system for detecting and identifying position sensing pattern feature information, 3D reconstruction of surgical instruments, acquisition and real-time positioning of surgical instrument tips and tip normals;

[0010] The real-time calibration system is used for three-dimensional reconstruction and registration of surgical instruments in the preoperative preparation stage, including: incremental reconstruction of non-planar three-dimensional models of position sensing patterns, three-dimensional reconstruction of surgical instruments, and calibration of surgical instrument tips and tip normal vectors.

[0011] Furthermore, the position sensing pattern is a checkerboard-like pattern formed by alternating black and white triangle blocks; the checkerboard-like pattern includes dense and non-repeated numbered positioning feature points distributed at each intersection.

[0012] Furthermore, the position sensing pattern is arranged at the tail end of the surgical instrument and fits onto the cylindrical and conical surfaces of the tail end of the surgical instrument.

[0013] Furthermore, the position sensing pattern is attached to the outer surface of the cylindrical and conical structures at the tail of the surgical instrument by laser engraving or matte self-adhesive stickers.

[0014] Furthermore, the visual positioning system includes a stereoscopic vision device consisting of two color industrial cameras of the same model. The stereoscopic vision device is fixed on a camera gimbal bracket and is used to establish a posture relationship between the stereo camera and the surgical instrument through a position sensing pattern.

[0015] Furthermore, the control system is composed of an industrial control computer and a high-definition display screen. The control system is used to reconstruct the model of the surgical instrument and obtain the surgical instrument tip and tip normal vector, and project the reconstructed model of the surgical instrument and the positioning result onto the display screen to achieve positioning visualization.

[0016] Furthermore, the industrial control computer is installed on a movable bracket, and the display screen is installed on a support rod of the movable bracket, thereby constructing an integrated structure of the stereoscopic vision device and the control system.

[0017] Furthermore, the incremental reconstruction of the non-planar three-dimensional model of the position sensing pattern establishes a positional relationship between the visual marker on the surgical instrument and the camera, including:

[0018] The visual positioning system uses the position sensing pattern to calibrate the camera parameters of the stereo vision device and establishes the camera coordinate system in space. camera};

[0019] The front-end vision device identifies the feature points of the position-aware pattern in the image at each viewing angle, calculates the 3D points of the feature points in the camera coordinate system, and screens the quality of each set of points. The high-quality 3D points at the new viewing angle are added to the image pool of the back-end system, or a set of low-quality points in the image pool is replaced.

[0020] The back-end system stitches the three-dimensional points of the feature points in the image pool, completes the three-dimensional reconstruction of the position-aware pattern feature point cloud, and establishes the point cloud coordinate system {O marker}.

[0021] Furthermore, the three-dimensional reconstruction of the surgical instrument includes:

[0022] The visual positioning system establishes a camera coordinate system by position sensing pattern {O camera} and point cloud coordinate system {O marker}; collect the contours of surgical instruments from different perspectives, and establish voxel blocks of size m×n×kd in the system; based on the pose estimation and contour information of the position-aware pattern, perform pixel carving on the voxel blocks, reconstruct the three-dimensional model of the surgical instrument, and perform texture mapping processing.

[0023] Furthermore, the surgical instrument tip and tip normal vector calibration are calibrated by using the three-dimensional model of the surgical instrument obtained in the three-dimensional reconstruction of the surgical instrument; the outer shell of the instrument point cloud model is extracted using a convex hull algorithm; the point on the shell farthest from the center of the point cloud is selected to determine the tip point; and then the center axis and normal vector of the tip are calculated based on the tip of the surgical instrument and the partial point cloud around the tip.

[0024] Compared with the existing technology, the present invention has the following beneficial effects: the present invention can reconstruct the position-aware pattern feature point cloud model and the surgical instrument model, calibrate the position of the surgical instrument tip, and project the reconstructed model and positioning results onto the display screen of the control system, providing reliable and accurate real-time navigation for clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram illustrating the implementation principle of a method for three-dimensional reconstruction and calibration of surgical instruments based on pixel engraving provided by an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of an implementation method for three-dimensional reconstruction and calibration of surgical instruments based on pixel engraving provided by an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of a position sensing pattern according to an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of a dental implant surgical instrument with a position sensing pattern according to an embodiment of the present invention;

[0029] Figure 5 Schematic diagram of the incremental position-aware pattern feature point cloud 3D reconstruction process in an embodiment of the present invention;

[0030] Figure 6This is a schematic diagram of a specific process for extracting the shape of a surgical instrument in an embodiment of the present invention;

[0031] Figure 7 2. It is a schematic diagram of the principle of three-dimensional reconstruction of surgical instruments in an embodiment of the present invention;

[0032] Figure 8 Schematic diagram of the calibration principle of the surgical instrument tip and tip normal vector in an embodiment of the present invention.

[0033] In the figure: 1. Visual positioning system (stereoscopic vision device); 2. Control system (industrial control computer + display screen); 3. Surgical instrument; 4. Position sensing pattern; 5. Feature point cloud model; 6. Surgical instrument reconstruction model. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0036] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0037] like Figure 1 As shown, the real-time registration system hardware of the present invention includes: a stereoscopic vision device, a position sensing pattern and a control system host and a display screen.

[0038] This embodiment provides a method for 3D reconstruction and calibration of surgical instruments based on pixel engraving, which provides a real-time calibration system. Figure 1 Shown include:

[0039] a position sensing pattern 4 , provided on the surgical instrument 3 , for assisting in the reconstruction and positioning of the surgical instrument;

[0040] A visual positioning system 1, used to detect the position sensing pattern on the surgical instrument and extract the outline of the surgical instrument;

[0041] Control system 2 is used for detecting and identifying position sensing pattern feature information, three-dimensional reconstruction of surgical instruments, acquisition and real-time positioning of surgical instrument tips and tip normal vectors.

[0042] The real-time calibration system is used for three-dimensional reconstruction and registration of surgical instruments in the preoperative preparation stage, including: incremental reconstruction of non-planar three-dimensional models of position sensing patterns, three-dimensional reconstruction of surgical instruments, and calibration of surgical instrument tips and tip normal vectors.

[0043] like Figure 2 As shown in the figure, the working process of the reconstruction and calibration system is: 1. Preparation stage, stereo vision device calibration, position perception pattern point cloud model 3D reconstruction; 2. Shape contour extraction stage, Canny edge extraction of surgical instrument image, region growing algorithm to extract surgical instrument shape image; 3. Pixel carving, position perception pattern feature detection, feature point recognition, feature point matching, surgical instrument shape pose estimation, voxel block based on shape pixel carving, and calibration of tip point and its normal vector based on instrument model.

[0044] like Figure 1 As shown, the visual positioning system 1 includes a stereoscopic vision device composed of two identical color industrial cameras. The stereoscopic vision device is fixed to a camera gimbal bracket and is used to establish the positional relationship between the stereoscopic camera and the surgical instrument through a position sensing pattern. The control system 2 is composed of an industrial control computer and a high-definition display screen. The control system is used to reconstruct a model of the surgical instrument and obtain the surgical instrument tip and tip normal vector. The reconstructed model of the surgical instrument and the positioning results are projected onto the display screen to achieve positioning visualization. The industrial control computer is mounted on a movable bracket, and the display screen is mounted on the support rod of the movable bracket, forming an integrated structure of the stereoscopic vision device and control system. Figure 1 Also shown are the reconstructed position-aware pattern feature point cloud model 5 and the reconstructed surgical instrument model 6.

[0045] like Figure 3 As shown, the position sensing pattern is a checkerboard pattern composed of alternating black and white triangle blocks.

[0046] In the checkerboard pattern composed of alternating black and white triangles, the pattern contains seamlessly stacked self-identifying marks. The intersections between the black and white triangles contain a high density of feature points, each with a unique number, such as Figure 3 shown.

[0047] The position sensing pattern is attached to the outer surface of the cylindrical and conical structure at the tail of the surgical instrument by laser engraving technology or made into a matte self-adhesive sticker, such as Figure 4 shown.

[0048] During the preoperative preparation stage, the control system 2 is used for calibration of the stereoscopic vision device, three-dimensional reconstruction of the surgical instrument, and calibration of the surgical instrument tip point and normal vector.

[0049] The incremental reconstruction of the non-planar three-dimensional model of the position sensing pattern establishes the position relationship between the visual marker on the surgical instrument and the camera, such as Figure 5 Shown, including:

[0050] Step 1: The visual positioning system uses a black and white checkerboard calibration plate to calibrate the camera parameters of the stereo vision device and establish the camera coordinate system in space. camera}.

[0051] Step 2: The front-end visual device identifies the feature points in the position perception pattern in the image at each viewing angle, calculates the three-dimensional points of the feature points in the image in the camera coordinate system and screens the quality of each group of points, and adds the three-dimensional points at the new viewing angle to the back-end image pool of the system, or replaces a group of low-quality points in the pool.

[0052] Step 3: The back-end system matches and splices the 3D points of multiple groups of feature points in the pool, completes the 3D reconstruction of the position-aware pattern feature point cloud and establishes the point cloud coordinate system {O marker}.

[0053] In this embodiment, the stereo vision device in step 1 takes about 32 sets of pictures of a black and white checkerboard calibration plate (20 mm / grid), calibrates the stereo vision device using the MATLAB dual-target calibration tool (stereoCameraCalibration), obtains camera parameters, and establishes a camera coordinate system in space {O camera}.

[0054] In this embodiment, the visual device described in step 2 captures the image of the surgical instrument, detects the position sensing pattern in the image, and identifies the feature points in the pattern. Combined with the camera parameters calibrated in step 1, the three-dimensional coordinates of the feature points in a set of left and right eye images captured by each camera in the camera coordinate system are calculated by triangulation. The front end of the control system detects and judges each set of new images. 1) If the image is blurred or the quality is too poor, it will be deleted; 2) If the image detection has a high similarity with the image in the pool, the image detection with higher quality or a larger number of feature points is selected and added to the pool. Therefore, the image pool can continuously optimize its internal quality without setting a threshold.

[0055] In this embodiment, the backend system in step 3 matches and splices the three-dimensional points of multiple groups of feature points in the pool, completes the three-dimensional reconstruction of the position-aware pattern feature point cloud and establishes a point cloud coordinate system {O marker The set of points containing the most feature points is considered the main group of points for subgroup matching. These subgroup points are then spliced ​​onto the main group of points. The spliced ​​subgroup points are then matched with other subgroup points for further splicing. This process continues until the number of feature points reaches saturation.

[0056] The three-dimensional reconstruction of the surgical instrument, such as Figure 7 Shown, including:

[0057] Step 1: Establish the camera coordinate system through the position-aware pattern camera} and point cloud coordinate system {O marker Collect the contours of surgical instruments from different perspectives.

[0058] Step 2: Create a voxel block of size m×n×kd in the reconstruction system. Based on the pose estimation and contour information of the position-aware pattern, pixel carving is performed on the voxel block to reconstruct the 3D model of the surgical instrument and perform texture mapping.

[0059] In this embodiment, the camera coordinate system {0 camera} and point cloud coordinate system {O marker The relative pose between the two images is obtained. The contours of surgical instruments under different viewing angles are collected. The image is smoothed using Gaussian filtering, and the Canny edge detection algorithm is combined with the region growing criterion to extract the clean contours of surgical instruments, such as Figure 6 shown.

[0060] In this embodiment, as described in step 2, a voxel block of size m×n×kd is established in the reconstruction system. Based on the pose estimation and contour information of the position-aware pattern, the voxel block is pixel-carved to reconstruct the three-dimensional model of the surgical instrument and perform texture mapping. The reconstruction method reconstructs the surgical instrument based on the intersection of the viewing cones within the camera field of view (FoV). In other words, the reconstructed model is the result of the volume intersection generated by the silhouette of each view, such as Figure 7 According to the different shapes, the area in the voxel block belonging to the surgical instrument is determined, and the three-dimensional model of the surgical instrument is gradually restored.

[0061] Figure 8 The schematic diagram of the calibration principle of the surgical instrument tip and tip normal vector in this embodiment. The calibration of the surgical instrument tip and tip normal vector includes: calibrating the surgical instrument tip and tip normal vector using the instrument 3D model obtained from the 3D reconstruction of the surgical instrument. The outer shell of the instrument point cloud model is extracted using a convex hull algorithm. The point on the shell farthest from the center of the point cloud is selected to determine the tip point P. tip Then, the central axis and normal vector α of the tip are calculated based on the point cloud of the tip of the surgical instrument and the parts around the tip.

[0062] Through the above method, the pixel engraving-based three-dimensional reconstruction and calibration method for surgical instruments provided in an embodiment of the present invention realizes the acquisition of the three-dimensional model of the surgical instrument, and calibrates the tip point and tip normal vector of the surgical instrument based on the surgical instrument model, providing a more reliable positioning effect for the surgical navigation system. The system projects the reconstructed model and positioning results onto the display screen of the control system through VisualStudio2019 and OpenCV library.

[0063] It should be noted that the above-mentioned three-dimensional reconstruction and calibration methods are all part of the system operation process. For clinicians, they only need to present the surgical instrument with the position sensing pattern attached to it under the field of view of the visual positioning system to complete the reconstruction and positioning of the surgical instrument.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for 3D reconstruction and calibration of surgical instruments based on pixel engraving, characterized in that: A real-time calibration system is provided, comprising: Position sensing patterns, provided on surgical instruments, to assist in the reconstruction and positioning of surgical instruments; A visual positioning system is used to detect the position sensing pattern on the surgical instrument and extract the contour of the surgical instrument; Control system for detecting and identifying position sensing pattern feature information, 3D reconstruction of surgical instruments, acquisition and real-time positioning of surgical instrument tips and tip normals; The real-time calibration system is used for three-dimensional reconstruction and registration of surgical instruments in the preoperative preparation stage, including: incremental reconstruction of non-planar three-dimensional models of position sensing patterns, three-dimensional reconstruction of surgical instruments, and calibration of surgical instrument tips and tip normal vectors.

2. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The position sensing pattern is a checkerboard-like pattern formed by alternating black and white triangle blocks; the checkerboard-like pattern includes dense and non-repeated numbered positioning feature points distributed at each intersection.

3. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The position sensing pattern is arranged at the tail end of the surgical instrument and fits onto the cylindrical and conical surfaces of the tail end of the surgical instrument.

4. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 3, characterized in that: The position sensing pattern is attached to the outer surface of the cylindrical and conical structures at the tail of the surgical instrument by laser engraving or matte self-adhesive stickers.

5. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The visual positioning system includes a stereoscopic vision device consisting of two color industrial cameras of the same model. The stereoscopic vision device is fixed on a camera pan-tilt bracket and is used to establish a posture relationship between the stereoscopic camera and the surgical instrument through a position sensing pattern.

6. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The control system consists of an industrial control computer and a high-definition display screen. The control system is used to reconstruct the model of the surgical instrument and obtain the surgical instrument tip and tip normal vector, and project the reconstructed model of the surgical instrument and the positioning result onto the display screen to achieve positioning visualization.

7. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 6, characterized in that: The industrial control computer is installed on a movable bracket, and the display screen is installed on a supporting rod of the movable bracket, so as to build an integrated structure of a stereoscopic vision device and a control system.

8. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The incremental reconstruction of the non-planar three-dimensional model of the position-aware pattern establishes a positional relationship between the visual marker on the surgical instrument and the camera, including: The visual positioning system uses the position sensing pattern to calibrate the camera parameters of the stereo vision device and establishes the camera coordinate system in space. camera }; The front-end vision device identifies the feature points of the position-aware pattern in the image at each viewing angle, calculates the 3D points of the feature points in the camera coordinate system, and screens the quality of each set of points. The high-quality 3D points at the new viewing angle are added to the image pool of the back-end system, or a set of low-quality points in the image pool is replaced. The back-end system stitches the three-dimensional points of the feature points in the image pool, completes the three-dimensional reconstruction of the position-aware pattern feature point cloud, and establishes the point cloud coordinate system {O marker }.

9. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The three-dimensional reconstruction of surgical instruments includes: The visual positioning system establishes a camera coordinate system by position sensing pattern {O camera } and point cloud coordinate system {O marker }; collect the contours of surgical instruments from different perspectives, and establish voxel blocks of size m×n×kd in the system; based on the pose estimation and contour information of the position-aware pattern, perform pixel carving on the voxel blocks, reconstruct the three-dimensional model of the surgical instrument, and perform texture mapping processing.

10. The method for 3D reconstruction and calibration of surgical instruments based on pixel engraving according to claim 1, characterized in that: The surgical instrument tip and tip normal vector calibration are performed by calibrating the surgical instrument three-dimensional model obtained from the surgical instrument three-dimensional reconstruction; a convex hull algorithm is used to extract the shell of the instrument point cloud model; the point on the shell farthest from the point cloud center is selected to determine the tip point; and then the center axis and normal vector of the tip are calculated based on the surgical instrument tip and the partial point cloud around the tip.

Images