Surgical instrument real-time registration system and method based on position sensing pattern

Abstract

The invention discloses a surgical instrument real-time registration system and method based on a position sensing pattern. The system and method are used for real-time registration of surgical instruments in an operation and in a preoperative preparation stage. The real-time registration system comprises a position sensing pattern used for assisting in positioning a surgical instrument; the visual positioning system is used for detecting a position sensing pattern on the surgical instrument; and the control system is used for real-time registration of the position sensing pattern feature information three-dimensional model and acquisition and real-time positioning of tip information of the surgical instrument. By means of the technology, the position sensing pattern three-dimensional point cloud model can be registered in real time, the tip pose of the surgical instrument can be fitted in real time, the result is displayed on a display screen of a control system, real-time registration and real-time positioning are achieved, and convenient and accurate real-time navigation is provided for clinical surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of computer-assisted surgical treatment, and in particular to a real-time registration system and method for surgical instruments based on position sensing patterns. Background Art

[0002] Minimally invasive total hip replacement surgery and oral dental implant surgery have the characteristics of small surgical area and poor exposure of the surgical area. At present, clinical surgery requires the use of special instruments such as endoscopes and optical instruments to assist doctors in completing the surgery, which is difficult and has a large invisible area. At the same time, doctors are also required to have rich clinical practical experience and on-the-spot decision-making ability. Therefore, many factors will lead to surgical errors and the incidence of postoperative complications, affecting the treatment effect.

[0003] At present, most computer-assisted surgeries use electromagnetic positioning, optical marker positioning or simple visual marker positioning to assist the navigation system in acquiring the position of the tip of the surgical instrument. Most of the existing optical or visual positioning markers are essentially positioning feature points with a single structure and are difficult to distinguish. It is difficult to cope with some scenes of occlusion and obstacles, which will lead to emergencies such as incorrect or even lost positioning information. On the other hand, surgical instruments are not a single tool. Surgeons need to replace different surgical tools during clinical operations. Therefore, a more convenient and real-time registration method is needed to assist doctors in locating temporarily replaced surgical instruments. Taking into account the above-mentioned problems such as poor positioning, weak anti-occlusion ability, and replacement of surgical instruments at any time, it is necessary to develop a real-time registration and positioning system and method for surgical instruments based on position sensing patterns to provide more accurate real-time navigation and positioning for surgical instruments. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a real-time registration system and method for surgical instruments based on position sensing patterns, so as to provide more accurate real-time navigation and positioning for surgical instruments.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solutions: a real-time registration system for surgical instruments based on position sensing patterns, which is used for real-time registration of surgical instruments during surgery and preoperative preparation, including: a position sensing pattern, which is used to assist in positioning the surgical instrument; a visual positioning system, which is used to detect the position sensing pattern on the surgical instrument; a control system, which is used for real-time registration of a three-dimensional model of position sensing pattern feature information and acquisition and real-time positioning of the surgical instrument tip information.

[0006] In a preferred embodiment, the position sensing pattern is specifically a checkerboard-like pattern composed of alternating black and white triangle blocks, which is arranged at the tail end of the surgical instrument and fits the cylindrical surface of the tail of the surgical instrument; the position sensing pattern is laser engraved or made into a matte self-adhesive sticker and attached to the outer surface of the cylindrical structure at the tail of the surgical instrument.

[0007] In a preferred embodiment, in the pattern of alternating black and white triangle blocks, The black and white triangle area contains There are dots distributed in the center of each black and white triangle. The intersection points between the black and white triangles in the pattern are the characteristic points of the pattern, and each characteristic point has a unique number.

[0008] In a preferred embodiment, the visual positioning system includes two industrial cameras of the same model, the two industrial cameras form a stereoscopic vision system device, and the stereoscopic vision device is fixed on the camera pan-tilt bracket; The visual positioning system is fixed on a movable bracket of the mobile cart, and establishes a posture relationship between the stereo camera and the surgical instrument through a position sensing pattern.

[0009] In a preferred embodiment, the control system is composed of an industrial control computer and a high-definition display screen. The registration system can assist the surgeon to register surgical instruments in real time and obtain and locate the tip information of the surgical instruments, and display the registration results and positioning results on the display screen of the control system.

[0010] In a preferred embodiment, the industrial control computer of the control system is installed inside the movable cart, and the display screen is installed on the support rod of the movable cart, so as to build an integrated structure of the stereoscopic vision device and the control device.

[0011] The present invention also provides a real-time registration method for surgical instruments based on position sensing patterns, which uses the real-time registration system for surgical instruments based on position sensing patterns, including: real-time registration of non-planar three-dimensional models of position sensing patterns, and acquisition and positioning of surgical instrument tips.

[0012] In a preferred embodiment, the real-time registration of the non-planar stereoscopic three-dimensional model of the position sensing pattern and the establishment of the positional relationship between the visual marker on the surgical instrument and the camera include: The visual positioning system uses a black and white checkerboard calibration plate to calibrate the camera parameters of the stereoscopic vision device and establish a camera coordinate system in space. The visual device detects the feature points in the position sensing pattern in the image in real time, calculates the three-dimensional coordinates of all feature points in the current image in the camera coordinate system and stores them in the system backend database; the backend system selects the three-dimensional coordinates of all high-quality feature points for splicing, completes the three-dimensional reconstruction of the feature point cloud of the position sensing pattern and establishes the point cloud coordinate system .

[0013] In a preferred embodiment, the surgical instrument tip acquisition and positioning includes: The visual positioning system establishes the camera coordinate system through the position perception pattern and point cloud coordinate system The relative position between them is that the surgical instrument with the position sensing pattern attached takes the tip as the center of the sphere, moves the surgical instrument, and the visual device detects the feature points on the position sensing pattern in real time to obtain Set camera coordinate system and point cloud coordinate system The motion transformation matrix between , and then solve the homogeneous transformation equation to obtain the position of the surgical instrument tip in the camera coordinate system The three-dimensional coordinates under the camera coordinate system The point cloud model and the three-dimensional coordinates of the tip complete the overall registration of the surgical instrument.

[0014] In a preferred embodiment, the visual mark on the surgical instrument is detected and the position sensing pattern is detected and positioned in real time, including: the visual positioning device detects the feature points on the position sensing pattern on the surgical instrument, obtains the two-dimensional coordinates of the feature points in the image, and uses the principle of stereoscopic vision to understand the three-dimensional coordinates of the feature points under the camera coordinates. , and then solve the point cloud coordinate system on the surgical instrument through the point cloud model and the camera coordinate system The relative posture between them is then used to obtain the real-time posture of the surgical instrument through the point cloud model of the registered surgical instrument and the position sensing pattern.

[0015] Compared with the prior art, the present invention has the following beneficial effects: the present invention can register the three-dimensional point cloud model of the position perception pattern and fit the tip posture of the surgical instrument in real time, and display the results on the display screen of the control system, thereby realizing real-time registration and real-time positioning, and providing convenient and accurate real-time navigation for clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of a method for real-time registration and positioning of surgical instruments based on position sensing patterns according to an embodiment of the present invention.

[0017] Figure 2The figure is a schematic diagram of the principle of a real-time registration and positioning method for surgical instruments based on position sensing patterns according to an embodiment of the present invention.

[0018] Figure 3 Schematic diagram of a position sensing pattern in a real-time registration and positioning method for a surgical instrument based on a position sensing pattern in an embodiment of the present invention.

[0019] Figure 4 Schematic diagram of a minimally invasive hip joint surgical instrument with a position sensing pattern in a real-time registration and positioning method for surgical instruments based on a position sensing pattern in an embodiment of the present invention.

[0020] Figure 5 The present invention is a schematic diagram of the integrated structure of a visual positioning device, a control host and a display screen in a real-time registration and positioning method for surgical instruments based on position sensing patterns in an embodiment of the present invention.

[0021] Figure 6 The figure is a schematic diagram of a specific flow chart of a method for real-time registration of surgical instruments based on position sensing patterns according to an embodiment of the present invention. Figure 7 Schematic diagram of the principle of real-time stitching of point cloud models in a real-time registration method for surgical instruments based on position-aware patterns in an embodiment of the present invention. Figure 8 The figure is a schematic diagram of a specific process of surgical instrument tip fitting in a real-time registration method of surgical instruments based on position sensing patterns according to an embodiment of the present invention. Fig. 9 This is an example diagram of real-time stitching of point cloud models in a real-time registration method for surgical instruments based on position-aware patterns in an embodiment of the present invention. Description of reference numerals: 1. Stereoscopic vision device (visual positioning system); 2. Host + display screen (control system); 3. Surgical instruments; 4. Position sensing pattern; 5. Registered point cloud model; DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0024] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0025] Please refer to the accompanying drawings. An embodiment of the technology of the present invention provides a real-time registration and positioning system and method for surgical instruments based on position sensing patterns.

[0026] The embodiment of the present disclosure provides a real-time registration system for surgical instruments based on position sensing patterns, such as Figure 1 and Figure 2 As shown, Figure 1 The figure is a flow chart of a method for real-time registration and positioning of surgical instruments based on position sensing patterns according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the principle of a real-time registration method for surgical instruments based on position sensing patterns according to an embodiment of the present invention. Figure 1 and Figure 2 It is used to help researchers and technicians in this field understand the technical process and technical content of this invention.

[0027] Depend on Figure 1-9 As shown, the real-time registration system of the present invention includes: a stereoscopic vision device, a position sensing pattern, a control system host and a display screen. Figure 2 As shown in the figure, the working principle of the real-time registration system is as follows: 1. In the preparation stage, the stereoscopic vision device is calibrated and the position sensing pattern is attached; 2. In the registration stage, the position sensing pattern feature detection, real-time image key frame acquisition and screening, point cloud tree splicing in the feature database, and surgical instrument tip fitting. The visual positioning system detects the position sensing pattern on the surgical instrument from multiple directions and calculates the motion transformation matrix between each position. , thereby calculating and splicing the position-aware pattern feature point cloud model.

[0028] Depend on Figure 1 As shown, the technology of the present invention provides a real-time registration system for surgical instruments based on position sensing patterns, including a stereoscopic vision device 1 of a visual positioning system, a host and display screen 2 of a control system, a surgical instrument 3 with a position sensing pattern, and a feature point cloud model 5 that is spliced ​​together in real time; wherein, the position sensing pattern 4 is used to assist in positioning; the stereoscopic vision device 1 is used to detect and locate the position sensing pattern on the surgical instrument; the control host and the display screen are respectively used to splice the feature point cloud model in real time and to fit the position of the tip of the surgical instrument in real time and present it on the display screen.

[0029] According to an embodiment of the present invention, Figure 3 As shown, the position sensing pattern is a checkerboard-like pattern composed of alternating black and white triangles. Figure 4 As shown, it is arranged at the tail end of the surgical instrument and fits the cylindrical surface of the tail end of the surgical instrument.

[0030] According to a further embodiment of the present invention, in the pattern composed of alternating black and white triangle blocks, The black and white triangle area contains The dots distributed in the center of each black and white triangle block, the intersection points between the black and white triangle blocks in the pattern are the characteristic points of the pattern, and each characteristic point has a unique number. Figure 3 As shown, the distribution density of feature points in the position sensing pattern is high, and any part thereof has an independent self-identification unit.

[0031] According to a further solution of the present invention, the position sensing pattern is attached to the outer surface of the cylindrical structure at the tail of the surgical instrument by laser engraving technology or made into a matte self-adhesive sticker, such as Figure 4 shown.

[0032] According to a further embodiment of the present invention, Figure 1 and Figure 5 As shown in FIG, the visual positioning system includes two industrial cameras of the same model, which form a stereoscopic vision system device, and the stereoscopic vision device is fixed on the camera gimbal bracket. Specifically, the stereoscopic vision device uses a pair of Hikvision industrial cameras of the same configuration and model, which are fixed on the camera gimbal bracket. Figure 5 As shown, it is fixed on the mechanical arm of the movable vehicle. Figure 1 As shown, the position relationship between the stereo camera and the surgical instrument is established through the position sensing pattern.

[0033] According to a further embodiment of the present invention, Figure 1 and Figure 5 As shown, the control system 2 is composed of an industrial control computer and a high-definition display screen. The registration system can assist surgeons in real-time registration of surgical instruments and acquisition and positioning of surgical instrument tip information, and display the registration results and positioning results on the display screen of the control system. The industrial control computer of the control system is installed inside the movable trolley, and the display screen is installed on the support rod of the movable trolley, so as to build an integrated structure of the stereoscopic vision device and the control device.

[0034] In the preoperative preparation stage, control system 2 is used to calibrate the stereoscopic vision device; in the intraoperative stage, control system 2 is used to complete the real-time detection of position perception patterns, real-time feature stitching, and real-time fitting and positioning of the tip posture of the surgical instrument by the visual positioning system.

[0035] According to a further embodiment of the present invention, the real-time registration of the non-planar stereoscopic three-dimensional model of the position sensing pattern establishes the position relationship between the visual marker on the surgical instrument and the camera, such as Figure 6 As shown, including: Step 1: The visual positioning system uses a black and white checkerboard calibration plate to calibrate the camera parameters of the stereo vision device and establish the camera coordinate system in space. .

[0036] Step 2: The visual device detects the feature points in the position perception pattern in the image in real time, and calculates the coordinates of all feature points in the current image in the camera coordinate system. The three-dimensional coordinates are stored in the system backend database.

[0037] Step 3: The backend system selects the 3D coordinates of all high-quality feature points for stitching, completes the 3D reconstruction of the position-aware pattern feature point cloud, and establishes the point cloud coordinate system .

[0038] According to an embodiment of the present invention, the stereo vision device in step 1 takes about 26 sets of pictures of a black and white checkerboard calibration plate (20 mm / grid), calibrates the stereo vision device through the MATLAB binocular calibration tool (stereoCameraCalibration), obtains the camera parameters, and establishes the camera coordinate system in space. .

[0039] According to an embodiment of the present invention, the visual positioning system in step 2 detects the pixel coordinates of all feature points in the current image in the image coordinate system from the surgical instrument image with the position sensing pattern. Combined with the camera parameters calibrated in step 1, the three-dimensional coordinates of the feature points in the camera coordinate system are calculated through triangulation. When the visual device shoots in real time from multiple positions and angles, the control system evaluates and screens the quality of each frame detection and stores the appropriate key frames in the database.

[0040] According to an embodiment of the present invention, the backend system in step 3 selects the best key frame as the start of tree splicing, and matches other key frames layer by layer in the database, such as Figure 7 As shown. By using the SVD decomposition method, the motion transformation matrix of the three-dimensional coordinates of the feature point cloud in the two key frames is calculated. , the feature point cloud is pieced into the optimal key frame layer by layer, and the model is continuously improved and output iteratively, such as Fig. 9 As shown in the example, the real-time registration of the location-aware pattern feature point cloud model is completed.

[0041] According to a further embodiment of the present invention, the surgical instrument tip is fitted and positioned as follows: Figure 8 As shown, including: Step 1: The visual positioning system establishes the camera coordinate system through the position perception pattern and point cloud coordinate system The relative posture between them is determined. The surgical instrument with the position sensing pattern attached takes the tip as the center of the sphere. The surgical instrument is moved, and the visual device detects the position sensing pattern in real time and calculates the three-dimensional coordinates of the feature points on the image.

[0042] Step 2: Record the feature information of each frame in real time, store it in the back-end database after screening, and wait for the back-end to fit the three-dimensional coordinates of the tip.

[0043] Step 3: Find Set camera coordinate system and point cloud coordinate system The motion transformation matrix between The tip of the surgical instrument in the camera coordinate system is solved by homogeneous transformation of the motion equation. The three-dimensional coordinates under the camera coordinate system The point cloud model and the three-dimensional coordinates of the tip complete the overall registration of the surgical instrument.

[0044] According to an embodiment of the present invention, the homogeneous transformation motion equation is as follows: (1) (2) in Serves 6 1, the first three elements of which are the required three-dimensional coordinates of the surgical instrument tip. .

[0045] According to a further solution of the present invention, the visual mark on the surgical instrument is detected and the position sensing pattern is detected and positioned in real time, including: a visual positioning device detects the position sensing pattern on the surgical instrument, obtains the two-dimensional coordinates of the feature point in the image, uses the stereo vision principle to understand the three-dimensional coordinates of the feature point in the camera coordinates, and uses the point cloud model to solve the point cloud coordinate system on the surgical instrument. and the camera coordinate system The relative posture between them is then used to obtain the real-time posture of the surgical instrument through the point cloud model of the registered surgical instrument and the position sensing pattern.

[0046] Through the above method, a real-time registration and positioning system for surgical instruments based on position sensing patterns provided by the technical embodiment of the present invention achieves the registration and navigation positioning of surgical instruments, and realizes the real-time requirements, providing convenience for doctors before and after the operation. The system displays the registration and positioning results on the display screen of the control system through Visual Studio 2022 and OpenCV library.

[0047] It should be noted that the above registration and positioning methods are all part of the system operation process. For clinicians, they only need to present the surgical instrument with the position sensing pattern attached to it under the field of view of the visual positioning system to complete the detection of the pattern and the positioning of the surgical instrument.

[0048] So far, the technology of the present invention has been described in detail in conjunction with the accompanying drawings. Although the present disclosure has been described in conjunction with the accompanying drawings, the embodiments disclosed in the accompanying drawings are intended to exemplify the preferred embodiments of the present disclosure and should not be construed as a limitation on the present disclosure. The size ratios in the accompanying drawings are merely schematic and should not be construed as limitations on the present disclosure. The above embodiments are only intended to illustrate the technical concept flow and features of the technology of the present invention. Their purpose is to enable technicians familiar with this technology to understand the content of the invention and implement it accordingly, and they cannot be used to limit the scope of protection of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be covered within the scope of protection of the technology of the present invention.

Claims

1. A real-time registration system for surgical instruments based on position sensing patterns, used for real-time registration of surgical instruments during surgery and preoperative preparation, characterized in that: include: Position sensing patterns to assist in positioning surgical instruments; Vision positioning systems to detect position-sensing patterns on surgical instruments; A control system is used for real-time registration of position sensing pattern feature information three-dimensional models and acquisition and real-time positioning of surgical instrument tip information.

2. A real-time registration system for surgical instruments based on position sensing patterns according to claim 1, characterized in that: The position sensing pattern is specifically a checkerboard-like pattern composed of alternating black and white triangle blocks, which is arranged at the tail end of the surgical instrument and fits the cylindrical surface of the tail of the surgical instrument; the position sensing pattern is laser engraved or made into a matte self-adhesive sticker and attached to the outer surface of the cylindrical structure at the tail of the surgical instrument.

3. A real-time registration system for surgical instruments based on position sensing patterns according to claim 2, characterized in that: In the pattern composed of alternating black and white triangles, The black and white triangle area contains There are dots distributed in the center of each black and white triangle. The intersection points between the black and white triangles in the pattern are the characteristic points of the pattern, and each characteristic point has a unique number.

4. A real-time registration system for surgical instruments based on position sensing patterns according to claim 1, characterized in that: The visual positioning system includes two industrial cameras of the same model, the two industrial cameras form a stereoscopic vision system device, and the stereoscopic vision device is fixed on the camera pan-tilt bracket; The visual positioning system is fixed on a movable bracket of the mobile cart, and establishes a posture relationship between the stereo camera and the surgical instrument through a position sensing pattern.

5. The real-time registration system for surgical instruments based on position sensing patterns according to claim 1, characterized in that: The control system is composed of an industrial control computer and a high-definition display screen. The registration system can assist surgical doctors in real-time registration of surgical instruments and acquisition and positioning of surgical instrument tip information, and display the registration results and positioning results on the display screen of the control system.

6. A real-time registration system for surgical instruments based on position sensing patterns according to claim 5, characterized in that: The industrial control computer of the control system is installed inside the movable trolley, and the display screen is installed on the supporting rod of the movable trolley, so as to build an integrated structure of the stereoscopic vision device and the control device.

7. A method for real-time registration of surgical instruments based on position sensing patterns, characterized in that: A real-time registration system for surgical instruments based on position sensing patterns as described in any one of claims 1 to 6 includes: real-time registration of non-planar three-dimensional models of position sensing patterns, acquisition and positioning of surgical instrument tips.

8. A method for real-time registration of surgical instruments based on position sensing patterns according to claim 7, characterized in that: The real-time registration of the non-planar stereoscopic three-dimensional model of the position sensing pattern and the establishment of the positional relationship between the visual marker on the surgical instrument and the camera include: The visual positioning system uses a black and white checkerboard calibration plate to calibrate the camera parameters of the stereoscopic vision device and establish a camera coordinate system in space. The visual device detects the feature points in the position sensing pattern in the image in real time, calculates the three-dimensional coordinates of all feature points in the current image in the camera coordinate system and stores them in the system backend database; the backend system selects the three-dimensional coordinates of all high-quality feature points for splicing, completes the three-dimensional reconstruction of the feature point cloud of the position sensing pattern and establishes the point cloud coordinate system .

9. A method for real-time registration of surgical instruments based on position sensing patterns according to claim 8, characterized in that: The surgical instrument tip acquisition and positioning comprises: The visual positioning system establishes the camera coordinate system through the position perception pattern and point cloud coordinate system The relative position between them is that the surgical instrument with the position sensing pattern attached takes the tip as the center of the sphere, moves the surgical instrument, and the visual device detects the feature points on the position sensing pattern in real time to obtain Set camera coordinate system and point cloud coordinate system The motion transformation matrix between , and then solve the homogeneous transformation equation to obtain the position of the surgical instrument tip in the camera coordinate system The three-dimensional coordinates under the camera coordinate system The point cloud model and the three-dimensional coordinates of the tip complete the overall registration of the surgical instrument.

10. A method for real-time registration of surgical instruments based on position sensing patterns according to claim 9, characterized in that: The method of detecting visual marks on the surgical instrument and performing real-time detection and positioning of the position sensing pattern includes: the visual positioning device detects feature points on the position sensing pattern on the surgical instrument, obtains the two-dimensional coordinates of the feature points in the image, and uses the principle of stereoscopic vision to understand the three-dimensional coordinates of the feature points under the camera coordinates. , and then solve the point cloud coordinate system on the surgical instrument through the point cloud model and the camera coordinate system The relative posture between them is then used to obtain the real-time posture of the surgical instrument through the point cloud model of the registered surgical instrument and the position sensing pattern.

Images