A system and method for real-time registration of surgical instruments based on a position-aware pattern

Abstract

The application discloses a kind of real-time registration system and method of surgical instrument based on position sensing pattern, for real-time registration of surgical instrument in surgery and preoperative preparation stage.The real-time registration system includes: position sensing pattern, for assisting positioning surgical instrument;Visual positioning system, for detecting the position sensing pattern on surgical instrument;Control system, for the real-time registration of position sensing pattern feature information three-dimensional model and the acquisition and real-time positioning of surgical instrument tip information.Utilize the present application technology, position sensing pattern three-dimensional point cloud model can be registered in real time, and the pose of surgical instrument tip is fitted in real time, and the result is displayed on the display screen of control system, realizes real-time registration and real-time positioning, and provides convenient and accurate real-time navigation for clinical surgery.

Description

Technical Field

[0001] This invention relates to the field of computer-assisted surgical treatment technology, and in particular to a real-time registration system and method for surgical instruments based on position-aware patterns. Background Technology

[0002] Minimally invasive total hip replacement surgery and dental implant surgery are characterized by their small surgical area and poor exposure. Currently, clinical surgery requires the assistance of specialized instruments such as endoscopes and optical instruments to assist surgeons in completing the procedure, which is quite challenging due to the large unseen area. Furthermore, it demands extensive clinical experience and decisive judgment from the surgeon. These factors can lead to surgical errors and postoperative complications, affecting the treatment outcome.

[0003] Currently, most computer-assisted surgeries rely on electromagnetic positioning, optical marker positioning, or simple visual marker positioning to assist navigation systems in acquiring the position and orientation of surgical instrument tips. However, most existing optical or visual positioning markers are essentially simple, indistinguishable positioning feature points, making them ill-suited for scenarios with occlusion and obstacles. This can lead to errors or even loss of positioning information. Furthermore, surgical instruments are not single tools; surgeons need to change instruments frequently during procedures. Therefore, a more convenient real-time registration method is needed to assist surgeons in locating temporarily changed instruments. Considering the aforementioned issues of poor positioning, weak resistance to occlusion, and the need for frequent instrument changes, it is necessary to develop a real-time registration and positioning system and method for surgical instruments based on position-aware patterns, providing higher-precision real-time navigation and positioning for surgical instruments. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a real-time registration system and method for surgical instruments based on position-aware patterns, so as to provide surgical instruments with higher accuracy real-time navigation and positioning.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a real-time registration system for surgical instruments based on position-aware patterns, used for real-time registration of surgical instruments during surgery and preoperative preparation, comprising: position-aware patterns for assisting in the positioning of surgical instruments; a visual positioning system for detecting position-aware patterns on surgical instruments; and a control system for real-time registration of the three-dimensional model of the feature information of the position-aware patterns and for acquiring and real-time positioning of the tip information of the surgical instruments.

[0006] In a preferred embodiment, the position sensing pattern is specifically a checkerboard-like pattern composed of alternating black and white triangular blocks, which is set at the tail end of the surgical instrument and conforms to the cylindrical surface of the tail end of the surgical instrument; the position sensing pattern is laser-engraved or made into a matte self-adhesive sticker and attached to the outer surface of the cylindrical structure at the tail end of the surgical instrument.

[0007] In a preferred embodiment, in a pattern composed of alternating black and white triangular blocks, The black and white triangular area contains A circle distributed at the center of each black and white triangular block The intersections between the black and white triangular blocks within the pattern are the pattern's feature points, and each feature point has a unique number.

[0008] In a preferred embodiment, the visual positioning system includes two industrial cameras of the same model, which together form a stereo vision system device, and the stereo vision device is fixed on a camera gimbal bracket.

[0009] The visual positioning system is fixed on a movable support of a mobile cart and establishes the positional relationship between the stereo camera and surgical instruments through position-aware patterns.

[0010] In a preferred embodiment, the control system consists of an industrial control computer and a high-definition display screen. The registration system can assist surgeons in registering surgical instruments in real time and obtaining and locating information of the tip of the surgical instruments, and display the registration results and positioning results on the display screen of the control system.

[0011] In a preferred embodiment, the industrial control computer of the control system is installed inside the mobile vehicle, and the display screen is installed on the support rod of the mobile vehicle, thus forming an integrated structure of the stereo vision device and the control device.

[0012] The present invention also provides a method for real-time registration of surgical instruments based on position-aware patterns, employing the aforementioned real-time registration system for surgical instruments based on position-aware patterns, including: real-time registration of a non-planar three-dimensional model of a position-aware pattern, and acquisition and positioning of the tip of the surgical instrument.

[0013] In a preferred embodiment, the real-time registration of the position-aware pattern non-planar three-dimensional model, establishing the pose relationship between visual markers on surgical instruments and the camera, includes:

[0014] The visual positioning system uses a checkerboard calibration plate to calibrate the camera parameters of the stereoscopic vision device and establish a camera coordinate system in space. The vision device detects feature points in a position-aware pattern in an image in real time, calculates the 3D coordinates of all feature points in the current image in the camera coordinate system, and stores them in the system's backend database. The backend system selects high-quality feature points and stitches their 3D coordinates together to complete the 3D reconstruction of the feature point cloud of the position-aware pattern and establishes a point cloud coordinate system. .

[0015] In a preferred embodiment, the acquisition and positioning of the surgical instrument tip includes:

[0016] The visual positioning system establishes a camera coordinate system through a position-aware pattern. and point cloud coordinate system Surgical instruments with relative poses and attached position-aware patterns are moved with their tips as the center of a sphere. A vision device detects feature points on the position-aware pattern in real time and calculates the relative poses. Camera coordinate system and point cloud coordinate system Motion transformation matrix between Then, by solving the homogeneous transformation equation, the position of the surgical instrument tip in the camera coordinate system can be determined. Record the camera coordinate system in three-dimensional coordinates. The entire registration of surgical instruments is completed using the lower point cloud model and the tip's three-dimensional coordinates.

[0017] In a preferred embodiment, the detection of visual markers on the surgical instruments, and the real-time detection and localization of position-aware patterns, includes: the visual localization device detecting feature points on the position-aware patterns on the surgical instruments, obtaining the two-dimensional coordinates of the feature points in the image, and using the principle of stereo vision to derive the three-dimensional coordinates of the feature points in the camera coordinate system. Then, the point cloud coordinate system on the surgical instruments is solved using the point cloud model. and camera coordinate system The relative poses between the surgical instruments are then used to obtain the real-time poses of the surgical instruments through a point cloud model of the registered surgical instruments and the position-aware pattern.

[0018] Compared with the prior art, the present invention has the following beneficial effects: the present invention can register the position-aware pattern three-dimensional point cloud model and fit the position pose of the surgical instrument tip in real time, and display the results on the display screen of the control system, realizing real-time registration and real-time positioning, and providing convenient and accurate real-time navigation for clinical surgery. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a real-time registration and positioning method for surgical instruments based on position-aware patterns, according to an embodiment of the present invention.

[0020] Figure 2This is a schematic diagram illustrating the principle of a real-time registration and positioning method for surgical instruments based on position-aware patterns, according to an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the position-aware pattern in a real-time registration and positioning method for surgical instruments based on position-aware patterns, as described in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of a minimally invasive hip joint surgical instrument with a position-aware pattern, used in a real-time registration and positioning method for surgical instruments based on position-aware patterns according to an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the integrated structure of the visual positioning device, control host, and display screen in a real-time registration and positioning method for surgical instruments based on position-aware patterns according to an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram illustrating the specific process of a real-time registration method for surgical instruments based on position-aware patterns according to an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram illustrating the real-time stitching principle of a point cloud model in a method for real-time registration of surgical instruments based on position-aware patterns, according to an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram illustrating the specific process of fitting the tip of a surgical instrument in a real-time registration method for surgical instruments based on a position-aware pattern, according to an embodiment of the present invention.

[0027] Figure 9 This is an example diagram of real-time stitching of point cloud models in a method for real-time registration of surgical instruments based on position-aware patterns according to an embodiment of the present invention.

[0028] Explanation of reference numerals in the attached figures:

[0029] 1. Stereo vision device (visual positioning system); 2. Main unit + display screen (control system); 3. Surgical instruments; 4. Position-aware pattern; 5. Registered point cloud model; Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0033] Please refer to the accompanying drawings. An embodiment of the present invention provides a real-time registration and positioning system and method for surgical instruments based on position-aware patterns.

[0034] The embodiments of this disclosure provide a real-time registration system for surgical instruments based on position-aware patterns, such as... Figure 1 and Figure 2 As shown, Figure 1 This is a schematic diagram of a real-time registration and positioning method for surgical instruments based on position-aware patterns according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the principle of a real-time registration method for surgical instruments based on position-aware patterns according to an embodiment of the present invention. Note: Figure 1 and Figure 2 This invention is intended to help researchers and technicians in the field understand its technical processes and content.

[0035] Depend on Figure 1-9 As shown, the real-time registration system of this invention includes the following hardware components: a stereoscopic vision device, a position-aware pattern, a control system host, and a display screen. Figure 2 As shown, the working principle of this real-time registration system is as follows: 1. Preparation stage: calibration of the stereo vision device and attachment of position-aware patterns; 2. Registration stage: feature detection of position-aware patterns, acquisition and filtering of real-time image keyframes, tree-like stitching of point clouds in the feature database, and fitting of surgical instrument tips. The visual positioning system calculates the motion transformation matrix between various positions by detecting the position-aware patterns on the surgical instruments from multiple angles. This allows for the calculation and splicing of a position-aware pattern feature point cloud model.

[0036] Depend on Figure 1 As shown, the present invention provides a real-time registration system for surgical instruments based on position-aware patterns, including a stereo vision device 1 of a visual positioning system, a host computer and a display screen 2 of a control system, surgical instruments 3 with position-aware patterns attached, and a feature point cloud model 5 stitched together in real time; wherein, the position-aware pattern 4 is used to assist in positioning; the stereo vision device 1 is used to detect and position the position-aware pattern on the surgical instrument; the host computer and the display screen are used to stitch together the feature point cloud model in real time and to fit the pose of the tip of the surgical instrument in real time and present it on the display screen.

[0037] According to embodiments of the present invention, by Figure 3 As shown, the position-aware pattern is a checkerboard-like pattern composed of alternating black and white triangular blocks. Figure 4 As shown, it is set at the tail end of the surgical instrument and fits the cylindrical surface of the tail end of the surgical instrument.

[0038] According to a further aspect of the invention, in the pattern composed of alternating black and white triangular blocks, in The black and white triangular area contains The pattern consists of dots distributed at the center of each black and white triangular block. The intersections between the black and white triangular blocks are the feature points of the pattern, and each feature point has a unique number. For example... Figure 3 As shown, the location-aware pattern has a high density of feature points, and any part of it has an independent self-identification unit.

[0039] According to a further aspect of the present invention, the position-sensing pattern is applied to the outer surface of the cylindrical structure at the tail of the surgical instrument using laser engraving technology or as a matte self-adhesive sticker, such as... Figure 4 As shown.

[0040] According to a further aspect of the invention, by Figure 1 and Figure 5 As shown, the visual positioning system includes two identical industrial cameras, which together form a stereo vision system. The stereo vision device is fixed to a camera pan-tilt mount. Specifically, the stereo vision device uses a pair of identical Hikvision industrial cameras, which are fixed to the camera pan-tilt mount. Meanwhile, as... Figure 5 As shown, it is fixed to the robotic arm of the movable cart. (By...) Figure 1 As shown, the pose relationship between the stereo camera and surgical instruments is established through position-aware patterns.

[0041] According to a further aspect of the invention, such as Figure 1 and Figure 5 As shown, the control system 2 consists of an industrial control computer and a high-definition display screen. The registration system assists surgeons in registering surgical instruments in real time and acquiring and locating the tip information of the surgical instruments, displaying the registration and positioning results on the display screen of the control system. The industrial control computer of the control system is installed inside the mobile cart, and the display screen is installed on the support rod of the mobile cart, forming an integrated structure of the stereoscopic vision device and the control device.

[0042] During the preoperative preparation stage, control system 2 is used for the calibration of the stereoscopic vision device; during the intraoperative stage, control system 2 is used to complete the real-time detection of position-aware patterns, real-time feature stitching, and real-time fitting and positioning of the surgical instrument tip pose by the visual positioning system.

[0043] According to a further aspect of the present invention, the real-time registration of the position-aware pattern non-planar three-dimensional model establishes the pose relationship between visual markers on surgical instruments and the camera, such as... Figure 6 As shown, it includes:

[0044] Step 1: The visual positioning system uses a black and white checkerboard calibration plate to calibrate the camera parameters of the stereo vision device and establish a camera coordinate system in space. .

[0045] Step 2: The vision device detects feature points in the position-aware pattern in the image in real time and calculates the coordinates of all feature points in the current image in the camera coordinate system. The three-dimensional coordinates are stored in the system's backend database.

[0046] Step 3: The backend system selects high-quality feature points and stitches their 3D coordinates together to complete the 3D reconstruction of the point cloud of the position-aware pattern feature and establish a point cloud coordinate system. .

[0047] According to an embodiment of the present invention, in step 1, the stereo vision device captures approximately 26 images of a black and white checkerboard calibration board (20mm / grid). The stereo vision device is then calibrated using MATLAB's stereoCameraCalibration tool to obtain camera parameters and establish a camera coordinate system in space. .

[0048] According to an embodiment of the present invention, in step 2, the visual positioning system detects the pixel coordinates of all feature points in the current image in the image coordinate system from the captured image of surgical instruments with position-aware patterns. Using the camera parameters calibrated in step 1, the three-dimensional coordinates of the feature points in the camera coordinate system are calculated through triangulation. When the vision device captures images in real time from multiple directions and angles, the control system evaluates and filters the quality of each detected frame, storing suitable keyframes in the database.

[0049] According to an embodiment of the present invention, in step 3, the backend system selects the optimal keyframe as the starting point for tree-like stitching, and matches other keyframes layer by layer in the database, such as... Figure 7 As shown, the motion transformation matrix of the 3D coordinates of the feature point cloud in two keyframes is calculated using the SVD decomposition method. The feature point cloud is stitched into the optimal keyframe layer by layer, and the model is continuously improved and output, such as... Figure 9 As shown in the example, real-time registration of a location-aware pattern feature point cloud model is completed.

[0050] According to a further aspect of the present invention, the tip fitting and positioning of the surgical instrument, such as... Figure 8 As shown, it includes:

[0051] Step 1: The visual positioning system establishes a camera coordinate system using a position-aware pattern. and point cloud coordinate system Surgical instruments with position-aware patterns attached to them are positioned relative to each other. The instruments are moved with their tips as the center of the ball. The vision device detects the position-aware patterns in real time and calculates the three-dimensional coordinates of feature points on the image.

[0052] Step 2: Record the feature information of each frame in real time, filter it, and store it in the backend database to wait for the backend to fit the tip's 3D coordinates.

[0053] Step 3: Find Camera coordinate system and point cloud coordinate system Motion transformation matrix between The equations of motion for the surgical instrument tip in the camera coordinate system are solved by homogeneous transformation. Record the camera coordinate system in three-dimensional coordinates. The entire registration of surgical instruments is completed using the lower point cloud model and the tip's three-dimensional coordinates.

[0054] According to an embodiment of the present invention, the homogeneous transformation equation of motion is as follows:

[0055] (1)

[0056] (2)

[0057] in It is 6 A matrix of tip coordinates, where the first three elements are the fitted three-dimensional coordinates of the surgical instrument tip. .

[0058] According to a further aspect of the present invention, the detection of visual markers on surgical instruments and the real-time detection and localization of position-aware patterns include: a visual positioning device detecting position-aware patterns on surgical instruments, obtaining the two-dimensional coordinates of feature points in an image, solving for the three-dimensional coordinates of feature points in camera coordinates using the principle of stereo vision, and solving for the point cloud coordinate system on the surgical instruments using a point cloud model. and camera coordinate system The relative poses between the surgical instruments are then used to obtain the real-time poses of the surgical instruments through a point cloud model of the registered surgical instruments and the position-aware pattern.

[0059] Through the above methods, the real-time registration and positioning system for surgical instruments based on position-aware patterns provided by the technical embodiments of the present invention achieves the registration and navigation positioning of surgical instruments and meets the real-time requirements, providing convenience for doctors before and after the surgical process. The system displays the registration and positioning results on the display screen of the control system through Visual Studio 2022 and OpenCV library.

[0060] It should be noted that the above registration and positioning methods are all part of the system's computation process. For clinicians, they only need to present the surgical instruments with the position-aware patterns attached to them within the field of view of the visual positioning system to complete the detection of the patterns and the positioning of the surgical instruments.

[0061] The present invention has now been described in detail with reference to the accompanying drawings. Although the present disclosure has been described in conjunction with the drawings, the embodiments disclosed in the drawings are intended to provide illustrative examples of preferred embodiments and should not be construed as limiting the present disclosure. The dimensions and proportions in the drawings are merely schematic and should not be construed as limiting the present disclosure. The above embodiments are only for illustrating the technical concept flow and features of the present invention, and are intended to enable those skilled in the art to understand the content of the invention and implement it accordingly, and should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made according to the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A real-time registration system for surgical instruments based on position-aware patterns, used for real-time registration of surgical instruments during surgery and preoperative preparation, characterized in that, include: Position-aware patterns are used to assist in the positioning of surgical instruments; A visual positioning system is used to detect position-aware patterns on surgical instruments; a control system is used for real-time registration of 3D models of position-aware pattern features and for acquiring and real-time positioning of surgical instrument tip information. The position-sensing pattern is specifically a checkerboard-like pattern composed of alternating black and white triangular blocks, set at the tail end of the surgical instrument and conforming to the cylindrical surface of the tail end of the surgical instrument; the position-sensing pattern is laser-engraved or made into a matte self-adhesive sticker and attached to the outer surface of the cylindrical structure at the tail end of the surgical instrument; In the pattern composed of alternating black and white triangular blocks, The black and white triangular area contains A circle distributed at the center of each black and white triangular block The intersections of the black and white triangular blocks within the pattern are the pattern's feature points, each with a unique number; The system's registration methods include: real-time registration of position-aware pattern non-planar 3D models, and acquisition and positioning of surgical instrument tips; Real-time registration of position-aware patterned non-planar 3D models, establishing the pose relationship between visual markers on surgical instruments and the camera, including: The visual positioning system uses a checkerboard calibration plate to calibrate the camera parameters of the stereoscopic vision device and establish a camera coordinate system in space. The vision device detects feature points in a position-aware pattern in an image in real time, calculates the 3D coordinates of all feature points in the current image in the camera coordinate system, and stores them in the system's backend database. The backend system selects high-quality feature points and stitches their 3D coordinates together to complete the 3D reconstruction of the feature point cloud of the position-aware pattern and establishes a point cloud coordinate system. ; Acquisition and positioning of surgical instrument tips, including: The visual positioning system establishes a camera coordinate system through a position-aware pattern. and point cloud coordinate system Surgical instruments with relative poses and attached position-aware patterns are moved with their tips as the center of a sphere. A vision device detects feature points on the position-aware pattern in real time and calculates the relative poses. Camera coordinate system and point cloud coordinate system Motion transformation matrix between Then, by solving the homogeneous transformation equation, the position of the surgical instrument tip in the camera coordinate system can be determined. Record the camera coordinate system in three-dimensional coordinates. The point cloud model and the tip's 3D coordinates are used to complete the overall registration of the surgical instruments; Detecting visual markers on surgical instruments and performing real-time detection and localization of position-aware patterns includes: the visual positioning system detecting feature points on the position-aware patterns on the surgical instruments, obtaining the two-dimensional coordinates of the feature points in the image, and using the principle of stereo vision to deduce the three-dimensional coordinates of the feature points in the camera coordinate system. Then, the point cloud coordinate system on the surgical instruments is solved using the point cloud model. and camera coordinate system The relative poses between the surgical instruments are then used to obtain the real-time poses of the surgical instruments through a point cloud model of the registered surgical instruments and the position-aware pattern.

2. The real-time registration system for surgical instruments based on position-aware patterns according to claim 1, characterized in that, The visual positioning system includes two industrial cameras of the same model. The two industrial cameras form the stereo vision device, and the stereo vision device is fixed on the camera gimbal bracket. The visual positioning system is fixed on a movable support of a mobile cart and establishes the positional relationship between the stereo camera and surgical instruments through position-aware patterns.

3. The real-time registration system for surgical instruments based on position-aware patterns according to claim 1, characterized in that, The control system consists of an industrial control computer and a high-definition display screen. The registration system can assist surgeons in registering surgical instruments in real time and obtaining and locating information of the tip of the surgical instruments, and display the registration results and positioning results on the display screen of the control system.

4. The real-time registration system for surgical instruments based on position-aware patterns according to claim 3, characterized in that, The industrial control computer of the control system is installed inside the mobile trolley, and the display screen is installed on the support rod of the mobile trolley, forming an integrated structure of stereo vision device and control device.

Images