A real-time tracking system for spinal motion and method thereof

Abstract

This invention relates to the field of image processing technology, specifically disclosing a real-time spinal motion tracking system and method. The system includes: a vertebral registration module for registering images of a target vertebra with preoperative images; a camera module for acquiring depth and visible light images of the registered target vertebra during surgery, and reconstructing a three-dimensional surface point cloud of the target vertebra based on the depth image; a feature point recognition module for performing two-dimensional feature point recognition on the visible light image acquired by the camera module to obtain three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra during surgery; an optical flow tracking module for setting position templates for the two-dimensional feature points and their corresponding three-dimensional associated points, and performing optical flow tracking between the current frame image acquired by the camera module and the previous frame image; and a motion tracking module for calculating the motion amplitude of the target vertebra based on the optical flow tracking. This invention can directly perform real-time motion tracking of the target vertebra itself, reducing computational complexity and ensuring the real-time performance of the algorithm. Low computational complexity ensures real-time algorithm performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a real-time spinal motion tracking system and method. Background Technology

[0002] In orthopedic spinal navigation surgery, a crucial step, such as the intraoperative registration of vertebral surface point clouds with preoperative CT or MRI point clouds, is how to track the movement of the target vertebra in real time and accurately during surgery. Current mainstream solutions typically fix the tracer to a vertebra or soft tissue adjacent to the target vertebra in the surgical area, assuming a rigid connection between the tracer and the body. An optical tracking system tracks the tracer's movement to replace the target vertebra's movement. However, the connections between vertebrae and between vertebrae and soft tissues in the spine are actually non-rigid. Relative movement may exist between the tracer and the target vertebra. Without effective motion compensation, the tracer's movement may not be able to replace the target vertebra's movement.

[0003] To address this, a real-time spinal motion tracking algorithm is proposed. This algorithm directly tracks the target vertebra itself, rather than tracking a tracer fixed around the target vertebra. Existing optical tracking schemes for direct tracking of the target vertebra typically employ a 3D camera to reconstruct the surface point cloud and directly identify and track the vertebra's unique bony features on the 3D point cloud. This approach is complex and difficult to implement. Summary of the Invention

[0004] Purpose of the invention: To address the above-mentioned shortcomings, this invention proposes a real-time spinal motion tracking system and method, which can directly track the motion of the target vertebra itself in real time, reducing computational complexity and ensuring the real-time performance of the algorithm.

[0005] Technical solution: This invention provides a real-time spinal motion tracking system, comprising:

[0006] The vertebral registration module is used to register intraoperative images of the target vertebra with preoperative images.

[0007] The camera module is used to acquire depth and visible light images of the target vertebra during surgery after registration, thereby setting the surgical area tracking region and reconstructing the three-dimensional surface point cloud of the target vertebra based on the depth images.

[0008] The feature point recognition module is used to perform two-dimensional feature point recognition on the visible light image of the target vertebra acquired by the camera module, and to obtain the three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra during surgery based on the parameters of the camera module.

[0009] The optical flow tracking module is used to set the position templates of two-dimensional feature points and their corresponding three-dimensional associated points, and to perform optical flow tracking between the current frame image and the previous frame image acquired by the camera module.

[0010] The motion tracking module calculates the motion amplitude of the target vertebra between adjacent frames based on the optical flow tracking of the optical flow tracking module, thus obtaining the motion amplitude of the target vertebra.

[0011] Specifically, it also includes an optical tracking module, which obtains the pose of the camera module through the optical tracking unit, and accordingly resets the surgical area tracking region on the image acquired by the camera module, and updates the position templates of the two-dimensional feature points and their corresponding three-dimensional associated points in combination with the position template set by the optical flow tracking module.

[0012] More specifically, the optical flow tracking module acquires the positional relationship between the optical tracking unit and the camera module in real time, resets the surgical area tracking region on the image acquired by the camera module according to the positional relationship, and updates the position templates of the two-dimensional feature points and their corresponding three-dimensional associated points according to the positional relationship.

[0013] Furthermore, since the camera module and the optical tracking unit are integrated into one and rigidly connected, the positional relationship between the optical tracking unit and the camera module can be obtained.

[0014] Alternatively, the camera module and the optical tracking unit can be set independently of each other. A camera tracer is installed on the camera module. The optical tracking unit acquires the pose of the camera tracer and calculates the positional relationship between the optical tracking unit and the camera module. At the same time, the optical tracking module monitors the spatial motion of the camera tracer relative to the reference tracer in real time, calculates the spatial motion of the camera module relative to the patient's affected area, and updates the corresponding position template accordingly.

[0015] Specifically, several marker points are set on the target vertebra of the patient, and the marker points in the preoperative image of the target vertebra are obtained. During the operation, the navigation probe touches the marker points on the target vertebra, and the vertebral registration module obtains the position information of the optical tracer array on the navigation probe through the optical tracking unit, calculates the position information of each marker point in the intraoperative image of the target vertebra, and registers the intraoperative image of the target vertebra with the preoperative image accordingly.

[0016] Specifically, the registration of the vertebral registration module includes initial registration and re-registration. The initial registration is the first registration at the start of the surgery, and the re-registration is the operation performed after the motion tracking module determines that the motion amplitude of the target vertebra exceeds a set threshold.

[0017] Specifically, the camera module selects the target vertebra and its adjacent area within a set range as the surgical area tracking region on the depth image and visible light image of the patient's affected area it acquires.

[0018] Specifically, the ratio between the number of points in the three-dimensional surface point cloud of the intraoperative target vertebra reconstructed by the camera module and the number of two-dimensional points in its visible light image is greater than 80%.

[0019] Specifically, the camera module uses a computer vision library to perform corner detection on visible light images, thereby completing the two-dimensional feature point recognition of the visible light image of the target vertebra acquired by the camera module.

[0020] Specifically, the feature point recognition module obtains the number of two-dimensional feature points with corresponding three-dimensional associated points on the visible light image of the target vertebra. If the ratio of this number to the total number of two-dimensional feature points on the visible light image of the target vertebra is lower than a set ratio, the intraoperative image and preoperative image of the target vertebra are re-registered by the vertebra registration module until the aforementioned ratio is higher than the set ratio.

[0021] Specifically, if the current frame image acquired by the camera module is the first frame image after initial registration or re-registration, the optical flow tracking module verifies whether there is a three-dimensional associated point at the corresponding coordinate of the three-dimensional point cloud associated with the pixel coordinate of the two-dimensional feature point in the visible light image. If there is, the three-dimensional point is considered valid and set as the position template of the corresponding three-dimensional associated point in the three-dimensional point cloud. At the same time, the corresponding two-dimensional feature point is set as the position template of the two-dimensional feature point. The position templates of the two-dimensional feature point and the three-dimensional associated point are not changed before the next registration.

[0022] If the current frame image acquired by the camera module is a non-first frame image after initial registration or re-registration, then the two-dimensional feature points in the previous frame image and their corresponding three-dimensional associated points in the three-dimensional point cloud are set as the corresponding position templates. The optical flow tracking module uses the position templates of the two-dimensional feature points in the previous frame image as the center and performs two-dimensional feature point detection in a search area of ​​a set size on the current frame image. If a two-dimensional feature point is detected, it means that the two-dimensional feature point is successfully tracked.

[0023] More specifically, the search area is set as a range of motion formed by expanding k times outward from the target vertebra in the directions of upward, downward, left, and right.

[0024] More specifically, the optical flow tracking module calculates the ratio between the number of successfully tracked two-dimensional feature points and the number of position templates of two-dimensional feature points in the previous frame image. When the ratio is less than a set proportion, the intraoperative image and preoperative image of the target vertebra are re-registered by the vertebral registration module until the ratio is higher than the set proportion.

[0025] Specifically, the optical flow tracking module employs sparse optical flow tracking to track the optical flow between the current frame image acquired by the camera module and the previous frame image.

[0026] Specifically, the motion tracking module calculates the Euclidean distance between corresponding three-dimensional points in adjacent frame images based on the optical flow tracking of the optical flow tracking module, and calculates the average value of their accumulated values ​​to obtain the motion amplitude of the target vertebra between adjacent frame images.

[0027] More specifically, when a two-dimensional feature point does not have a corresponding three-dimensional point, the motion tracking module calculates the Euclidean distance between the two-dimensional feature point and its corresponding two-dimensional feature point in the previous frame image and replaces it with the distance in the cumulative calculation.

[0028] The present invention also provides a method for real-time spinal motion tracking based on the aforementioned real-time spinal motion tracking system, comprising the following steps:

[0029] S1, the vertebral registration module registers the intraoperative images of the target vertebra with the preoperative images;

[0030] S2. Use a camera module to acquire depth and visible light images of the target vertebra during surgery, set the surgical area tracking region accordingly, and reconstruct its three-dimensional surface point cloud based on the depth image to obtain the three-dimensional surface point cloud of the target vertebra during surgery.

[0031] S3, the feature point recognition module performs feature point recognition on the visible light image of the target vertebra obtained in S2, and obtains the three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra obtained in S2 according to the parameters of the camera module.

[0032] S4. The optical flow tracking module determines whether the current frame image captured by the camera module is the first frame image after registration by S1.

[0033] If so, then set the two-dimensional feature points in the visible light image and the corresponding three-dimensional associated points in the three-dimensional point cloud as the corresponding position templates;

[0034] Otherwise, the two-dimensional feature points in the visible light image of the previous frame image acquired by the camera module and the corresponding three-dimensional associated points in the three-dimensional point cloud are set as the corresponding position templates, and optical flow tracking is performed between the current frame image acquired by the camera module and the previous frame image.

[0035] S5. The motion tracking module calculates the motion amplitude of the target vertebra between adjacent frames based on the optical flow tracking in S4, thus obtaining the motion amplitude of the target vertebra.

[0036] Beneficial effects: This invention performs optical flow tracking based on two-dimensional feature point recognition on visible light images, and then uses the spatial motion of the corresponding three-dimensional point cloud directly indexed by the two-dimensional feature points to replace the motion of the vertebrae, constructing a 2D-3D collaborative tracking system that can directly track the real-time motion of the target vertebrae themselves. Compared with direct bony feature recognition and tracking on the three-dimensional point cloud, two-dimensional feature point recognition and tracking greatly reduces computational complexity, thus ensuring the real-time performance of the algorithm. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 An example image showing the fusion of a visible light image with its reconstructed 3D surface point cloud, where... Figure 1 (a) is an example image of a sheep scorpion in visible light. Figure 1 (b) is an example image of the reconstructed 3D surface point cloud of a sheep scorpion. Figure 1 (c) is an example image of the fusion of a visible light image of a sheep scorpion and its reconstructed 3D surface point cloud;

[0039] Figure 2 An example image is provided to correlate two-dimensional feature points in a visible light image with corresponding three-dimensional points in the reconstructed three-dimensional surface point cloud. Figure 2 The left image is Figure 1 (a) An example diagram of the surgical area tracking region defined thereon. Figure 2 The right image is Figure 1 (c) An example diagram of the surgical area tracking region set thereon;

[0040] Figure 3 This is a structural example diagram of a real-time tracking of spinal motion according to an embodiment of the present invention;

[0041] Figure 4 This is a structural example diagram of real-time spinal motion tracking according to another embodiment of the present invention;

[0042] Figure 5 This is a flowchart of the real-time spinal motion tracking method of the present invention.

[0043] In the figure, 1. Optical tracking unit, 2. Patient, 3. Reference tracer, 4. Anatomical region, 5. Target vertebra, 6. Bed, 7. RGB-D camera, 8. Camera tracer;

[0044] 71. Depth camera; 72. Visible light camera;

[0045] A. Surgical area tracking region, P1. Two-dimensional feature point, P2. Three-dimensional point associated with the two-dimensional feature point. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings. It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this invention should have the ordinary meaning understood by those skilled in the art.

[0047] The real-time spinal motion tracking system of the present invention includes:

[0048] The vertebral registration module is used to register the intraoperative images of the target vertebra with the preoperative images of the target vertebra.

[0049] The camera module is used to acquire depth and visible light images of the target vertebra during surgery after registration, thereby setting the surgical area tracking region and reconstructing the three-dimensional surface point cloud of the target vertebra based on the depth images.

[0050] The feature point recognition module is used to perform two-dimensional feature point recognition on the visible light image of the target vertebra acquired by the camera module, and to obtain the three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra during surgery based on the parameters of the camera module.

[0051] The optical flow tracking module is used to set the position templates of two-dimensional feature points and their corresponding three-dimensional associated points, and to perform optical flow tracking between the current frame image and the previous frame image acquired by the camera module.

[0052] The motion tracking module calculates the motion amplitude between adjacent frames based on the optical flow tracking module, thus obtaining the motion amplitude of the target vertebra.

[0053] The present invention also includes an optical tracking module, which obtains the pose of the camera module through the optical tracking unit, thereby resetting the surgical area tracking region on the image acquired by the camera module, and updating the position template of the two-dimensional feature points and their corresponding three-dimensional associated points in combination with the position template set by the optical flow tracking module.

[0054] In this invention, several marker points are set on the target vertebra of the patient, and the marker points in the preoperative image of the target vertebra are obtained. During the operation, the navigation probe touches the marker points on the target vertebra of the patient. The vertebra registration module obtains the position information of the optical tracer array on the navigation probe through the optical tracking unit, and then calculates the position information of each marker point and displays it in the intraoperative image of the target vertebra. Based on this, the intraoperative image of the target vertebra is registered with the preoperative image.

[0055] In this invention, an optical tracer array is set on the navigation probe, and the navigation probe is positioned on each marker point. The optical tracking module can obtain the position information of the navigation probe tip by acquiring the pose of the optical tracer array, and then obtain the position of all marker points.

[0056] In this invention, after the vertebral registration module acquires the positions of the marker points on the surface of the target vertebra acquired by the navigation probe during surgery and displays them in the intraoperative image of the target vertebra, it performs coarse registration by matching the marker points in the preoperative image of the target vertebra. Fine registration is then performed using the ICP algorithm. Furthermore, this invention allows for the random selection of verification points on the surface of the target vertebra by the navigation probe to verify the registration accuracy, ensuring precise matching between the intraoperative and preoperative images of the target vertebra and meeting the accuracy requirements of the optical flow tracking and motion tracking modules of this invention.

[0057] In this invention, the preoperative image of the target vertebra can be a preoperative CT image or MRT point cloud of the target vertebra, obtained in advance. The intraoperative image of the target vertebra can be a CBCT point cloud of the target vertebra acquired during the operation using a C-arm CT scanner or a depth point cloud acquired using a depth camera.

[0058] In this invention, real-time spinal motion tracking can be applied to various spinal navigation surgeries. This embodiment only uses the registration of the intraoperative three-dimensional point cloud of the patient's affected area, i.e., the target vertebra, with the preoperative CT image or MRT point cloud as an example for illustration.

[0059] In this invention, the preoperative image of the patient's affected area is generally a processed pure vertebral point cloud. However, during surgery, only a portion of the vertebrae and various tissues are exposed at the affected area, which does not correspond to the pure vertebral region in the preoperative image. Therefore, in this invention, during registration, the target vertebra of the patient during surgery should be registered with the corresponding vertebral region in the preoperative image of the affected area. Figure 1 As shown in the diagram, in current mainstream methods, doctors clean surface tissues with specific bony features, such as spinous processes and lateral processes, to improve registration accuracy and lay the foundation for subsequent tracking algorithms. Generally speaking, the larger the area of ​​tissue cleaned from the vertebral surface, the higher the registration accuracy, and the higher the accuracy of subsequent real-time spinal motion tracking.

[0060] In this invention, the vertebral registration module involves initial registration and re-registration. Initial registration is the first registration at the start of surgery, while re-registration occurs after the target vertebra's motion amplitude exceeds a set threshold. Theoretically, the two are only different in time sequence; the consistency of the registration method is not specifically required, and it is not limited to any current registration method that meets the required registration accuracy. In this invention, the set threshold can be determined according to actual needs.

[0061] In this invention, in actual surgical scenarios, after the surgeon moves the camera module to a suitable working distance and angle, its region of interest will include the entire anatomical area. To accelerate the reconstruction of the three-dimensional surface point cloud of the patient's affected area and reduce interference from non-bone features when identifying two-dimensional feature points in visible light images, the target vertebra and its adjacent areas within a defined range can be selected on the depth and visible light images of the patient's affected area as the surgical area tracking region, based on the allowable range of vertebral movement. Figure 2 The area referred to by A in the middle.

[0062] In this invention, due to the high precision requirements of spinal registration, the target vertebra in the spine is generally only allowed to move at the millimeter level after registration. Therefore, the precision of the camera module used in this invention should also reach the sub-millimeter level, and the points in the reconstructed three-dimensional surface point cloud should be dense. Specifically, the requirement that can be met is that the ratio between the number of points in the reconstructed three-dimensional surface point cloud and the number of two-dimensional points on the visible light image of the target vertebra should be greater than 80%.

[0063] In this invention, the feature point recognition module performs two-dimensional feature point recognition on the visible light image of the target vertebra by: using computer vision libraries such as OpenCV, performing corner detection on the visible light image to obtain the corresponding two-dimensional feature points; the corner point type is not limited to robust corner points such as Harris and Shi-Tomasi. In this invention, the selected corner point type should be compatible with the subsequent optical flow tracing algorithm.

[0064] Furthermore, the present invention can further solve for the corresponding sub-pixel corner points based on the corner detection results, thereby obtaining more accurate two-dimensional feature points, which can improve the accuracy of subsequent optical flow tracking algorithms and achieve good tracking results.

[0065] Existing camera modules, such as RGB-D cameras, undergo camera calibration before leaving the factory. They generally already implement visible light image and depth image calibration. This means there's a mapping function—the parameters of the aforementioned RGB-D camera—that associates the pixel points of the two-dimensional image in the visible light image with the coordinates of points in the three-dimensional surface point cloud of the depth image. This allows the 3D associated points of the two-dimensional feature points in the visible light image to be obtained in the 3D surface point cloud of the target vertebra during surgery. Figure 2 As shown in P2. Based on the mapping function, each 3D point in the 3D surface point cloud of the target vertebra during surgery can find a unique corresponding 2D pixel in the 2D image of the visible light image, such as... Figure 2As shown in P1, since the depth camera and visible light camera in the camera module are rigidly connected, the mapping function remains unchanged once determined, provided the mechanical structure is not altered. Therefore, the pixel coordinates of the two-dimensional feature points detected in each frame of the visible light image can be directly obtained from the coordinates of the corresponding three-dimensional associated points in the three-dimensional surface point cloud through the mapping function, thus saving the time spent calculating bony features on the three-dimensional surface point cloud each time. It should be noted that due to factors such as the reconstruction angle, the number of three-dimensional points in the three-dimensional surface point cloud reconstructed by the camera module is not exactly the same as the number of two-dimensional feature points in the visible light image. Generally, the number of three-dimensional points in the three-dimensional surface point cloud is less than or equal to the number of two-dimensional feature points in the visible light image. Therefore, it is necessary to further determine whether there are three-dimensional associated points in the three-dimensional surface point cloud corresponding to the two-dimensional feature points in the visible light image. When the proportion of non-existent points reaches a certain level, for safety reasons, the intraoperative image and preoperative image of the target vertebra should be re-registered. At this time, the position of the camera module can be adjusted appropriately, and the intraoperative image and preoperative image of the target vertebra can be re-registered through the vertebra registration module until the aforementioned proportion is higher than the set ratio.

[0066] In this invention, if the current frame image acquired by the camera module is the first frame image after initial registration or re-registration, the optical flow tracking module verifies whether there is a three-dimensional associated point at the corresponding coordinate of the three-dimensional point cloud associated with the pixel coordinates of the two-dimensional feature points in the visible light image. If there is, the three-dimensional point is considered valid and set as the position template of the corresponding three-dimensional associated point in the three-dimensional point cloud. At the same time, the corresponding two-dimensional feature point is set as the position template of the two-dimensional feature point. The position templates of the two-dimensional feature point and the three-dimensional associated point are not changed before the next registration.

[0067] In this invention, if the current frame image acquired by the camera module is a non-first frame image after initial registration or re-registration, then the two-dimensional feature points in the previous frame image and the corresponding three-dimensional associated points in the three-dimensional point cloud are set as the corresponding position templates. The optical flow tracking module uses the position templates of the two-dimensional feature points in the previous frame image as the center and performs two-dimensional feature point detection within a search area of ​​a set size on the current frame image. If a two-dimensional feature point is detected, it indicates that the two-dimensional feature point is successfully tracked.

[0068] In a specific embodiment of the present invention, the search area of ​​a set size can be a rectangle or a circle.

[0069] In this invention, to improve recognition accuracy, the search area is generally not set too large or too small. This is to avoid frequent re-registration due to surgical interruptions caused by the failure to recognize a large number of two-dimensional feature points during tracking, or tracking abnormalities caused by a large number of corner point recognition errors, which could lead to medical accidents. In this invention, the search area is generally set as a range of motion formed by expanding k times outward from the target vertebra in the upward, downward, left, and right directions, respectively, where k can be 2-4.

[0070] In this invention, the optical flow tracking module calculates the ratio between the number of successfully tracked two-dimensional feature points and the number of position templates of two-dimensional feature points in the previous frame image. When the ratio is less than a set ratio, the intraoperative image of the target vertebra and the preoperative image of the target vertebra need to be re-registered by the vertebral registration module until the ratio is higher than the set ratio.

[0071] In this invention, sparse optical flow tracking is used for optical flow tracing. Compared with dense optical flow tracing, sparse optical flow tracing selects key pixels to replace the whole, which has the advantages of low computational cost, fast computation speed, and suitability for scenarios with high real-time requirements. However, it may not be as accurate as dense optical flow tracing when dealing with large-scale object motion. But in the application scenario of this invention, only small-scale movement of the target vertebra is generally allowed within millimeters, so sparse optical flow tracing can be used.

[0072] Furthermore, the present invention can employ LK optical flow tracking to process moving objects of different scales, thereby reducing noise interference and exhibiting good robustness.

[0073] In this invention, during the operation, the doctor may intentionally or unintentionally move the camera module, such as the RGB-D camera, in order to obtain a better surgical position or perform other operations. At this time, the template position of the two-dimensional feature points and their corresponding three-dimensional associated points set in the registration process changes with the relative position of the camera module. If it is not corrected, it may lead to the interruption of the operation.

[0074] The specific revisions are as follows:

[0075] The optical flow tracking module acquires the positional relationship between the optical tracking unit and the camera module in real time. Based on this positional relationship, it resets the surgical area tracking region on the image acquired by the camera module and updates the position templates of the two-dimensional feature points and their corresponding three-dimensional associated points based on this positional relationship.

[0076] Furthermore, it can be referred to Figure 3In one embodiment of the present invention, the patient 2 is placed on a hospital bed 6, and an anatomical region 4 is formed at the patient's affected area. Within the anatomical region 4 is the target vertebra 5. A reference tracer 3 is set at the patient's affected area. An RGB-D camera 7 and an optical tracking unit 1 are integrated into one unit, which integrates the binocular camera of the optical tracking unit 1 and the depth camera 71 and visible light camera 72 of the RGB-D camera 7. The two are rigidly connected, so the positional relationship between the optical tracking unit 1 and the RGB-D camera 7 is fixed and known. At the same time, the optical tracking unit 1 can acquire the pose of the reference tracer 3.

[0077] Furthermore, it can be referred to Figure 4 Patient 2 is placed on bed 6, and an anatomical region 4 is formed at the patient's affected area. Within the anatomical region 4 is the target vertebra 5. A reference tracer 3 is set at the patient's affected area. The RGB-D camera 7 and the optical tracking unit 1 are set independently. A camera tracer 8 is installed on the RGB-D camera 7. The optical tracking unit 1 can obtain the pose of the camera tracer 8 through the binocular camera and its optical sensor, and then calculate the positional relationship between the optical tracking unit 1 and the RGB-D camera 7. At the same time, the spatial movement of the camera tracer 8 relative to the reference tracer 3 is monitored in real time, that is, the spatial movement of the RGB-D camera 7 relative to the patient's affected area is obtained. The optical tracking module can make corrections based on this to update the position template.

[0078] In this invention, the motion tracking module calculates the Euclidean distance between corresponding three-dimensional points in adjacent frame images based on the optical flow tracking of the optical flow tracking module, and calculates the average value of their accumulated values ​​to obtain the motion amplitude of the target vertebra between adjacent frame images.

[0079] Furthermore, the motion tracking module can also determine whether the motion amplitude of the target vertebra between adjacent frames is greater than the aforementioned set threshold. If it is greater than the set threshold, the intraoperative image of the target vertebra and the preoperative image of the target vertebra need to be re-registered through the vertebra registration module.

[0080] In this invention, since not every two-dimensional feature point has a corresponding three-dimensional point, the motion tracking module can calculate the Euclidean distance between the two-dimensional feature point and its corresponding two-dimensional feature point in the previous frame and replace it to participate in the accumulation.

[0081] This invention also provides a method for real-time tracking of spinal motion, such as... Figure 5 As shown, the steps include:

[0082] S1. The intraoperative images of the target vertebra are registered with the preoperative images of the target vertebra using the vertebral registration module.

[0083] S2. Use a camera module to acquire depth and visible light images of the target vertebra during surgery, set the surgical area tracking region accordingly, and reconstruct its three-dimensional surface point cloud based on the depth image to obtain the three-dimensional surface point cloud of the target vertebra during surgery.

[0084] S3, the feature point recognition module performs feature point recognition on the visible light image of the target vertebra obtained in S2, and obtains the three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra obtained in S2 according to the parameters of the camera module.

[0085] S4. The optical flow tracking module determines whether the current frame image captured by the camera module is the first frame image after registration by S1.

[0086] If so, then set the two-dimensional feature points in the visible light image and the corresponding three-dimensional associated points in the three-dimensional point cloud as the corresponding position templates;

[0087] Otherwise, the two-dimensional feature points in the visible light image of the previous frame image acquired by the camera module and the corresponding three-dimensional associated points in the three-dimensional point cloud are set as the corresponding position templates, and optical flow tracking is performed between the current frame image acquired by the camera module and the previous frame image.

[0088] S5. The motion tracking module calculates the motion amplitude between adjacent frames based on the optical flow tracking in S4, thus obtaining the motion amplitude of the target vertebra.

[0089] This invention's camera module adds a visible light camera to a depth camera, simultaneously acquiring visible light images based on 3D surface point cloud reconstruction. Optical flow tracking is then performed on these visible light images based on 2D feature point recognition. The spatial motion of the corresponding 3D point cloud, directly indexed by the 2D feature points, is then used to replace the vertebral motion, constructing a 2D-3D collaborative tracking system. Compared to directly performing bony feature recognition and tracking on the 3D point cloud, 2D feature point recognition and tracking significantly reduces computational complexity, thus ensuring the algorithm's real-time performance.

[0090] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the invention as described above, which are not provided in the details for the sake of brevity.

[0091] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this invention should be included within the protection scope of this invention.

Claims

1. A real-time spinal motion tracking system, characterized in that, include: The vertebral registration module is used to register intraoperative images of the target vertebra with preoperative images. The camera module is used to acquire depth and visible light images of the target vertebra during surgery after registration, thereby setting the surgical area tracking region and reconstructing the three-dimensional surface point cloud of the target vertebra based on the depth images. The feature point recognition module is used to perform two-dimensional feature point recognition on the visible light image of the target vertebra acquired by the camera module, and to obtain the three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra during surgery based on the parameters of the camera module. The optical flow tracking module is used to set the position templates of two-dimensional feature points and their corresponding three-dimensional associated points, and to perform optical flow tracking between the current frame image and the previous frame image acquired by the camera module. If the current frame image acquired by the camera module is the first frame image after initial registration or re-registration, the optical flow tracking module verifies whether there is a three-dimensional associated point at the corresponding coordinate of the three-dimensional point cloud associated with the pixel coordinate of the two-dimensional feature point in the visible light image. If there is, the three-dimensional point is considered valid and set as the position template of the corresponding three-dimensional associated point in the three-dimensional point cloud. At the same time, the corresponding two-dimensional feature point is set as the position template of the two-dimensional feature point. The position templates of the two-dimensional feature point and the three-dimensional associated point will not be changed before the next registration. If the current frame image acquired by the camera module is a non-first frame image after initial registration or re-registration, then the two-dimensional feature points in the previous frame image and the corresponding three-dimensional associated points in the three-dimensional point cloud are set as the corresponding position templates. The optical flow tracking module uses the position templates of the two-dimensional feature points in the previous frame image as the center and performs two-dimensional feature point detection in a search area of ​​a set size on the current frame image. If a two-dimensional feature point is detected, it means that the two-dimensional feature point is successfully tracked. The motion tracking module calculates the motion amplitude of the target vertebra between adjacent frames based on the optical flow tracking of the optical flow tracking module, thus obtaining the motion amplitude of the target vertebra.

2. The real-time spinal motion tracking system according to claim 1, characterized in that, It also includes an optical tracking module, which obtains the pose of the camera module through the optical tracking unit, and accordingly resets the surgical area tracking region on the image acquired by the camera module, and updates the position templates of the two-dimensional feature points and their corresponding three-dimensional associated points in combination with the position template set by the optical flow tracking module.

3. The real-time spinal motion tracking system according to claim 2, characterized in that, The optical flow tracking module acquires the positional relationship between the optical tracking unit and the camera module in real time, and resets the surgical area tracking region on the image acquired by the camera module according to the positional relationship. At the same time, it updates the position templates of two-dimensional feature points and their corresponding three-dimensional associated points according to the positional relationship.

4. The real-time spinal motion tracking system according to claim 3, characterized in that, The camera module and the optical tracking unit are integrated into one unit and rigidly connected, so the positional relationship between the optical tracking unit and the camera module can be obtained. Alternatively, the camera module and the optical tracking unit are set independently of each other. A camera tracer is installed on the camera module. The optical tracking unit acquires the pose of the camera tracer and calculates the positional relationship between the optical tracking unit and the camera module. The optical tracking module monitors the spatial motion of the camera tracer relative to the reference tracer in real time, calculates the spatial motion of the camera module relative to the patient's affected area, and updates the corresponding position template accordingly.

5. The real-time spinal motion tracking system according to claim 1, characterized in that, Several marker points are set on the target vertebra of the patient, and the marker points in the preoperative image of the target vertebra are obtained. During the operation, the navigation probe touches the marker points on the target vertebra, and the vertebra registration module obtains the position information of the optical tracer array on the navigation probe through the optical tracking unit. The position information of each marker point in the intraoperative image of the target vertebra is calculated, and the intraoperative image of the target vertebra is registered with the preoperative image.

6. The real-time spinal motion tracking system according to claim 1, characterized in that, The vertebral registration module includes initial registration and re-registration. The initial registration is the first registration at the start of the surgery, and the re-registration is the operation performed after the motion tracking module determines that the motion amplitude of the target vertebra exceeds a set threshold.

7. The real-time spinal motion tracking system according to claim 1, characterized in that, The camera module selects the target vertebra and its adjacent area within a set range as the surgical area tracking region on the depth and visible light images of the patient's affected area it acquires.

8. The real-time spinal motion tracking system according to claim 1, characterized in that, The ratio between the number of points in the three-dimensional surface point cloud of the intraoperative target vertebra reconstructed by the camera module and the number of two-dimensional points in its visible light image is greater than 80%.

9. The real-time spinal motion tracking system according to claim 1, characterized in that, The camera module is based on a computer vision library and performs corner detection on visible light images to obtain two-dimensional feature point recognition of the visible light images of the target vertebra acquired by the camera module.

10. The real-time spinal motion tracking system according to claim 1, characterized in that, The feature point recognition module obtains the number of two-dimensional feature points with corresponding three-dimensional associated points on the visible light image of the target vertebra. If the ratio of this number to the total number of two-dimensional feature points on the visible light image of the target vertebra is lower than a set ratio, the intraoperative image and preoperative image of the target vertebra are re-registered by the vertebra registration module until the ratio is higher than the set ratio.

11. The real-time spinal motion tracking system according to claim 1, characterized in that, The search area is set as a range of motion formed by expanding k times outward from the target vertebra in the directions above, below, left, and right.

12. The real-time spinal motion tracking system according to claim 11, characterized in that, The optical flow tracking module calculates the ratio between the number of successfully tracked two-dimensional feature points and the number of position templates of two-dimensional feature points in the previous frame image. When the ratio is less than a set ratio, the intraoperative image and preoperative image of the target vertebra are re-registered by the vertebral registration module until the ratio is higher than the set ratio.

13. The real-time spinal motion tracking system according to claim 1, characterized in that, The optical flow tracking module employs sparse optical flow tracking to track the optical flow between the current frame image captured by the camera module and the previous frame image.

14. The real-time spinal motion tracking system according to claim 1, characterized in that, The motion tracking module calculates the Euclidean distance between corresponding three-dimensional points in adjacent frame images based on the optical flow tracking of the optical flow tracking module, and calculates the average value of their accumulated values ​​to obtain the motion amplitude of the target vertebra between adjacent frame images.

15. The real-time spinal motion tracking system according to claim 14, characterized in that, When a two-dimensional feature point does not have a corresponding three-dimensional point, the motion tracking module calculates the Euclidean distance between the two-dimensional feature point and its corresponding two-dimensional feature point in the previous frame image and replaces it, which is then added to the accumulation.

16. A method for real-time tracking of spinal motion based on the real-time spinal motion tracking system according to any one of claims 1-15, characterized in that, Including the following steps: S1, the vertebral registration module registers the intraoperative images of the target vertebra with the preoperative images; S2. Use a camera module to acquire depth and visible light images of the target vertebra during surgery, set the surgical area tracking region accordingly, and reconstruct its three-dimensional surface point cloud based on the depth image to obtain the three-dimensional surface point cloud of the target vertebra during surgery. S3, the feature point recognition module performs feature point recognition on the visible light image of the target vertebra obtained in S2, and obtains the three-dimensional associated points in the three-dimensional surface point cloud of the target vertebra obtained in S2 according to the parameters of the camera module. S4. The optical flow tracking module determines whether the current frame image captured by the camera module is the first frame image after registration by S1. If so, then set the two-dimensional feature points in the visible light image and the corresponding three-dimensional associated points in the three-dimensional point cloud as the corresponding position templates; Otherwise, the two-dimensional feature points in the visible light image of the previous frame image acquired by the camera module and the corresponding three-dimensional associated points in the three-dimensional point cloud are set as the corresponding position templates, and optical flow tracking is performed between the current frame image acquired by the camera module and the previous frame image. S5. The motion tracking module calculates the motion amplitude of the target vertebra between adjacent frames based on the optical flow tracking in S4, thus obtaining the motion amplitude of the target vertebra.

Images