Ankle arthroscope noninvasive navigation method based on anatomical constraint and binocular vision

Abstract

The invention discloses an ankle arthroscope noninvasive navigation method based on anatomical constraint and binocular vision. The ankle arthroscope noninvasive navigation method mainly comprises the following steps: reconstructing a three-dimensional skeleton triangular mesh model; extracting a talus pulley joint axis; defining body surface mark points of the patient and deploying a binocular vision environment; detecting mark points on the surface of the ankle skin on the operation side of the patient in real time, and performing three-dimensional mapping; calculating a physiologic effective rotation angle based on the anatomical constrained real-time posture of the ankle; and performing digital twinning synchronization and auxiliary decision making, and finally realizing non-invasive navigation of the ankle arthroscope. The method is completely non-invasive and non-radiation, and does not need to see through or implant markers in an operation, so that the operation complexity is effectively reduced. By introducing a manifold projection algorithm and a biomechanical constraint mechanism, noise data generated by skin sliding can be identified and stripped, and effective inhibition of STA is realized in noninvasive tracking. In addition, the real-time performance of the system is ensured by adopting a lightweight posture recognition network, expensive optical tracking equipment is not needed, the deployment cost is reduced, and the system has clinical popularization value.

Description

Technical Field

[0001] This invention relates to the fields of computer-assisted surgery (CAS) and medical image processing, and in particular to a non-invasive computer vision positioning, posture perception and augmented reality (AR) navigation method for use in ankle arthroscopic minimally invasive surgery. Background Technology

[0002] Ankle arthroscopy offers advantages such as minimal trauma and rapid recovery. However, the ankle joint space is extremely narrow, and the surgical procedure relies on two-dimensional endoscopic images, lacking depth perception. This makes it difficult for surgeons to establish precise surgical approaches, easily leading to cartilage or neurovascular damage. Without the assistance of a navigation system, the accuracy and safety of the surgery depend entirely on the surgeon's clinical experience, resulting in a long learning curve and the potential for errors in approach or incomplete lesion removal in challenging cases.

[0003] The existing radiographic-assisted and optical positioning navigation technologies in the field of foot and ankle orthopedics have the following main drawbacks:

[0004] (1) Radiation and invasive risks. Traditional fluoroscopic navigation involves cumulative radiation exposure; bone needle-based optical navigation requires drilling markers into the bone shaft, increasing the risk of additional trauma and infection.

[0005] (2) Precision failure caused by soft tissue artifacts (STA). Existing non-invasive surface tracking methods are limited by the relative sliding between the skin and deep bones (i.e., STA), which often results in tracking errors under static and dynamic conditions that exceed the clinically acceptable range and are difficult to meet the precision requirements of surgery.

[0006] (3) The system deployment cost is high. Traditional navigation relies on expensive infrared optical tracking equipment, which is bulky and complicated to operate, making it difficult to popularize in primary hospitals. Summary of the Invention

[0007] In view of the above-mentioned prior art, the present invention provides a non-invasive ankle arthroscopic navigation system and method based on anatomical constraints and binocular vision. It can effectively filter out STA through algorithms and achieve high-precision dynamic tracking of the patient's ankle joint without the need for bone screws (non-invasive) and complex tracking equipment.

[0008] To address the aforementioned technical problems, this invention proposes a non-invasive ankle arthroscopic navigation method based on anatomical constraints and binocular vision, the main steps of which are as follows:

[0009] Step 1) Reconstruct the 3D skeletal triangular mesh model;

[0010] Step 2) Extract the talus trochlear joint axis;

[0011] Step 3) Defining patient surface landmarks and deploying binocular visual environment;

[0012] Step 4) Real-time detection of landmarks on the skin surface of the patient's surgical side foot and ankle, and three-dimensional mapping;

[0013] Step 5) Calculate the physiological effective rotation angle based on the real-time foot and ankle posture under anatomical constraints;

[0014] Step 6) Digital twin synchronization and decision support ultimately achieve non-invasive ankle arthroscopic navigation.

[0015] In this invention, the specific content of step 1) is as follows:

[0016] Step 1-1) Obtain the patient's preoperative ankle CT image data, and use the open-source software 3D slicer to extract the tibia, talus, fibula, calcaneus, navicular, cuboid, metatarsals and cuneiform from the ankle CT image data through threshold segmentation and region growth algorithms;

[0017] Steps 1-2) The extracted bone segmentation results will be reconstructed using the open-source software 3D slicer to generate a three-dimensional bone triangular mesh model, including: a three-dimensional bone triangular mesh model of the talus; a three-dimensional bone triangular mesh model composed of the tibia and fibula, referred to as the leg model; and a three-dimensional bone triangular mesh model composed of the talus, calcaneus, scaphoid, cuboid, metatarsals and cuneiformes, referred to as the foot model.

[0018] Virtual surgical planning information is preset in one or both of the leg model and foot model. This information includes preset surgical approach points, key anatomical points, and anatomical structure warning areas.

[0019] In this invention, step 2) specifically includes the following:

[0020] Step 2-1) Sampling point cloud data of the three-dimensional skeletal triangular mesh model of the talus, and automatically segmenting the point cloud of the talus trochlear articular surface using normal filtering and curvature filtering based on anatomical principles;

[0021] Step 2-2) Based on the aforementioned point cloud of the talus articular surface, and according to the anatomical characteristics of the approximately conical talus articular surface, construct an objective function to calculate the optimal conical fit.

[0022]

[0023] in, Radial distance, This is the axial distance. It is the angle of a semi-cone.

[0024] Conical parameters include the cone's semi-cone angle. This determines the opening amplitude of the cone; the axial distance It is a point in a point cloud. To the top of the cone In the axial direction The projected distance on, i.e. The radial distance It is a point From and The vertical distance from the defined central axis;

[0025] Steps 2-3) Use the BFGS quasi-Newton optimization algorithm to solve for the optimal cone parameters, thereby extracting the cone's central axis vector. ; the central axis vector of the cone As the fitted talus joint axis vector This completes the extraction of the talus trochlear joint axis.

[0026] In this invention, step 3) specifically includes the following:

[0027] Step 3-1) Predefine a group of landmarks on the patient's surgical side foot and ankle. The group of landmarks includes 6 landmarks, namely the first tibial point, the second tibial point, the medial malleolus, the navicular bone, the calcaneus, and the first metatarsal bone.

[0028] Step 3-2) Deploy the visual environment by using a sterile medical pen to draw custom geometric patterns at six marker points on the patient's skin surface; deploy a binocular camera and, through coordinate system transformation, unify the marker point observation data in the camera coordinate system to the digital twin model coordinate system.

[0029] In this invention, step 4) is specifically described as follows:

[0030] Step 4-1) In the visual environment deployed above, the binocular camera captures RGB video stream and synchronized depth image in real time; using a pre-trained lightweight pose recognition neural network, the two-dimensional pixel coordinates of 6 marker points are extracted from the original RGB color image;

[0031] Step 4-2) Combining the information from the synchronized depth image, map the two-dimensional point set corresponding to the two-dimensional pixel coordinates of the six marker points to the original three-dimensional keypoint sequence in the camera coordinate system. ;

[0032] In this invention, step 5) is specifically described as follows:

[0033] Step 5-1) Process the original 3D keypoint sequence A sliding window moving average filter is applied to eliminate high-frequency noise from the stereo camera, resulting in a smoothed 3D keypoint sequence. ;

[0034] Step 5-2) Based on the kinematics of two rigid bodies, the smoothed 3D keypoint sequence is... The model is divided into a lower leg group and a foot group. The lower leg group includes landmarks such as the first tibial point, the second tibial point, and the medial malleolus. The foot group includes landmarks such as the scaphoid bone, the first metatarsal bone, and the calcaneus. The Kabsch algorithm is used to calculate the preliminary rotation matrices of the lower leg group and the foot group relative to the lower leg model and foot model, respectively. and This allows us to determine the relative motion, including soft tissue artifacts. ,in, This represents the real-time relative rotation matrix of the foot model relative to the lower leg model, including soft tissue slippage errors;

[0035] Step 5-3) Using Lie algebras Projection properties, including the relative motion of soft tissue artifacts mentioned above. Mapped to the fitted talus joint axis vector On the defined single-degree-of-freedom rotating manifold, calculate the physiological effective rotation angle between the lower leg model and the foot model. ;

[0036] In this invention, step 6) is specifically described as follows:

[0037] Step 6-1) Construct a digital twin foot and ankle model in the open-source 3D software Blender based on the lower leg model and foot model; the digital twin foot and ankle model receives the effective rotation angle. The real-time script interface deployed in Blender receives data and uses the corrected angle parameters to drive the movement of the foot model relative to the lower leg model in the Blender scene in real time, thereby achieving synchronous high-fidelity mapping of the patient's real-time foot and ankle posture in the Blender digital twin foot and ankle model.

[0038] Step 6-2) The real-time images of the patient's actual foot and ankle at the surgical site are combined with the real-time motion state of the digital twin foot and ankle model obtained in Step 6-1) and the virtual surgical planning information preset in Step 1-2) are dynamically overlaid in real time by computer and displayed in the doctor's AR glasses field of view.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] (1) Completely non-invasive and radiation-free, using a combination of binocular vision and surface feature points, eliminating the need for intraoperative fluoroscopy and bone nail markers, achieving truly non-invasive operation while ensuring navigation accuracy and reducing surgical risks.

[0041] (2) This invention effectively suppresses STA during non-invasive tracking by introducing a manifold projection algorithm and a biomechanical anatomical constraint mechanism. The algorithm can automatically identify and remove non-rigid noise data caused by skin sliding, and map the observation points on the body surface to a motion manifold space that conforms to the anatomical characteristics of the human body.

[0042] (3) A lightweight attitude recognition network model is adopted, which reduces the requirements for computing hardware while ensuring real-time performance. The system does not require expensive large optical trackers, which reduces the procurement cost and deployment difficulty for departments and has clinical promotion value. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the anatomical landmarks on the surface of the foot and ankle and their corresponding detection indexes;

[0044] Figure 2 This is a flowchart of the method of the present invention, which shows the entire process from offline modeling to online processing and calculation output;

[0045] Figure 3 This is a flowchart of the data flow and algorithm logic of this invention (please specify which part);

[0046] Figure 4 This is a schematic diagram illustrating the geometric principle of extracting the talus trochlear joint axis based on the optimal conic fitting algorithm in this invention.

[0047] Figure 5 This is a real-time running effect diagram of the embodiment, where A - original RGB color image, B - synchronized depth image, C - key point recognition, and D - model pose synchronization. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.

[0049] The present invention proposes a non-invasive ankle arthroscopic navigation method based on anatomical constraints and binocular vision, which mainly includes four stages.

[0050] Phase 1: Offline personalized anatomical modeling and constraint extraction, such as Figure 2 and Figure 3 As shown.

[0051] 1) Personalized 3D bone reconstruction, namely, reconstructing a 3D triangular mesh model of the patient's ankle joint.

[0052] Preoperative ankle CT images of the patient were acquired. The open-source software 3D Slicer was used to extract the tibia, talus, fibula, and related structures of the calcaneus, navicular, cuboid, metatarsals, and cuneiform from these images using thresholding and region growing algorithms. The extracted bone segmentation results were then used to reconstruct patient-specific three-dimensional skeletal triangular mesh models using 3D Slicer, including: a talus 3D skeletal triangular mesh model; a 3D skeletal triangular mesh model composed of the tibia and fibula, designated as the leg model; and a 3D skeletal triangular mesh model composed of the talus, calcaneus, navicular, cuboid, metatarsals, and cuneiform, designated as the foot model. Virtual surgical planning information was preset in one or both of the leg and foot models, including preset surgical approach points, key anatomical points, and anatomical structure warning areas. The talus 3D skeletal triangular mesh model serves as the basis for subsequent extraction of the talus trochlear joint axis, while the leg and foot models serve as the foundation for digital twin-driven models.

[0053] 2) Extract the talus trochlear joint axis (TJA).

[0054] Point cloud sampling was performed on the reconstructed three-dimensional skeletal triangular mesh model of the talus, and point clouds of the articular surfaces of the talus trochlea were automatically segmented using normal filtering and curvature filtering based on anatomical principles.

[0055] Based on the point cloud of the talus articular surface, and taking into account the approximately conical anatomical characteristics of the talus articular surface, an objective function is constructed to calculate the optimal conical fit.

[0056]

[0057] in, Radial distance, is a point From and The vertical distance from the defined central axis; The axial distance is the distance between points in the point cloud. To the top of the cone In the axial direction The projected distance on, i.e. ; The semi-cone angle of the cone determines the opening range of the cone;

[0058] The optimal cone parameters, including cone apex position, axial direction, and semi-cone angle, were obtained using the BFGS quasi-Newton optimization algorithm; the cone central axis vector was then extracted. ; the central axis vector of the cone As the fitted talus joint axis vector This completes the extraction of the talus trochlear joint axis, as follows: Figure 4As shown, the central axis of the cone serves as a specific physiological degree of freedom constraint reference for this patient, and is used to subsequently filter out non-physiological pose errors caused by skin sliding.

[0059] Phase Two: Establishing Anatomical Correspondences and System Initialization, such as Figure 1 and Figure 2 As shown.

[0060] 3) Establishment and perception configuration of patient surface landmarks (i.e., deployment of binocular visual environment).

[0061] Predefine a group of landmarks and select six anatomical landmarks on the patient's surgical side foot and ankle that have high subcutaneous bone adhesion and are relatively less affected by soft tissue sliding: such as Figure 1 As shown, in this embodiment, the six marker points include the first tibial point 0 located above the anterior crest of the tibia, the second tibial point 1 located below the anterior crest of the tibia, the medial malleolus point 2, the point at the tuberosity of the scaphoid bone 4, the head of the first metatarsal bone 4, and the calcaneal tuberosity 5. For visual environment deployment, a custom geometric pattern is drawn at each of the six marker points on the patient's skin using a sterile medical pen: a solid circle at the second tibial point 0, a hollow triangle at the second tibial point 1, a hollow circle at the medial malleolus 2, an intersecting cross at the scaphoid bone 3, a hollow square at the first metatarsal bone 4, and a solid square at the calcaneus 5. When deploying the binocular camera, the marker point observation data in the camera coordinate system is aligned with the coordinate system of the digital twin model through coordinate system transformation or hand-eye calibration.

[0062] The third stage: Intraoperative real-time visual perception and noise reduction, such as Figure 2 and Figure 3 As shown.

[0063] 4) Real-time detection of landmarks on the skin surface of the patient's surgical side foot and ankle, and three-dimensional mapping.

[0064] In the aforementioned deployed visual environment, a binocular camera captures RGB video streams and synchronized depth images in real time. Using a pre-trained lightweight pose recognition neural network, the two-dimensional pixel coordinates of six marker points are extracted from the RGB color image. Combined with information from the synchronized depth image, the two-dimensional point set corresponding to the two-dimensional pixel coordinates of these six marker points is mapped to the original three-dimensional keypoint sequence in the camera coordinate system. .

[0065] 5) Real-time foot and ankle posture based on anatomical constraints, with refined calculation of physiological effective rotation angles.

[0066] To address the accuracy degradation caused by Static Accuracy Detection (STA) mentioned in the background art, this invention implements a three-layer denoising strategy, such as... Figure 2 and Figure 3 As shown.

[0067] Temporal smoothing: the original 3D keypoint sequence A sliding window moving average filter is applied to eliminate high-frequency noise from the stereo camera, resulting in a smoothed 3D keypoint sequence. .

[0068] Rigid body registration: Based on two rigid body kinematics, the smoothed 3D keypoint sequence is registered. The model was divided into a lower leg group and a foot group. The lower leg group included landmarks such as the first tibial point 0, the second tibial point 1, and the medial malleolus 2. The foot group included landmarks such as the scaphoid bone 3, the calcaneus 4, and the first metatarsal bone 5. The Kabsch algorithm was used to calculate the preliminary rotation matrices of the lower leg group and the foot group relative to the preoperative lower leg model. Preliminary rotation matrix of the foot model This allows us to determine the relative motion, including soft tissue artifacts. ,in, This represents the real-time relative rotation matrix of the foot model relative to the lower leg model, including soft tissue slippage errors.

[0069] Manifold projection correction: using Lie algebra Projection properties, including the relative motion of soft tissue artifacts mentioned above. Mapped to the fitted talus joint axis vector On the defined single-degree-of-freedom rotating manifold, calculate the physiological effective rotation angle between the lower leg model and the foot model. .

[0070] Phase 4: Visual guidance.

[0071] 6) Digital twin synchronization and decision support.

[0072] In the open-source 3D software Blender, a digital twin foot and ankle model is constructed based on the aforementioned lower leg and foot models. This digital twin foot and ankle model receives the aforementioned effective rotation angle. The real-time script interface deployed in Blender receives data and uses the corrected pose angle parameters to synchronously drive the movement of the foot model relative to the lower leg model in the Blender scene in real time. This achieves a high-fidelity synchronous mapping of the patient's real-time foot and ankle posture to the Blender digital twin foot and ankle model. Figure 5 As shown.

[0073] Surgical planning and warning information are overlaid and displayed in real time on AR glasses or a surgical monitor via a computing device. In this embodiment, the computer dynamically overlays real-time images of the patient's actual foot and ankle at the surgical site with the real-time movement state of the digital twin foot and ankle model, combined with previously preset virtual surgical planning information, and displays it in the doctor's AR glasses field of view, thereby achieving non-invasive ankle arthroscopy navigation.

[0074] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many improvements and changes under the guidance of the present invention without departing from the spirit of the present invention, and these improvements and changes are all within the protection scope of the present invention.

Claims

1. A non-invasive ankle arthroscopic navigation method based on anatomical constraints and binocular vision, characterized in that, Includes the following steps: Step 1) Reconstruct the 3D skeletal triangular mesh model: Step 1-1) Obtain the patient's preoperative ankle CT image data, and use the open-source software 3D slicer to extract the tibia, talus, fibula, calcaneus, navicular, cuboid, metatarsals and cuneiform from the ankle CT image data through threshold segmentation and region growth algorithms; Steps 1-2) The extracted bone segmentation results will be reconstructed using the open-source software 3D slicer to generate a three-dimensional bone triangular mesh model, including: a three-dimensional bone triangular mesh model of the talus; a three-dimensional bone triangular mesh model composed of the tibia and fibula, referred to as the leg model; and a three-dimensional bone triangular mesh model composed of the talus, calcaneus, scaphoid, cuboid, metatarsals and cuneiformes, referred to as the foot model. Virtual surgical planning information is preset in one or both of the leg model and foot model. This information includes preset surgical approach points, key anatomical points, and anatomical structure warning areas. Step 2) Extract the talar trochlear joint axis (TJA): Step 2-1) Sampling point cloud data of the three-dimensional skeletal triangular mesh model of the talus, and automatically segmenting the point cloud of the talus trochlear articular surface using normal filtering and curvature filtering based on anatomical principles; Step 2-2) Based on the aforementioned point cloud of the talus articular surface, and according to the anatomical characteristics of the approximately conical talus articular surface, construct an objective function to calculate the optimal conical fit. in, Radial distance, It is the axial distance. It is the angle of a semi-cone. Conical parameters include the cone's semi-cone angle. This determines the opening amplitude of the cone; the axial distance It is a point in a point cloud. To the top of the cone In the axial direction The projected distance on, i.e. The radial distance It is a point From and The vertical distance from the defined central axis; Steps 2-3) Use the BFGS quasi-Newton optimization algorithm to solve for the optimal cone parameters, thereby extracting the cone's central axis vector. ; the central axis vector of the cone As the fitted talus joint axis vector This completes the extraction of the talus trochlear joint axis. Step 3) Defining patient surface landmarks and deploying binocular visual environment: Step 3-1) Predefine a group of landmarks on the patient's foot and ankle on the operated side. The group of landmarks includes 6 landmarks, namely the first tibial point (0), the second tibial point (1), the medial malleolus (2), the navicular bone (3), the first metatarsal bone (4), and the calcaneus (5). Step 3-2) Deploy the visual environment by using a sterile medical pen to draw custom geometric patterns at six marker points on the patient's skin surface; deploy a binocular camera and, through coordinate system transformation, unify the marker point observation data in the camera coordinate system to the digital twin model coordinate system; Step 4) Real-time detection of landmarks on the skin surface of the patient's surgical side ankle, and 3D mapping: Step 4-1) In the visual environment deployed above, the binocular camera captures RGB video stream and synchronized depth image in real time; using a pre-trained lightweight pose recognition neural network, the two-dimensional pixel coordinates of 6 marker points are extracted from the original RGB color image; Step 4-2) Combining the information from the synchronized depth image, map the two-dimensional point set corresponding to the two-dimensional pixel coordinates of the six marker points to the original three-dimensional keypoint sequence in the camera coordinate system. ; Step 5) Based on the real-time foot and ankle posture under anatomical constraints, calculate the physiological effective rotation angle: Step 5-1) Process the original 3D keypoint sequence A sliding window moving average filter is applied to eliminate high-frequency noise from the stereo camera, resulting in a smoothed 3D keypoint sequence. ; Step 5-2) Based on the kinematics of two rigid bodies, the smoothed 3D keypoint sequence is... The model is divided into a lower leg group and a foot group. The lower leg group includes the first tibial point (0), the second tibial point (1), and the medial malleolus (2) as landmarks. The foot group includes the navicular bone (3), the calcaneus (4), and the first metatarsal bone (5) as landmarks. The Kabsch algorithm is used to calculate the preliminary rotation matrices of the lower leg group and the foot group relative to the lower leg model and the foot model, respectively. and This allows us to determine the relative motion, including soft tissue artifacts. ,in, This represents the real-time relative rotation matrix of the foot model relative to the lower leg model, including soft tissue slippage errors; Step 5-3) Using Lie algebras Projection properties, including the relative motion of soft tissue artifacts mentioned above. Mapped to the fitted talus joint axis vector On the defined single-degree-of-freedom rotating manifold, calculate the physiological effective rotation angle between the lower leg model and the foot model. ; Step 6) Digital Twin Synchronization and Decision Support Step 6-1) Construct a digital twin foot and ankle model in the open-source 3D software Blender based on the lower leg model and foot model; the digital twin foot and ankle model receives the effective rotation angle. The real-time script interface deployed in Blender receives data and uses the corrected angle parameters to drive the movement of the foot model relative to the lower leg model in the Blender scene in real time, thereby achieving synchronous high-fidelity mapping of the patient's real-time foot and ankle posture in the Blender digital twin foot and ankle model. Step 6-2) The real-time images of the patient's foot and ankle at the surgical site are combined with the real-time motion state of the digital twin foot and ankle model obtained in Step 6-1) and the virtual surgical planning information preset in Step 1-2) by computer in real time and displayed in the doctor's AR glasses field of view, thereby realizing non-invasive navigation of ankle arthroscopy.

Images