Standard artificial temporomandibular joint installation navigation system based on augmented reality

Abstract

The invention discloses an augmented reality-based standard artificial temporomandibular joint installation navigation system, which comprises the following steps: S1, carrying out image registration on a preoperative CT image of a patient and a target operative site to obtain a registration result, and S2, updating the registration result in real time, and readjusting registration based on AR display equipment to obtain a registration result; virtual-actual transformation between the virtual image and the actual model is realized; s3, calibrating a surgical instrument, respectively aligning the surgical instrument with the TMJ surgical site and the virtual model, and tracking the position and posture of a patient entity in real time; and S4, navigating the artificial joint installation path in real time based on the updated position and posture of the patient entity.

Description

Technical Field

[0001] The present invention relates to the field of oral and maxillofacial surgery visualization, and particularly to a standard artificial temporomandibular joint installation navigation system based on augmented reality. Background Art

[0002] The augmented reality technology AR can directly display virtual models, images or text information on the patient through an optically transparent head-mounted display, providing surgeons with the function of obtaining sufficient visual input and more intuitive and convenient real-time navigation without having to shift their line of sight from the surgical area, and accurately displaying the virtual pre-operative planned prosthetic implant at the actual surgical site of the patient.

[0003] Currently, clinically, a method of using a digital guide plate to assist in bone contour and prosthesis positioning is adopted. Through computer-aided surgery technology, we can simulate the placement of the glenoid fossa prosthesis and the mandibular prosthesis during the pre-operative digital design process, and achieve bone trimming during the operation through a 3D-printed digital guide plate.

[0004] Due to the smooth curved surfaces of the zygomatic arch and the lateral side of the mandibular ramus, the digital guide plate may shift. In addition, the larger the digital guide plate, the larger the surgical incision and the wider soft tissue dissection are required to ensure correct positioning, which may cause greater surgical trauma, and the larger surgical scar also affects the patient's appearance.

[0005] In view of this, the present invention provides a standard artificial temporomandibular joint installation navigation system based on augmented reality to solve the above problems. Summary of the Invention

[0006] The purpose of the present invention is to provide a standard artificial temporomandibular joint installation navigation system based on augmented reality. By wearing an AR device, surgeons can directly view the pre-operative planned osteotomy plane and prosthesis placement position at the patient's TMJ surgical site within the field of view of the head-mounted display, thereby improving the safety and accuracy of prosthesis installation and replacing the digital guide plate to obtain a smaller surgical incision.

[0007] To achieve the above purpose, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a standard artificial temporomandibular joint installation navigation system based on augmented reality, which is characterized by comprising:

[0009] An optical tracking device for real-time tracking of the positions and postures of a plurality of reflective spheres arranged on a registration and calibration detection tool to obtain the pose information of the corresponding surgical instrument and the patient reference frame in space;

[0010] An augmented reality display device is used to display a virtual three-dimensional model of a temporomandibular joint prosthesis and spatially superimpose it on the actual anatomical structure of the patient's maxillofacial region to achieve visual navigation guidance of virtual-real fusion;

[0011] A registration and calibration detection tool component, including a probe, a patient reference frame, an instrument calibration block, and an AR calibration block, is used to establish the spatial transformation relationship between the medical image coordinate system, the tracker coordinate system, and the augmented reality coordinate system, and output the corresponding translation-rotation matrix;

[0012] A preoperative planning module, based on the three-dimensional reconstructed maxillofacial CT data and the standard prosthesis geometric model, combines key anatomical point extraction and point cloud analysis algorithms to automatically generate the placement direction, position, and rotation matrix of the prosthesis, and outputs the three-dimensional pose matching result;

[0013] A navigation control module, based on the real-time tracking data and the preoperative planning result, embeds the virtual prosthesis model into the augmented reality view and performs real-time navigation update and deviation prompt according to the intraoperative instrument movement state.

[0014] As a preferred technical solution of the first aspect of the present invention, it further includes an interface interaction module, which supports multi-view synchronous display of the virtual prosthesis, the bone surface structure, and the detection tool trajectory, and provides functions of error correction, model adjustment, and multi-scheme comparison.

[0015] In a second aspect, the present invention provides an automated placement navigation method for a temporomandibular joint prosthesis based on augmented reality, including the following steps:

[0016] Step S1: Image registration is performed on the patient's preoperative CT image and the target surgical site to obtain the registration result:

[0017] Step S2: The registration result is updated in real time, and the registration is readjusted based on the AR display device to achieve the virtual-real transformation between the virtual image and the actual model;

[0018] Step S3: Surgical instrument calibration is performed to align the surgical instruments with the TMJ surgical site and the virtual model respectively, and the position and posture of the patient's entity are tracked in real time;

[0019] Step S4: Based on the updated position and posture of the patient's entity, the installation path of the artificial joint is navigated in real time.

[0020] As a preferred technical solution of the second aspect of the present invention, the translation-rotation matrix is the transformation matrix from coordinate system A to coordinate system B; it includes a rotation matrix and a translation matrix. The rotation matrix represents the rotation direction of coordinate system A relative to coordinate system B, and the translation matrix represents the position of the origin of coordinate system A in coordinate system B.

[0021] As a preferred technical solution of the second aspect of the present invention, the acquisition logic of the translation-rotation matrix is as follows:

[0022] Collect a set of corresponding point pairs that can be expressed in both coordinate systems when the probe clicks on the anatomical points.

[0023] Based on the objective function of the alignment algorithm registration, solve the rotation and translation amounts under the minimum error, and use the rotation and translation amounts under the minimum error as the translation-rotation matrix.

[0024] As a preferred technical solution of the second aspect of the present invention, image registration is performed on the preoperative CT image of the patient and the target surgical site, including the following steps:

[0025] Step S101: Based on the NDI navigation head, provide the world coordinate system O Tracker ;

[0026] Step S102: Based on the HoloLens device, provide the coordinate system O in the virtual field of view Holo ;

[0027] Step S103: For the 3D model reconstructed from the CT image, provide the prosthesis coordinate system O Image ;

[0028] Step S104: For the surgical site of the patient, its coordinate system O PatientRef is provided by the patient's reference frame, and its position relative to the patient remains unchanged during the operation;

[0029] Step S105: For the surgical instrument, its coordinate system O HandRef is provided by the reference frame fixed on the handle during the operation.

[0030] As a preferred technical solution of the second aspect of the present invention, the specific steps for the step S2 to achieve the virtual-real transformation between the virtual image and the actual model are as follows:

[0031] Initial registration initialization, use the probe to click on the anatomical landmark points on the patient's jawbone; collect the positions of the anatomical landmark points in the tracker coordinate system and the image space, and obtain the transformation matrix between the image coordinate system and the patient reference coordinate system;

[0032] Construct the initial virtual-real transformation matrix to transform the virtual image from the image space to the HoloLens coordinate space.

[0033] Real-time tracking and error detection, continuously monitor the changes of the reference frame and the reflective sphere of the detection tool in the NDI navigation system, and when it is recognized that the patient's head moves or the reference frame is slightly offset, obtain the detection registration error;

[0034] If the set error threshold is exceeded, the system automatically or prompts for registration update, re - collects a set of reference points or calls the previously collected point set, recalculates the transformation matrix between the image and the reality based on the current NDI tracking result, and relocates, rotates, and aligns the virtual image in the AR view in HoloLens;

[0035] Use the updated transformation matrix to convert the three - dimensional virtual model to the correct position in the AR coordinate system, realizing the precise overlap of the virtual prosthesis and the real bone surface in the AR view.

[0036] As a preferred technical solution of the second aspect of the present invention, the specific steps for calibrating the surgical instrument and aligning it with the TMJ surgical site and the virtual model are as follows:

[0037] Obtain the correspondence between the virtual image and the patient model in real - time and establish the correspondence between the surgical instrument and the patient head model;

[0038] Use the instrument calibration block for calibration. Insert the probe tip into the known hole positions of the calibration block, record the position of the reflective sphere under the NDI system, obtain the transformation matrix of the probe rigid body, and track the three - dimensional position and orientation of the tip / blade in real - time;

[0039] Establish a spatial correspondence between the virtual image and the actual model. Install a reference frame with a reflective sphere in the zygomatic arch or frontal bone area of the patient; use the probe to click on more than 3 anatomical landmark points in sequence and match them with the corresponding points in the CT model;

[0040] Map the real - time position of the instrument end to the image coordinate system or the AR view to construct a spatial transformation chain:

[0041] Track and update the position of the instrument end in the image coordinates or HoloLens coordinates in real - time; display the movement trajectory of the instrument model in HoloLens to assist the doctor in judging whether it deviates from the trajectory; realize the alignment of the surgical instrument with the TMJ surgical site and the virtual model.

[0042] As a preferred technical solution of the second aspect of the present invention, use a reference system with a reflective sphere. The reflective sphere is fixed on the handle and remains constant with the handle and the drill bit throughout the TMJ replacement process. The calibration block is used to align the tips and axes of the drill bit and the probe;

[0043] Obtain the tip position and axis of the surgical drill bit relative to the patient model in real - time, obtain the positional relationship between the surgical drill bit and the probe through tip - axis alignment calibration, and determine the translation - rotation matrix between the drill bit tip coordinate system and the patient reference coordinate system. This calibration method.

[0044] As a preferred technical solution of the second aspect of the present invention, the display and hiding of the virtual model are in HoloLens 2, which displays the positions of the prosthesis, cutting plane, and inserted screws in the preoperative plan relative to the patient's TMJ, and drills the handle along these positions into the patient's head mold;

[0045] The relative positions of the inserted screws and the surgical drill or drill bit, or the relative positions of the cutting plane and the oscillating saw, as well as the real-time depth, distance, and angle deviations are simultaneously displayed in the navigation software interface of the computer.

[0046] In the above technical solution, the technical effects and advantages provided by the present invention are as follows:

[0047] The present invention assists in the replacement of the temporomandibular joint with a standard artificial joint. This surgical navigation system realizes the integration of the prosthesis placement position in the preoperative plan and the surgical site of the patient's temporomandibular joint, achieves high-precision visualization of virtual design, and its accuracy has no significant difference from that of the digital guide plate. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0049] Figure 1 It is the working flow chart of the navigation system of the present invention.

[0050] Figure 2 It is the hardware composition diagram of the navigation system of the present invention.

[0051] Figure 3 It is the registration diagram of the virtual reality and the real scene of the present invention.

[0052] Figure 4 It is the schematic diagram of the registration process of the surgical oscillating saw of the present invention.

[0053] Figure 5 It is the diagram of the TMJ joint replacement surgery under real-time navigation of the present invention.

[0054] Figure 6 It is the schematic diagram of the prosthesis installation error of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted.

[0056] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the example embodiments of the present disclosure. However, those skilled in the art will recognize that one or more of the specific details may be omitted in practicing the technical solutions of the present disclosure, or other methods, components, steps, etc. may be used. In other cases, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0057] Artificial temporomandibular joint replacement is an important means for treating end-stage joint diseases. Standard artificial joint prostheses are designed according to the skeletal morphology of Caucasians. When applied to Chinese patients, a large amount of bone mass usually needs to be removed from the zygomatic arch and the lateral side of the mandibular ramus of the patient to ensure the correct installation and positioning of the prosthesis. If the prosthesis is placed improperly, it is very likely to damage the external auditory canal, alveolar nerve, or cranial base. Therefore, how to accurately achieve the placement position of the preoperative planned standard prosthesis during the operation is very important. Augmented reality technology can realize the visualization of the preoperative design scheme during the operation. The surgeon can view the osteotomy plane, prosthesis placement position, and key anatomical structures of the preoperative plan at the patient's TMJ surgical site, thereby improving the safety and accuracy of prosthesis installation and replacing the digital guide plate to obtain a smaller surgical incision.

[0058] Example 1

[0059] As Figure 1 shown, the present invention provides a navigation system for the installation of a standard artificial temporomandibular joint based on augmented reality, which is used for the implantation of a standard artificial joint prosthesis. The hardware system used includes:

[0060] An optical tracking device for tracking the position of the reflective balls placed on the registration and calibration detection tools, thereby recording the current position and angle of these detection tools in real time;

[0061] Specifically, as Figure 2 I is NDIPolaris Spectra, abbreviated as NDI navigation head, which can achieve sub-millimeter accuracy and track the reflective balls in real time. The reflector is installed on;

[0062] An AR display device for displaying a virtual model and aligning it with the surgical site of an actual patient;

[0063] Specifically, such as Figure 2 E is an OST-HMD device based on Microsoft HoloLens 2, supporting SLAM and virtual-real fusion; using a probe ( Figure 2 A), patient reference system ( Figure 2 B), handle reference system ( Figure 2 F), drill bit and drill bit calibration block ( Figure 2 L), etc. registration and calibration detection tools to determine the translation-rotation matrix of the spatial transformation between objects. The patient prosthesis and virtual-real calibration calibration block ( Figure 2 K) is made by 3D printing (WEILAI 8000 resin, Shenzhen Future Factory Technology Co., Ltd., China), and the processing error is 200μm. The surgical detection tool includes a tool head ( Figure 2 H), a sagittal bone saw, a drill, a grinding head ( Figure 2 J). The tool head is used as a clamping detection tool seen, the drill bit and the drill provide the power for cutting and drilling. The sagittal bone saw blade is used to perform osteotomy by horizontal swinging, the drill needle prepares the titanium nail hole channel, and the grinding head grinds the uneven bone surface of the mandibular ramus and the glenoid fossa. Finally, the prosthesis implanted in the TMJ consists of two parts: the glenoid fossa prosthesis and the mandibular prosthesis, which are respectively fixed to the outside of the glenoid fossa and the outside of the mandibular ramus with titanium nails.

[0064] A software platform, based on multiple open-source platforms, assists in the automated placement navigation of standard artificial joint prostheses. The development framework includes Unity (3D rendering), VisualStudio (code development), Qt (UI interaction), VTK / ITK (medical image processing: segmentation, registration), MRTK (mixed reality interaction design), and HoloLensARToolKit (AR calibration and spatial anchoring).

[0065] Specifically, by integrating three-dimensional image processing, AR visualization, optical tracking, and spatial registration technologies, the full-process digital navigation of the TMJ prosthesis from "preoperative design → intraoperative guidance → postoperative verification" is realized, greatly improving the accuracy and efficiency of the surgery. The following work processes are realized based on the software platform:

[0066] Stage 1: Preoperative preparation, using a calibration block to determine the translation-rotation matrix of the NDI coordinate system of detection tools such as drill bits and probes;

[0067] Collect the patient's maxillofacial CT data and reconstruct the three-dimensional bone model;

[0068] Design the position and posture of the standard TMJ prosthesis on the model;

[0069] Import the above design data into the augmented reality navigation system ARNavig-sTMJP.

[0070] Phase 2: Calibration and registration. Establish a consistent coordinate relationship between the surgical design in the virtual space and the patient and instruments in reality.

[0071] HoloLens registration: Enable the augmented reality headset (such as HoloLens 2) to recognize and superimpose virtual images.

[0072] Image-patient registration: Use detection tools such as probes to perform point cloud registration on CT images and patient anatomy.

[0073] Instrument calibration: Identify the position and orientation of surgical instruments (such as drills, saw blades).

[0074] Registration verification: If the registration of the three is good (the error is within the acceptable range), proceed to the next stage; otherwise, re-calibration is required.

[0075] Phase 3: Intraoperative navigation. Register the TMJ surgical site with the image, implant the prosthesis according to the navigation plan, and use AR overlay to verify the bone surface fit and screw angle.

[0076] AR real-time navigation: Display the virtual model in HoloLens to guide the doctor's precise operation.

[0077] Prosthesis implantation: Saw the bone, drill holes, and fix the standard TMJ prosthesis according to the navigation.

[0078] Postoperative evaluation: Perform postoperative image comparison and analysis on the prosthesis implantation position to verify the accuracy.

[0079] Embodiment 2

[0080] Based on the hardware device provided in Embodiment 1, this embodiment is an augmented reality-based standard artificial temporomandibular joint installation navigation system for standard TMJ prosthesis implantation, including the following steps:

[0081] Step S1: Perform image registration on the patient's preoperative CT image and the target surgical site to obtain the registration result:

[0082] Specifically, the target surgical site is the TMJ surgical site, and TMJ is the temporomandibular joint. That is, in temporomandibular joint surgery, image registration is a technical process of aligning preoperative medical images (such as CT / MRI) with the patient's actual anatomical structure in real time during the operation. Its purpose is to ensure that virtual surgical plans (such as prosthesis position, osteotomy line) can be accurately mapped to the real patient.

[0083] Exemplary illustration, in order to achieve the coincidence of the virtual image of the TMJ and the real object in the surgeon's field of view, registration of the TMJ surgical site and the image is required. The prerequisite for real-time navigation is to unify the coordinates of the objects in the system under the same coordinate system. The unification of spatial coordinates is achieved by obtaining the spatial transformation between objects and forming a closed loop, which is realized by a series of 4*4 matrices, called the translation-rotation matrix, that is: the homogeneous transformation matrix that uniformly processes rotation and translation in three-dimensional space. For the coordinate P A =(x A ,y A ,z A ,1) T and P B =(x B ,y B ,z B ,1) T in the space of any two objects A and B, the translation-rotation matrix can be represented by as Equation (1):

[0084]

[0085] where: represents the transformation matrix from coordinate system A to B; is a 3*3 rotation matrix, representing the rotation direction of coordinate system A relative to coordinate system B, is a 3*1 translation matrix, representing the position of the origin of coordinate system A in coordinate system B.

[0086] In augmented reality, to achieve virtual-real alignment, the coordinates of all objects in the system must be unified under the same reference coordinate system through the homogeneous transformation matrix to achieve precise navigation and surgical guidance.

[0087] The purpose of registration is to calculate the translation-rotation matrix The commonly used alignment algorithm is the SVD algorithm, which reverses the translation-rotation matrix between two different coordinate systems through the coordinates of the same point in the two coordinate systems. These points are called fiducial points. Define a set of fiducial points as S=(P1 P2 … P n ). Then, the coordinates in coordinate system O A can be expressed as The coordinates in coordinate system O B can also be expressed as Define the objective function of SVD registration as It reflects the error of the transformation between two coordinate systems A and B. The goal of the SVD algorithm is to obtain the Satisfy the following equation (2):

[0088]

[0089] Among all possible combinations of rotation + translation for the coordinates in all coordinate systems O B and the coordinates in coordinate system O A find the one that minimizes the overall error.

[0090] More specifically, image registration of the patient's preoperative CT image and the target surgical site includes the following steps:

[0091] Step S101: Provide the world coordinate system O Tracker based on the NDI navigation head;

[0092] Step S102: Provide the coordinate system O Holo under the virtual field of view based on the HoloLens device;

[0093] Step S103: Provide the prosthesis coordinate system O Image from the 3D model reconstructed from the CT image;

[0094] Step S104: The patient's surgical site, whose coordinate system O PatientRef is provided by the patient's reference frame, and its position relative to the patient remains unchanged during the operation;

[0095] Step S105: The surgical instrument, whose coordinate system O HandRef is provided by the reference frame fixed on the handle during the operation;

[0096] Among the above objects, the NDI navigation head can track and obtain the translation-rotation matrix between the world coordinate system and the reference system in real time and while the translation-rotation matrix Image between O PatientRef and O O Holo and the translation-rotation matrix Tracker between O and the translation-rotation matrix HandRef between the tip of the surgical detection tool (in direct contact with the patient) and O These three translation-rotation matrices need to be achieved through registration and calibration operations.

[0097] To calculate Select a series of anatomical landmarks that maintain a fixed anatomical position during the operation, such as the cusp tip point, the anterior nasal spine point, and the nasion point, as the reference points for registration. The reference point set Image under O The coordinates. Then use a probe to determine the set of fiducial points By placing the probe tip on the fiducial points. In the coordinate system of the probe, the position of the tip is fixed, and the NDI navigation head can use a translation-rotation matrix to obtain directly. When the probe tip coincides with one of the fiducial points, the constraint is satisfied. Therefore, O PatientRef The coordinates of the selected fiducial points can be obtained as follows:

[0098]

[0099] After obtaining the fiducial point coordinates in O Image and O PatientRef the translation-rotation matrix can be calculated by substituting the coordinates of S Im and S PRef into Equation (2), thus achieving registration between the TMJ surgical site and the image, as follows:

[0100]

[0101] Step S2: Update the registration result in real time, readjust the registration based on the AR display device, and achieve virtual-real transformation between the virtual image and the actual model;

[0102] Exemplary illustration, to more intuitively represent the pre-operative planned prosthesis at the patient's temporomandibular joint surgical site, use the HoloLens 2 AR device to project the virtual 3D model reconstructed from the medical image onto the patient's head mold, thus achieving the fusion of the virtual and actual scenes through the HoloLens 2 device. Define the coordinates of the virtual model under the HoloLens2 device as O VModel , under the HoloLens camera, the spatial transformation of the CT image from the right coordinate system O Image to the left coordinate system O VModel of the model The image can be represented by a mirror transformation on the x-axis, and the diagonal matrix is diag(-1,1,1,1). Therefore, by continuously transforming the coordinate system using the following Equation (5), the translation-rotation matrix related to the virtual model and the HoloLens device can be obtained

[0103]

[0104] In the above Equation (5), can be obtained in real time using the NDI navigation head while can be derived separately from Equation (4). Therefore, the key to realizing virtual reality scene fusion and AR-based real-time navigation lies in calculating the translation-rotation matrix between the HoloLens 2 device and the NDI navigation head which can also be achieved by the SVD registration algorithm.

[0105] To calculate we use a cube with a reflective sphere on top and black and white blocks on its surface as the registration detection tool, where the reflective sphere and the black and white blocks can be recognized and located by the NDI navigation head and the HoloLens 2 camera respectively. Define the cube vertices as the reference points registered by the HoloLens tracker, and then calculate the set of reference points under the NDI navigation head S Tracker and the coordinates of the set of reference points of the HoloLens 2 camera S Holo . Therefore, by substituting S Holo and S Tracker into Equation (2), we can obtain which helps with the registration of the HoloLens Tracker as follows:

[0106]

[0107] Figure 3 shows the complete matrix transformation operation for registration. It realizes the virtual-real transformation between the virtual image and the actual model by creating a closed loop using the translation-rotation matrix, integrating the NDI optical tracker with the HoloLens augmented reality device for accurately aligning the patient's real anatomical structure with the preoperative medical image and the virtual model. Among them: O Tracker is the origin of the tracker coordinates: is the tracker coordinate system;

[0108] is the position of the patient reference frame under the tracker; is the position of the probe under the tracker; is the position of the calibration block under the tracker; is the position of the HoloLens under the tracker; is the transformation matrix from the image coordinates to the virtual patient coordinates; is the tracker coordinate system;

[0109] is the transformation matrix from the patient reference frame to the image coordinates;

[0110] O Image is the origin of the medical image coordinate system (usually CT or MRI image): P Im is the reference point (anatomical structure) in the image; O PatientRef is the origin of the patient reference frame: PRef is an anatomical landmark point (such as the zygomatic point): P Ref is a fiducial point (such as the zygomatic point); ProbeTip (Fiducial Point) is the probe tip (fiducial point); O Probe is the origin of the probe coordinates: O Cube is the origin of the calibration block coordinates: O Holo is the origin of the AR headset coordinates: P Holo is the origin of the AR headset coordinates.

[0111] The optical tracker is an NDI optical tracker that real-time tracks and monitors the detection tools to obtain the positions of various targets. The detection tools include a probe, a patient reference frame, and an AR calibration block, and output the spatial transformation matrix of each detection tool in the tracker coordinate system (such as ); The operator uses the probe to touch the anatomical landmark point (P Ref ) on the skull; Record the reference point P of the anatomical landmark point in the image space (CT model) Im ; Calculate the coordinate transformation matrix between the image coordinates and the real position of the patient through a registration algorithm (such as ICP)

[0112] The patient reference frame (O PatientRef ) is fixed on the skull and serves as the spatial reference coordinates for the entire operation;

[0113] The positions of all intraoperative instruments (such as drills, saw blades, etc.) will be located relative to the patient reference frame; It plays the role of "anchoring" the real patient space and avoids coordinate drift during the operation.

[0114] The AR calibration block is fused with the HoloLens coordinates for HoloLens spatial anchoring; The NDI tracking system simultaneously tracks the position of the ARCube construction calibration block under the tracker HoloLens determines its own position in the tracker coordinate system by recognizing the Cube Realize the spatial coincidence display of the virtual model and the real patient's bones in HoloLens, and use the augmented reality headset to real-time display the virtual prosthesis model, osteotomy line, and drilling path; Realize the doctor's visual surgical guidance in the AR view, greatly improving the positioning accuracy and operation efficiency.

[0115] Step S3: Surgical instrument calibration, align the surgical instruments with the TMJ surgical site and the virtual model respectively, and real-time track the position and posture of the patient entity.

[0116] Exemplary description. To achieve navigation in real-time surgery, in addition to obtaining the correspondence between the virtual image and the patient model in real-time, it is also necessary to establish the correspondence between the surgical instrument and the patient head mold. Therefore, it is necessary to calibrate the surgical instrument. In temporomandibular joint replacement surgery, the surgical instruments include a saw, a drill, and drill bits. The bone saw is used to cut bone, and the drill and drill bits are used to insert screws to fix the prosthesis in place.

[0117] For the drill and drill bits, it is necessary to obtain the tip position and axis of the surgical drill relative to the patient model in real-time. Therefore, the position relationship between the surgical drill and the probe is obtained through tip-axis alignment calibration, and the drill tip coordinate system O DrillTip and the patient reference coordinate system O PatientRef The translation-rotation matrix between them This calibration method uses a reference system with a reflective sphere. The reflective sphere is fixed to the handle and remains constant with the handle and drill bit throughout the TMJ replacement process. The calibration block is used to align the tips and axes of the drill bit and the probe.

[0118] The probe coordinate system O Probe and the handle reference coordinate system O HandRef to the navigation tracker device O Tracker The translation-rotation matrix can be automatically obtained as and Assume there is a virtual point P in space. The coordinates of P under O Probe and O HandRef are P Prb =(x Prb ,y Prb ,z Prb ) T and P DrT =(x DrT ,y DrT ,z DrT ) T . Since the probe and the drill bit have coincident tips and opposite axes, the relationship between the drill tip and the probe can be described as Therefore, The drill tip can be obtained through the following equation (7), which reflects the tip position and direction of the drill bit:

[0119]

[0120] After obtaining the translation-rotation matrix of the drill tip relative to the handle reference system, the transformation matrix of the drill tip relative to the patient reference system can also be calculated according to equation (8) through the following equation, so that the relative position of the drill tip relative to the patient head mold can be tracked by the navigation device and displayed on the screen in real-time, thereby being used to track the surgical drill needle in real-time.

[0121]

[0122] This study presents an algorithm for calibrating the cutting plane of a reciprocating saw, which is used to calculate the translation-rotation matrix of the endpoints of the cutting plane of the surgical reciprocating saw relative to the surgical site of the patient's temporomandibular joint in real time. Similarly, in order to calculate a handle reference system with a fixed position relative to the cutting plane of the reciprocating saw is also required during the entire surgical process, and the translation-rotation matrix between the coordinate system of the handle reference system and the coordinate system of the endpoints of the cutting plane of the reciprocating saw is calculated to implement the calibration operation of the saw.

[0123] Use the probe to point to the left end, right end in front of the saw, and any point on the cutting plane of the reciprocating saw, numbered 1, 2, and 3 in sequence. Similar to the alignment calibration with the tip axis, the coordinates of these three points in the surgical instrument reference frame O HandRef can all be obtained according to Equation (7). Assume that the coordinates of these three points in the coordinate system O HandRef are P1=(x1, y1, z1) T , P2=(x2, y2, z2) T , P3=(x3, y3, z3) T . That is to say, point 1 is the origin of the coordinate system, the line between point 1 and point 2, that is, the cutting line of the reciprocating saw is the y-axis, the perpendicular line from point 3 to the line between point 1 and point 2, that is, the advancing direction of the reciprocating saw is the x-axis, and the normal vector of the plane formed by points 1, 2, and 3 is the z-axis, as Figure 4 shown in A.

[0124] The vector from point 2 to point 1 is P 21 =(x2 - x1, y2 - y1, z2 - z1) T , the vector from point 3 to point 1 is P 31 =(x3 - x1, y3 - y1, z3 - z1) T , and we can obtain a set of orthogonal bases (e Saw , e x , e y , e z ) parallel to the O Saw coordinate axes, as shown in Equation (9)

[0125]

[0126] Normalize the orthogonal bases (e x , e y , e z ) to obtain the orthonormal basis which constitutes the rotation matrix from the coordinate system O HandRef to the coordinate system O Saw Since the defined point 1 is O Sawthe origin, so the translation matrix

[0127] Then, the translation matrix between the cutting surface and the handle of the finally calibrated oscillating saw is as follows:

[0128]

[0129] where and P1 are both 3×1 matrices.

[0130] After obtaining the transformation relationship of the saw surface of the surgical instrument with respect to the handle reference system it is also possible to calculate according to Equation (8) so as to obtain the position and axial correspondence between the cutting surface of the oscillating saw and the cutting surface of the patient's temporomandibular joint, and complete the calibration of the cutting surface of the oscillating saw.

[0131] After calibrating the cutting plane of the oscillating saw, a closed loop is formed among the oscillating saw, the TMJ surgical site of the patient, and the navigation device, enabling the navigation device to obtain the position and attitude of the cutting plane of the surgical oscillating saw with respect to the patient's surgical area in real time, and realizing the real-time tracking of the position and posture of the surgical instrument with respect to the patient's entity through the navigation system. Figure 4 Figure B shows the navigation interface after calibrating the cutting plane of the oscillating saw. The cutting plane of the oscillating saw is displayed on the navigation screen in real time and aligned with the cutting line of the pre-operative plan.

[0132] Step S4: Based on updating the position and posture of the patient entity, navigate the artificial joint installation path in real time.

[0133] Exemplarily, through a series of operations of virtual and actual device alignment, image registration, and surgical instrument calibration, the translation-rotation matrices between all objects are derived and a closed loop is formed. At this time, the surgeon can observe that the virtual model overlaps with the patient's head mold under the HoloLens 2 device, as Figure 5 shown in Figure A.

[0134] After starting the TMJ replacement, in order to make the display clearer, the display of the virtual model is hidden in the HoloLens2, and only the positions of the pre-operative planned prosthesis, cutting plane, and inserted screws with respect to the patient's TMJ are shown, and the handle is drilled into the patient's head mold along these positions. As Figure 5 shown in Figure B, the relative positions of the inserted screws and the surgical drill or bit, or the relative positions of the cutting plane and the oscillating saw, as well as the real-time depth, distance, and angle deviations are simultaneously displayed in the navigation software interface of the computer.

[0135] Example 3

[0136] The standard artificial temporomandibular joint installation navigation system based on augmented reality developed in Example 1 and Example 2 provides the following test data:

[0137] A series of phantom experiments were conducted on the phantoms of 10 patients to verify these metrics. On each phantom, 6 landmarks (anatomical landmarks or bone screw tips) were selected as the reference points for image-phantom registration and the given points for calculating TRE, DCE, and HRE. FRE was calculated through image-phantom registration and HoloLens tracker registration for each phantom. TRE and DCE were calculated and averaged at these six reference points. The measurement of HRE was to use a probe to locate the position of the reference point of the virtual model under the field of view of the HoloLens and calculate the deviation from the corresponding reference point in the actual phantom, which can be used to verify whether the virtual model and the actual phantom are well aligned, especially in occluded scenarios.

[0138] The mean values, standard deviations (SD), medians, and confidence intervals at the 95% confidence level (95% CI) of FRE, TRE, DCE, SCDE, SCAE, and HRE of ARNavig-sTMJP are shown in Table 1 below.

[0139] Table 1 System errors of ARNavig-sTMJP

[0140]

[0141]

[0142] By fitting the errors of the patient models of the guiding group and the AR group measured separately from the preoperative design, there was no significant difference in the anteroposterior distance error of the glenoid fossa prosthesis between the digital guide group and the AR group (0.84 ± 0.21 mm vs. 1.14 ± 0.69 mm, p > 0.05), and there was no significant difference in the medial-lateral error in the horizontal upward direction (0.91 ± 0.62 mm vs. 1.05 ± 0.60 mm, p > 0.05); there was no significant difference in the angle error of the ramus prosthesis (2.58 ± 1.94° vs. 2.07 ± 1.41°, p > 0.05) (as Figure 6 shown).

[0143] Only some exemplary embodiments of the present invention have been described by way of illustration. Undoubtedly, for those of ordinary skill in the art, the described embodiments can be modified in various different ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. An augmented reality-based standard artificial temporomandibular joint installation navigation system, characterized in that, Comprising: An optical tracking device for real-time tracking of the positions and postures of multiple reflecting spheres arranged on a registration and calibration detection tool to obtain the pose information of the corresponding surgical instrument and patient reference frame in space; An augmented reality display device for displaying a virtual three-dimensional model of a temporomandibular joint prosthesis and spatially superimposing it on the actual anatomical structure of the patient's maxillofacial region to achieve visual navigation guidance with virtual-real fusion; A registration and calibration detection tool assembly, including a probe, a patient reference frame, an instrument calibration block, and an AR calibration block, for establishing the spatial transformation relationship between the medical image coordinate system, the tracker coordinate system, and the augmented reality coordinate system, and outputting the corresponding translation-rotation matrix; A preoperative planning module, based on the three-dimensional reconstructed maxillofacial CT data and the standard prosthesis geometric model, combined with key anatomical point extraction and point cloud analysis algorithms, automatically generates the placement direction, position, and rotation matrix of the prosthesis, and outputs the three-dimensional pose matching result; A navigation control module, based on real-time tracking data and preoperative planning results, embeds the virtual prosthesis model into the augmented reality view, and performs real-time navigation update and deviation prompt according to the intraoperative instrument motion state.

2. The augmented reality-based standard artificial temporomandibular joint installation navigation system according to claim 1, wherein It further includes an interface interaction module, which supports multi-view synchronous display of the virtual prosthesis, bone surface structure, and detection tool trajectory, and provides functions such as error correction, model adjustment, and multi-scheme comparison.

3. An automated placement navigation method for temporomandibular joint prostheses based on augmented reality, characterized in that, Comprising the following steps: Step S1: Perform image registration on the patient's preoperative CT image and the target surgical site to obtain the registration result; Step S2: Real-time update the registration result, and readjust the registration based on the AR display device to achieve the virtual-real transformation between the virtual image and the actual model; Step S3: Calibrate the surgical instrument, align the surgical instrument with the TMJ surgical site and the virtual model respectively, and real-time track the position and posture of the patient entity; Step S4: Based on the updated position and posture of the patient entity, real-time navigate the artificial joint installation path.

4. The automated placement navigation method for a temporomandibular joint prosthesis based on augmented reality according to claim 3, wherein The augmented reality-based standard artificial temporomandibular joint installation navigation system according to claims 1-2, wherein the translation-rotation matrix is the transformation matrix from coordinate system A to coordinate system B; it includes a rotation matrix and a translation matrix, the rotation matrix represents the rotation direction of coordinate system A relative to coordinate system B, and the translation matrix represents the position of the origin of coordinate system A in coordinate system B.

5. A method for automatically placing and navigating a temporomandibular joint prosthesis based on augmented reality according to claim 4, characterized in that, The acquisition logic of the translation-rotation matrix: Collect a set of corresponding point pairs that can be expressed in both coordinate systems when the probe clicks on the anatomical points; Based on the objective function of the alignment algorithm registration, solve the rotation and translation amounts under the minimum error, and use the rotation and translation amounts under the minimum error as the translation-rotation matrix.

6. The automated placement navigation method for a temporomandibular joint prosthesis based on augmented reality according to claim 5, wherein, Performing image registration on the patient's preoperative CT image and the target surgical site includes the following steps: Step S101: Provide the world coordinate system O based on the NDI navigation head Tracker ; Step S102: Provide a coordinate system O in the virtual field of view based on the HoloLens device Holo ; Step S103: Provide a prosthesis coordinate system O for the 3D model reconstructed from the CT image Image ; Step S104: The surgical site of the patient, with its coordinate system O PatientRef is provided by the patient's reference frame, and its position relative to the patient remains unchanged throughout the surgical procedure; Step S105: A surgical instrument, whose coordinate system O HandRef is provided by a reference frame fixed to the handle during the surgery.

7. A method for automatically placing and navigating a temporomandibular joint prosthesis based on augmented reality according to claim 6, characterized in that, The specific steps for the step S2 to achieve the virtual-real transformation between the virtual image and the actual model are: Initial registration initialization, use the probe to click on the anatomical landmark points on the patient's jawbone; collect the positions of the anatomical landmark points in the tracker coordinate system and the image space, and obtain the transformation matrix between the image coordinate system and the patient reference coordinate system; Construct an initial virtual-real transformation matrix to transform the virtual image from the image space to the HoloLens coordinate space. Real-time tracking and error detection, continuously monitor the changes of the reference frame and the reflective sphere of the detection tool in the NDI navigation system, identify the movement of the patient's head or slight deviation of the reference frame, and obtain the detection registration error; If the set error threshold is exceeded, the system automatically or prompts for registration update, re-collect a set of reference points or call the collected point set, based on the current NDI tracking results, recalculate the transformation matrix between the image and reality, and re-position, rotate, and align the virtual image in the AR view in HoloLens; Use the updated transformation matrix to convert the three-dimensional virtual model to the correct position in the AR coordinate system, and achieve the precise overlap of the virtual prosthesis and the real bone surface in the AR view.

8. A method for automatically placing and navigating a temporomandibular joint prosthesis based on augmented reality according to claim 7, characterized in that, The specific steps for aligning the surgical instrument with the TMJ surgical site and the virtual model are as follows: Obtain the correspondence between the virtual image and the patient model in real time and establish the correspondence between the surgical instrument and the patient head model; Use the instrument calibration block for calibration, insert the probe tip into the known hole positions of the calibration block, record the position of the reflective sphere under the NDI system, obtain the transformation matrix of the probe rigid body, and continuously track the three-dimensional position and orientation of the tip / blade; Establish a spatial correspondence between the virtual image and the actual model, and install a reference frame with a reflective sphere in the zygomatic arch or frontal bone area of the patient; Use the probe to click on more than 3 anatomical landmark points in sequence and match them with the corresponding points in the CT model; Map the real-time position of the instrument end to the image coordinate system or the AR view to construct a spatial transformation chain: Continuously track and update the position of the instrument end in the image coordinate or HoloLens coordinate; display the movement trajectory of the instrument model in HoloLens to assist the doctor in judging whether it deviates from the trajectory; achieve the alignment of the surgical instrument with the TMJ surgical site and the virtual model.

9. The automated placement navigation method for a temporomandibular joint prosthesis based on augmented reality according to claim 8, wherein Use a reference system with a reflective sphere, the reflective sphere is fixed on the handle and remains constant with the handle and the drill bit during the entire TMJ replacement process, and the calibration block is used to align the tips and axes of the drill bit and the probe; Obtain the tip position and axis of the surgical drill relative to the patient model in real time, obtain the position relationship between the surgical drill and the probe through tip-axis alignment calibration, and determine the translation-rotation matrix between the drill tip coordinate system and the patient reference coordinate system.

10. A method for automatically placing and navigating a temporomandibular joint prosthesis based on augmented reality according to claim 9, characterized in that, The display of the virtual model is hidden in HoloLens 2, and the position of the prosthesis, cutting plane, and inserted screws in the preoperative plan relative to the patient's TMJ is displayed, and the handle is drilled into the patient head model along these positions; The relative positions of the inserted screws and the surgical drill or drill bit, or the relative positions of the cutting plane and the oscillating saw, as well as the real-time depth, distance, and angle deviations are simultaneously displayed on the navigation software interface of the computer.

Images