Oral surgery navigation system and method based on multi-anchor point coordination and mandibular kinematic constraints

Abstract

This invention discloses an oral surgery navigation system and method based on multi-anchor point collaboration and mandibular kinematic constraints, belonging to the field of augmented reality surgical navigation technology. The system employs two smart wearable devices fixed to the patient's mandible and maxilla (or skull) respectively as dual anchor points, and utilizes an observation terminal and a fixed reference terminal to construct a dual-view visual measurement network, achieving real-time acquisition of the relative pose of the craniomaxilla. The system further integrates inertial measurement units, magnetometers, ultra-wideband communication modules, and environmental point cloud map positioning information to provide continuous pose estimation even when vision is partially or completely lost. Simultaneously, a personalized temporomandibular joint kinematic constraint model is established based on preoperative images to constrain mandibular movement and reduce inertial recursion errors. Through multi-source sensor fusion and augmented reality display, stable navigation is achieved during complex oral surgeries.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality (AR) and oral and maxillofacial surgery navigation technology, specifically to an oral surgery navigation system and method that utilizes multiple consumer-grade smart wearable and mobile devices, combined with personalized mandibular kinematic constraints and environmental perception, to achieve high robustness and high precision. Background Technology

[0002] In surgeries such as dental implants and jaw osteotomy, surgeons often need to precisely overlay a pre-reconstructed 3D model of the jawbone onto the patient's actual surgical area to avoid important anatomical structures. Existing AR surgical navigation systems mostly use optical markers fixed on the patient's teeth and track their position and posture through an external camera. However, due to the confined surgical space and frequent instrument obstructions, purely visual solutions are prone to losing lock, leading to navigation interruptions.

[0003] To overcome occlusion, existing solutions incorporate inertial measurement units (IMUs) for short-term pose recursion. However, these solutions generally treat the patient's head-jaw region as a rigid body, neglecting the mandibular movements independent of the skull, such as opening and closing the mouth and swallowing, which are common during surgery. When vision fails, inertial recursion cannot distinguish between overall head movement and independent mandibular movement, leading to severe drift in the stacked model. Furthermore, the movement of the observation device (such as the doctor's handheld mobile device) itself can be incorrectly included in the patient's movement, further reducing accuracy.

[0004] Some approaches attempt to use markerless depth cameras for skull tracking on the patient's face, but depth cameras are limited by working distance and lighting conditions, resulting in insufficient stability in complex surgical environments. Another approach is to increase the number of IMUs, but consumer-grade IMUs drift quickly and still cannot meet the sub-millimeter accuracy requirements under prolonged occlusion. Therefore, there is an urgent need for a navigation system that can fundamentally separate the relative movements of the mandible and maintain high accuracy under all working conditions. Summary of the Invention

[0005] The purpose of this invention is to provide an oral surgery navigation system and method based on multi-anchor point collaboration and mandibular kinematic constraints. Through the organic integration of multiple technologies such as dual intelligent anchor points, dual-view visual network, ultra-wideband spatial positioning, temporomandibular joint model constraints and environmental point cloud map, high-precision navigation is achieved under fully occluded conditions.

[0006] Technical solution:

[0007] This invention first discloses an oral surgery navigation system based on multi-anchor point collaboration and mandibular kinematic constraints, comprising:

[0008] A first smart wearable device is used to be fixed to the patient's mandibular dentition via a first custom rigid connection plate. The first smart wearable device has at least a first display screen, a first inertial measurement unit, a first magnetometer, and a first ultra-wideband communication module. The first display screen is used to display a first dynamic optical graphic.

[0009] A second smart wearable device is used to be fixed to the patient's maxilla or skull region via a second customized connection device. The second smart wearable device has at least a second display screen, a second inertial measurement unit, a second magnetometer, and a second ultra-wideband communication module. The second display screen is used to display a second dynamic optical graphic.

[0010] The first mobile terminal, serving as an observation terminal, includes at least a first rear-facing camera, a third inertial measurement unit, a third magnetometer, a third ultra-wideband communication module, and a first environmental perception sensor. The first rear-facing camera is used to simultaneously or time-divisionally capture the first dynamic optical image and the second dynamic optical image. The first mobile terminal is held by the surgeon or assistant or mounted on an adjustable bracket, with its camera's visual axis always facing the patient's oral surgical area, ensuring that at least one of the first or second dynamic optical images can be observed. The first mobile terminal is allowed to move within a preset observation range, but its pose changes are compensated in real time by the inertial measurement unit, the ultra-wideband communication module, and visual measurement.

[0011] The second mobile terminal, as a reference terminal, is fixedly installed on a support next to the operating table, an instrument table support or other stable structure during navigation, and its installation position remains unchanged; it has at least a second rear camera, a fourth inertial measurement unit, a fourth magnetometer, a fourth ultra-wideband communication module and a second environmental perception sensor, and the second rear camera is used to capture the first dynamic optical image and the second dynamic optical image simultaneously or in time-division manner.

[0012] The multi-source fusion module, running at the application layer of the first mobile terminal and / or the second mobile terminal, is used for:

[0013] Receive visual pose data about the first smart wearable device and the second smart wearable device obtained by the first mobile terminal and the second mobile terminal respectively, and directly calculate the six-degree-of-freedom relative pose of the mandible relative to the maxilla.

[0014] It synchronously receives high-frequency motion data from all inertial measurement units, heading data from all magnetometers, and ranging and / or angle of arrival data between all ultra-wideband communication modules;

[0015] During normal visual tracking, the spatial transformation relationship between each coordinate system and the offset and scale parameters of each inertial measurement unit are calibrated online based on the visual pose data.

[0016] During periods of partial or complete failure of visual tracking, the environmental point cloud is collected in real time using the environmental perception sensors of the second mobile terminal and / or the first mobile terminal, and matched with the pre-built environmental point cloud map to obtain positioning information. Combined with the spatial positioning constraints of angle of arrival-range fusion composed of at least three ultra-wideband communication modules, differential motion data of multiple inertial measurement units, heading constraints of multiple magnetometers, and a personalized temporomandibular joint kinematic constraint model based on preoperative patient image data, the current pose of the mandible relative to the maxilla is recursively estimated to maintain the stable superposition of navigation images.

[0017] The augmented reality visualization module is used to overlay the preoperatively reconstructed three-dimensional anatomical model onto the surgical field of view in real time according to the relative pose.

[0018] The error feedback module is used to calculate the spatial deviation between the surgical instruments and the planned target point in real time and output it in a visual manner.

[0019] Preferably, the first customized rigid connection plate of the first smart wearable device is a personalized 3D printed connection plate designed based on the three-dimensional scanning data of the patient's mandibular dentition. It is fixed to the mandibular dentition by tooth surface snap-fit, bonding or magnetic attraction to ensure that the first smart wearable device and the mandible maintain a stable spatial relationship.

[0020] The second connection device of the second smart wearable device is a 3D-printed fixation plate or non-invasive headband that is personalized based on the patient's maxillary dentition or skull surface anatomy. Its connection with the skull is rigid or semi-rigid to ensure that the transformation relationship between the optical coordinate system and the maxillary coordinate system remains constant during the operation.

[0021] Preferably, the first dynamic optical pattern and the second dynamic optical pattern are dynamic QR codes or flashing patterns containing timestamp codes and / or device identifiers, used to achieve frame-level time synchronization and device differentiation in multi-view visual measurements.

[0022] Preferably, the multi-source fusion module obtains the personalized temporomandibular joint kinematic constraint model in the following manner:

[0023] A three-dimensional model of the mandible is segmented from preoperative CBCT or CT images, the rotation center of the bilateral condyles is identified and located, and the rotation axis of the temporomandibular joint connecting the bilateral rotation centers is established.

[0024] The pose of the rotation axis is registered to the coordinate system of the second smart wearable device or the maxilla;

[0025] During visual failure, only the angular change of the mandible relative to the maxilla along the rotation axis is estimated as a state variable, instead of the unconstrained integral over the six degrees of freedom of the entire state.

[0026] Preferably, the environmental perception sensors of the first mobile terminal and / or the second mobile terminal are lidar and / or time-of-flight depth sensors. The pre-built environmental point cloud map is generated by the second mobile terminal scanning the surgical environment before the surgery begins and includes environmental anchor points with significant geometric features. When visual tracking fails and ultra-wideband data is temporarily abnormal, the first mobile terminal collects environmental point clouds in real time through its own environmental perception sensors and matches them with the pre-built environmental point cloud map to restore its absolute pose in space.

[0027] Preferably, when the multi-source fusion module utilizes the ultra-wideband communication module, it simultaneously acquires the ultra-wideband signal angle of arrival and time-of-flight ranging values ​​among at least three of the first mobile terminal, the second mobile terminal, and the two smart wearable devices. By triangulation and angle of arrival intersection, it calculates the three-dimensional position of each device in space, thus forming a spatial positioning constraint independent of optics.

[0028] Preferably, the multi-source fusion module executes a hierarchical degradation fusion strategy based on the current availability status of each sensor, specifically including:

[0029] When both optical patterns can be clearly captured by at least one of the two mobile terminals, the visual solution of the six degrees of freedom pose is the primary method, while other sensors are used for online calibration and redundancy verification.

[0030] When only one optical pattern can be captured, the visible anchor point provides the absolute pose reference, while the invisible anchor point is recursively derived based on the latest calibrated inertial parameters and ultrawide spatial constraints, and combined with a personalized temporomandibular joint kinematic constraint model to limit the recursive divergence.

[0031] When all optical images are blocked, the inertial measurement unit differential recursion, ultra-wideband angle-of-arrival-range fusion and magnetometer heading constraint work together and forcibly apply temporomandibular joint kinematic constraints to maintain pose estimation.

[0032] When ultra-wideband signals are also severely interfered with, environmental point cloud map relocation is used as a fallback.

[0033] Preferably, the first mobile terminal and the second mobile terminal calibrate their relative attitude and position in real time through ultra-wideband ranging and resection to maintain the consistency of the dual-view visual network measurement.

[0034] Preferably, the multi-source fusion module utilizes the neural engine built into the first mobile terminal, the second mobile terminal, or the smart wearable device to run a trained recurrent neural network model. This model learns the bias change pattern of habitual measurement units and the personalized temporal characteristics of the patient's mandibular movements during the visual availability phase, and adaptively compensates and predicts the inertial data during the visual failure phase.

[0035] This invention also discloses an oral surgery navigation method based on multi-anchor point collaboration and mandibular kinematic constraints, comprising the following steps:

[0036] S1: A first smart wearable device is fixed in the patient's mandibular dentition, and a second smart wearable device is fixed in the maxilla or skull region. The first and second smart wearable devices respectively display different dynamic optical graphics.

[0037] S2: Set the first mobile terminal as the observation end and aim it at the surgical area, and set the second mobile terminal to a stable position in the surgical environment. The rear cameras of the two mobile terminals are configured to capture the optical graphics displayed by the two smart wearable devices simultaneously or in turn.

[0038] S3: Based on the pose of the two optical images acquired by the dual-view visual measurement network, the initial six-degree-of-freedom relative pose of the mandible relative to the maxilla is directly calculated. The coordinate system extrinsic parameters of each inertial measurement unit and the ultra-wideband module are simultaneously calibrated online. A kinematic constraint model containing the position of the personalized temporomandibular joint rotation axis is imported from the preoperative image.

[0039] S4: During navigation, real-time time-synchronized acquisition of multi-source sensor data, including visual pose, all inertial measurement unit data, all magnetometer data, and all ultra-wideband ranging and / or angle of arrival data;

[0040] S5: Determine the availability of visual information. When all or part of the vision is available, perform fusion output based on the relative pose calculated by vision and continuously update the calibration parameters.

[0041] S6: When visual tracking fails, lock the last valid relative pose of the last frame before the failure, switch to inertial recursion mode, and integrate the personalized temporomandibular joint kinematic constraint model as a process model into the recursion. At the same time, integrate the differential motion data of multiple inertial measurement units, ultra-wideband spatial positioning constraints, multi-magnetic meter heading constraints and pre-built environmental point cloud map positioning information to recursively estimate the current mandibular relative pose.

[0042] S7: When vision is restored, compare the deviation between the visual repositioning pose and the recursive pose, perform smoothing correction and update the inertial measurement unit bias estimate, and switch to a fusion mode that mainly solves the relative pose based on vision.

[0043] S8: Throughout the entire surgical procedure, the preoperative 3D model is superimposed on the surgical image on the observation terminal in real time based on the relative pose, and the error between the surgical instruments and the planned target point is calculated and displayed.

[0044] The beneficial effects of this invention are:

[0045] Achieving precise superposition of true anatomical relationships under the condition of relative movement between the maxilla and mandible eliminates model misalignment caused by traditional rigid assumptions.

[0046] Under prolonged and extensive visual occlusion, relying on temporomandibular joint (TMJ) kinematic constraints, ultra-wideband (UWB) spatial positioning network, and environmental maps, the inertial recursive cumulative error is significantly reduced, navigation stability is improved during visual failure, and navigation is uninterrupted.

[0047] It is entirely based on consumer-grade devices, making it cost-effective and easy to promote in clinical settings.

[0048] Through layered degradation, the system is highly robust and has no risk of single point of failure. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall system structure.

[0050] Figure 2 This is a schematic diagram of the fixed positions of two smart wearable devices and the layout of two terminals.

[0051] Figure 3 This is a logical block diagram of a multi-sensor hierarchical fusion architecture.

[0052] Figure 4 Flowchart for establishing and using the kinematic constraint model of the temporomandibular joint.

[0053] Figure 5 This is a flowchart illustrating the pose derivation process based on the fusion of TMJ constraints and UWB spatial network during visual failure. Detailed Implementation

[0054] The present invention will be further described below with reference to embodiments, but the scope of protection of the present invention is not limited thereto:

[0055] Example 1: System Hardware Configuration and Fixing

[0056] like Figure 1 and Figure 2As shown, this embodiment provides an oral surgery navigation system based on multi-anchor point collaboration and mandibular kinematic constraints. The system includes a first smart wearable device, a second smart wearable device, a first mobile terminal, a second mobile terminal, and a multi-source fusion module. The first smart wearable device is a smartwatch, fixed to the buccal side of the patient's mandibular canine to molar region via a 3D-printed resin connecting plate using magnetic attraction. The second smart wearable device is another smartwatch, fixed to a stable area of ​​the skull via an elastic headband / transparent fixation frame customized to the patient's forehead or zygomatic arch shape as a second customized connection device. Both the first and second smart wearable devices display encoded dynamic QR codes with embedded synchronization timestamps on their screens. The first mobile terminal (observation end) is a smartphone equipped with a rear camera, LiDAR, inertial measurement unit, magnetometer, and ultra-wideband communication chip, held by an assistant or fixed to an adjustable bracket. The second mobile terminal (reference end) is the same model smartphone, mounted on a fixed bracket beside the operating table, ensuring its rear camera's field of view covers the entire surgical area where both smart wearable devices are located. Both the first and second smart wearable devices integrate ultra-wideband (UWB) communication modules and establish data connections with the first and second mobile terminals via UWB wireless communication links. Before the operation, the reference device first scans the operating room with lidar to generate and save a three-dimensional environmental point cloud map.

[0057] During system initialization, a global reference coordinate system is established based on the location of the fixed reference end. The first smart wearable device, the second smart wearable device, the observation mobile terminal, and the reference mobile terminal each establish their own local coordinate systems, and initial spatial position association is completed through visual recognition results and ultra-wideband communication results. The dynamic QR codes displayed by the two smart wearable devices contain device numbers and time synchronization information to ensure that data acquisition across multiple devices is based on a unified time reference. The system internally employs a unified timestamp caching mechanism to synchronize and manage visual, inertial measurement unit, magnetometer, and ultra-wideband data.

[0058] Example 2: Establishment and Registration of a Personalized Temporomandibular Joint Kinematic Constraint Model

[0059] like Figure 4 As shown, this embodiment establishes a personalized temporomandibular joint kinematic constraint model based on the patient's preoperative CBCT data. A three-dimensional model of the mandible is segmented from the patient's preoperative CBCT data, and the rotation centers of the bilateral condyles are located. The straight line connecting these two points is the patient's personalized temporomandibular joint rotation axis. The system registers the established temporomandibular joint rotation axis with the spatial position of a second smart wearable device, ensuring that the rotation axis is stably bound to the skull reference coordinate system.

[0060] During visual impairment, mandibular movement is constrained to rotation about the personalized temporomandibular joint rotation axis, and its motion state can be represented as follows:

[0061] θ∈[θ min ,θ max ]

[0062] Where θ represents the rotation angle of the mandible around the axis of rotation of the temporomandibular joint; θ min and θ max These represent the minimum and maximum mouth opening angles obtained based on the patient's preoperative imaging data or dynamic calibration, respectively.

[0063] During navigation, when visual tracking is normal, the system continuously updates the relative spatial relationship between the mandible and the skull; when vision fails partially or completely, the system no longer performs a fully unconstrained six-degree-of-freedom recursion on the mandible, but instead, based on the rotation axis, only allows the mandible to perform finite-degree-of-freedom motion estimation along the opening and closing direction, thereby reducing the positional error caused by inertial drift.

[0064] Simultaneously, the system allows for setting compensation ranges for lateral mandibular movements and slight translational movements based on the patient's actual mouth-opening habits, thereby improving adaptability in complex oral surgery scenarios. These compensation ranges can be obtained through preoperative CBCT imaging measurements (such as extracting the geometric margin of the temporomandibular joint space) or through real-time dynamic calibration during system initialization guided by the patient's mouth-opening and closing movements.

[0065] Example 3: Dual-view visual network and online calibration

[0066] like Figure 3 As shown, the multi-source fusion module adopts a layered architecture that fuses visual, inertial measurement, ultra-wideband positioning, and environmental perception information. At the start of the procedure, the rear cameras of both mobile terminals capture dynamic QR codes on the first and second smart wearable devices at a frequency of 60Hz. At the application layer, the six-DOF pose of each device relative to each smart wearable device is calculated using an augmented reality framework or a custom visual algorithm.

[0067] The system calculates the positional relationship of the mandible relative to the maxilla based on visual measurement results:

[0068] Tjaw maxilla=(Tcamera maxilla) -1 ·Tcamera jaw

[0069] Where: Tcamera jaw is the pose of the mandibular anchor point in the camera coordinate system; Tcamera maxilla is the pose of the maxillary anchor point in the camera coordinate system; Tjaw maxilla is the pose transformation matrix of the mandible relative to the maxilla.

[0070] The system utilizes synchronously acquired visual pose data, along with all inertial measurement units (IMUs) and ultra-wideband data, for joint initialization: solving for the fixed transformation matrix between the reference and observation ends, the rigid pose transformation matrix between each smart wearable device IMU and its optical coordinate system, and estimating the bias of each IMU. The multi-source fusion module constructs a state estimator based on the extended Kalman filter (EKF) or graph optimization algorithm, using the pose calculated from dual-view vision as the observation update vector and the kinematic integral of the IMU as the state prediction vector, achieving spatiotemporal calibration in multiple coordinate systems.

[0071] The state prediction process of the extended Kalman filter is represented as follows:

[0072] x k =f(x k -1,u k )+w k

[0073] The observation update process is represented as follows:

[0074] z k =h(x k )+v k

[0075] Where: x k The current state vector; u k Input for the inertial measurement unit; z k For visual, ultrawideband, and magnetometer observations; w k and v k These represent process noise and observation noise, respectively.

[0076] Because of the use of dual-view observation, the visual pose can be optimized through epipolar constraints and triangulation principles, thereby improving the absolute accuracy of the initial registration.

[0077] The system uses a fixed reference point as a unified spatial reference and simultaneously observes two smart wearable devices via dual terminals to unify the spatial relationships between the observation point, the skull anchor point, and the mandibular anchor point. The vision module performs real-time detection of dynamic QR code corner points and, combined with camera parameters, calculates the relative pose relationships between the devices. During navigation, each smart wearable device and each mobile terminal interacts with wireless pulse signals in real time via a UWB wireless communication link to calculate the relative distance and radio frequency characteristic data between the devices. This data is then transmitted in real-time to the multi-source fusion module via this link. The system further utilizes ultra-wideband ranging results to correct the visual calculation results, reducing errors caused by monocular vision under occlusion and large-angle deflection conditions.

[0078] During the initialization phase, the system synchronously calibrates the correspondence between the camera coordinate system, the smart wearable device coordinate system, the inertial measurement unit coordinate system, and the world reference coordinate system. The multi-source fusion module operates using a hierarchical thread structure, including a vision processing thread, an inertial data acquisition thread, an ultra-wideband communication thread, and a fusion computing thread. All threads are synchronized using a unified timestamp.

[0079] Example 4: Layered Fusion Navigation for All Operating Conditions

[0080] like Figure 5 As shown, the system executes a hierarchical degradation fusion strategy based on the real-time status of the vision, ultra-wideband, and inertial measurement units. During navigation, the multi-source fusion module continuously monitors the point rate within visual feature points. When the observation end camera is obstructed by the device, but the reference end can still clearly capture the optical graphics displayed by the first and second smart wearable devices, the system directly updates the relative pose of the mandible using the visual calculation of the reference end. The observation end receives the update through the calibrated transformation matrix, and navigation switches seamlessly. When the observation end can still clearly capture the optical graphics displayed by the first and second smart wearable devices, but the reference end is temporarily obstructed, the system uses the visual calculation result of the observation end as the source of the current pose update. The reference end uses the most recent valid calibration result to maintain coordinate system consistency and recalibrates the spatial relationship between the two terminals after visual recovery. When both ends are obstructed, the system switches to the inertial recursive mode: differential elimination eliminates the motion of the observer and the reference end themselves, extracts the incremental motion of the mandibular wearable device's inertial measurement unit relative to the cranial wearable device's inertial measurement unit, and forcibly applies a temporomandibular joint rotation axis constraint—only integrating the angular component around the rotation axis and ignoring noise disturbances in other directions. During this process, the ultra-wideband modules of the four devices continuously communicate, and calculate the three-dimensional configuration of the four devices in space by fusing the angle of arrival and ranging, which serves as the input filter for absolute position constraints.

[0081] The ultrawideband ranging result between any two devices can be expressed as:

[0082] d ij =√[(x i -x j ) 2 +(y i -y j ) 2 +(z i -z j ) 2 ]

[0083] Where: d ij This represents the distance between device i and device j; (x i ,y i ,z i ) and (x j ,y j ,zj ) represent the spatial coordinates of the corresponding devices.

[0084] The magnetometer provides a ground-based heading difference, further limiting rotational drift. If the ultra-wideband signal abruptly changes due to large-area metal obstruction, the system automatically degrades to a pure inertial measurement unit with temporomandibular joint constraint mode, and triggers the observation-end lidar to perform an instantaneous environmental point cloud scan, matching it with a pre-built map to restore its own pose, thereby maintaining global consistency.

[0085] The system dynamically adjusts the fusion weights of each sensor based on the current number of visual feature points, the quality of ultra-wideband communication, and the stability of inertial data. When visual information is complete, the visual localization result is used as the primary pose source; when visual information is partially occluded, visible anchor points are used as absolute references, combined with inertial recursion to maintain short-term continuity; when vision completely fails, the system automatically switches to a joint mode of inertial, ultra-wideband, and kinematic constraints. The entire switching process employs a gradual state transition method to avoid obvious jumps or flickering of the augmented reality model on the observation terminal.

[0086] Example 5: Visual Restoration and Correction

[0087] Once the visual occlusion is removed, the system re-identifies the dynamic QR code and resumes visual pose measurement. The system first compares the spatial deviation between the current visual result and the inertial recursive result.

[0088] The pose deviation after visual recovery can be expressed as:

[0089] e=T vision -T inertial

[0090] Wherein: T vision Indicates the visual relocalization result; T inertial represents the result of inertial recursion; e represents the pose error between the two.

[0091] When the deviation is within the allowable range, the system employs a progressive smoothing correction method to slowly return the augmented reality model to the visual measurement position, avoiding sudden model jumps that could affect the doctor's observation. When the deviation exceeds the preset range, the system further verifies the consistency between the LiDAR environmental point cloud and the pre-built environmental map. If it is confirmed that inertial recursion has resulted in cumulative drift, global relocalization is performed, and the inertial state parameters are reinitialized. Simultaneously, the system records the inertial motion trajectory, visual recovery error, and mandibular movement changes during the current occlusion period, and uses this information to update the inertial bias compensation model built based on Long Short-Term Memory (LSTM) networks or Gated Recurrent Units (GRUs) to improve the stability of pose prediction during subsequent occlusion periods.

[0092] The model takes high-frequency data sequences from historical inertial measurement units and mandibular kinematic constraint parameters as inputs and the predicted inertial drift bias as output. It fine-tunes the network weights online by continuously feeding back the residuals during visual recovery, thereby improving the stability of pose prediction during subsequent occlusion periods.

[0093] The embodiments described above are only for illustrating the technical concept of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. An oral surgery navigation system based on multi-anchor point collaboration and mandibular kinematic constraints, characterized in that, include: A first smart wearable device is used to be fixed to the patient's mandibular dentition via a first custom rigid connection plate. The first smart wearable device has at least a first display screen, a first inertial measurement unit, a first magnetometer, and a first ultra-wideband communication module. The first display screen is used to display a first dynamic optical graphic. A second smart wearable device is used to be fixed to the patient's maxilla or skull region via a second customized connection device. The second smart wearable device has at least a second display screen, a second inertial measurement unit, a second magnetometer, and a second ultra-wideband communication module. The second display screen is used to display a second dynamic optical graphic. The first mobile terminal, serving as an observation terminal, includes at least a first rear-facing camera, a third inertial measurement unit, a third magnetometer, a third ultra-wideband communication module, and a first environmental perception sensor. The first rear-facing camera is used to simultaneously or time-divisionally capture the first dynamic optical image and the second dynamic optical image. The first mobile terminal is held by the surgeon or assistant or mounted on an adjustable stand. Its camera's visual axis is always directed towards the patient's oral surgical area, ensuring that it can observe at least one of the first or second dynamic optical images. The first mobile terminal is allowed to move within a preset observation range, but its pose changes are compensated in real time by the inertial measurement unit, the ultra-wideband communication module, and visual measurement. The second mobile terminal, serving as a reference terminal, is fixedly mounted on a support frame, instrument table support, or other stable structure beside the operating table during navigation, and its mounting position remains unchanged. It includes at least a second rear-facing camera, a fourth inertial measurement unit, a fourth magnetometer, a fourth ultra-wideband communication module, and a second environmental perception sensor. The second rear-facing camera is used to simultaneously or in time-division multiplexing capture the first dynamic optical image and the second dynamic optical image. The multi-source fusion module, running at the application layer of the first mobile terminal and / or the second mobile terminal, is used for: Receive visual pose data about the first smart wearable device and the second smart wearable device obtained by the first mobile terminal and the second mobile terminal respectively, and directly calculate the six-degree-of-freedom relative pose of the mandible relative to the maxilla. It synchronously receives high-frequency motion data from all inertial measurement units, heading data from all magnetometers, and ranging and / or angle of arrival data between all ultra-wideband communication modules; During normal visual tracking, the spatial transformation relationship between each coordinate system and the offset and scale parameters of each inertial measurement unit are calibrated online based on the visual pose data. During periods of partial or complete failure of visual tracking, the environmental point cloud is collected in real time using the environmental perception sensors of the second mobile terminal and / or the first mobile terminal, and matched with the pre-built environmental point cloud map to obtain positioning information. Combined with the spatial positioning constraints of angle of arrival-range fusion composed of at least three ultra-wideband communication modules, differential motion data of multiple inertial measurement units, heading constraints of multiple magnetometers, and a personalized temporomandibular joint kinematic constraint model based on preoperative patient image data, the current pose of the mandible relative to the maxilla is recursively estimated to maintain the stable superposition of navigation images. The augmented reality visualization module is used to overlay the preoperatively reconstructed three-dimensional anatomical model onto the surgical field of view in real time according to the relative pose. The error feedback module is used to calculate the spatial deviation between the surgical instruments and the planned target point in real time and output it in a visual manner.

2. The system according to claim 1, characterized in that, The first customized rigid connection plate of the first smart wearable device is a personalized 3D printed connection plate designed based on the three-dimensional scanning data of the patient's mandibular dentition. It is fixed to the mandibular dentition by tooth surface snap-fit, bonding or magnetic attraction to ensure a stable spatial relationship between the first smart wearable device and the mandible. The second connection device of the second smart wearable device is a 3D-printed fixation plate or non-invasive headband that is personalized based on the patient's maxillary dentition or skull surface anatomy. Its connection with the skull is rigid or semi-rigid to ensure that the transformation relationship between the optical coordinate system and the maxillary coordinate system remains constant during the operation.

3. The system according to claim 1, characterized in that, The first dynamic optical pattern and the second dynamic optical pattern are dynamic QR codes or flashing patterns containing timestamp codes and / or device identifiers, used to achieve frame-level time synchronization and device differentiation in multi-view visual measurements.

4. The system according to claim 1, characterized in that, The multi-source fusion module obtains the personalized temporomandibular joint kinematic constraint model in the following manner: A three-dimensional model of the mandible is segmented from preoperative CBCT or CT images, the rotation center of the bilateral condyles is identified and located, and the rotation axis of the temporomandibular joint connecting the bilateral rotation centers is established. The pose of the rotation axis is registered to the coordinate system of the second smart wearable device or the maxilla; During visual failure, only the angular change of the mandible relative to the maxilla along the rotation axis is estimated as a state variable, instead of the unconstrained integral over the six degrees of freedom of the entire state.

5. The system according to claim 1, characterized in that, The environmental perception sensors of the first mobile terminal and / or the second mobile terminal are lidar and / or time-of-flight depth sensors. The pre-built environmental point cloud map is generated by the second mobile terminal scanning the surgical environment before the surgery begins and includes environmental anchor points with significant geometric features. When visual tracking fails and ultra-wideband data is temporarily abnormal, the first mobile terminal collects environmental point clouds in real time through its own environmental perception sensors and matches them with the pre-built environmental point cloud map to restore its absolute pose in space.

6. The system according to claim 1, characterized in that, When utilizing the ultra-wideband communication module, the multi-source fusion module simultaneously acquires the ultra-wideband signal angle of arrival and time-of-flight ranging values ​​among at least three of the first mobile terminal, the second mobile terminal, and the two smart wearable devices. By triangulation and angle of arrival intersection, the three-dimensional position of each device in space is calculated, thus forming a spatial positioning constraint independent of optics.

7. The system according to claim 1, characterized in that, The multi-source fusion module executes a hierarchical degradation fusion strategy based on the current availability status of each sensor, specifically including: When both optical patterns can be clearly captured by at least one of the two mobile terminals, the visual solution of the six degrees of freedom pose is the primary method, while other sensors are used for online calibration and redundancy verification. When only one optical pattern can be captured, the visible anchor point provides the absolute pose reference, while the invisible anchor point is recursively derived based on the latest calibrated inertial parameters and ultrawide spatial constraints, and combined with a personalized temporomandibular joint kinematic constraint model to limit the recursive divergence. When all optical images are blocked, the inertial measurement unit differential recursion, ultra-wideband angle-of-arrival-range fusion and magnetometer heading constraint work together and forcibly apply temporomandibular joint kinematic constraints to maintain pose estimation. When ultra-wideband signals are also severely interfered with, environmental point cloud map relocation is used as a fallback.

8. The system according to claim 1, characterized in that, The first mobile terminal and the second mobile terminal calibrate their relative attitude and position in real time through ultra-wideband ranging and resection to maintain the consistency of dual-view visual network measurements.

9. The system according to claim 1, characterized in that, The multi-source fusion module utilizes the neural engine built into the first mobile terminal, the second mobile terminal, or the smart wearable device to run a trained recurrent neural network model. This model learns the bias change pattern of habitual measurement units and the personalized temporal characteristics of the patient's jaw movements during the visual availability phase, and adaptively compensates and predicts inertial data during visual failure.

10. An oral surgery navigation method based on multi-anchor point collaboration and mandibular kinematic constraints, characterized in that, Includes the following steps: S1: A first smart wearable device is fixed in the patient's mandibular dentition, and a second smart wearable device is fixed in the maxilla or skull region. The first and second smart wearable devices respectively display different dynamic optical graphics. S2: Set the first mobile terminal as the observation end and aim it at the surgical area, and set the second mobile terminal to a stable position in the surgical environment. The rear cameras of the two mobile terminals are configured to capture the optical graphics displayed by the two smart wearable devices simultaneously or in turn. S3: Based on the pose of the two optical images acquired by the dual-view visual measurement network, the initial six-degree-of-freedom relative pose of the mandible relative to the maxilla is directly calculated. The coordinate system extrinsic parameters of each inertial measurement unit and the ultra-wideband module are simultaneously calibrated online. A kinematic constraint model containing the position of the personalized temporomandibular joint rotation axis is imported from the preoperative image. S4: During navigation, real-time time-synchronized acquisition of multi-source sensor data, including visual pose, all inertial measurement unit data, all magnetometer data, and all ultra-wideband ranging and / or angle of arrival data; S5: Determine the availability of visual information. When all or part of the vision is available, perform fusion output based on the relative pose calculated by vision and continuously update the calibration parameters. S6: When visual tracking fails, lock the last valid relative pose of the last frame before the failure, switch to inertial recursion mode, and integrate the personalized temporomandibular joint kinematic constraint model as a process model into the recursion. At the same time, integrate the differential motion data of multiple inertial measurement units, ultra-wideband spatial positioning constraints, multi-magnetic meter heading constraints and pre-built environmental point cloud map positioning information to recursively estimate the current mandibular relative pose. S7: When vision is restored, compare the deviation between the visual repositioning pose and the recursive pose, perform smoothing correction and update the inertial measurement unit bias estimate, and switch to a fusion mode that mainly solves the relative pose based on vision. S8: Throughout the entire surgical procedure, the preoperative 3D model is superimposed on the surgical image on the observation terminal in real time based on the relative pose, and the error between the surgical instruments and the planned target point is calculated and displayed.

Images