Real-time ar navigation system for dental surgery based on multimodal fusion and deep learning registration

Abstract

The present application relates to the technical field of computer-aided medical treatment, and discloses a dental surgery real-time AR navigation system based on multi-modal fusion and deep learning registration, comprising: a data acquisition and preprocessing module, which is used for acquiring CT data and double-channel real-time binocular video streams collected by a micro binocular camera module at the end of a surgical instrument, and performing stereo correction and space-time alignment; a double-flow feature coding network module, which is used for extracting three-dimensional anatomical geometric features and binocular vision texture features in parallel; a feature fusion and initial measurement module, which is used for fusing cross-modal features by using an attention mechanism and predicting an initial pose in combination with sensor prior information; a rendering verification and optimization module, which is used for constructing a binocular viewpoint differentiable rendering feedback loop; and an augmented reality navigation module, which is used for realizing accurate superposition of a virtual model and double-channel real-time video based on the optimized pose. Through binocular vision perception, cross-modal deep fusion and self-supervised closed-loop optimization, the present application effectively overcomes weak texture and occlusion interference in an oral environment.

Description

Technical Field

[0001] This invention relates to the field of computer-aided medical technology, specifically to a real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration. Background Technology

[0002] Dental implant and maxillofacial surgery demand extremely high precision in instrument positioning and angle. With the development of computer-aided surgical techniques, augmented reality (AR) navigation systems are increasingly being applied clinically. These systems aim to provide surgeons with intuitive anatomical visualization and path guidance by overlaying preoperatively planned three-dimensional anatomical models with real intraoperative images of the patient. Currently, mainstream dental navigation systems typically employ infrared optics-based positioning schemes. This requires wearing a tracking splint on the patient's teeth or implanting positioning screws in the jawbone, along with an external optical tracker to establish a spatial mapping between preoperative CT images and the patient's head during surgery.

[0003] Existing optical positioning solutions typically require the use of contact positioning devices, which limits the operating space within the oral cavity and increases the complexity of clinical procedures. Label-free registration techniques based on computer vision attempt to achieve positioning through image feature matching, but their application in the oral clinical environment faces challenges. The oral cavity environment is typically characterized by indistinct tooth surface textures, reflective properties, and the presence of body fluids and instrument occlusion during surgery. In these conditions, texture-based visual feature extraction algorithms are prone to insufficient or unevenly distributed effective feature points, affecting the stability and continuity of registration.

[0004] On the other hand, some deep learning-based registration methods use pre-trained model parameters for inference. Due to differences in individual patient anatomy and the variability of intraoperative imaging angles and lighting conditions, models relying solely on offline training may lack adaptability when handling complex scenarios for specific individuals. Furthermore, in surgical scenarios lacking external ground truth, if the system lacks an online calibration mechanism based on physical consistency or stereo geometric constraints, initial registration deviations or accumulated errors from long-term tracking are difficult to correct in a timely manner, thus affecting the final positioning accuracy of the navigation system. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration. This system solves the problems of existing dental navigation technologies, which mainly rely on invasive markers for positioning, have low visual registration accuracy in oral environments with weak textures and interference from blood and saliva, and lack an intraoperative adaptive online calibration mechanism, leading to drift errors during long-term tracking.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: a real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration, including a data acquisition and preprocessing module, a dual-stream feature coding network module, a feature fusion and initial testing module, a rendering verification and optimization module, and an augmented reality navigation module.

[0007] The data acquisition and preprocessing module acquires preoperative oral cone-beam computed tomography (CBCT) data from the patient via a medical image communication interface and performs isotropic resampling. It then uses a trilinear interpolation algorithm to reconstruct the two-dimensional slice sequence into an isotropic three-dimensional voxel matrix. Simultaneously, the module performs intensity normalization on the three-dimensional voxel matrix according to a preset Hounsfield unit threshold range, filtering out soft tissue and atmospheric background interference. The module also acquires dual-channel real-time binocular video streams from a miniature binocular camera module integrated into the surgical instrument during surgery. Using pre-stored intrinsic and extrinsic parameter matrices of the left and right cameras, distortion coefficient vectors, and extrinsic parameter matrices, it remaps the pixel coordinates of the original images using a distortion mapping table and epipolar correction algorithm to eliminate radial and tangential distortion and complete stereo correction. Finally, it synchronizes the dual-channel real-time binocular video streams with data from external positioning sensors based on the binocular reference view exposure time.

[0008] The dual-stream feature encoding network module comprises a parallel 3D spatial feature encoding unit and a dynamic visual feature encoding unit. The 3D spatial feature encoding unit employs a deep 3D convolutional neural network, extracting a 3D anatomical geometric feature tensor containing the spatial topology of the dental arch and the gradient of bone density distribution from a 3D voxel matrix through multiple cascaded 3D convolutional layers and 3D residual connectives. The dynamic visual feature encoding unit employs a temporal-based convolutional neural network, receiving a dual-channel binocular reference view sequence containing the current frame and adjacent historical frames. It extracts a binocular visual texture feature tensor containing texture, edge, dynamic change trends, and disparity depth information using 3D convolutional kernels or temporal shift operations, and enhances the rigid structure feature response through a channel attention mechanism.

[0009] The feature fusion and initial testing module constructs an attention fusion unit based on a query-key-value mechanism. It flattens the spatial dimension of the binocular visual texture feature tensor and performs a linear transformation to generate a query matrix. It also flattens the voxel spatial dimension of the 3D anatomical geometric feature tensor and performs a linear transformation to generate a key matrix and a value matrix. The module calculates the scaled dot product attention between the query matrix and the key matrix, generating an attention weight matrix representing the correlation between binocular video pixel positions and 3D voxel positions. This attention weight matrix is ​​then used to weight and aggregate the value matrix, generating a fused feature tensor that preserves the 2D spatial layout. Furthermore, the module uses a multilayer perceptron to map the homogeneous transformation matrix of the external positioning sensor into a geometric embedding vector. This geometric embedding vector is then concatenated with the vector of the fused feature tensor after global average pooling, and input into the regression prediction unit. The output is a surgical instrument initial pose matrix containing translation vectors and rotation quaternions.

[0010] The rendering verification and optimization module incorporates a differentiable renderer based on a ray-stepping algorithm. According to the binocular camera viewpoint defined by the initial pose matrix, it performs integral sampling of the 3D voxel data along the left and right viewing directions, extracting the rigid structure surface and projecting it onto the left and right 2D imaging planes to generate binocular virtual projection images. Simultaneously, it generates binarized mask images identifying high-density rigid structure regions. The module calculates the binocular stereo consistency loss or geometric feature difference between the binocular virtual projection image and the dual-channel real-time binocular video stream under the weighted binarized mask image. During the inference phase, the module freezes the neural network weight parameters, calculates the partial derivative of the consistency loss with respect to the pose parameters, and uses a gradient descent algorithm to iteratively fine-tune the initial pose matrix online until the loss converges or a preset number of iterations is reached, outputting the optimized final pose matrix.

[0011] Based on the optimized final pose matrix, the augmented reality navigation module performs rigid body space transformation on the pre-planned 3D virtual model. Using alpha channel blending technology, the transformed virtual model is precisely overlaid and displayed on dual-channel real-time binocular video streams. Simultaneously, the augmented reality navigation module calculates in real-time the Euclidean distance between the center point of the surgical instrument tip and the surface of key anatomical structures in 3D space, as well as the angular deviation between the instrument axis and the planned path. A visual alarm is triggered when the Euclidean distance is less than a preset safety threshold or the angular deviation exceeds the allowable range. Furthermore, the augmented reality navigation module continuously compares the spatial consistency of the optimized final pose matrix with the data from external positioning sensors. If the deviation exceeds the tolerance range, a tracking anomaly is identified, and an alarm is issued.

[0012] This invention provides a real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration. It has the following beneficial effects:

[0013] 1. This invention extracts 3D anatomical geometric features and binocular visual texture features in parallel through a dual-stream feature coding network module, and calculates a cross-modal weight matrix using a query-key-value attention mechanism in the feature fusion and initial testing modules, achieving depth alignment of pixel-level and voxel-level features. This design utilizes the rigid geometric constraints of CT data combined with the depth parallax information contained in binocular images to compensate for information loss in the intraoperative binocular video stream under weak texture, reflective, or partially occluded environments, thus improving the robustness of the registration algorithm in complex oral surgical scenarios.

[0014] 2. This invention constructs a closed-loop feedback mechanism based on physical consistency through a rendering verification and optimization module. It utilizes a differentiable renderer to generate virtual projections corresponding to the binocular viewpoints and calculates the stereo consistency loss between these projections and the dual-channel real-time binocular video streams. During the inference phase, the system freezes the network weights and uses a gradient descent algorithm to iteratively fine-tune the initial pose matrix online. This allows for adaptive correction of the initial estimation error using binocular stereo geometric constraints without the need for additional manual labeling, ensuring the tracking accuracy of surgical navigation.

[0015] 3. This invention solves the problem of traditional external optical positioning's line of sight being easily obstructed by medical personnel or instruments by integrating a miniature binocular camera module into the end of surgical instruments. Simultaneously, the augmented reality navigation module performs dual consistency checks on the visual algorithm and external sensors, and monitors the safe distance between surgical instruments and key anatomical structures in real time. Through a redundancy and cross-calibration mechanism of multimodal data, it reduces the surgical risks caused by single-system failures. Attached Figure Description

[0016] Figure 1 This is a system framework diagram of the present invention;

[0017] Figure 2 This is a detailed processing flowchart of the data acquisition and preprocessing module of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of the dual-stream feature coding network module of the present invention;

[0019] Figure 4 This is a logical architecture diagram of the feature fusion and preliminary testing module of the present invention;

[0020] Figure 5 This is a schematic diagram of the processing flow of the rendering verification and optimization module of the present invention;

[0021] Figure 6 This is a schematic diagram of the processing flow of the augmented reality navigation module of the present invention.

[0022] The module includes: 100, Data Acquisition and Preprocessing Module; 200, Dual-Stream Feature Coding Network Module; 300, Feature Fusion and Preliminary Testing Module; 400, Rendering Verification and Optimization Module; and 500, Augmented Reality Navigation Module. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See attached document Figure 1 , Figure 1 This is a schematic diagram of the overall architecture of a real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration, according to an embodiment of the present invention. The present invention provides a real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration, comprising:

[0025] The data acquisition and preprocessing module 100 is configured to acquire the patient's preoperative three-dimensional computed tomography scan data and the intraoperative dual-channel real-time binocular video stream acquired by the miniature binocular camera module integrated into the end of the surgical instrument; the data acquisition and preprocessing module 100 performs voxelization processing on the three-dimensional computed tomography scan data and performs distortion correction, stereo correction and time synchronization on the dual-channel real-time binocular video stream;

[0026] The dual-stream feature coding network module 200 is configured to extract anatomical geometric features from the three-dimensional computed tomography data using a three-dimensional convolutional neural network, and to extract binocular visual texture features from the dual-channel real-time binocular video streams in parallel using a two-dimensional convolutional neural network.

[0027] The feature fusion and initial measurement module 300 is configured to calculate the attention weight matrix between the binocular visual texture features and the three-dimensional anatomical geometric features, use the weight matrix to generate a fused feature vector, and combine the initial prior information of the external positioning sensor to regress and predict the initial pose matrix of the surgical instrument with the integrated miniature binocular camera module relative to the world coordinate system.

[0028] The rendering verification and optimization module 400 is configured to construct a differentiable rendering function based on the initial pose matrix, generate a virtual digital reconstructed image corresponding to the binocular viewpoint, calculate the binocular stereo consistency loss between the virtual digital reconstructed image and the dual-channel real-time binocular video stream, and iteratively optimize the pose matrix online using the gradient descent algorithm.

[0029] The augmented reality navigation module 500 is configured to project and transform the pre-planned surgical path and anatomical structure model onto the current binocular reference view coordinate system based on the optimized final pose matrix, and then overlay and display it on the dual-channel real-time binocular video stream.

[0030] To more clearly illustrate the technical solution of the present invention, the specific structure, working principle and implementation details of the above modules will be described in detail below with reference to the accompanying drawings.

[0031] See attached document Figure 2 , Figure 2 A detailed processing flow of a data acquisition and preprocessing module 100 according to an embodiment of the present invention is illustrated. The data acquisition and preprocessing module 100 is configured to format and spatiotemporally align multi-source heterogeneous data, and its processing specifically includes the following steps:

[0032] Step S110: Construct a three-dimensional voxel representation of the static anatomical data. The data acquisition and preprocessing module 100 acquires the patient's preoperative oral cone-beam computed tomography (CBCT) scan data through a medical imaging communication interface. Since the raw data is typically stored as a series of two-dimensional DICOM slices, and the slice thickness varies between different devices, the data acquisition and preprocessing module 100 performs isotropic resampling. Specifically, the data acquisition and preprocessing module 100 sets a uniform spatial resolution and uses a trilinear interpolation algorithm to reconstruct the two-dimensional slice sequence into a three-dimensional voxel matrix to support subsequent generation of a three-dimensional virtual model containing the dentition, bones, and key neural canals based on volume rendering technology. Furthermore, to eliminate intensity differences caused by different scanning devices and adapt to the input distribution of the neural network, the data acquisition and preprocessing module 100 normalizes the voxel intensity. The data acquisition and preprocessing module 100 first sets the threshold range of interest based on Hounsfield units. This threshold range covers the density range of tooth enamel, dentin, and alveolar bone, while filtering out soft tissue and air background. For any point in the voxel matrix... Its normalized intensity value Calculate using the following formula:

[0033]

[0034] in, The original voxel intensity value. and These are the preset lower and upper limits of the region of interest intensity threshold, respectively. Less than hour, Set to 0; when Greater than hour, Set to 1. The processed data is cropped into a fixed-size 3D tensor centered on the preoperatively planned implantation point coordinates, which serves as the input for subsequent 3D feature extraction. Reading and reconstructing DICOM data are standard techniques in medical image processing and will not be elaborated upon here.

[0035] Step S120: Acquire and calibrate dynamic binocular vision data. This step S120 relies on an improved surgical instrument hardware device, namely, a miniature binocular camera module integrated at the end of the surgical instrument. This module includes a pair of 2mm miniature lenses with a fixed baseline distance, used to simultaneously acquire left-eye binocular video streams and right-eye binocular video streams. The data acquisition and preprocessing module 100 reads the pre-stored intrinsic parameter matrices, distortion coefficient vectors, and extrinsic parameter matrices calibrated by the binocular system for each of the left and right cameras.

[0036] Specifically, for dual-camera acquisition applications, the data acquisition and preprocessing module 100 performs dual geometric correction: First, it uses a distortion mapping table to remap the pixel coordinates of the original images of the left and right eyes respectively, eliminating radial and tangential distortions caused by the wide-angle lens; Second, it performs stereo rectification processing, using a binocular extrinsic matrix to reproject the left and right image planes to a coplanar and row-aligned state, ensuring that the projection of the same point in space on the left and right images is on the same horizontal scan line.

[0037] Subsequently, the data acquisition and preprocessing module 100 stitches or stacks the corrected dual-channel images, adjusts the resolution to the size required for network input, and performs normalization processing to generate a standardized binocular input tensor containing parallax geometric constraints.

[0038] Step S130: Synchronize auxiliary positioning data. While acquiring the binocular video stream, the data acquisition and preprocessing module 100 reads the real-time pose data of the surgical instrument sent by the NDI optical positioning system at high frequency through the communication interface. This real-time pose data characterizes the spatial position of the surgical instrument's end effector relative to the optical tracker base. Since the visual acquisition frequency and the optical positioning frequency are usually inconsistent, the data acquisition and preprocessing module 100 performs a timestamp alignment operation. Using the exposure center time of the binocular reference view as a reference, the data acquisition and preprocessing module 100 calculates the interpolated pose of the binocular reference view's exposure time using a linear interpolation algorithm based on the positioning data of two adjacent moments. The aligned positioning data includes position coordinates. and attitude quaternions The data acquisition and preprocessing module 100 converts it into homogeneous transformation matrix This matrix will subsequently be used as prior information input into the neural network, rather than being used directly as the final navigation result.

[0039] By executing steps S110 to S130, the data acquisition and preprocessing module 100 converts raw source data with different physical dimensions, sampling frequencies and data structures into standardized tensor and matrix forms, thereby providing a spatiotemporally aligned data foundation for the subsequent deep feature fusion of the dual-stream feature coding network module 200.

[0040] See attached document Figure 3 , Figure 3 This is a schematic diagram of a dual-stream feature coding network module 200 according to an embodiment of the present invention. The present invention provides a dual-stream feature coding network module 200, which is configured to establish a feature extraction architecture based on a multimodal deep learning network. The dual-stream feature coding network module 200 processes three-dimensional voxel data and dual-channel real-time binocular video streams from the data acquisition and preprocessing module 100 in parallel, extracting high-dimensional anatomical geometric features and stereoscopic spatiotemporal visual features containing depth information through a three-dimensional spatial feature coding unit and a dynamic visual feature coding unit, respectively.

[0041] For processing preoperative static modal data, the 3D spatial feature encoding unit receives preprocessed 3D computed tomography voxel tensors. The 3D spatial feature encoding unit employs a deep 3D convolutional neural network (3D-CNN) as its backbone architecture, configured to extract multi-level, semantically rich 3D anatomical features from the CT volume data. Specifically, the 3D spatial feature encoding unit comprises multiple cascaded 3D convolutional layers, batch normalization layers, and nonlinear activation function layers. To avoid the vanishing gradient problem while increasing the number of network layers and to effectively capture the complex spatial topology of the dentition, skeleton, and key neural canals, the 3D spatial feature encoding unit introduces 3D residual connective blocks. For the input feature tensor... The output of the residual block Defined as:

[0042]

[0043] in, This represents the residual mapping function consisting of multiple stacked convolutional layers. For the first Layer weight parameters, This is an identity mapping. Through this skip connection mechanism, the network can learn subtle changes in the input data, thereby more effectively extracting the bone density distribution gradient of the alveolar bone, the spatial topology of the dentition, and the tubular geometric features of the inferior alveolar nerve canal. After multiple layers of downsampling convolution operations with a stride of 2, the 3D spatial feature encoding unit compresses the input high-resolution voxels into a feature channel number of... Spatial dimension is 3D feature tensor The three-dimensional feature tensor The global anatomical features inside the patient's oral cavity were encoded, providing rigid geometric constraints for subsequent cross-modal registration.

[0044] For processing intraoperative dynamic modal data, the Dynamic Visual Feature Encoding Unit (DVIU) receives dual real-time binocular video streams from the miniature binocular camera module. To fully utilize the geometric advantages of binocular vision, the DVIU is configured to employ a weight-shared Siamese CNN or a dual-channel stacked architecture, receiving left and right eye binocular reference view sequences containing the current time and adjacent historical time points. This unit not only extracts texture and edge information from the image plane, but more importantly, it extracts implicit disparity and depth features by analyzing the pixel correspondence between the left and right views.

[0045] Specifically, the Dynamic Visual Feature Encoding Unit (DVI) performs stereo matching and temporal aggregation during feature extraction, utilizing 3D convolutional kernels to simultaneously capture spatial stereo geometric features and dynamic trends over time. To address the complex environment within the oral cavity, the DVI embeds a stereo channel attention module within the convolutional layers. This module is configured to automatically learn the weight coefficients of the feature channels, focusing on enhancing the feature response to rigid objects with significant three-dimensional structures (such as teeth), and utilizing binocular consistency constraints to suppress noise interference caused by specular reflections and soft tissue occlusion from a monocular perspective. The DVI ultimately outputs a stereo spatiotemporal feature tensor containing the corresponding feature channels and planar dimensions. This feature tensor integrates visual texture, temporal dynamics, and local depth geometry information constructed from binocular parallax.

[0046] Through the parallel processing described above, the dual-stream feature coding network module 200 maps heterogeneous medical image data and optical video data into high-dimensional feature embedding spaces. The output of the three-dimensional spatial feature coding unit... With the output of the dynamic visual feature encoding unit Although they differ in spatial dimensions, they are mapped to a unified feature channel dimension. This provides a structured data foundation for the feature fusion and initial testing module 300 to perform pixel-disparity-voxel-level feature alignment based on an attention mechanism. For the specific layer settings, kernel size selection, and training parameter initialization of the convolutional neural network, those skilled in the art can perform conventional configurations based on actual GPU memory size and computing power, and will not be elaborated upon here.

[0047] See attached document Figure 4 , Figure 4This is a logical architecture diagram of a feature fusion and preliminary measurement module 300 according to an embodiment of the present invention. The feature fusion and preliminary measurement module 300 is configured as the core computing hub of the system, performing depth alignment of binocular visual texture features and three-dimensional anatomical geometric features, and combining auxiliary sensor data to regress and predict the spatial pose of surgical instruments. The feature fusion and preliminary measurement module 300 achieves pixel-level fusion and pose preliminary measurement of cross-modal information through the following specific steps:

[0048] Step S310: Construct cross-modal attention mapping and feature fusion. The feature fusion and initial testing module 300 receives the stereo spatiotemporal feature tensor and the three-dimensional feature tensor from the two-stream feature coding network module 200. To address the mismatch between the dimensions of binocular projection images and the three-dimensional voxel space, and to enable the dynamic projection and mapping of three-dimensional structural information from CT data onto the corresponding pixel regions of the binocular reference view, the feature fusion and initial measurement module 300 constructs an attention fusion unit based on a query-key-value mechanism.

[0049] Specifically, the feature fusion and initial measurement module 300 first performs a spatial dimension reshaping operation: reshaping the spatial dimension of the three-dimensional spatiotemporal feature tensor. Flatten the matrix and generate a query matrix through linear transformation. Simultaneously, the three-dimensional feature tensor voxel space dimension Flattening, and generating the bond matrix through linear transformation. Sum matrix Query matrix The key matrix represents each pixel in the binocular video stream and its implicit disparity depth features. Sum matrix It characterizes the geometric features of each voxel point in the three-dimensional anatomical space.

[0050] Based on this, the feature fusion and initial testing module 300 calculates the scaled dot product attention of the query matrix and the key matrix, generating a dimension of The attention weight matrix is ​​used. Each element in this weight matrix quantifies the feature correlation between a specific pixel location in the binocular reference view and a specific anatomical point in the CT voxel space. The module then uses this weight matrix to adjust the value matrix. Perform weighted aggregation to generate a fused feature tensor. The computational model for this process is defined by the following formula:

[0051]

[0052] in, The query matrix originates from the stereo video stream. and The key matrix and value matrix are derived from CT data. is the feature dimension of the key vector, used to scale the dot product result to prevent gradient vanishing. For normalized exponential functions, This represents the matrix transpose operation. Through the above operations, the feature fusion and initial measurement module 300 achieves pixel-level depth fusion, and the output is... It maintains the same two-dimensional spatial layout as the binocular reference view, but the feature vector of each pixel aggregates the geometrically corresponding three-dimensional anatomical features (such as alveolar bone boundaries and nerve canal locations) retrieved through an attention mechanism and uses binocular disparity information for geometric consistency weighting. This mechanism enables the network to automatically assign higher weights to anatomical landmarks (such as cusps) with high texture discriminativeness and significant stereo geometric features, while suppressing low-confidence features caused by soft tissue occlusion or light reflection.

[0053] Step S320: Embed prior information for auxiliary localization. The feature fusion and preliminary measurement module 300 receives the NDI homogeneous transformation matrix from the data acquisition and preprocessing module 100. NDI data, as coarse pose information, provides initial guidance for the network's global search space. The feature fusion and initial testing module 300 will... The transformation matrix is ​​flattened into a one-dimensional vector and input into a multilayer perceptron (MLP) for nonlinear feature mapping. This MLP is configured to map low-dimensional physical coordinate data into high-dimensional geometric embedding vectors. The feature dimensions of the geometric embedding vector are configured to match the fused feature tensor. The channel dimensions are matched to achieve semantic space alignment between physical sensor data and depth visual features.

[0054] Step S330: Regress and predict the initial pose matrix. The feature fusion and initial measurement module 300 processes the fused feature tensor generated in step S310. Perform global average pooling to extract the binocular vision-anatomy joint feature vector representing the overall state of the current frame, and then combine it with the geometric embedding vector generated in step S320. Channel concatenation is performed to form a multimodal aggregated feature vector. This multimodal aggregated feature vector is then input into the regression prediction unit. The regression prediction unit contains two parallel fully connected layer branches:

[0055] Translation regression branch and rotation regression branch. The translation regression branch outputs a three-dimensional translation vector. The rotational regression branch outputs a quaternion vector. The output quaternions are then subjected to L2 normalization to ensure the orthogonality of the rotation matrices. The feature fusion and preliminary measurement module 300 ultimately determines the final result based on the output... and Reconstruct the initial pose matrix of the jaws relative to the world coordinate system in the current binocular reference view. Initial pose matrix By integrating binocular visual texture details, stereo depth geometric constraints, anatomical geometric constraints, and prior position information from NDI sensors, robust estimation of instrument poses in real surgical scenarios is achieved.

[0056] See attached document Figure 5 , Figure 5 This is a schematic diagram of the processing flow of the rendering verification and optimization module 400 according to an embodiment of the present invention. The rendering verification and optimization module 400 calculates the stereoscopic photometric or geometrical differences between the binocular virtual projection and the dual-path real observations to achieve closed-loop accuracy verification without manual annotation. This ensures that the system achieves sub-millimeter-level tracking accuracy in surgical scenarios lacking true pose values. The steps include:

[0057] Step S410: Generate a binocular differentiable virtual projection. The rendering verification and optimization module 400 receives static CT voxel data and the intrinsic parameter matrices of the left and right cameras from the data acquisition and preprocessing module 100, and receives the initial pose matrix from the feature fusion and initial measurement module 300. The rendering verification and optimization module 400 has a built-in differentiable renderer based on a ray stepping algorithm. The renderer is configured to adjust according to the initial pose matrix. The defined binocular camera viewpoint (combined with the baseline distance calibrated by binocular extrinsic parameters) is used to perform volume rendering or isosurface extraction on the 3D CT voxel data. Specifically, the rendering verification and optimization module 400 performs integral sampling of the CT voxel density along the left and right viewing directions, extracts rigid structural surfaces using preset enamel and dentin density thresholds, and projects the 3D surface onto the left and right 2D imaging planes using the camera intrinsic parameter matrix to generate binocular virtual projection images. Simultaneously, the rendering verification and optimization module 400 generates a binarized mask image corresponding to the binocular viewpoint during the rendering process. The regions with a pixel value of 1 in the mask image correspond to high-density rigid structures in the CT data. Teeth, bones Areas with a pixel value of 0 correspond to low-density areas. Soft tissue, atmospheric background This enables automatic removal of soft tissue occlusion areas. Since the rendering process is fully differentiable—meaning the gradient of the projected image pixel values ​​relative to the pose parameters can be analytically calculated—this step supports backpropagation of the error gradient, providing a mathematical basis for subsequent pose iteration optimization.

[0058] Step S420: Calculate the consistency difference in binocular stereo vision. The rendering verification and optimization module 400 acquires the current frame dual-channel real surgical video images (left / right images) from the data acquisition and preprocessing module 100. To eliminate the modal difference between the voxel grayscale values ​​of CT data and the pixel grayscale values ​​of the optical camera, the rendering verification and optimization module uses 400 pairs of dual-channel real video images. and binocular virtual projection images Sobel edge detection or conversion to surface normal maps is performed separately to extract common geometric features. Subsequently, the rendering verification and optimization module 400 calculates the stereo consistency loss between the two methods. This loss function quantifies the degree of matching between the binocular projection and the binocular observation, and its calculation formula is as follows:

[0059]

[0060] in, Represents the pixel coordinates of the image plane. The valid pixel region of the image. For dual-channel real video images at pixel level The value at that location, The values ​​of the binocular virtual projection images at the same point. This represents the conversion of an image into structured features. Such as gradient magnitude Feature extraction function, The weights of the binary mask generated in step S410, Represents L1 norm distance Absolute error Using this formula, the system transforms the 3D pose estimation problem into a problem of maximizing the structural similarity of binocular stereo images, and utilizes binary mask weights. It effectively shields the matching accuracy from non-rigid areas such as gingival bleeding and instrument obstruction.

[0061] Step S430: Perform closed-loop pose optimization and verification. The rendering verification and optimization module 400 calculates the binocular stereo consistency loss based on step S420. Execute feedback adjustment. During the real-time inference phase of surgical navigation, the rendering verification and optimization module 400 freezes the weight parameters of the neural network and uses the gradient descent algorithm to adjust only the initial pose matrix of the input. The translation vectors and rotation quaternions in the module are iteratively updated. Specifically, the module calculates the total loss function. The partial derivatives of the pose parameters are taken with respect to the gradient descent direction, and the pose parameters are finely adjusted until the loss value converges or the preset number of iterations is reached, at which point the final optimized pose matrix is ​​output. Furthermore, if the optimized eventual consistency loss... If the value is still higher than the preset safety threshold, the rendering verification and optimization module 400 will generate a positioning anomaly signal, indicating a serious topological conflict between the NDI positioning data and the visual data. This closed-loop mechanism based on binocular rendering-stereo comparison-optimization constitutes the core feedback loop of this invention, enabling the system to self-correct initial measurement errors caused by lens distortion or illumination changes by utilizing the geometric constraints of CT priors and binocular parallax constraints.

[0062] See attached document Figure 6 , Figure 6 This is a schematic diagram of the processing flow of an augmented reality navigation module 500 according to an embodiment of the present invention. The augmented reality navigation module 500 is configured to apply the precise six-degree-of-freedom pose matrix calculated by the rendering verification and optimization module 400 to the visualization output end, and to perform spatially precise registration and overlay display of the preoperative planned three-dimensional virtual model and the intraoperative real-time dual-channel video stream through augmented reality technology, providing three-dimensional and intuitive navigation guidance for surgical operations.

[0063] Step S510: Perform binocular spatial transformation and rendering of the virtual model. The augmented reality navigation module 500 calls the 3D virtual model containing the dental arch, bones, and key neural canals, generated preoperatively using high-resolution CT data through volume rendering technology. Based on the final optimized pose matrix output by the rendering verification and optimization module 400, the augmented reality navigation module 500 performs a rigid body spatial transformation operation on the 3D virtual model. That is, using the rotation and translation components in the pose matrix, combined with the extrinsic baseline data of the binocular camera, the coordinates of each vertex in the 3D virtual model are transformed from the world coordinate system to the left and right camera coordinate systems of the current miniature binocular camera module. Subsequently, the augmented reality navigation module 500 uses a projection matrix consistent with the optical intrinsics of the miniature binocular camera integrated on the surgical instrument to project the transformed 3D model onto the left and right 2D image planes respectively. During the rendering process, in order to distinguish different anatomical structures and display internal details, the augmented reality navigation module 500 assigns preset color and transparency attributes to different tissues. For example, it sets the alveolar bone surface as a semi-transparent material and renders the inferior alveolar nerve canal and the preoperatively planned implant path as a bright opaque entity, thereby generating a pair of binocular virtual anatomical structure images that perfectly match the current binocular surgical perspective.

[0064] Step S520: Generate binocular augmented reality overlay images. The augmented reality navigation module 500 receives dual real-time video streams captured by a miniature binocular camera module integrated on the surgical instrument and uses them as a background layer. The augmented reality navigation module 500 uses the binocular virtual anatomical structure image generated in step S510 as a foreground layer and overlays it onto the left and right real-time video streams respectively using alpha channel blending technology. Based on the high-precision pose estimation achieved by the aforementioned module, the virtual neural tube and tooth root structures in the foreground can be accurately mapped to the corresponding anatomical positions in the background binocular video, achieving pixel-level spatial alignment of the two viewpoints. The augmented reality navigation module 500 outputs the synthesized binocular augmented reality image through a stereoscopic display device, enabling key anatomical structures hidden inside soft tissues or bones to be directly presented in a visual manner on the real dental arch video image, and providing the doctor with depth perception.

[0065] Step S530: Implement real-time navigation monitoring and redundancy safety verification. During the synchronization process of augmented reality display, the augmented reality navigation module 500 calculates in real-time the Euclidean distance between the center point of the surgical instrument tip and key anatomical structures (such as the surface of the nerve canal) in three-dimensional space, as well as the angular deviation between the instrument axis direction and the pre-planned path direction. When the calculated distance is less than a preset safety threshold, or the angular deviation exceeds the allowable range, the augmented reality navigation module 500 controls the navigation aid elements on the display interface to change color (e.g., from green to red) and triggers an alarm signal.

[0066] Furthermore, to ensure system reliability, the augmented reality navigation module 500 implements an NDI redundancy check mechanism. Although the system primarily relies on deep learning networks for high-precision tracking, the module continuously reads real-time pose data provided by the NDI optical positioning system. The augmented reality navigation module 500 calculates the spatial consistency between the pose matrix output by its real-time visual algorithm and the pose matrix output by the NDI system, specifically calculating the difference in magnitude of the translation vectors and the difference in angles of the rotation quaternions. If the deviation exceeds the system's set tolerance range, the augmented reality navigation module 500 determines that the current visual tracking may be severely obstructed or interfered with by ambient lighting, and then automatically switches the display mode or issues a system consistency anomaly alarm, prompting the operator to check. This utilizes the redundancy characteristics of multimodal data to ensure the safety of surgical navigation.

[0067] In a specific embodiment of the present invention, the process of using the above system for oral implant surgery navigation is as follows:

[0068] The data acquisition and preprocessing module 100 first performs preoperative preparation and real-time data access tasks. During the surgical planning phase, the module acquires cone-beam computed tomography (CBCT) data of the patient's oral cavity, performs isotropic resampling and normalization, and constructs a three-dimensional voxel matrix with uniform resolution. Based on this three-dimensional data, the module further segments the anatomical structures of teeth, alveolar bone, and inferior alveolar nerve canal, and records the implant placement path planned by the surgeon. Upon entering the surgical phase, the module simultaneously acquires left and right dual-channel real-time video streams of the oral cavity at a rate of 30 frames per second using a miniature binocular camera module (containing a pair of 2mm wide-angle lenses with a fixed baseline distance) integrated into the implant handpiece. Simultaneously, the module acquires optical marker data attached to the implant handpiece at high frequency through the NDI optical positioning system and outputs a homogeneous transformation matrix containing initial pose information.

[0069] Subsequently, the dual-stream feature coding network module 200 processes the multimodal data output by the data acquisition and preprocessing module 100 in parallel. For static computed tomography (CT) scan data, the 3D spatial feature coding unit in the dual-stream feature coding network module 200 uses a deep 3D convolutional neural network to extract a high-dimensional anatomical geometric feature tensor containing the spatial topology of the dental arch, bone density gradient, and neural canal orientation. For dynamic real-time dual-channel video streams, the dynamic visual feature coding unit in the dual-stream feature coding network module 200 uses a Siamese convolutional neural network or a dual-channel architecture to process the left and right image sequences containing the current frame and historical frames. During this process, the dual-stream feature coding network module 200 utilizes the stereo channel attention mechanism embedded within the network to automatically suppress noise signals generated by saliva reflection, surgical instrument occlusion, and lens distortion, and focuses on extracting high-frequency textures, edge features, and binocular parallax depth features of the tooth surface to generate a stereo spatiotemporal visual feature tensor.

[0070] Next, the feature fusion and initial measurement module 300 performs depth alignment and initial pose estimation of cross-modal data. The feature fusion and initial measurement module 300 maps stereo spatiotemporal visual features to query vectors, maps 3D anatomical geometric features to key and value vectors, and calculates the dot product similarity between the query vector and the key vector to generate an attention weight matrix. Using this weight matrix, the feature fusion and initial measurement module 300 dynamically projects and aggregates the 3D anatomical features to the corresponding pixel positions in the binocular reference view, achieving pixel-level feature fusion based on disparity geometric constraints. The feature fusion and initial measurement module 300 then concatenates the fused feature vector with the NDI coarse pose features encoded by a multilayer perceptron and inputs it to the regression prediction unit. The feature fusion and initial measurement module 300 directly outputs the initial six-DOF pose of the current moment's miniature binocular camera module center relative to the patient's dental coordinate system through a fully connected layer. This pose includes a 3D translation vector and a rotation quaternion.

[0071] To correct potential errors in the initial regression, the rendering verification and optimization module 400 initiates a self-supervised closed-loop feedback process. Utilizing its built-in differentiable renderer, the module 400 projects the rigid structure in the preoperative computed tomography data into two virtual anatomical structures based on the initial pose output by the feature fusion and preliminary measurement module 300 and the intrinsic and extrinsic parameter matrices of the binocular camera. This generates two corresponding virtual anatomical structure projection images (left and right). The module 400 calculates the difference in stereoscopic structural consistency between this binocular virtual image pair and the actual dual-channel surgical video in terms of gradient magnitude or surface normal vectors. If a difference exists, the module 400, while freezing the neural network weights, iteratively fine-tunes the input pose parameters using a gradient descent algorithm until the binocular virtual projection and the actual dual-channel observations are highly geometrically identical, thus obtaining the final optimized pose after binocular stereo geometric verification.

[0072] Finally, the Augmented Reality Navigation Module 500 applies the final optimized pose to the visualization output. Using this pose matrix, the Augmented Reality Navigation Module 500 transforms the pre-planned 3D virtual model (including the semi-transparent alveolar bone, the highlighted inferior alveolar nerve canal, and the preset implant path) to the current left and right camera coordinate systems, and precisely overlays it onto the real-time dual-channel video stream using Alpha blending technology. Through the stereoscopic image output by the Augmented Reality Navigation Module 500, the doctor can intuitively see the anatomical structures hidden beneath the soft tissue and the planned path. Simultaneously, the Augmented Reality Navigation Module 500 calculates the Euclidean distance between the instrument tip and the nerve canal, as well as the angular deviation from the planned path, in real time. When the distance is less than a safety threshold or the deviation is too large, the Augmented Reality Navigation Module 500 triggers a visual alarm. Furthermore, the Augmented Reality Navigation Module 500 continuously compares the pose calculated by the visual algorithm with the pose provided by the NDI system. If the deviation exceeds the tolerance range, the Augmented Reality Navigation Module 500 indicates a system anomaly, thus providing multiple safety redundancies while ensuring sub-millimeter-level tracking accuracy.

Claims

1. A real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration, characterized in that, include: The data acquisition and preprocessing module is configured to acquire the patient's preoperative three-dimensional computed tomography (CT) scan data and intraoperative dual-channel real-time binocular video streams acquired by a miniature binocular camera module integrated into the end of the surgical instrument; and to perform voxelization processing on the three-dimensional CT scan data and distortion correction, stereo correction and time synchronization on the dual-channel real-time binocular video streams. The dual-stream feature encoding network module, connected to the data acquisition and preprocessing module, is configured to extract anatomical geometric features from the three-dimensional computed tomography data using a three-dimensional convolutional neural network, and to extract binocular visual texture features from the dual-channel real-time binocular video stream in parallel using a two-dimensional convolutional neural network. The feature fusion and initial measurement module, connected to the dual-stream feature coding network module, is configured to calculate the attention weight matrix between two-dimensional visual texture features and three-dimensional anatomical geometric features, use the weight matrix to generate a fused feature vector, and combine it with the initial prior information from the external positioning sensor to regress and predict the initial pose matrix of the surgical instrument with the integrated miniature binocular camera module relative to the world coordinate system. The rendering verification and optimization module, connected to the feature fusion and initial test module, is configured to construct a differentiable rendering function based on the initial pose matrix, generate a virtual digital reconstructed image corresponding to the binocular viewpoint, and optimize the initial pose matrix online by calculating the binocular stereo consistency loss between the virtual digital reconstructed image and the dual-channel real-time binocular video stream using the gradient descent algorithm. The augmented reality navigation module, connected to the rendering verification and optimization module, is configured to project and transform the pre-planned surgical path and anatomical structure model onto the binocular reference view coordinate system based on the optimized final pose matrix, and then overlay and display it on the real-time binocular video stream.

2. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 1, characterized in that, The data acquisition and preprocessing module is configured to perform the following operations: set a uniform spatial resolution, reconstruct the two-dimensional slice sequence of the three-dimensional computed tomography data into an isotropic three-dimensional voxel matrix using a trilinear interpolation algorithm, and normalize the voxel intensity according to a preset Hounsfield unit threshold interval; read the pre-stored intrinsic parameter matrices, distortion coefficient vectors, and extrinsic parameter matrices of the left and right cameras of the miniature binocular camera module, and remap the original image pixel coordinates of the dual-channel real-time binocular video stream using a distortion mapping table and an epipolar correction algorithm to perform distortion correction and stereo correction operations.

3. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 1, characterized in that, The dual-stream feature coding network module includes parallel three-dimensional spatial feature coding units and dynamic visual feature coding units: The dynamic visual feature encoding unit employs a time-based convolutional neural network to receive a dual-path binocular reference view sequence containing the current frame and adjacent historical frames, and extracts a spatiotemporal feature tensor containing texture, edge, dynamic changes and disparity depth information. The dynamic visual feature encoding unit employs a time-series-based convolutional neural network to receive a binocular reference view sequence containing the current frame and adjacent historical frames, and extracts spatiotemporal feature tensors containing texture, edges, and dynamic changes.

4. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 3, characterized in that, The three-dimensional spatial feature encoding unit includes three-dimensional residual connection blocks, which capture subtle changes in anatomical structures through a skip connection mechanism; the dynamic visual feature encoding unit embeds a channel attention module in the convolutional layer, which is configured to automatically learn the weight coefficients of the feature channels, enhance the feature response to rigid structures and suppress noise interference caused by soft tissue occlusion and illumination changes.

5. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 1, characterized in that, The feature fusion and initial testing module is configured to construct an attention fusion unit based on a query-key-value mechanism: The spatial dimension of the binocular visual texture features is flattened and linearly transformed into a query matrix, and the voxel spatial dimension of the three-dimensional anatomical geometric features is flattened and linearly transformed into a key matrix and a value matrix. Calculate the scaled dot product attention between the query matrix and the key matrix to generate the attention weight matrix, and use the attention weight matrix to perform weighted aggregation on the value matrix to generate a fusion feature tensor that maintains the two-dimensional spatial layout.

6. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 5, characterized in that, The feature fusion and initial measurement module is also configured to receive the homogeneous transformation matrix from the external optical positioning system as the initial prior information, map the homogeneous transformation matrix into a geometric embedding vector using a multilayer perceptron, and concatenate the geometric embedding vector with the vector obtained by global average pooling of the fused feature tensor, and input the concatenation to the regression prediction unit to output the translation vector and rotation quaternion in the initial pose matrix.

7. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 1, characterized in that, The rendering verification and optimization module has a built-in differentiable renderer based on the ray stepping algorithm. The differentiable renderer is configured to perform integral sampling of the voxel density of the three-dimensional computed tomography data along the left and right line of sight according to the binocular camera viewpoint defined by the initial pose matrix, extract the rigid structure surface and project it onto the left and right two-dimensional imaging planes to generate a binocular virtual projection image, and at the same time generate a binary mask image. The binary mask image is used to identify high-density rigid structure areas and mask low-density areas.

8. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 7, characterized in that, The rendering verification and optimization module is configured to perform closed-loop pose optimization during the real-time inference phase: calculate the photometric consistency loss or geometric feature difference between the binocular virtual projection image and the dual-channel real-time binocular video stream under the weighting of the binarized mask image; under the premise of freezing the neural network weight parameters, calculate the partial derivative of the consistency loss with respect to the pose parameters, and use the gradient descent algorithm to iteratively fine-tune the input initial pose matrix until the consistency loss converges or reaches the preset number of iterations, and output the optimized pose matrix.

9. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 1, characterized in that, The augmented reality navigation module is configured to perform rigid body space transformation on the preoperatively generated 3D virtual model using the optimized final pose matrix, and to superimpose the transformed virtual model onto the dual-channel real-time binocular video stream using Alpha channel blending technology; at the same time, it calculates the Euclidean distance between the center point of the surgical instrument tip and the key anatomical structure and the angular deviation from the planned path in real time, and triggers a visual alarm when the Euclidean distance is less than the safety threshold or the angular deviation exceeds the allowable range.

10. The real-time AR navigation system for dental surgery based on multimodal fusion and deep learning registration according to claim 9, characterized in that, The augmented reality navigation module is also configured to perform redundancy security checks: Continuously read real-time pose data provided by external positioning sensors and calculate the spatial consistency between the optimized final pose matrix and the real-time pose data provided by external positioning sensors. If the deviation between the two exceeds the preset fault tolerance range, the visual tracking is judged to be abnormal and a system consistency abnormality alarm is issued.

Images