Registration system based on multi-modal data fusion and tooth key points and storage medium
By detecting key points of tooth anatomy using an improved YOLO network and STN module, and combining a multi-branch structure and spatiotemporal feature alignment module, the system solves the problems of increased surgical complexity due to sensors and markers and poor robustness of traditional registration algorithms. This results in a high-precision, low-cost dental surgical navigation system suitable for minimally invasive surgeries such as orthodontics and dental implants.
Patent Information
- Application Number
- CN202510980993.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-16
- Publication Date
- 2025-11-25
AI Technical Summary
Existing deep learning surgical navigation technologies in oral implant surgery increase surgical complexity and cost due to the use of sensors and markers. Furthermore, traditional registration algorithms are computationally complex and have poor robustness, making it difficult to meet the requirements for real-time operation and high precision during surgery.
A registration system based on multimodal data fusion and key tooth points is adopted. The improved YOLO network and STN module are used to detect key points of tooth anatomy. The registration is performed by combining multi-branch structure and spatiotemporal feature alignment module, which reduces the use of markers and sensors and improves robustness and accuracy.
It achieves low-cost, low-complexity, high-precision, and high-robust surgical navigation, reducing surgical complexity and cost, improving surgical safety, and is suitable for minimally invasive surgical scenarios such as orthodontics and dental implants that require high-precision positioning.
Smart Images
Figure CN121010633A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of surgical navigation system technology, specifically relating to a registration system and storage medium based on multimodal data fusion and key tooth points. Background Technology
[0002] In dental implant surgery, precise tooth positioning and instrument navigation are crucial for success. Traditional dental implant surgery relies on the dentist's experience and preoperative planning, lacking real-time navigation and feedback mechanisms. However, with the development of deep learning technology, real-time positioning and registration have become possible. Deep learning technology can improve the accuracy and safety of the surgery, facilitating dentists in designing surgical plans before surgery and enabling timely adjustments and real-time feedback during the procedure.
[0003] However, existing technologies for surgical navigation using deep learning are still not mature enough, and they mainly have the following problems:
[0004] First, deep learning algorithms require complex sensors and markers for image recognition, some of which need to be placed around the patient's mouth or on their teeth. This increases the complexity and cost of the surgery and also affects the surgical field of view. Therefore, minimizing the need for sensors and markers during surgical navigation is a key issue in improving the practicality of surgical navigation systems and reducing their adverse effects on the surgery.
[0005] On the other hand, surgical navigation systems typically involve registering preoperative and intraoperative data. Traditional registration algorithms are computationally complex, lack robustness, and struggle to meet the real-time and high-precision requirements of intraoperative procedures. Intraoperative bleeding, saliva, or instrument obstruction can easily lead to inaccurate registration. Therefore, developing new registration methods to improve robustness is a pressing issue that surgical navigation systems need to address. Summary of the Invention
[0006] To address the problems of existing technologies, this invention provides a registration system and storage medium based on multimodal data fusion and tooth key points.
[0007] A registration system based on multimodal data fusion and tooth key points includes:
[0008] The input module is configured to accept preoperative three-dimensional oral cavity data and real-time image data acquired during the operation.
[0009] The region detection module is configured to segment an oral cavity region image from the image data using a region detection model;
[0010] The key point detection module is configured to detect key anatomical points of teeth in the oral cavity region image using a key point detection model.
[0011] The registration module is configured to register the oral cavity three-dimensional data and image data using the key points of the anatomical structure as reference points.
[0012] The keypoint detection model uses an improved YOLO network algorithm, which is improved as follows:
[0013] Connect an STN module to the YOLO network front end;
[0014] A multi-branch fusion structure is set up in the YOLO network, which includes: a classification detection branch for locating the position and category of teeth, a key point prediction branch for outputting the location information of key points in the anatomical structure of teeth, and an uncertainty branch for outputting the probability distribution standard deviation σ of the location information of key points in the anatomical structure.
[0015] Preferably, the preoperative oral three-dimensional data is selected from one of the following data or data obtained by fusing and registering them: intraoral scan data, oral CBCT data.
[0016] Preferably, the algorithm of the region detection model is selected from the YOLO v8 network.
[0017] Preferably, the improved YOLO network is an improved YOLOv8 network, with the following structure:
[0018] The oral cavity region image is input into an STN module. The STN module is sequentially connected to a first convolutional layer, a second convolutional layer, and a first C2f module. The second convolutional layer is connected to a first C2f module, which is then connected to a third convolutional layer. The third convolutional layer is connected to the second C2f module, which is then connected to a fourth convolutional layer. The fourth convolutional layer is connected to the third C2f module, which is then connected to a fifth convolutional layer. The fifth convolutional layer is connected to the fourth C2f module, which is then connected to an SPPF module. The SPPF module is connected to a first upsampling layer, which is then connected to a first stitching layer. This first stitching layer is used to fuse the features of the first upsampling layer and the third C2f module. The first stitching layer is connected to... The system is connected to a fifth C2f module, with a second upsampling layer connected to its back end. The second upsampling layer is then connected to its back end by a second stitching layer, which fuses the outputs of the second upsampling layer and the second C2f module. A sixth C2f module is connected to its back end, with a sixth convolutional layer connected to its back end. A third stitching layer is connected to its back end, fusing the outputs of the sixth convolutional layer and the fifth C2f module. A seventh C2f module is connected to its back end, with a seventh convolutional layer connected to its back end. A fourth stitching layer is connected to its back end, fusing the outputs of the seventh convolutional layer and the SPPF module. Finally, an eighth C2f module is connected to its back end.
[0019] The sixth, seventh, and eighth C2f modules are each connected to an output layer at their back ends, which is used to output targets at different resolutions. The output content of the output layer includes the location information of key points of the anatomical structure, the location of a single tooth, the category of a single tooth, and the standard deviation σ of the probability distribution of the location information of key points of the anatomical structure.
[0020] Preferably, the improved YOLOv8 network parameters are set as follows:
[0021] The STN module includes a 3x3 convolutional layer, a max pooling layer, and a 2x3 affine transformation matrix output connected in sequence.
[0022] The parameter settings for the first convolutional layer include: kernel = 6x6, stride = 2, padding = 2, out_channels = 32, activation = SiLU;
[0023] The parameter settings for the second convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 64, BatchNorm + SiLU;
[0024] The parameter settings for the first C2f module include: Bottleneck×3, out_channels=64, shortcut=True, group=4;
[0025] The parameter settings for the third convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 128;
[0026] The parameter settings for the second C2f module include: Bottleneck × 6, out_channels = 128, attention = CBAM;
[0027] The parameter settings for the fourth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 256;
[0028] The parameter settings for the third C2f module include: Bottleneck × 6, out_channels = 256, attention = SE;
[0029] The parameter settings for the fifth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 512;
[0030] The parameter settings for the fourth C2f module include: Bottleneck×3, out_channels=512, DeformableConv=True;
[0031] The parameter settings for the SPPF module include: pool_size = 5, three parallel MaxPool → concat, out_channels = 512;
[0032] The parameter settings for the first upsampling layer include: scale_factor = 2, mode = 'nearest';
[0033] The parameter settings for the first splicing layer include: concat_dim = 1;
[0034] The parameter settings for the fifth C2f module include: Bottleneck × 3, out_channels = 256;
[0035] The parameter settings for the second upsampling layer include: scale_factor = 2, mode = 'bilinear';
[0036] The parameter settings for the second splicing layer include: concat_dim = 1;
[0037] The parameter settings for the sixth C2f module include: Bottleneck × 3, out_channels = 128;
[0038] The parameter settings for the sixth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 128;
[0039] The parameter settings for the seventh C2f module include: Bottleneck × 3, out_channels = 25;
[0040] The parameter settings for the seventh convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 256;
[0041] The parameter settings for the eighth C2f module include: Bottleneck × 3, out_channels = 512.
[0042] Preferably, the algorithm used for the registration is selected from the spatiotemporal feature alignment module.
[0043] Preferably, the image data is patient facial and oral images acquired using a multi-view camera array.
[0044] Preferred options also include:
[0045] The preoperative planning module is configured to perform preoperative planning based on preoperative three-dimensional oral data;
[0046] The surgical navigation module is configured to calculate the 6DoF pose of the surgical instruments in the preoperative model coordinate system by combining the markers set on the surgical instruments with the marker algorithm.
[0047] The visualization module is configured to visualize preoperative planning and surgical instruments in a coordinate system established based on preoperative oral 3D data.
[0048] Preferred options also include:
[0049] The postoperative verification module is configured to record the intraoperative registration log, reconstruct the surgical process, and generate an accuracy report for physician review.
[0050] The present invention also provides a computer-readable storage medium having stored thereon a computer program for implementing the above-described registration system based on multimodal data fusion and dental key points.
[0051] In this invention, the "key points of anatomical structure" refer to key points of the tooth's anatomical structure, and the definitions of these key points can be determined according to existing technologies. The selection of key points of anatomical structure can be determined according to existing technologies; for example, in embodiments of this invention, cusps and proximal contact points are used as key points of anatomical structure.
[0052] This invention constructs a deep learning-based surgical navigation (particularly dental surgical navigation) method and registration system. The aim is to register multimodal data (including preoperative three-dimensional oral data and intraoperative images) to visualize preoperative planning, the surgical site, and the real-time position of surgical instruments in the same coordinate system using augmented reality (AR) and other technologies. Based on the technical solution of this invention, the following beneficial technical effects can be achieved:
[0053] 1. This invention replaces physical markers with key points of the tooth's anatomical structure (such as cusps and proximal contact points), thereby reducing the use of markers or sensors, and in particular, reducing or eliminating markers placed around the patient's mouth or on the teeth. This reduces the complexity and cost of the surgery and minimizes the impact of markers on the surgical field of view.
[0054] 2. To address the issue of reducing the impact of markers on the performance of surgical navigation systems, this invention provides a novel registration algorithm. This algorithm utilizes an improved YOLOv8 algorithm to detect oral regions and key tooth points. Through two-stage inference, it enhances robust detection and computational complexity under dynamic occlusion.
[0055] 3. In the preferred embodiment, the present invention combines the new registration algorithm (and the new registration system) with the function of surgical navigation, which can achieve high-precision (navigation accuracy can reach the millimeter level, <1mm) and highly robust surgical navigation function.
[0056] In summary, this invention provides a low-cost, low-complexity, high-precision, and highly robust surgical navigation technology, which is particularly suitable for minimally invasive surgical scenarios such as orthodontics and dental implants that require high-precision positioning, surgical instrument navigation, and dynamic path correction. It can significantly improve surgical safety, reduce the risk of postoperative complications, and has a promising application prospect.
[0057] Obviously, based on the above description of the present invention, and according to common technical knowledge and conventional methods in the field, various other modifications, substitutions or alterations can be made without departing from the basic technical concept of the present invention.
[0058] The following detailed embodiments further illustrate the above-described content of the present invention. However, this should not be construed as limiting the scope of the present invention to the following examples. All technologies implemented based on the above-described content of the present invention fall within the scope of the present invention. Attached Figure Description
[0059] Figure 1 This is a schematic diagram of the key points of each anatomical structure in Example 1;
[0060] Figure 2 Example images of the patient's face and oral cavity acquired by the multi-view camera array in Example 1;
[0061] Figure 3 This is an example image of the image before transformation by the STN module in Example 1;
[0062] Figure 4 This is an example image of the image transformed by the STN module in Example 1;
[0063] Figure 5 This is a diagram of the improved YOLOv8 network framework in Example 1. Detailed Implementation
[0064] It should be noted that the algorithms for data acquisition, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structures and circuit connections not specifically described, can all be implemented using content already disclosed in the prior art.
[0065] Example 1: Registration System Based on Multimodal Data Fusion and Tooth Key Points
[0066] This embodiment provides a registration system for dental surgical navigation, which includes:
[0067] The input module is configured to accept preoperative three-dimensional oral cavity data and real-time image data acquired during the operation.
[0068] The region detection module is configured to segment an oral cavity region image from the image data using a region detection model;
[0069] The key point detection module is configured to detect key anatomical points of teeth in the oral cavity region image using a key point detection model.
[0070] The registration module is configured to register the oral cavity three-dimensional data and image data using the key points of the anatomical structure as reference points.
[0071] The preoperative planning module is configured to perform preoperative planning based on preoperative three-dimensional oral data. Taking implant placement as an example, the preoperative planning may include: setting the implant placement position, depth, and tilt angle, and generating an ideal path.
[0072] The surgical navigation module is configured to calculate the 6DoF pose of the surgical instruments in the preoperative model coordinate system by combining the markers set on the surgical instruments with the marker algorithm.
[0073] The visualization module is configured to visualize preoperative planning and surgical instruments in a coordinate system established based on preoperative oral 3D data;
[0074] The postoperative verification module is configured to record the intraoperative registration log, reconstruct the surgical process, and generate an accuracy report for physician review.
[0075] The preoperative three-dimensional oral data is selected from one of the following data or data obtained by fusing and registering them: intraoral scan data and oral CBCT data. As a preferred embodiment, the specific acquisition process of the preoperative three-dimensional oral data is as follows:
[0076] 1) An intraoral scanner acquires high-precision 3D data of the patient's teeth, gums, and bones, and extracts and calibrates key anatomical points of the teeth. In this embodiment, the key anatomical points are selected from the cusps of all teeth moving clockwise from the first premolar on the left side of the mandible to the first premolar on the right side of the mandible. The definitions of each key anatomical point on the model are as follows: Figure 1 As shown.
[0077] 2) Collect oral CBCT data, perform 3D modeling on the CBCT data, and convert the CBCT data into point cloud data.
[0078] 3) The two 3D models are registered using the ICP algorithm (Iterative Closest Point Algorithm) to align the global coordinate systems of the two datasets.
[0079] The intraoperative real-time image data consists of images of the patient's face and mouth acquired using a multi-view camera array (example image shown). Figure 2 (As shown).
[0080] The algorithm for the region detection model can be any existing image segmentation algorithm. As a preferred approach, this embodiment uses the YOLOv8 network for implementation. The input of the YOLOv8 network is the image data acquired in real time during the operation (1080p@60fps), and the output is an image of the oral cavity region (640*640).
[0081] The training data for the region detection model is collected as follows: a large amount of oral cavity region data from different camera angles is collected, and the selected areas of the oral cavity region are manually marked.
[0082] The keypoint detection model uses an improved YOLOv8 network as its algorithm, and the improvement is as follows:
[0083] 1) A learnable STN (Spatial Transformation Network) module was added to learn the transformation matrix between teeth and a standard position. This matrix transforms the original image into an aligned image for further recognition. The STN network and YOLOv8 are trained simultaneously. Affine transformations are used to align cases of excessively large mouth openings or angles to a standard position, and non-rigid deformation parameters of teeth (such as gingival compression deformation) are learned. The image before and after transformation by the STN module is shown below. Figure 3 , 4 As shown.
[0084] 2) Includes a multi-branch structure that enables joint optimization through multi-task learning:
[0085] Classification and detection branch: Outputs the minimum bounding rectangle (center point + width and height) of each tooth to accurately locate the position and category of each tooth;
[0086] Keypoint prediction branch: Outputs the location information of the anatomical keypoints of each tooth (X, Y, ...).
[0087] Z);
[0088] Uncertainty Branch: The uncertainty prediction branch for each tooth key point outputs the standard deviation σ of the probability distribution for each coordinate point.
[0089] Specifically, the network structure is as follows: Figure 5 As shown, its structure is as follows:
[0090] The oral cavity region image is input into an STN module. The STN module is sequentially connected to a first convolutional layer, a second convolutional layer, and a first C2f module. The second convolutional layer is connected to a first C2f module, which is then connected to a third convolutional layer. The third convolutional layer is connected to the second C2f module, which is then connected to a fourth convolutional layer. The fourth convolutional layer is connected to the third C2f module, which is then connected to a fifth convolutional layer. The fifth convolutional layer is connected to the fourth C2f module, which is then connected to an SPPF module. The SPPF module is connected to a first upsampling layer, which is then connected to a first stitching layer. This first stitching layer is used to fuse the features of the first upsampling layer and the third C2f module. The first stitching layer is connected to... The system is connected to a fifth C2f module, with a second upsampling layer connected to its back end. The second upsampling layer is then connected to its back end by a second stitching layer, which fuses the outputs of the second upsampling layer and the second C2f module. A sixth C2f module is connected to its back end, with a sixth convolutional layer connected to its back end. A third stitching layer is connected to its back end, fusing the outputs of the sixth convolutional layer and the fifth C2f module. A seventh C2f module is connected to its back end, with a seventh convolutional layer connected to its back end. A fourth stitching layer is connected to its back end, fusing the outputs of the seventh convolutional layer and the SPPF module. Finally, an eighth C2f module is connected to its back end.
[0091] The sixth, seventh, and eighth C2f modules are each connected to an output layer at their backends, used to output targets at different resolutions. The output layer outputs Pose, Detect, CLS, and Sigma. Pose represents the location information of key points in the anatomical structure (belonging to the key point prediction branch), Detect represents the location of a single tooth (belonging to the classification detection branch), CLS represents the category of a single tooth (belonging to the classification detection branch), and Sigma represents the standard deviation σ of the probability distribution of the output anatomical key point location information (belonging to the classification detection branch).
[0092] The improved YOLOv8 network parameters are set as follows:
[0093] The STN module includes a 3x3 convolutional layer, a max pooling layer, and a 2x3 affine transformation matrix output connected in sequence.
[0094] The parameter settings for the first convolutional layer include: kernel = 6x6, stride = 2, padding = 2, out_channels = 32, activation = SiLU;
[0095] The parameter settings for the second convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 64, BatchNorm + SiLU;
[0096] The parameter settings for the first C2f module include: Bottleneck×3, out_channels=64, shortcut=True, group=4;
[0097] The parameter settings for the third convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 128;
[0098] The parameter settings for the second C2f module include: Bottleneck × 6, out_channels = 128, attention = CBAM;
[0099] The parameter settings for the fourth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 256;
[0100] The parameter settings for the third C2f module include: Bottleneck × 6, out_channels = 256, attention = SE;
[0101] The parameter settings for the fifth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 512;
[0102] The parameter settings for the fourth C2f module include: Bottleneck×3, out_channels=512, DeformableConv=True;
[0103] The parameter settings for the SPPF module include: pool_size = 5, three parallel MaxPool → concat, out_channels = 512;
[0104] The parameter settings for the first upsampling layer include: scale_factor = 2, mode = 'nearest';
[0105] The parameter settings for the first splicing layer include: concat_dim = 1;
[0106] The parameter settings for the fifth C2f module include: Bottleneck × 3, out_channels = 256;
[0107] The parameter settings for the second upsampling layer include: scale_factor = 2, mode = 'bilinear';
[0108] The parameter settings for the second splicing layer include: concat_dim = 1;
[0109] The parameter settings for the sixth C2f module include: Bottleneck × 3, out_channels = 128;
[0110] The parameter settings for the sixth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 128;
[0111] The parameter settings for the seventh C2f module include: Bottleneck × 3, out_channels = 25;
[0112] The parameter settings for the seventh convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 256;
[0113] The parameter settings for the eighth C2f module include: Bottleneck × 3, out_channels = 512.
[0114] The training process of the key point detection model includes: using the 3D model rendering of the oral scanning model to generate a large amount of data with key points of anatomical structure for pre-training, and then using manually labeled real photo data for fine-tuning.
[0115] In the model training process of this application, the loss function is constructed by comprehensively considering several key aspects to ensure that the model can accurately detect tooth positions, estimate key points, and perform classification. The complete description of the loss function is as follows:
[0116] 1. Loss of dental examination frame (L) bbox ):
[0117] It consists of two parts: center point loss and width and height loss.
[0118] L bbox =L center +L wh
[0119] Center point loss (L) center ):
[0120]
[0121] Among them, c i To predict the coordinates of the center point, These are the coordinates of the true center point.
[0122] The center point loss uses smooth L1 loss, which is sensitive to small deviations and can effectively handle the difference between the predicted value and the true value.
[0123] Width and height loss (L) wh ):
[0124] L wh =1―IoU((w,h),(w * ,h * ))
[0125] Where w,h are the width and height of the predicted bounding box, w * ,h * This represents the actual frame width and height.
[0126] The width and height loss uses IoU loss to maintain scale invariance and ensure the matching degree between the width and height of the predicted box and the width and height of the ground truth box.
[0127] 2. Key Point Loss (L) kpt ):
[0128]
[0129] Among them, u j To predict coordinates, These are the coordinates of the truth value.
[0130] The keypoint loss function employs Laplace negative log-likelihood (LL Loss). Based on the Laplace distribution assumption, this loss is suitable for handling sudden errors that may occur in surgical scenarios. The model automatically learns the uncertainty σ; the larger the predicted value, the lower the confidence level of the point (e.g., σ increases when the keypoint is obscured by blood).
[0131] 3. Tooth classification loss (L) kpt ):
[0132]
[0133] in, For real labels, p k This represents the predicted probability.
[0134] The classification loss uses the cross-entropy loss function to measure the difference between the predicted probability and the true label.
[0135] 4. Loss weight allocation:
[0136] The total loss is obtained by weighted summation of the bounding box loss, keypoint loss, and classification loss:
[0137] Total loss = 0.5 * L bbox +1.0*L kpt +0.2*L kpt
[0138] As a preferred approach, dynamic weighting is used in the total loss to automatically reduce the weight of high-σ prediction points.
[0139] In the registration module of this embodiment, the algorithm used for registration is selected from the spatiotemporal feature alignment module. During the registration process, this embodiment utilizes a multi-camera collaborative system, a camera array system, and the spatiotemporal feature alignment module to obtain the position and orientation of the teeth relative to the cameras using the reprojection error method. Specifically, the spatiotemporal feature alignment module is used for cross-viewpoint and cross-temporal feature matching and tracking to ensure the consistency of multi-camera observation data. Subsequently, BA (Bundle Adjustment) optimization based on binocular SLAM is used to improve accuracy and robustness.
[0140] The visualization module utilizes existing augmented reality (AR) technology to overlay the expected drill path and safety boundaries (such as distance warnings from the nerve canal) onto the navigation interface during implantation surgery. When the drill approaches the nerve canal, the AR navigation interface triggers a red warning and limits the motor speed.
[0141] As can be seen from the above embodiments, the present invention realizes a low-cost, low-complexity, high-precision, and highly robust surgical navigation technology, which is particularly suitable for minimally invasive surgical scenarios such as orthodontics and dental implants that require high-precision positioning, surgical instrument navigation, and dynamic path correction. It can significantly improve surgical safety, reduce the risk of postoperative complications, and has a very good application prospect.
Claims
1. A registration system based on multimodal data fusion and tooth key points, characterized in that, include: The input module is configured to accept preoperative three-dimensional oral cavity data and real-time image data acquired during the operation. The region detection module is configured to segment an oral cavity region image from the image data using a region detection model; The key point detection module is configured to detect key anatomical points of teeth in the oral cavity region image using a key point detection model. The registration module is configured to register the oral cavity three-dimensional data and image data using the key points of the anatomical structure as reference points. The keypoint detection model uses an improved YOLO network algorithm, which is improved as follows: Connect an STN module to the YOLO network front end; A multi-branch fusion structure is set up in the YOLO network, which includes: a classification detection branch for locating the position and category of teeth, a key point prediction branch for outputting the location information of key points in the anatomical structure of teeth, and an uncertainty branch for outputting the probability distribution standard deviation σ of the location information of key points in the anatomical structure.
2. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that: The preoperative oral three-dimensional data are selected from one of the following data or data obtained by fusing and registering them: intraoral scan data, oral CBCT data.
3. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that: The algorithm for the region detection model is selected from the YOLO v8 network.
4. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that: The improved YOLO network is an improved YOLOv8 network, and its structure is as follows: The oral cavity region image is input into an STN module. The STN module is sequentially connected to a first convolutional layer, a second convolutional layer, and a first C2f module. The second convolutional layer is connected to a first C2f module, which is then connected to a third convolutional layer. The third convolutional layer is connected to the second C2f module, which is then connected to a fourth convolutional layer. The fourth convolutional layer is connected to the third C2f module, which is then connected to a fifth convolutional layer. The fifth convolutional layer is connected to the fourth C2f module, which is then connected to an SPPF module. The SPPF module is connected to a first upsampling layer, which is then connected to a first stitching layer. This first stitching layer is used to fuse the features of the first upsampling layer and the third C2f module. The first stitching layer is connected to... The system is connected to a fifth C2f module, with a second upsampling layer connected to its back end. The second upsampling layer is then connected to its back end by a second stitching layer, which fuses the outputs of the second upsampling layer and the second C2f module. A sixth C2f module is connected to its back end, with a sixth convolutional layer connected to its back end. A third stitching layer is connected to its back end, fusing the outputs of the sixth convolutional layer and the fifth C2f module. A seventh C2f module is connected to its back end, with a seventh convolutional layer connected to its back end. A fourth stitching layer is connected to its back end, fusing the outputs of the seventh convolutional layer and the SPPF module. Finally, an eighth C2f module is connected to its back end. The sixth, seventh, and eighth C2f modules are each connected to an output layer at their back ends, which is used to output targets at different resolutions. The output content of the output layer includes the location information of key points of the anatomical structure, the location of a single tooth, the category of a single tooth, and the standard deviation σ of the probability distribution of the location information of key points of the anatomical structure.
5. The registration system based on multimodal data fusion and tooth key points according to claim 4, characterized in that: The improved YOLOv8 network parameters are set as follows: The STN module includes a 3x3 convolutional layer, a max pooling layer, and a 2x3 affine transformation matrix output connected in sequence. The parameter settings for the first convolutional layer include: kernel = 6x6, stride = 2, padding = 2. out_channels=32,activation=SiLU; The parameters for the second convolutional layer are: kernel = 3x3, stride = 2, padding = 1. out_channels=64,BatchNorm+SiLU; The parameter settings for the first C2f module include: Bottleneck × 3, out_channels = 64. shortcut=True,group=4; The parameter settings for the third convolutional layer include: kernel = 3x3, stride = 2, padding = 1. out_channels = 128; The parameter settings for the second C2f module include: Bottleneck × 6, out_channels = 128, attention = CBAM; The parameter settings for the fourth convolutional layer include: kernel = 3x3, stride = 2, padding = 1. out_channels = 256; The parameter settings for the third C2f module include: Bottleneck × 6, out_channels = 256. attention = SE; The parameter settings for the fifth convolutional layer include: kernel = 3x3, stride = 2, padding = 1, out_channels = 512; The parameter settings for the fourth C2f module include: Bottleneck×3, out_channels=512, DeformableConv=True; The parameter settings for the SPPF module include: pool_size = 5, three parallel MaxPool → concat, out_channels = 512; The parameter settings for the first upsampling layer include: scale_factor = 2, mode = 'nearest'; The parameter settings for the first splicing layer include: concat_dim = 1; The parameter settings for the fifth C2f module include: Bottleneck × 3, out_channels = 256; The parameter settings for the second upsampling layer include: scale_factor = 2, mode = 'bilinear'; The parameter settings for the second splicing layer include: concat_dim = 1; The parameter settings for the sixth C2f module include: Bottleneck × 3, out_channels = 128; The parameter settings for the sixth convolutional layer include: kernel = 3x3, stride = 2, padding = 1. out_channels = 128; The parameter settings for the seventh C2f module include: Bottleneck × 3, out_channels = 25; The parameter settings for the seventh convolutional layer include: kernel = 3x3, stride = 2, padding = 1. out_channels = 256; The parameter settings for the eighth C2f module include: Bottleneck × 3, out_channels = 512.
6. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that: The algorithm used for the registration is selected from the spatiotemporal feature alignment module.
7. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that: The image data consists of images of the patient's face and mouth acquired using a multi-view camera array.
8. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that, Also includes: The preoperative planning module is configured to perform preoperative planning based on preoperative three-dimensional oral data; The surgical navigation module is configured to calculate the 6DoF pose of the surgical instruments in the preoperative model coordinate system by combining the markers set on the surgical instruments with the marker algorithm. The visualization module is configured to visualize preoperative planning and surgical instruments in a coordinate system established based on preoperative oral 3D data.
9. The registration system based on multimodal data fusion and tooth key points according to claim 1, characterized in that, Also includes: The postoperative verification module is configured to record the intraoperative registration log, reconstruct the surgical process, and generate an accuracy report for physician review.
10. A computer-readable storage medium, characterized in that, It stores a computer program for implementing the registration system based on multimodal data fusion and tooth key points as described in any one of claims 1-9.