A multi-modal fusion data-based oral curve three-dimensional automatic fitting method and system

Abstract

The application discloses a kind of oral cavity curve three-dimensional automatic fitting method and system based on multi-modal fusion data, it is related to orthodontic technical field, including: collecting the CBCT image of patient oral cavity and digitized dental model;The CBCT image and digitized dental model are registered and fused, and registration result is obtained;Respectively obtain the coordinate of mark point on CBCT image and digitized dental model, and obtain the mark point detection result of skull and tooth;According to the registration result and the mark point detection result of skull and tooth, three-dimensional curve is directly fitted in three-dimensional space.This application realizes the automatic point identification and accurate fitting measurement of maxillary compensation curve, and can also accurately monitor the change of maxillary compensation curve in treatment process, provides scientific basis for formulating individualized treatment plan, maintaining normal physiological curvature of dentition and function.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of orthodontics, more particularly to a method and system for automatically fitting a three-dimensional curve in the oral cavity based on multi-modal fusion data. BACKGROUND

[0002] In the field of orthodontics, the study of occlusal relationship has always been one of the core issues, and the Spee curve and its related compensation curve concept are crucial for understanding the arrangement of dentition, occlusal efficiency, and orthodontic treatment strategy. However, previous studies on the measurement of the upper compensation curve are almost in a blank state.

[0003] In orthodontic treatment, the coordination of the upper compensation curve and the lower Spee curve is crucial for ensuring the stability of the dental arch and the efficiency of mastication. Traditional measurement methods are limited by cephalometric films or plaster models, making it difficult to accurately measure complex oral and maxillofacial structures.

[0004] Therefore, how to provide a method and system for automatically fitting a three-dimensional curve in the oral cavity based on multi-modal fusion data, which can automatically identify and accurately measure the points of the upper compensation curve, and accurately monitor the changes in the upper compensation curve during treatment, is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a method and system for automatically fitting a three-dimensional curve in the oral cavity based on multi-modal fusion data, which can automatically identify and accurately measure the points of the upper compensation curve, and accurately monitor the changes in the upper compensation curve during treatment, providing a scientific basis for developing personalized treatment plans, maintaining normal physiological curvature of dentition, and functionality.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: a method for automatically fitting a three-dimensional curve in the oral cavity based on multi-modal fusion data, comprising: collecting CBCT images and digital dental models of a patient's oral cavity;

[0007] Registering and fusing the CBCT images and digital dental models to obtain a registration result;

[0008] Respectively obtaining the coordinates of the landmark points on the CBCT images and digital dental models to obtain the landmark point detection results of the skull and teeth;

[0009] According to the registration result and the landmark point detection results of the skull and teeth, a three-dimensional curve is directly fitted in a three-dimensional space.

[0010] Preferably, the registration and fusion of the CBCT images and digital dental models comprises:

[0011] Correct the head position of the CBCT image, segment the teeth of the CBCT image by using a U-Net model to obtain a tooth segmentation result;

[0012] Segment the dental crown of the digital dental arch model by using a graph neural network model to obtain a dental crown segmentation result;

[0013] Based on the tooth segmentation result and the dental crown segmentation result, perform registration fusion by using an iterative closest point and a FilterReg algorithm based on a probability model.

[0014] Preferably, the coordinates of the landmark points of the CBCT image are obtained, including:

[0015] The following landmark point coordinates of the skull are calibrated and exported from the CBCT image after correcting the head position: bilateral infraorbital points, right ear points, nasal root points, cranial base points, anterior nasal spine points, and posterior nasal spine points, and a reference plane is established.

[0016] Preferably, the coordinates of the landmark points on the digital dental arch model are obtained, including:

[0017] Based on PointDGCN, the automatic positioning of the tooth landmark points of the digital dental arch model is performed;

[0018] The dental crown landmark points of the maxillary dentition include 11C, 12C, 14B, 15B, 16MB, 16DB, 17MB, 17DB, 21C, 22C, 24B, 25B, 26MB, 26DB, 27MB, and 27DB; wherein C represents the incisal edge midpoint, B represents the buccal cusp point, MB represents the mesial buccal cusp point, and DB represents the distal buccal cusp point.

[0019] Preferably, according to the registration result and the landmark point detection result of the skull and teeth, a three-dimensional curve is directly fitted in a three-dimensional space, including:

[0020] The polynomial method is used to fit the maxillary posterior Spee curve, the maxillary anterior Spee curve, the maxillary right whole Spee curve, and the maxillary left whole Spee curve in a three-dimensional space, respectively, to obtain the final three-dimensional curve.

[0021] Preferably, the U-Net model is used to segment the teeth of the CBCT image, including:

[0022] The CBCT original body data is preprocessed;

[0023] A voxel-level segmentation network based on a 3D U-Net architecture is used, the input is a fixed-size voxel block, and the output is a corresponding voxel label probability map, which performs multi-class segmentation on each tooth and background tissue;

[0024] Based on the voxel-level segmentation result, the isosurface of each tooth voxel region is extracted by the Marching Cubes algorithm to generate a structured triangular mesh model, and the CBCT tooth segmentation result is obtained.

[0025] Preferably, the structure of the graph neural network model comprises: a feature encoding module, a spatial transformation module, a dynamic edge convolution module and a full dentition level attention pooling module connected in sequence.

[0026] The graph neural network model is used for tooth crown segmentation of the digital dental arch model, comprising:

[0027] Based on the mesh data in the digital dental arch model, first, a graph structure is constructed based on the K-nearest neighbor algorithm in the preprocessing stage to obtain initial face sheet features;

[0028] The feature encoding module is composed of a multi-layer perceptron and an instance normalization and LeakyReLU activation function combination, and the initial face sheet features are encoded by the feature encoding module;

[0029] The spatial transformation module learns the feature transformation matrix of each sample;

[0030] Local feature extraction is performed based on the dynamic edge convolution module;

[0031] The full dentition level attention pooling module is used to aggregate the global features of the upper and lower jaws of each sample to obtain the full dentition level semantic context, and feed it back to the node level;

[0032] Each face sheet node is classified into multiple categories, and the corresponding tooth category is output.

[0033] Preferably, the PointDGCN is used for automatic positioning of tooth landmark points of the digital dental arch model, comprising:

[0034] Based on the tooth crown segmentation result, a local subgraph is constructed for each tooth as a candidate region;

[0035] The PointDGCN network is used for landmark point prediction; the PointDGCN network comprises a feature encoding module, a spatial transformation module, a dynamic edge convolution module and a multi-head regression module connected in sequence.

[0036] The normalized positions of multiple landmark points are output for each node;

[0037] According to the response of each type of landmark point on all nodes, the specific position of the landmark point in the original dental arch model is determined by using a weighted average method, and the three-dimensional coordinate restoration of the landmark point is completed.

[0038] Preferably, a three-dimensional automatic fitting system for oral curves based on multi-modal fusion data comprises:

[0039] a clinical data collection module configured to collect a CBCT image and a digital dental model of a patient's oral cavity;

[0040] a registration fusion module configured to perform registration fusion on the CBCT image and the digital dental model to obtain a registration result;

[0041] a landmark point detection module configured to obtain landmark point coordinates on the CBCT image and the digital dental model, respectively, to obtain a landmark point detection result of a skull and teeth;

[0042] a fitting module configured to directly fit a three-dimensional curve in a three-dimensional space according to the registration result and the landmark point detection result of the skull and teeth.

[0043] According to the technical solution described above, compared with the prior art, the present disclosure provides a three-dimensional automatic fitting method and system for oral curve based on multi-modal fusion data. The present disclosure fuses multi-modal data of a cone beam CT (CBCT) image and a digital dental model, automatically identifies a marker point, and performs three-dimensional fitting, thereby achieving three-dimensional automatic fitting and accurate measurement and analysis of the maxillary compensation curve. This technical solution not only overcomes the limitations of traditional measurement methods, but also fully utilizes the advantages of modern three-dimensional measurement technology, greatly improving the accuracy and precision of measurement.

[0044] The present disclosure provides a brand-new quantitative measurement method for maxillary compensation curve, filling the gap in this field in previous research. The application of this method provides a more accurate and comprehensive evaluation tool for orthodontic treatment, which has great significance for promoting the development of orthodontic treatment technology and improving treatment effect. At the same time, the present disclosure can be directly applied in clinical practice to provide accurate maxillary compensation curve measurement data for doctors, thereby guiding the formulation and implementation of orthodontic treatment plans. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0046] Figure 1 A clinical data schematic diagram of an example sample provided for the embodiments of the present disclosure.

[0047] Figure 2 A multi-modal data fusion schematic diagram provided for the embodiments of the present disclosure.

[0048] Figure 3A right maxillary curve fitting schematic diagram of a three-dimensional space provided for an embodiment of the present application.

[0049] Figure 4 A left maxillary curve fitting schematic diagram of a three-dimensional space provided for an embodiment of the present application.

[0050] Figure 5 A reference plane schematic diagram in a CBCT image provided for an embodiment of the present application.

[0051] Figure 6 A right Spee curve related measurement item schematic diagram provided for an embodiment of the present application.

[0052] Figure 7 A curvature direction schematic diagram provided for an embodiment of the present application.

[0053] Figure 8 A three-dimensional automatic fitting method flowchart of an oral curve based on multi-modal fusion data provided for an embodiment of the present application.

[0054] Figure 9 A 3D U-Net architecture schematic diagram provided for an embodiment of the present application.

[0055] Figure 10 A self-defined graph neural network structure CellDGCN structure schematic diagram provided for an embodiment of the present application.

[0056] Figure 11 A PointDGCN structure schematic diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0058] The embodiments of the present application disclose a three-dimensional automatic fitting method of oral curves based on multi-modal fusion data, as shown in the following formula (I), which comprises the following steps: Figure 8

[0059] Collecting CBCT images and digital dental arch models of a patient's oral cavity;

[0060] Registering and fusing the CBCT images and the digital dental arch models to obtain a registration result;

[0061] ​The existing technology mainly depends on a single CBCT image or a two-dimensional lateral film for measurement, and there is a lack of precision and a limitation of two-dimensional representation, and the embodiment of the present application adopts multi-modal data fusion, fuses the CBCT image and the digital dental model, combines the respective advantages of the two, retains the three-dimensional information of the CBCT, and utilizes the high-precision characteristics of the digital dental model, and high-precision three-dimensional measurement of the maxillary compensation curve is realized.

[0062] The coordinates of the mark points on the CBCT image and the digital dental model are acquired respectively to obtain the mark point detection results of the skull and the teeth; after registration and unification of the coordinate system based on segmentation, the detection of the mark points of the skull and the teeth is performed;

[0063] According to the registration result and the mark point detection result of the skull and the teeth, a three-dimensional curve is directly fitted in a three-dimensional space.

[0064] Based on the traditional measurement method, manual operation is required, and the operation is complex and inefficient; meanwhile, the two-dimensional representation cannot comprehensively reflect the three-dimensional shape. The advanced image processing algorithm and the automatic point recognition technology are adopted in the embodiment of the present application, three-dimensional automatic fitting and analysis of the maxillary compensation curve are realized, the measurement process is greatly simplified, the efficiency is improved, and a more intuitive and comprehensive three-dimensional evaluation tool is provided.

[0065] Based on the existing technology, the treatment is often based on experience, and the individual differences of patients are not fully considered. The embodiment of the present application can more accurately judge the arrangement of the patient's dentition and the occlusion relationship by accurately measuring and analyzing the maxillary compensation curve, and provides strong support for doctors to develop personalized treatment plans, and promotes the individualization and precision development of orthodontic treatment.

[0066] The embodiment of the present application ensures its leading position and commercial value in the field of maxillary compensation curve measurement through multi-modal data fusion, three-dimensional automatic fitting and analysis, and individualization and precision treatment.

[0067] Specifically, the CBCT image and the digital dental model are registered and fused, including:

[0068] The head position of the CBCT image is corrected, the U-Net model is adopted to segment the teeth of the CBCT image, and the tooth segmentation result is obtained;

[0069] The graph neural network model is adopted to segment the dental crown of the digital dental model, and the dental crown segmentation result is obtained;

[0070] Based on the tooth segmentation result and the dental crown segmentation result, the iterative closest point and the FilterReg algorithm based on the probability model are adopted for registration and fusion.

[0071] The head position of the CBCT image is manually corrected using Dolphin Imaging software (version 11.8; Dolphin Imaging and Management Systems, Chatsworth, CA). The tooth segmentation of the CBCT is realized based on a U-Net model, the tooth crown segmentation of the corresponding dental model (obtained by oral scanning) is realized based on a graph neural network model, and then based on the tooth segmentation result and the tooth crown segmentation result, the high-precision registration of the CBCT and the digital dental model and the unification of the coordinate system are realized by using an Iterative Closest Point (ICP) algorithm and a FilterReg algorithm based on a probability model.

[0072] Specifically, the coordinates of the landmark points of the CBCT image are obtained, including:

[0073] The three-dimensional coordinates of the following skull landmark points of the CBCT image after correcting the head position are calibrated and exported: bilateral infraorbital points, right ear points, nasal root points, cranial base points, anterior nasal spine points, and posterior nasal spine points. A reference plane is established based on these landmark points, and the specific operation is as follows:

[0074] The CBCT image after correcting the head position is imported into 3D Slicer 5.0.3 (www.sclicer.org) software in DICOM format, as shown in Figure 5 The coordinates of the following landmark points are manually calibrated and exported: bilateral infraorbital points (OrR, OrL), right ear points (PoR), nasal root points (N), cranial base points (Ba), anterior nasal spine points (ANS), and posterior nasal spine points (PNS). The following reference planes are established:

[0075] The reference plane includes:

[0076] ① A horizontal plane H is established: the horizontal plane H is constructed by the bilateral orbital points (OrR, OrL) and the right ear point (PoR);

[0077] ② A sagittal plane S is established: passing through the N point and the Ba point and perpendicular to the horizontal plane;

[0078] ③ A coronal plane C is established: passing through the N point and perpendicular to the horizontal plane and the sagittal plane;

[0079] ④ A PP plane is established: passing through the ANS point and the PNS point and perpendicular to the sagittal plane.

[0080] Specifically, the U-Net model is used for tooth segmentation of the CBCT image, including:

[0081] The CBCT raw data is preprocessed;

[0082] A voxel-level segmentation network based on 3D U-Net architecture is adopted, the input is a fixed size voxel block, and the output is a corresponding voxel label probability map, and multi-class segmentation of each tooth and background tissue is performed.

[0083] Based on the voxel-level segmentation result, the isosurface of each tooth voxel region is extracted by the Marching Cubes algorithm to generate a structured triangular mesh model, and the CBCT tooth segmentation result is obtained.

[0084] In one specific embodiment of the present application, the CBCT tooth segmentation method based on 3D U-Net has the following specific steps:

[0085] 1.1 Data preprocessing

[0086] For the original CBCT volume data, first, standard preprocessing operations such as gray scale normalization and spatial resampling are performed to ensure the consistency of the input data in terms of scale and value range, thereby improving the stability and convergence efficiency of the segmentation model training.

[0087] 1.2 Model structure

[0088] A voxel-level segmentation network based on 3D U-Net architecture is adopted, as shown in Figure 9 The network as a whole adopts an encoder-decoder structure, which can fully capture multi-scale spatial features. The input is a fixed size voxel patch, and the network output is a corresponding voxel label probability map, realizing multi-class segmentation of each tooth and background tissue. To improve the feature expression ability and training stability of the model, residual connections are introduced in the network to alleviate the gradient vanishing problem of deep networks by passing information across layers, accelerate the convergence speed of the model, and enhance the preservation and transmission of detailed features, thereby improving the accuracy and robustness of segmentation.

[0089] 1.3 Loss function and training strategy

[0090] A joint optimization strategy of cross-entropy loss and Dice loss is adopted to balance the class distinction ability and spatial overlap accuracy. Specifically, cross-entropy loss helps to improve the overall classification accuracy and is suitable for handling a large number of classes with relatively balanced distribution; while Dice loss is more sensitive to uneven sample quantity, which can effectively improve the recognition ability of the model in tooth adhesion area and boundary fuzzy area. The model is trained using the Adam optimizer with an initial learning rate of 0.001, and combined with the cosine annealing learning rate adjustment strategy to gradually reduce the learning rate to stabilize the convergence process.

[0091] 1.4 Three-dimensional reconstruction

[0092] Based on the voxel-level segmentation result, the isosurface of each tooth voxel region is extracted by the Marching Cubes algorithm to generate a structured triangular mesh model. This method realizes the conversion from high-density voxel data to lightweight geometric representation, ensures that the CBCT tooth segmentation result and the dental model have good registration conditions on the basis of unified triangular mesh data format, and facilitates subsequent accurate registration and fusion processing.

[0093] Specifically, the structure of the graph neural network model comprises: a feature encoding module, a spatial transformation module, a dynamic edge convolution module and a full dental arch level attention pooling module connected in sequence.

[0094] The graph neural network model is used for crown segmentation of the digital dental model, comprising:

[0095] Based on the grid data in the digital dental model, first, in the preprocessing stage, a graph structure is constructed based on the K-Nearest Neighbor (KNN) algorithm to obtain initial face sheet features.

[0096] A multi-layer perceptron and an instance normalization, Leaky ReLU activation function combination are used to form a feature encoding module, and the initial face sheet features are encoded by the feature encoding module.

[0097] A feature transformation matrix of each sample is learned by the spatial transformation module.

[0098] Local feature extraction is performed based on the dynamic edge convolution module.

[0099] The full dental arch level attention pooling module is used to aggregate the global features of the upper and lower jaws of each sample to obtain the full dental arch level semantic context, and feed it back to the node level.

[0100] Each face sheet node is classified into multiple categories, and the corresponding tooth category is output.

[0101] In one embodiment of the present application, the CellDGCN-based dental model crown segmentation method comprises the following specific steps:

[0102] 2.1 Graph construction and input features

[0103] For the grid data in the dental model, first, in the preprocessing stage, a graph structure is constructed based on the K-Nearest Neighbor (KNN) algorithm. Each node in the graph corresponds to a face sheet (cell), and the edges connect the spatially adjacent face sheets to describe their geometric adjacency relationship. The input feature of each node is a 15-dimensional vector, including the face sheet center point coordinates (3-dimensional), the three vertex coordinates (a total of 9-dimensional), and the normal vector (3-dimensional), to fully capture the local geometry and spatial structure information.

[0104] 2.2 Network structure design

[0105] A custom graph neural network structure CellDGCN for dental crown segmentation task is constructed, as shown in the following figure, mainly including the following modules: Figure 10

[0106] Feature encoding module

[0107] The initial patch features are encoded by a Linear Sequential structure composed of a multi-layer perceptron (MLP) and an instance normalization (Instance Normalization) and a Leaky ReLU activation function, to enhance their expression ability.

[0108] Spatial transformation module (STNkd)

[0109] A spatial transformation network (STN) is introduced to learn the feature transformation matrix of each sample, to realize the spatial alignment of the input features, and to improve the robustness of the model to tooth deformation under different poses.

[0110] Dynamic edge convolution module (Edge Conv)

[0111] Based on Edge Conv, local feature extraction is realized to capture the geometric differences and spatial relationships between nodes. This module is stacked in multiple layers in the network and uses residual connections to improve the local modeling capability.

[0112] Full dentition level attention pooling module

[0113] The GlobalAttention module is used to aggregate the global features of the upper and lower jaws of each sample, to obtain the full dentition level semantic context, and to feed it back to the node level to enhance the node features through concatenation and improve the segmentation discrimination ability.

[0114] 2.3 Loss function and training strategy

[0115] Dice Loss is used as the loss function, which can directly optimize the overlap between the prediction and the true label, effectively alleviating the gradient vanishing problem of small class samples in training. In the training process, the model input is a feature vector based on the patch, and the output is the tooth class probability distribution corresponding to each patch. The Adam optimizer is used, with an initial learning rate of 0.001, and the learning rate is reduced using the cosine annealing strategy when the performance on the validation set does not improve.

[0116] 2.4 Output

[0117] The final model performs multi-class classification on each patch node and outputs the corresponding tooth class, realizing fine segmentation of the tooth region.​

[0118] Specifically, the coordinates of the landmark points on the digitized dental arch model are obtained, including:

[0119] Based on PointDGCN, the automatic positioning of the tooth landmark points of the digitized dental arch model is performed;

[0120] The crown landmark points of the maxillary dentition include 11C, 12C, 14B, 15B, 16MB, 16DB, 17MB, 17DB, 21C, 22C, 24B, 25B, 26MB, 26DB, 27MB, 27DB; wherein C represents the incisal edge midpoint, B represents the buccal cusp point, MB represents the mesial buccal cusp point, and DB represents the distal buccal cusp point.

[0121] Based on the existing graph neural network model (GNNs), the segmentation of the teeth and the automatic positioning of the landmark points can be realized.

[0122] In one specific embodiment of the present application, CBCT is registered with the dental arch model, and the specific steps are as follows:

[0123] 3.1 Coarse registration (ICP)

[0124] The centroid coordinates of each tooth in the CBCT and the dental arch model and the coarse outline are extracted, the initial positions are aligned, and the classic iterative closest point (ICP) algorithm is used for preliminary rigid body registration.

[0125] 3.2 Fine registration (FilterReg)

[0126] Based on the coarse registration result, the FilterReg algorithm based on the probabilistic graph model is used for non-rigid fine registration, the joint density matching between the two sets of point clouds is maximized, and the boundary and surface fitting accuracy is improved.

[0127] 3.3 Coordinate system unification

[0128] The CBCT and the dental arch model are mapped to the same three-dimensional coordinate system, providing a basis for subsequent anatomical measurement, model fusion and landmark point prediction.

[0129] Specifically, based on PointDGCN, the automatic positioning of the tooth landmark points of the digitized dental arch model is performed, including:

[0130] Based on the crown segmentation result, a local subgraph is constructed for each tooth as a candidate region;

[0131] PointDGCN network is used for landmark point prediction; the PointDGCN network includes a feature encoding module, a spatial transformation module, a dynamic edge convolution module and a multi-head regression module connected in turn;

[0132] predicting the normalized positions of multiple landmarks for each node;

[0133] According to the response of each type of landmark on all nodes, the specific position of the landmark in the original dental model is determined by using a weighted average method, and the three-dimensional coordinate restoration of the landmark is completed.

[0134] In one specific embodiment of the present application, the PointDGCN-based dental landmark positioning method for dental models is as shown in Figure 11 The specific steps are as follows:

[0135] 4.1 Candidate region extraction

[0136] After completing the crown segmentation, based on the prediction results of CellDGCN, a local subgraph is constructed for each tooth as a candidate region for subsequent landmark positioning. Specifically, the face sheet set corresponding to each tooth is extracted from the segmentation results, and a subgraph is constructed according to its spatial adjacency relationship to constrain the landmark search space, reduce non-target region interference, and thus improve the positioning accuracy and stability.

[0137] 4.2 Network structure design

[0138] The landmark positioning task uses a custom graph neural network architecture PointDGCN designed specifically for regression form landmark prediction, which mainly includes the following modules:

[0139] Feature encoding module

[0140] The input features are the same as CellDGCN, which are 15-dimensional vectors for each face sheet node, including the center point coordinates (3-dimensional), the three vertex coordinates (9-dimensional in total), and the face sheet normal vector (3-dimensional). The feature encoding process is realized by a Linear Sequential module, which combines linear mapping, instance normalization (Instance Normalization) and LeakyReLU activation function, effectively improving the expression ability of the initial geometric features.

[0141] Spatial transformation module (STNkd)

[0142] The STNkd module is introduced to enhance the robustness of the model to geometric transformations such as rotation and translation. This module can learn the transformation matrix in the input feature space, align the features, and reduce the sensitivity of the model to input poses.

[0143] Dynamic edge convolution module (EdgeConv)

[0144] Dynamic edge convolution layers are used to extract the structural relationship between the facets. Each layer of EdgeConv dynamically constructs the edge feature representation, capturing the complex geometric dependence between nodes, and enhances the local feature representation through graph convolution.

[0145] Multi Head Regressor

[0146] Based on the extracted node features, a multi-channel coordinate regression is performed using a multi-layer perceptron (MLP) structure. Each channel corresponds to a type of landmark point. This structure can regress multiple landmark point coordinates in parallel, completing the positioning task of multiple types of landmark points.

[0147] 4.3 Loss function and training strategy

[0148] The mean squared error loss function (MSE Loss) is used as the optimization objective to regress and predict the three-dimensional coordinates of each type of landmark point in the normalized space. During training, the model receives the facet feature vectors of the local subgraph of each tooth and outputs the normalized position coordinates of multiple landmark points. The Adam optimizer is used with an initial learning rate of 0.0005, and the cosine annealing strategy is used to reduce the learning rate when the performance on the validation set does not improve.

[0149] 4.4 Output

[0150] The model finally outputs the normalized position prediction of multiple landmark points for each node. According to the response of each type of landmark point on all nodes, a weighted average (Weighted Average) method is used to determine its specific position in the original dental model, completing the restoration of the three-dimensional coordinates of the landmark points.

[0151] Specifically, according to the registration result and the landmark point detection result of the skull and teeth, a three-dimensional curve is directly fitted in three-dimensional space, including:

[0152] A polynomial method is used to fit the upper posterior Spee curve, the upper anterior Spee curve, the upper right whole Spee curve, and the upper left whole Spee curve in three-dimensional space, respectively, to obtain the final three-dimensional curve.

[0153] Specifically, the Spee curve of the posterior segment of the maxilla is directly fitted in three-dimensional space: the curve T_R_PB is fitted by points 15B, 16MB, 16DB, 17MB, 17DB using the polynomial method; the Spee curve of the anterior segment of the maxilla is directly fitted in three-dimensional space: the curve T_R_AB is fitted by points 11C, 12C, 14B, 15B using the polynomial method; the Spee curve of the right side of the maxilla is directly fitted in three-dimensional space: the curve T_R_SpB is fitted by points 11C, 12C, 14B, 15B, 16MB, 16DB, 17MB, 17DB using the polynomial method. The direct fitting of the Spee curve of the left side of the maxilla in three-dimensional space is analogous to the right side, and the posterior segment obtains the curve T_L_PB, the anterior segment obtains the curve T_L_AB, and the whole segment obtains the curve T_L_SpB.

[0154] Specifically, the output is measured:

[0155] The right side Spee curve related measurement items (as shown in Figure 6

[0156] (1) The curvature of the curve T_R_PB T_R_CPB (including the maximum curvature T_R_CPB and the average curvature T_R_ACPB), the curvature of the curve T_R_AB T_R_CAB (including the maximum curvature T_R_CAB and the average curvature T_R_ACAB), and the curvature of the curve T_R_SpB T_R_CSpB (including the maximum curvature T_R_CSpB, the average curvature T_R_ACSpB, and the curvature at the distobuccal cusp of the first molar T_R_6CSpB).

[0157] (2) The vertical distances of points 11C, 15B, 17DB to the PP plane 11-PP, 15B-PP, 17DB-PP, and the ratios of the vertical distances 11-PP / 15B-PP, 17DB-PP / 15B-PP are calculated.

[0158] Left side Spee curve related measurement items:

[0159] (1) The curvature of the curve T_L_PB T_L_CPB (including the maximum curvature T_L_CPB and the average curvature T_L_ACPB), the curvature of the curve T_L_AB T_L_CAB (including the maximum curvature T_L_CAB and the average curvature T_L_ACAB), and the curvature of the curve T_L_SpB T_L_CSpB (including the maximum curvature T_L_CSpB, the average curvature T_L_ACSpB, and the curvature at the distobuccal cusp of the first molar T_L_6CSpB).

[0160] ​(2) The vertical distances 21-PP, 25B-PP, 27DB-PP of the points 21C, 25B, 27DB to the PP plane are calculated, and the ratios of the vertical distances 21-PP / 25B-PP, 27DB-PP / 25B-PP are calculated.

[0161] Wherein, the direction of curvature is defined as follows (as shown in Figure 7

[0162] The yellow line is the PP plane, the purple line is the posterior Spee curve, the orange line is the anterior Spee curve, and the blue line is the whole Spee curve. ① The posterior extension line of the posterior Spee curve intersects the PP plane, and the intersection point is located on the distal side of the second molar. When the curvature of the curve is positive; no intersection or intersection point on the mesial side of the second molar, the curvature of the curve is negative. ② The anterior extension line of the anterior Spee curve intersects the PP plane, and the intersection point is located on the anterior part of the central incisor. When the curvature of the curve is positive; no intersection or intersection point on the posterior part of the central incisor, the curvature of the curve is negative. ③ When the anterior extension line and the posterior extension line of the whole Spee curve both intersect the PP plane, the curvature of the curve is positive; no intersection or one intersection, the curvature of the curve is negative.

[0163] In one specific embodiment of the present application, a three-dimensional automatic fitting system for oral curves based on multi-modal fusion data comprises:

[0164] A clinical data collection module is configured to collect CBCT images and digital dental models of a patient's oral cavity.

[0165] A registration and fusion module is configured to register and fuse the CBCT images and the digital dental models to obtain a registration result.

[0166] A landmark point detection module is configured to obtain landmark point coordinates on the CBCT images and the digital dental models respectively to obtain a landmark point detection result of the skull and the teeth.

[0167] A fitting module is configured to directly fit a three-dimensional curve in a three-dimensional space according to the registration result and the landmark point detection result of the skull and the teeth.

[0168] The embodiments of the present application have the following beneficial effects:

[0169] Technical level

[0170] ① High-precision measurement: Fusion of multi-modal data: By fusing CBCT images and digital dental models, high-precision measurement of the maxillary compensation curve is achieved. The high-precision characteristics (usually up to microns) of digital oral scanning technology make up for the deficiency (about 0.3mm) of CBCT in the accuracy of the dental model.

[0171] ​Improving measurement accuracy: This multi-modal data fusion approach can more accurately capture the three-dimensional morphology and subtle changes of the maxillary compensation curve, reducing errors and uncertainties in traditional measurement methods.

[0172] ②Three-dimensional automatic fitting and analysis: Automated process: Advanced image processing algorithms and automatic point recognition technology are used to achieve three-dimensional automatic fitting of the maxillary compensation curve. This automated process reduces human intervention and improves measurement and analysis efficiency. Three-dimensional visualization: The three-dimensional visualization interface allows doctors to more intuitively observe the morphology and positional relationship of the maxillary compensation curve, helping to more accurately assess the patient's oral condition.

[0173] Advantages of clinical application

[0174] ①Development of personalized treatment plans: Precise assessment: By accurately measuring and analyzing the maxillary compensation curve, doctors can more accurately determine the patient's tooth arrangement and occlusion relationship problems, thereby developing personalized treatment plans for patients. Improve treatment effect: Personalized treatment plans can better meet the needs of patients, improve treatment effectiveness and patient satisfaction.

[0175] ②Improve doctors' work efficiency and decision-making ability: Efficient measurement: Automated measurement and analysis processes reduce the workload of doctors and improve work efficiency. Assist decision-making: High-precision, three-dimensional measurement data provide more comprehensive and accurate reference information for doctors, helping them make more accurate clinical decisions.

[0176] Contribution to the field of orthodontics

[0177] ①Promote the development of orthodontic technology: A new method for measuring the maxillary compensation curve is proposed, filling a gap in the field. Promote technological progress: This innovative measurement method is expected to promote research and development in the field of orthodontics, leading to the continuous emergence of new technologies and methods.

[0178] ②Improve the overall level of orthodontic treatment: Standardized measurement: With the popularization and application of the invention, a unified and standardized measurement method and standard for the maxillary compensation curve can be established, improving the overall level and reliability of orthodontic treatment. Improve patient experience: By improving treatment effectiveness and patient satisfaction, the invention helps improve patients' oral health status and quality of life.

[0179] In summary, the invention has significant advantages and contributions in terms of technology, clinical application, and development in the field of orthodontics. It not only improves the measurement accuracy and precision of the maxillary compensation curve, but also promotes the development of personalized and precise orthodontic treatment, making important contributions to the progress of orthodontic practice.

[0180] In one embodiment of the present application, as shown in Figure 1 Fig. 3a, a is the CBCT image; b is the digital dental model; the example sample is a 16-year-old Angle II high-angle adolescent patient, and the clinical data including CBCT image and digital dental model are collected.

[0181] The head position of the CBCT image was manually corrected using Dolphin Imaging software (version 11.8; Dolphin Imaging and Management Systems, Chatsworth, CA). Based on the U-Net model, tooth segmentation of CBCT was realized, and based on the graph neural network model, tooth crown segmentation of the corresponding dental model was realized. Then, based on the tooth segmentation result and the tooth crown segmentation result, the Iterative Closest Point (ICP) algorithm and the FilterReg algorithm based on the probability model were used to realize high-precision registration of CBCT and the dental model and unify the coordinate system, as shown in Figure 2 Fig. 3b, where a is the digital dental model; b is the tooth crown segmentation of the dental model; c is the tooth segmentation of the CBCT image; and d is the high-precision registration of CBCT and the dental model. Through calculation, the average registration error is 0.23±0.03 mm, which meets the accuracy requirement.

[0182] The CBCT image after correction of the head position was imported into 3D Slicer 5.0.3 (www.sclicer.org) software in DICOM format, and the coordinates of the following landmark points were manually calibrated and exported: bilateral infraorbital points (OrR, OrL), right ear point (PoR), nasal root point (N), cranial base point (Ba), anterior nasal spine point (ANS), and posterior nasal spine point (PNS), and a reference plane was established.

[0183] Based on the existing graph neural network model (GNNs), tooth segmentation and automatic positioning of landmark points can be realized. Through verification, the distance error of the landmark points obtained by automatic identification and calibration using the existing model (0.40±0.25 mm) and the distance error between three orthodontic physicians (0.44±0.44 mm) had no statistical difference (t=-1.86, P=0.06), so the algorithm was used to automatically identify and calibrate the crown landmark points, including 11C, 12C, 14B, 15B, 16MB, 16DB, 17MB, 17DB, 21C, 22C, 24B, 25B, 26MB, 26DB, 27MB, and 27DB. The marker point coordinates were imported into the automatic fitting program for curve fitting, Figure 3 Fig. 4a is a schematic diagram of right maxillary curve fitting in three-dimensional space, Figure 4The left maxillary curve fitting diagram in three-dimensional space is shown, and the measurement items (as shown in Table 1) are output.

[0184] Table 1 Output item summary table

[0185]

[0186] The embodiment of the present application aims to provide a three-dimensional automatic fitting method and system for oral curve based on multi-modal fusion data, an automatic point recognition method and a maxillary compensation curve automatic fitting measurement method based on multi-modal fusion data (CBCT + digital dental model). The purposes of the embodiment of the present application specifically include the following aspects:

[0187] ① Improve the measurement accuracy: by fusing the multi-modal data of CBCT image and digital dental model, the high precision characteristics (micron level) of digital oral scanning technology are used to make up for the deficiency of CBCT in tooth model precision, so as to realize the accurate quantitative measurement of the maxillary compensation curve.

[0188] ② Realize three-dimensional automatic fitting: by using advanced image processing algorithm and automatic point recognition technology, the maxillary compensation curve can be automatically fitted in three-dimensional space, which fully reflects its three-dimensional shape and positional relationship. This provides more accurate and comprehensive evaluation tools for orthodontists, which helps to develop more accurate treatment plans.

[0189] ③ Establish a unified standard: establish a unified and standardized measurement method and standard for maxillary compensation curve. It is helpful for comparison and verification between different research institutions and doctors, and promotes the popularization and application of maxillary compensation curve measurement technology.

[0190] ④ Promote the precision of orthodontic treatment: promote the transformation of orthodontic treatment from traditional experience-based treatment to precision and individualized treatment. Through accurate measurement and analysis of the maxillary compensation curve, doctors can more accurately judge the arrangement of the patient's dentition and occlusion relationship, develop more personalized treatment plans, and improve treatment effect and patient satisfaction.

[0191] In the present specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0192] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatic three-dimensional fitting of oral cavity curves based on multimodal fusion data, characterized in that, include: Collect CBCT images of the patient's oral cavity and digital dental models; The CBCT images and the digital dental model are registered and fused to obtain the registration result; The coordinates of landmarks on CBCT images and digital dental models were obtained to obtain the landmark detection results of the skull and teeth. Based on the registration results and the detection results of the landmark points of the skull and teeth, a three-dimensional curve is directly fitted in three-dimensional space. The CBCT images and digital dental models are registered and fused, including: The head position of the CBCT image was corrected, and the U-Net model was used to segment the teeth of the CBCT image to obtain the tooth segmentation results; A graph neural network model was used to segment the crowns of the digital dental model to obtain the crown segmentation results. Based on the tooth segmentation results and the crown segmentation results, registration and fusion are performed using iterative nearest point and FilterReg algorithm based on a probability model.

2. The method for automatic three-dimensional fitting of oral cavity curves based on multimodal fusion data according to claim 1, characterized in that, Obtain the coordinates of landmarks in the CBCT image, including: The coordinates of the following skull landmarks were determined and exported from the CBCT images after head position correction: bilateral infraorbital points, right ear point, nasal root point, skull base point, anterior nasal spine point, and posterior nasal spine point, and a reference plane was established. Obtain the coordinates of landmark points on the digital dental model, including: Automatic positioning of tooth landmarks in the digital dental model is performed based on PointDGCN. The crown landmarks of the maxillary dentition include 11C, 12C, 14B, 15B, 16MB, 16DB, 17MB, 17DB, 21C, 22C, 24B, 25B, 26MB, 26DB, 27MB, and 27DB; where C represents the midpoint of the incisal edge, B represents the buccal apex, MB represents the mesial buccal apex, and DB represents the distal buccal apex.

3. The method for automatic three-dimensional fitting of oral cavity curves based on multimodal fusion data according to claim 1, characterized in that, Based on the registration results and the detection results of the landmark points of the skull and teeth, a three-dimensional curve is directly fitted in three-dimensional space, including: The Spee curves of the posterior maxilla, the anterior maxilla, the entire right maxilla, and the entire left maxilla were fitted in three-dimensional space using the polynomial method to obtain the final three-dimensional curves.

4. The method for automatic three-dimensional fitting of oral cavity curves based on multimodal fusion data according to claim 1, characterized in that, The U-Net model is used to segment teeth in the CBCT images, including: For the raw CBCT data, preprocessing operations are performed on the raw CBCT data; A voxel-level segmentation network based on the 3D U-Net architecture is adopted. The input is a voxel block of fixed size, and the output is the corresponding voxel label probability map. Multi-class segmentation is performed on each tooth and background tissue. Based on the voxel-level segmentation results, the contour surfaces of each tooth voxel region are extracted using the Marching Cubes algorithm to generate a structured triangular mesh model, thus obtaining the CBCT tooth segmentation results.

5. The method for automatic three-dimensional fitting of oral cavity curves based on multimodal fusion data according to claim 1, characterized in that, The structure of the graph neural network model includes: a feature encoding module, a spatial transformation module, a dynamic edge convolution module, and a full-dental level attention pooling module connected in sequence; The digital dental model is segmented using a graph neural network model, including: Based on the mesh data in the digital dental model, a graph structure is first constructed using the K-nearest neighbor algorithm in the preprocessing stage to obtain initial patch features; A feature encoding module is constructed by combining a multilayer perceptron with instance normalization and the LeakyReLU activation function. The initial patch features are encoded through the feature encoding module. The feature transformation matrix of each sample is learned through the spatial transformation module; Local feature extraction is performed based on a dynamic edge convolution module; The full dentition level attention pooling module is used to perform global feature aggregation on the upper and lower jaws of each sample to obtain the full dentition level semantic context, and then feed it back to the node level. Perform multi-category classification on each face node and output its corresponding tooth category.

6. The method for automatic three-dimensional fitting of oral cavity curves based on multimodal fusion data according to claim 2, characterized in that, Automatic localization of tooth landmarks in the digital dental model based on PointDGCN includes: Based on the crown segmentation results, a local sub-image is constructed for each tooth as a candidate region; The PointDGCN network is used for marker prediction; the PointDGCN network includes a feature encoding module, a spatial transformation module, a dynamic edge convolution module, and a multi-head regression module connected in sequence. Predict the normalized positions of multiple markers output by each node; Based on the response of each type of marker point at all nodes, a weighted average method is used to determine its specific location in the original dental model, thus completing the three-dimensional coordinate reconstruction of the marker point.

7. A three-dimensional automatic fitting system for oral cavity curves based on multimodal fusion data, employing the three-dimensional automatic fitting method for oral cavity curves based on multimodal fusion data as described in any one of claims 1-6, characterized in that, include: The clinical data collection module is used to collect CBCT images of the patient's oral cavity and digital dental models; The registration and fusion module is used to register and fuse the CBCT images and the digital dental model to obtain the registration result. The landmark detection module is used to acquire the coordinates of landmarks on CBCT images and digital dental models, respectively, to obtain the landmark detection results of the skull and teeth. The fitting module is used to directly fit a three-dimensional curve in three-dimensional space based on the registration results and the detection results of the landmark points of the skull and teeth.

Images