A method for constructing a three-dimensional facial median sagittal plane based on an intelligent registration algorithm

Abstract

The application relates to a three-dimensional facial median sagittal plane construction method based on an intelligent registration algorithm, which has the following steps: (1) constructing a variable graph structure neural network algorithm for registration of a three-dimensional facial data ontology and mirror image point cloud, and the algorithm steps are as follows: 1) constructing feature vectors of key points in the ontology and mirror image point cloud data X and Y; 2) obtaining the corresponding relationship of the key points in the point cloud X and Y based on the feature vectors; and 3) calculating the rotation and translation matrix R and t of the mirror image data through singular value decomposition; (2) constructing a three-dimensional facial median sagittal plane through a principal component analysis algorithm; and the application can realize accurate and efficient construction of a three-dimensional facial median sagittal plane, and provides a new median sagittal plane construction solution for digital diagnosis and treatment in the oral clinic.

Description

Technical Field

[0001] This invention relates to a method for constructing a symmetrical reference plane for the face, specifically a method for constructing a three-dimensional facial midsagittal plane based on an intelligent registration algorithm. Background Technology

[0002] 1. Background

[0003] Facial symmetry and harmony are crucial factors in facial aesthetics and attractiveness, and three-dimensional facial asymmetry analysis is a fundamental clinical issue of common concern across various disciplines of oral medicine. In areas such as oral and maxillofacial surgery, trauma and oncology surgery planning, orthodontic treatment design, and aesthetic restoration design, three-dimensional facial asymmetry analysis is a vital step, serving as the primary basis for treatment planning and surgical design. Accurate determination of the three-dimensional median sagittal plane (MSP) is the prerequisite and foundation for facial asymmetry analysis.

[0004] 2. Current Status and Development of Research at Home and Abroad

[0005] With the increasing development of digital dentistry, reducing reliance on experts has become a real need, and automated algorithms for constructing the 3D facial midsagittal plane have attracted widespread attention from scholars. Therefore, algorithms for constructing a symmetrical reference plane based on the overlapping and correlation of a 3D facial ontology model and its mirror model have become a research hotspot, known as the "ontology-mirror correlation method." The core of the ontology-mirror correlation method is the optimal overlap (registration) algorithm between the 3D facial ontology and its mirror model. Currently, domestic and international research on 3D overlap algorithms mainly falls into two categories: the Iterative Closest Point (ICP) algorithm without reference markers and the Procrustes Analysis (PA) algorithm with reference markers. When applied to the midsagittal plane construction of 3D facial data without significant deformities, the ICP and PA correlation methods can basically meet the needs of clinical oral diagnosis and treatment, exhibiting high stability and accuracy. However, for clinical facial deformity data, the ontology-mirror overlap effect of these two algorithms still lags behind expert experience. In recent years, deep learning algorithms have been studied in point cloud registration of industrial geometric models. However, medical 3D point cloud data has a large amount of information and complex shape, and no relevant deep learning algorithms have been applied to the ontology-mirror point cloud registration of 3D facial data.

[0006] 3. Review and Summary

[0007] Based on the review of previous research, it can be concluded that the current development direction for constructing a three-dimensional facial midsagittal plane is characterized by high automation, low reliance on clinical experience, and accuracy and stability. The core of this approach is the registration of three-dimensional facial ontology with mirrored point cloud data. Existing research on various semi-automated and automated algorithms is striving towards this goal to varying degrees. However, research on semi-automated registration algorithms based on ICP and PA algorithms is still limited by the manual selection of facial regions and the manual definition of important facial anatomical landmarks, resulting in limited automation. Currently, none of these algorithms have achieved automatic, intelligent, accurate, and stable construction of the three-dimensional facial midsagittal plane.

[0008] References

[0009] [1]BaudouinJY,TiberghienG.Symmetry,averageness,andfeaturesizeinthefacial attractiveness of women[J].ActaPsychol(Amst),2004,117(3):313-332.

[0010] [2]Ferrario VF, Sforza C, Ciusa V, et al. The effect of sex and age onfacial asymmetry in healthy subjects: a cross-sectional study from adolescence to mid-adulthood[J]. J Oral Maxillofac Surg, 2001, 59(4): 382-388.

[0011] [3]Shaner DJ, Peterson AE, Beattie OB, et al.Assessment of softtissue facial asymmetry in medically normal and syndrome-affected individualsby analysis of landmarks and measurements[J].AmJMedGenet,2000,93(2):143-154.

[0012] [4]Burke P H,Healy M J.A serial study ofnormal facial asymmetry inmonozygotic twins[J].Ann HumBiol,1993,20(6):527-534.

[0013] [5]Lo L J,Yang C T,Ho C T,et al.Automatic Assessment of3-DimensionalFacial Soft Tissue Symmetry Before and After Orthognathic Surgery Using aMachine Learning Model:A Preliminary Experience[J].AnnPlastSurg,2021,86(3SSuppl2):S224-S228.

[0014] [6]Lin H,Chiang W,Yang C,et al.On construction of transfer learningfor facial symmetry assessmentbefore andafter orthognathic surgery[J].ComputerMethods andPrograms inBiomedicine,2021,200:105928.

[0015] [7]Duran,Gokhan,Serhat,etal.Accuracyandreliabilityof3Dstereophotogrammetry:Acomparison todirectanthropometryand2Dphotogrammetry[J].AngleOrthod,2016,86(3):487-494.

[0016] [8]Weinberg S M,Naidoo S,Govier D P,et al.Anthropometric precisionand accuracy ofdigital three-dimensional photogrammetry:comparing the Genexand 3dMD imaging systems with one anotherandwithdirectanthropometry.[J].JCraniofac Surg,2006,17(3):477-483.

[0017] [9]BenzM,LaboureuxX,MaierT,etal.The SymmetryofFaces[Z].200243-50.

[0018]

[10] Hartmann J,Meyer-Marcotty P,Benz M,et al.Reliability of a Methodfor Computing Facial Symmetry Plane and Degree of Asymmetry Based on 3D-data[J].J Orofac Orthop,2007,68(6):477-490.

[0019]

[11] Klingenberg C P,Barluenga M,Meyer A.Shape analysis of symmetricstructures:quantifying variationamong individuals andasymmetry[J].Evolution,2002,56(10):1909-1920.

[0020]

[12] Xiong Y,Zhao Y,Yang H,et al.Comparison Between InteractiveClosest Point and Procrustes Analysis for Determining the Median SagittalPlane ofThree-Dimensional Facial Data[J].J Craniofac Surg,2016,27(2):441-444.

[0021]

[13] Liu Xiaojing, Li Qianqian, Wang Xiaoxia, et al. Automatic construction method of three-dimensional cranial midsagittal plane based on ontology-mirror association [J]. Chinese Journal of Orthodontics, 2014, 21(3):148-150.

[0022]

[14] Djordjevic J,Pirttiniemi P,Harila V,et al.Three-dimensional longitudinal assessment offacial symmetryinadolescents[J].EurJOrthod,2013,35(2):143-151.

[0023]

[15] DGCNN variable graph network:

[0024] Wang,Y.,et al.,Dynamic graph cnn for learning on point clouds.AcmTransactions On Graphics(tog),2019.38(5):p.1-12.

[0025]

[16] Gumbel-Softmax normalized exponential function:

[0026] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. In International Conference on Learning Representations (ICLR), 2017.

[0027]

[17] Transformer network:

[0028] Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need[J].Advances in neural information processing systems,2017,30.

[0029]

[18] Principal component analysis algorithm:

[0030] Wang Lei. A Symmetry Detection Algorithm for 3D Point Cloud Models Based on Weighted PCA Analysis [J]. Journal of Shandong University, 2014(9):166-170. Summary of the Invention

[0031] (a) Technical problems to be solved

[0032] This invention addresses the need for three-dimensional facial asymmetry analysis in digital oral clinical diagnosis and treatment. It establishes a method for constructing the three-dimensional facial midsagittal plane based on intelligent registration technology, which can achieve accurate and efficient construction of the three-dimensional facial midsagittal plane, providing a new midsagittal plane construction solution for digital oral clinical diagnosis and treatment.

[0033] (II) Technical Solution

[0034] The present invention provides a method for constructing a three-dimensional facial median sagittal plane based on an intelligent registration algorithm, comprising the following steps:

[0035] (1) Construct a variable graph structure neural network algorithm for registering 3D facial data ontology with mirrored point clouds. The algorithm steps are as follows:

[0036] 1) Construct feature vectors for key points in the ontology and mirror point cloud data X and Y;

[0037] 2) Based on the above feature vectors, obtain the correspondence between key points in point clouds X and Y;

[0038] 3) Calculate the rotation and translation matrices R and t of the mirrored data through singular value decomposition;

[0039] (2) Principal component analysis algorithm is used to construct the three-dimensional midsagittal plane of the face;

[0040] Based on the registration results of the above 3D face ontology and mirror image data, the joint point cloud data of "ontology-mirror image" is obtained. This data has symmetric characteristics. The covariance matrix of the joint point cloud matrix of "ontology-mirror image" is calculated by principal component analysis algorithm. The eigenvalues ​​and eigenvectors of the covariance matrix are solved by eigenvalue decomposition method. The eigenvector corresponding to the largest eigenvalue is selected as the normal vector of the median sagittal plane. The geometric center of the point cloud is calculated as the center point of the median sagittal plane, and the median sagittal plane function expression of the 3D face data is obtained.

[0041] in:

[0042] The specific algorithm used in the algorithm steps is as follows:

[0043] Step 1): Construct keypoint feature vectors Φ using a variable graph network and a Transformer network. X Φ Y ;

[0044] Step 2): Based on Φ X Φ Y The correspondence between key points is calculated using the normalized exponential function (Gumbel-Softmax) and the temperature parameter λ:

[0045]

[0046] Step 3): Given the correspondence and coordinates of the key points, calculate the transformation matrix through singular value decomposition, which includes rotation and translation matrices R and t.

[0047]

[0048] In the above formula, V and U are two matrices obtained by singular value decomposition; It is the predicted rotation and translation matrix.

[0049] The transformation matrix is ​​used to transform the mirror point cloud Y into Y'. The steps of obtaining the key point feature vector, calculating the correspondence between key points, and performing matrix singular value decomposition are iterated on X and Y' until the number of iterations reaches the hyperparameter and the iteration terminates.

[0050] The training process of the variable graph structure neural network uses 800 cases of 3D facial data collected from oral clinical practice. The rotation and translation matrices based on the ICP algorithm ontology and mirror point cloud registration are used as the ground truth for supervised learning. The dataset for training and testing the 800 cases of 3D facial data collected clinically is divided into training, validation, and test sets in an 8:1:1 ratio, with a male-to-female ratio of 1:1. The overall loss function of the variable graph structure neural network is given by formula ③.

[0051]

[0052] L represents the overall loss, L p L is the total loss at the p-th iteration. p It consists of three parts: Rigid motion loss, Cyclic consistency loss, Global feature alignment loss, as shown in formulas ④, ⑤, and ⑥ respectively:

[0053]

[0054]

[0055]

[0056] In the above formula, For the truth values ​​of the rotation and translation matrices, It is the transpose of the predicted rotation matrix. It is the predicted translation matrix. Let X be the rotation and translation matrices from point cloud X to point cloud Y. Let Y be the rotation and translation matrix from point cloud Y to point cloud X. For the global representation of point cloud X, Let Y be the global representation of the point cloud.

[0057] (III) Beneficial Effects

[0058] The intelligent registration algorithm of this invention can automatically construct the three-dimensional facial midsagittal plane, effectively reducing the workload of clinical data annotation, shortening the digital diagnostic design time, reducing reliance on expert experience, and improving the efficiency and effectiveness of clinical diagnosis and treatment. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the variable graph structure neural network framework of the present invention;

[0060] In the diagram: X: Body point cloud; Y: Mirror point cloud; Φ X : Eigenvector; Φ Y : Eigenvector; m p The correspondence between key points. Detailed Implementation

[0061] The following examples are used to illustrate the present invention, but are not intended to limit the scope of the invention.

[0062] like Figure 1 As shown:

[0063] The present invention provides a method for constructing a three-dimensional facial median sagittal plane based on an intelligent registration algorithm, comprising the following steps:

[0064] (1) Construct a variable graph structure neural network algorithm for registering 3D facial data ontology with mirrored point clouds. The algorithm steps are as follows:

[0065] 1) Construct feature vectors for key points in the ontology and mirror point cloud data X and Y;

[0066] 2) Based on the above feature vectors, obtain the correspondence between key points in point clouds X and Y;

[0067] 3) Calculate the rotation and translation matrices R and t of the mirrored data through singular value decomposition;

[0068] (2) Principal component analysis algorithm is used to construct the three-dimensional midsagittal plane of the face;

[0069] Based on the registration results of the above 3D face ontology and mirror image data, the joint point cloud data of "ontology-mirror image" is obtained. This data has symmetric characteristics. The covariance matrix of the joint point cloud matrix of "ontology-mirror image" is calculated by principal component analysis algorithm. The eigenvalues ​​and eigenvectors of the covariance matrix are solved by eigenvalue decomposition method. The eigenvector corresponding to the largest eigenvalue is selected as the normal vector of the median sagittal plane. The geometric center of the point cloud is calculated as the center point of the median sagittal plane, and the median sagittal plane function expression of the 3D face data is obtained.

[0070] in:

[0071] The specific algorithm used in the algorithm steps is as follows:

[0072] Step 1): Construct keypoint feature vectors Φ using a variable graph network and a Transformer network. X Φ Y ;

[0073] Step 2): Based on Φ X Φ Y The correspondence between key points is calculated using the normalized exponential function (Gumbel-Softmax) and the temperature parameter λ:

[0074]

[0075] Step 3): Given the correspondence and coordinates of the key points, calculate the transformation matrix through singular value decomposition, which includes rotation and translation matrices R and t.

[0076]

[0077] In the above formula, V and U are two matrices obtained by singular value decomposition; It is the predicted rotation and translation matrix.

[0078] The transformation matrix is ​​used to transform the mirror point cloud Y into Y'. The steps of obtaining the key point feature vector, calculating the correspondence between key points, and performing matrix singular value decomposition are iterated on X and Y' until the number of iterations reaches the hyperparameter and the iteration terminates.

[0079] The training process of the variable graph structure neural network uses 800 cases of 3D facial data collected from oral clinical practice. The rotation and translation matrices based on the ICP algorithm ontology and mirror point cloud registration are used as the ground truth for supervised learning. The dataset for training and testing the 800 cases of 3D facial data collected clinically is divided into training, validation, and test sets in an 8:1:1 ratio, with a male-to-female ratio of 1:1. The overall loss function of the variable graph structure neural network is given by formula ③.

[0080]

[0081] L represents the overall loss, L p L is the total loss at the p-th iteration. p It consists of three parts: Rigid motion loss, Cyclic consistency loss, Global feature alignment loss, as shown in formulas ④, ⑤, and ⑥ respectively:

[0082]

[0083]

[0084]

[0085] In the above formula, For the truth values ​​of the rotation and translation matrices, It is the transpose of the predicted rotation matrix. It is the predicted translation matrix. Let X be the rotation and translation matrices from point cloud X to point cloud Y. Let Y be the rotation and translation matrix from point cloud Y to point cloud X. For the global representation of point cloud X, Let Y be the global representation of the point cloud.

[0086] Advantages of this invention:

[0087] (1) This invention is the first to achieve intelligent registration and overlap of three-dimensional face ontology and mirror data based on a variable graph structure neural network algorithm;

[0088] (2) This invention is the first to realize the construction of the midsagittal plane of “ontology-mirror” joint data based on principal component analysis algorithm.

[0089] As described above, the present invention can be sufficiently realized. The above description is merely a reasonable embodiment of the present invention, and the scope of protection of the present invention includes, but is not limited to, these embodiments. Any non-substantial modifications or alterations made by those skilled in the art based on the technical solutions of the present invention are included within the scope of the present invention.

Claims

1. A method for constructing a three-dimensional facial median sagittal plane based on an intelligent registration algorithm, characterized in that: (1) Construct a variable graph structure neural network algorithm for registering 3D facial data ontology with mirrored point clouds. The algorithm steps are as follows: 1) Construct feature vectors for key points in the ontology and mirror point cloud data X and Y; 2) Based on the above feature vectors, obtain the correspondence between key points in point clouds X and Y; 3) Calculate the rotation and translation matrices R and t of the mirrored data through singular value decomposition; (2) Principal component analysis algorithm is used to construct the three-dimensional midsagittal plane of the face; Based on the registration results of the above 3D face ontology and mirror image data, joint point cloud data of "ontology-mirror image" is obtained. This data has symmetric properties. Principal component analysis algorithm is used to calculate the covariance matrix of the joint point cloud matrix of "ontology-mirror image". The eigenvalues ​​and eigenvectors of the covariance matrix are solved based on the eigenvalue decomposition method. The eigenvector corresponding to the largest eigenvalue is selected as the normal vector of the median sagittal plane. The geometric center of the point cloud is calculated as the center point of the median sagittal plane, and the median sagittal plane function expression of the 3D face data is obtained. The specific algorithm used in the algorithm steps is as follows: Step 1): Construct keypoint feature vectors Φ using a variable graph network and a Transformer network. X Φ Y ; Step 2): Based on Φ X Φ Y The correspondence between key points is calculated using the normalized exponential function (Gumbel-Softmax) and the temperature parameter λ: Step 3): Given the correspondence and coordinates of the key points, calculate the transformation matrix through singular value decomposition, which includes rotation and translation matrices R and t. In the above formula, V and U are two matrices obtained by singular value decomposition; It is the predicted rotation and translation matrix. The transformation matrix is ​​used to transform the mirror point cloud Y into Y'. The steps of obtaining the key point feature vector, calculating the correspondence between key points, and performing matrix singular value decomposition are iterated on X and Y' until the number of iterations reaches the hyperparameter and the iteration terminates. The training process of the variable graph structure neural network uses 800 cases of 3D facial data collected from oral clinical practice. The rotation and translation matrices based on the ICP algorithm ontology and mirror point cloud registration are used as the ground truth for supervised learning. The dataset for training and testing the 800 cases of 3D facial data collected clinically is divided into training, validation, and test sets in an 8:1:1 ratio, with a male-to-female ratio of 1:

1. The overall loss function of the variable graph structure neural network is given by formula ③. L represents the overall loss, L p L is the total loss at the p-th iteration. p It consists of three parts: Rigid motion loss, Cyclic consistency loss, Global feature alignment loss, as shown in formulas ④, ⑤, and ⑥ respectively: In the above formula, For the truth values ​​of the rotation and translation matrices, It is the transpose of the predicted rotation matrix. It is the predicted translation matrix. Let X be the rotation and translation matrices from point cloud X to point cloud Y. Let Y be the rotation and translation matrix from point cloud Y to point cloud X. For the global representation of point cloud X, Let Y be the global representation of the point cloud.

Images