A three-dimensional facial symmetry reference plane intelligent construction method

Abstract

The present application relates to a kind of three-dimensional face symmetry reference plane intelligent construction method, including: for obtaining three-dimensional face data region of interest, first, three-dimensional face data is projected to two-dimensional plane image, the key point of the key point of the image in the anatomic region is obtained by extracting two-dimensional image face key point, then the key point on the image is mapped back to three-dimensional face, obtain the face key point of three-dimensional face data, and the region of interest of three-dimensional face data is obtained by region expansion, further establish plane reflection symmetry neural network intelligent construction three-dimensional face symmetry reference plane;The present application can realize accurate, efficient construction in line with three-dimensional face symmetry reference plane, provide a new symmetry reference plane construction solution for oral clinical digital diagnosis and treatment.

Description

Technical Field

[0001] This invention relates to a method for constructing a symmetric reference plane, specifically a method for intelligently constructing a three-dimensional facial symmetric reference plane. background

[0002] Facial aesthetic assessment and facial symmetry analysis are fundamental issues of common concern across various dental disciplines, and are particularly crucial in fields with high medical and aesthetic requirements, such as orthodontics, oral and maxillofacial surgery, and prosthodontics. Facial asymmetry analysis plays a vital role in clinical diagnosis, surgical planning, and efficacy evaluation in dentistry. Asymmetry analysis is often based on the facial symmetry reference plane (SRP), also known as the median sagittal plane (MSP). The accuracy of constructing the symmetry reference plane directly affects the accuracy and reliability of subsequent diagnostic and treatment outcomes.

[0003] Due to innate or environmental factors, no human face is perfectly symmetrical. Therefore, constructing a symmetry reference plane based on the complex and irregularly distributed asymmetrical geometric facial information of natural organisms is not unique. Thus, researching intelligent and automated algorithms for three-dimensional facial symmetry reference planes can provide a more efficient and accurate solution for symmetry analysis of three-dimensional facial data in dental clinics.

[0004] 2. Current Status and Development of Research at Home and Abroad

[0005] Previous literature on the construction of 3D facial symmetry reference planes mainly falls into three categories: manually defined methods, semi-automated algorithms, and automated algorithms. Manually defined methods often construct symmetry reference planes based on important anatomical landmarks on a 3D digital model of the face, typically including midline and bilateral anatomical landmarks. Regarding the selection criteria for these landmarks, different scholars hold differing opinions, resulting in poor reproducibility of symmetry reference planes constructed by different users and research subjects. Semi-automated algorithms, also known as "ontology-mirror association algorithms," are methods for constructing 3D facial symmetry reference planes based on both manual intervention and software computation. The "ontology-mirror association algorithm" is a semi-automated algorithm that uses software to construct symmetry reference planes. Based on the principle of overlap matching between the ontology model and the mirror model, it can be divided into Protodyakonov analysis algorithms using reference landmarks and iterative nearest-point algorithms without reference landmarks. While the ontology-mirror association algorithm improves the automation of symmetry reference plane construction, it still requires manual selection of landmarks or regions to achieve a satisfactory overlap effect.

[0006] In recent years, artificial intelligence (AI) technology has been widely applied in image processing, disease diagnosis, and decision-making in the field of oral medicine. Deep learning algorithms based on artificial neural networks possess powerful feature extraction and learning capabilities, can reasonably handle complex morphological information, and learn implicit patterns from it, demonstrating superior performance in various intelligent tasks and possessing enormous application potential. Deep learning algorithms have provided new ideas for constructing symmetry reference planes for 3D models. Liu Xinguo et al. proposed the Sym-Net network for point cloud symmetry analysis, using the PointNet++ network for feature extraction. This supervised learning network model classifies the "midline" and "bilateral" point clouds of regular geometric models, determining the symmetry plane from the "bilateral" points. Xu Kai et al. proposed the SymmetryNet network for predicting the symmetry plane of 3D models based on a single RGB-D image. This supervised learning network model determines the correspondence between two points in the point cloud and the symmetry plane by predicting the rotational and reflection symmetry of the model. The above models have good robustness to local shapes; however, supervised learning requires expert-annotated data, and training on large datasets reveals its limitations. There has been research on using deep learning algorithms to construct symmetric reference planes for 3D geometric models, but there has been no research on directly using deep learning algorithms to intelligently construct symmetric reference planes based on 3D facial data.

[0007] 3. Review and Summary

[0008] Based on the review of previous research, it can be concluded that the current development direction is to construct a highly automated, clinically experience-dependent, accurate, and stable 3D facial symmetry reference plane algorithm. Existing research on various semi-automated and automated algorithms is striving towards this goal to varying degrees. Semi-automated algorithm research based on ICP and PA algorithms is still limited by the manual selection of facial regions and the manual definition of important facial anatomical landmarks, resulting in limited automation. Automated algorithms have been studied in fields such as computer vision for regular geometric data, but research on irregular 3D facial data in oral medicine has not yet been reported. None of the aforementioned algorithms have yet achieved automatic, intelligent, accurate, and stable construction of a 3D facial symmetry reference plane.

[0009] References

[0010] [1]Rhodes,Gillian.The Evolutionary Psychology of Facial Beauty[J].2006,57(1):199-226.

[0011] [2]Burriss R P,Roberts S C,Welling L,et al.Heterosexual RomanticCouples Mate Assortatively for Facial Symmetry,But Not Masculinity[J].Personality and Social Psychology Bulletin,2011,37(5):601-613。

[0012] [3]Thiesen G,Gribel B F,Freitas M P M.Facial asymmetry:a currentreview[J].Dental press journal of orthodontics,2015,20(6):110-125。

[0013] [4]Weinberg S M,Naidoo S,Govier D P,et al.Anthropometric precisionand accuracy of digital three-dimensional photogrammetry:comparing the Genexand 3dMD imaging systems with one another and with direct anthropometry.[J].Journal of Craniofacial Surgery,2006,17(3):477。

[0014] [5]Duran,Gokhan,Serhat,et al.Accuracy and reliability of 3Dstereophotogrammetry:A comparison to direct anthropometry and 2Dphotogrammetry[J].Angle Orthodontist,2016。

[0015] [6]Lo L,Yang C,Ho C,et al.Automatic Assessment of 3-DimensionalFacial Soft Tissue Symmetry Before and After Orthognathic Surgery Using aMachine Learning Model:A Preliminary Experience[J].Annals of Plastic Surgery,2021,86(3S)。

[0016] [7]Lin H,Chiang W,Yang C,et al.On construction of transfer learningfor facial symmetry assessment before and after orthognathic surgery[J].Computer Methods and Programs in Biomedicine。2021,200:105928。

[0017] [8]Murakami D,Inada E,Saitoh I,et al.Morphological differences offacial soft tissue contours from child to adult of Japanese males:a three-dimensional cross-sectional study[J].Arch Oral Biol。2014,59(12):1391-1399。

[0018] [9]Nur R B,Cakan D G,Arun T.Evaluation of facial hard and soft tissueasymmetry using cone-beam computed tomography[J].Am J Orthod DentofacialOrthop,2016,149(2):225-237。

[0019]

[10] Benz M,Laboureux X,Maier T,et al.The Symmetry of Faces.[Z].200243-50。

[0020]

[11] Verhoeven T, Xi T, Schreurs R, et al. Quantification of facial asymmetry: A comparative study of landmark-based and surface-based registrations[J]. Journal of Cranio-Maxillofacial Surgery, 2016, 44(9): 1131-1136.

[0021]

[12] Klingenberg CP, Barluenga M, Meyer A. Shape analysis of symmetricstructures: quantifying variation among individuals and asymmetry[J]. Evolution, 2002, 56(10): 1909-1920.

[0022]

[13] Tian Kaiyue. Digital correction scheme design for mandibular prognathism and deviation deformity [D]. Peking University School of Medicine, 2015.

[0023]

[14] Zhu Y, Zheng S, Yang G, et al. A novel method for 3D face symmetryreference plane based on weighted Procrustes analysis algorithm[J]. BMC OralHealth, 2020, 20(1):319.

[0024]

[15] Dot G,Schouman T,Chang S,et al.Automatic 3-DimensionalCephalometric Landmarking via Deep Learning[J].JOURNAL OF DENTAL RESEARCH,2022,101(11):1380-1387.

[0025]

[16] Hao J,Liao W,Zhang Y L,et al.Toward Clinically Applicable 3-Dimensional Tooth Segmentation via Deep Learning[J].JOURNAL OF DENTALRESEARCH,2022,101(3):304-311。

[0026]

[17] Liu J L,Li S H,Cai Y M,et al.Automated Radiographic Evaluation ofAdenoid Hypertrophy Based on VGG-Lite[J].JOURNAL OF DENTAL RESEARCH,2021,100(12):1337-1343。

[0027]

[18] Wang H,Minnema J,Batenburg K J,et al.Multiclass CBCT ImageSegmentation for Orthodontics with Deep Learning[J].JOURNAL OF DENTALRESEARCH,2021,100(9):943-949。

[0028]

[19] Lecun Y,Bengio Y,Hinton G.Deep learning[J].NATURE,2015,521(7553):436-444。

[0029]

[20] Ji P,Liu X.A fast and efficient 3D reflection symmetry detectorbased on neural networks[J].MULTIMEDIA TOOLS AND APPLICATIONS,2019,78(24):35471-35492。

[0030]

[21] Qi CR, Yi L, Su H, et al. PointNet plus plus: Deep HierarchicalFeature Learning on Point Sets in a Metric Space [Z]. 31st Annual Conference on Neural Information Processing Systems (NIPS): 2017.30.

[0031]

[22] Shi Y, Huang J, Zhang H, et al. SymmetryNet: Learning to PredictReflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-DImages[J]. ACM TRANSACTIONS ON GRAPHICS, 2020, 39(6). Summary of the Invention

[0032] (a) Technical problems to be solved

[0033] This invention addresses the need for three-dimensional facial asymmetry analysis in digital oral clinical diagnosis and treatment by establishing a method for intelligently constructing a three-dimensional facial symmetry reference plane. This method includes two steps: extraction of the region of interest from three-dimensional facial data and construction of a planar reflection symmetry neural network. It can achieve accurate and efficient construction of a three-dimensional facial symmetry reference plane, providing a new solution for constructing a symmetry reference plane in digital oral clinical diagnosis and treatment.

[0034] (II) Technical Solution

[0035] The present invention provides a method for intelligently constructing a three-dimensional facial symmetry reference plane, comprising the following steps:

[0036] (1) Automatic extraction of regions of interest from 3D facial data:

[0037] First, the region of interest is automatically extracted from the 3D facial data:

[0038] To obtain the region of interest (ROI) of 3D facial data, the 3D facial data is first projected onto a 2D planar image. Key points of the anatomical regions in the image are obtained by extracting facial key points from the 2D image. These key points are then mapped back onto the 3D face to obtain the key points of the 3D facial data. Finally, the 3D facial data ROI is obtained through region expansion. This process is as follows: Figure 1 As shown;

[0039] (2) Constructing a three-dimensional facial symmetric reference plane using a planar reflection-symmetric neural network:

[0040] Based on the extraction of the three-dimensional facial data interest region in step (1), a planar reflection symmetry neural network is further established to intelligently construct a three-dimensional facial symmetry reference plane; the neural network framework is as follows: Figure 2 As shown, the planar reflection symmetric neural network first uniformly samples the surface of the grid in the region of interest of the 3D facial data to obtain 3D point cloud data with the same number of points, resulting in 10240 points. The 3D point cloud data is then processed by a point cloud depth network—PonitNet—for global feature extraction. These global features are input into a fully connected network to predict the plane parameters of the symmetric reference plane. The symmetric reference plane is represented by the general expression Ax + By + Cz + D = 0. The entire neural network structure is trained in a supervised manner. The loss function of the neural network is set to the mean square error between the predicted plane parameters and the plane parameters from the expert database that establishes the true plane parameters. The expression for the loss function is as follows: Where A, B, C, and D are the parameters of the expert database plane Ax + By + Cz + D = 0. These are the planar parameters of a planar reflection symmetric neural network that predicts a symmetric reference plane.

[0041] in:

[0042] The specific method for extracting facial key points from 2D images employs a dense face alignment algorithm (3D Dense Face Alignment, or 3DDFA for short). This algorithm detects and labels facial key points based on 3D facial data, extracting 51 key points from the 2D facial image (such as...). Figure 1 As shown), the facial key points are distributed across the eyebrows, eye area, nose area, lip area, and jawline from the bilateral tragus to the chin. The area where the facial key points are located covers the main areas of facial aesthetic analysis: the feature representation of the point cloud for facial asymmetry analysis of the eye area, nose area, and lip area. The facial key points on the two-dimensional image are back-projected to obtain the corresponding facial key points on the three-dimensional facial data.

[0043] in:

[0044] To further expand the point cloud of the area near the key facial points in the 3D facial data and construct a region of interest that conforms to clinical diagnosis and analysis, other vertices in the 3D facial data with a diameter less than 20mm from the key facial points are selected for region expansion. The expanded area is the region of interest of the 3D facial data. Based on the above 3D facial interest extraction method, the periorbital area, nasal area, and lip area can all be completely included in the selection area.

[0045] (III) Beneficial Effects

[0046] This invention presents an intelligent construction method for establishing a three-dimensional facial symmetry reference plane based on the extraction of interest regions from three-dimensional facial data and a planar reflection symmetry neural network. This method can automatically construct a three-dimensional facial symmetry reference plane, effectively reducing the workload of clinical data annotation, shortening the design time for digital diagnosis, reducing reliance on expert experience, and improving the efficiency and effectiveness of clinical diagnosis and treatment. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the selection of the region of interest in three-dimensional facial data in this invention;

[0048] Figure 2 This is a schematic diagram of the planar reflection symmetric neural network framework of the present invention.

[0049] In the figure: PoinNet: a point cloud deep network; 1024, 512, 256, 64, and 4 represent the number of image channels, respectively. Detailed Implementation

[0050] The following examples are used to illustrate the present invention, but are not intended to limit the scope of the invention.

[0051] The present invention provides a method for intelligently constructing a three-dimensional facial symmetry reference plane, comprising the following steps:

[0052] (1) Automatic extraction of regions of interest from 3D facial data:

[0053] First, the region of interest is automatically extracted from the 3D facial data:

[0054] To obtain the region of interest (ROI) of 3D facial data, the 3D facial data is first projected onto a 2D planar image. Key points of the anatomical regions in the image are obtained by extracting facial key points from the 2D image. These key points are then mapped back onto the 3D face to obtain the key points of the 3D facial data. Region expansion is then used to obtain the 3D facial data ROI, and subsequently, a planar reflection symmetry neural network is established to intelligently construct a 3D facial symmetry reference plane. This process is as follows: Figure 1 As shown;

[0055] (2) Constructing a three-dimensional facial symmetric reference plane using a planar reflection-symmetric neural network:

[0056] Based on the extraction of the three-dimensional facial data interest region in step (1), a planar reflection symmetry neural network is further established to intelligently construct a three-dimensional facial symmetry reference plane; the neural network framework is as follows: Figure 2As shown, the planar reflection symmetric neural network first uniformly samples the mesh surface of the region of interest in the 3D facial data to obtain 3D point cloud data with the same number of points, resulting in 10240 points. The 3D point cloud data is then processed by a point cloud depth network—PonitNet—for global feature extraction. These global features are input into a fully connected network to predict the planar parameters of the symmetric reference plane. The symmetric reference plane is represented by the general expression Ax + By + Cz + D = 0. The entire neural network structure is trained in a supervised manner. The neural network loss function is set to the mean square error between the predicted planar parameters and the planar parameters in the expert database that establishes the labeled true planar parameters. The loss function expression is as follows: Where A, B, C, and D are the parameters of the expert database plane Ax + By + Cz + D = 0. These are the planar parameters of a planar reflection symmetric neural network that predicts a symmetric reference plane.

[0057] in:

[0058] The specific method for extracting facial key points from 2D images employs a dense face alignment algorithm (3D Dense Face Alignment, or 3DDFA for short). This algorithm detects and labels facial key points based on 3D facial data, extracting 51 key points from the 2D facial image (such as...). Figure 1 As shown), the facial key points are distributed across the eyebrows, eye area, nose area, lip area, and jawline from the bilateral tragus to the chin. The area where the facial key points are located covers the main areas of facial aesthetic analysis: the feature representation of the point cloud for facial asymmetry analysis of the eye area, nose area, and lip area. The facial key points on the two-dimensional image are back-projected to obtain the corresponding facial key points on the three-dimensional facial data.

[0059] in:

[0060] To further expand the point cloud of the area near the key facial points in the 3D facial data and construct a region of interest that conforms to clinical diagnosis and analysis, other vertices in the 3D facial data with a diameter less than 20mm from the key facial points are selected for region expansion. The expanded area is the region of interest of the 3D facial data. Based on the above 3D facial interest extraction method, the periorbital area, nasal area, and lip area can all be completely included in the selection area.

[0061] This invention relates to a planar reflection-symmetric neural network training method. It utilizes three-dimensional facial data from 400 subjects collected in oral clinical settings. Based on an expert database plane with labeled ground truth parameters, the data is categorized into four types according to the degree of deformity: normal, lower facial deformity, mid-facial deformity, and upper facial deformity. The training and test sets for each category are allocated in a 4:1 ratio, with a male-to-female ratio of 1:1. By optimizing the aforementioned loss function and combining key regions and attention modules, the planar reflection-symmetric neural network can effectively learn the features of three-dimensional facial data and predict the three-dimensional facial symmetry reference plane. Evaluation using clinical data from subjects without significant facial deformities shows that the average angular error between the three-dimensional facial symmetry reference plane constructed by this method and the expert database plane is less than 1°, and the prediction accuracy exceeds 90%. The construction of the symmetric reference plane for 40 cases of three-dimensional facial data takes only 4 seconds, significantly improving construction efficiency, and the accuracy meets the application requirements of oral clinical settings.

[0062] Advantages of this invention:

[0063] (1) A method for constructing a three-dimensional facial symmetric reference plane based on a planar reflection symmetric neural network model was first realized;

[0064] (2) For the first time, the automatic extraction of interest regions based on three-dimensional facial data and the combination of supervision signal algorithm are realized to improve the robustness of the construction of symmetric reference plane.

[0065] As described above, the present invention can be sufficiently realized. The above description is merely a reasonable embodiment of the present invention, and the scope of protection of the present invention includes, but is not limited to, these embodiments. Any non-substantial modifications or alterations made by those skilled in the art based on the technical solutions of the present invention are included within the scope of the present invention.

Claims

1. A method for intelligently constructing a three-dimensional facial symmetric reference plane, characterized in that... The steps are as follows: (1) Automatic extraction of regions of interest from 3D facial data: First, the region of interest is automatically extracted from the 3D facial data: To obtain the region of interest of the three-dimensional facial data, the three-dimensional facial data is first projected onto a two-dimensional planar image. By extracting the facial key points of the two-dimensional image, the key points of the anatomical region in the image are obtained. Then, the key points on the image are mapped back to the three-dimensional face to obtain the facial key points of the three-dimensional facial data. The region of interest of the three-dimensional facial data is obtained by region expansion. (2) Constructing a three-dimensional facial symmetric reference plane using a planar reflection symmetric neural network: Based on the extraction of the three-dimensional facial data interest region in step (1), a planar reflection symmetric neural network is further established to intelligently construct a three-dimensional facial symmetric reference plane. The planar reflection symmetric neural network first performs uniform sampling on the grid surface of the three-dimensional facial data interest region to obtain three-dimensional point cloud data with the same number of points, resulting in 10240 points of three-dimensional point cloud data. The three-dimensional point cloud data is then processed by a point cloud depth network - PonitNet network for global feature extraction. The global features are input into a fully connected network to predict the plane parameters of the symmetric reference plane. The symmetric reference plane is represented by a general expression equation. The entire neural network structure is trained in a supervised manner; the loss function of the neural network is set as the mean square error between the predicted plane parameters and the plane parameters in the expert database that establishes the true plane parameters; the expression of the loss function is as follows: ； Where A, B, C, and D are the parameters of the plane Ax+By+Cz+D=0 in the expert database. ： These are the planar parameters of a planar reflection symmetric neural network that predicts a symmetric reference plane.

2. The intelligent construction method for a three-dimensional facial symmetry reference plane as described in claim 1, characterized in that... include: The specific method for extracting facial key points from 2D images employs a dense face alignment algorithm. This algorithm detects and marks facial key points based on 3D facial data, extracting 51 key points from 2D facial images. These key points are distributed across the eyebrows, eye area, nose area, lip area, and the jawline from the bilateral tragus to the chin. The areas containing these facial key points cover the main areas of facial aesthetic analysis: the feature representation of the point cloud for facial asymmetry analysis of the eye area, nose area, and lip area. The facial key points on the 2D images are then back-projected to obtain the corresponding facial key points on the 3D facial data.

3. The intelligent construction method for a three-dimensional facial symmetry reference plane as described in claim 2, characterized in that... include: To further expand the point cloud of the area near the key facial points in the three-dimensional facial data, a region of interest that conforms to clinical diagnosis and analysis is constructed; Other vertices on the 3D facial data whose diameter is less than 20mm from the key points of the face are selected for region expansion. The expanded region is the region of interest of the 3D facial data. Based on the above 3D facial data and extraction method, the periorbital area, nasal area and lip area can be completely included in the selection area.

Images