Facial deformation prediction method and device
By integrating 3DMM and neural networks for personalized face prediction models, the method addresses the limitations of existing face prediction techniques in orthognathic surgery, achieving high precision and efficiency in predicting soft tissue changes and improving surgical outcomes.
Patent Information
- Application Number
- CN202411223179.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2025-03-21
AI Technical Summary
Current face prediction methods in orthognathic surgery, particularly in the field of maxillofacial surgery, face limitations such as low accuracy, high data dependency, and complexity in data processing, which hinder precise preoperative assessments and postoperative evaluations, and lack individualization in predicting soft tissue changes.
The method employs a combination of three-dimensional morphable models (3DMM) and neural network algorithms to construct personalized face prediction models, focusing on soft tissue changes and incorporating texture correction using CNNs to enhance prediction accuracy and efficiency, reducing reliance on extensive training data and accommodating individual surgeon styles.
This approach enables high-precision, efficient, and individualized face prediction, improving surgical outcomes and patient satisfaction by directly modeling soft tissue changes and reducing computational complexity, thus facilitating real-time and accurate predictions.
Smart Images
Figure CN119671927A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of information technology, and in particular to a facial deformation prediction method and device. Background Art
[0002] Orthognathic surgery is a specialized field that surgically corrects jaw deformities and related facial asymmetries. Jaw deformities not only affect the patient's bite function, but can also have a significant impact on facial aesthetics, which can negatively affect the patient's psychological and social interactions. Orthognathic surgery can effectively improve the patient's bite relationship and facial appearance by precisely cutting and repositioning the upper and lower jaws.
[0003] However, the complexity and high risk of orthognathic surgery require accurate facial shape prediction and postoperative effect evaluation before surgery. At present, clinical practice mainly relies on the doctor's experience and two-dimensional imaging data (such as X-rays) for preoperative evaluation and postoperative prediction. However, two-dimensional imaging data cannot fully reflect the relationship between bones and soft tissues in three-dimensional space, which limits the accuracy of prediction.
[0004] In recent years, with the development of computer technology and medical imaging technology, the application of three-dimensional imaging data (such as CT and MRI) and computer-aided design (CAD) in orthognathic surgery has gradually increased. The facial shape prediction method based on three-dimensional images can more accurately simulate the surgical process and its effects. However, the existing three-dimensional facial shape prediction methods still have certain shortcomings in terms of calculation accuracy, processing speed and ease of use, and further improvement and optimization are urgently needed.
[0005] In addition, the interaction between soft tissue and hard tissue involved in orthognathic surgery is complex, and the morphological changes of soft tissue after surgery are difficult to predict, which further increases the difficulty of facial shape prediction. Therefore, developing an efficient, accurate and easy-to-operate method for predicting facial shape in orthognathic surgery is of great significance for improving the success rate of surgery and patient satisfaction.
[0006] In orthognathic surgery, facial shape prediction technology mainly includes prediction methods based on two-dimensional images and three-dimensional images.
[0007] 1. Prediction method based on two-dimensional images
[0008] Traditional orthognathic surgery facial shape prediction methods mainly rely on two-dimensional imaging data, such as X-rays, photos, and cephalograms. These methods use the doctor's experience and expertise to evaluate the patient's facial and jaw conditions before surgery and predict the postoperative results. The doctor uses these two-dimensional imaging data to design surgical plans and predict facial shape. The specific steps are as follows:
[0009] 1) Obtain a skull X-ray or photograph of the patient.
[0010] 2) Measure important anatomical landmarks using manual measurements or computer-assisted software.
[0011] 3) Design surgical plan and predict postoperative appearance based on measurement results and doctor’s experience.
[0012] However, the flatness of two-dimensional images limits their accurate reflection of the relationship between bones and soft tissues in three-dimensional space, thus affecting the accuracy of the prediction results. In addition, it relies on the doctor's experience, is highly subjective, and the results are easily affected by the doctor's level and lack standardization. Therefore, it is impossible to accurately simulate the changes in soft tissues and the prediction effect is limited.
[0013] 2. Prediction method based on 3D images
[0014] With the development of computer technology and medical imaging technology, the application of three-dimensional imaging data (such as CT and MRI) in orthognathic surgery has gradually increased. The facial shape prediction method based on three-dimensional imaging uses computer-aided design (CAD) software to convert the patient's three-dimensional CT data into a three-dimensional model, and then predicts the postoperative facial changes by simulating the surgical process. Compared with two-dimensional imaging, this method can provide more comprehensive and accurate preoperative evaluation and postoperative prediction. The method mainly includes the following steps:
[0015] 1) Obtain the patient's three-dimensional CT or MRI imaging data.
[0016] 2) Use computer-aided design (CAD) software to reconstruct a three-dimensional head model of the patient.
[0017] 3) Perform surgical simulation in the 3D model to adjust the position of the upper and lower jaws.
[0018] 4) Based on the changes in bone position after surgery, the soft tissue simulation algorithm is used to predict the morphological changes of facial soft tissue.
[0019] Compared with the two-dimensional imaging method, the three-dimensional imaging method can fully reflect the relationship between bones and soft tissues in three-dimensional space, improving the accuracy of prediction. At the same time, the use of computer technology for surgical simulation and facial shape prediction reduces the influence of doctors' subjective factors. Compared with the two-dimensional imaging facial shape prediction method, it can more accurately simulate the changes in soft tissues and improve the prediction effect.
[0020] In recent years, there are three main models of soft tissue simulation algorithms that have been reported more frequently:
[0021] 1) Mass-spring model, which simulates the changes of soft and hard tissues into an elastic model. Its advantages are simple structure and fast operation speed, but the deformation of the model is mainly determined by the elastic coefficient, which cannot reflect the biological characteristics and the prediction accuracy is insufficient.
[0022] 2) Finite element model: It is a relatively accurate biomechanical model recognized clinically for predicting the posterior postoperative shape after orthognathic surgery. It can introduce personalized differences such as gender and age of different patients to meet the different deformation conditions of different anatomical regions. However, its disadvantages are high computational cost, large memory usage and low prediction efficiency.
[0023] 3) Mass tensor model: This model has the advantages of simple structure of mass spring model while maintaining the biomechanical relevance of finite element model. However, its prediction accuracy still cannot meet clinical needs and it is still not widely used.
[0024] Therefore, the existing 3D image prediction methods also have some shortcomings: First, the 3D image data processing and the construction of 3D models are computationally complex and time-consuming, which affects the convenience of clinical application. Second, the existing 3D image prediction methods still have large errors in predicting the morphological changes of soft tissues and cannot fully and accurately simulate the postoperative facial morphology. Third, users need to have certain computer and medical imaging knowledge, and the operation is complicated and not convenient enough.
[0025] 3. Prediction methods based on artificial intelligence
[0026] In recent years, with the rapid development of artificial intelligence technology, AI-based facial shape prediction methods have gradually emerged in the field of orthognathic surgery. These methods use machine learning and deep learning algorithms, combined with a large amount of medical imaging data, to automatically learn and predict postoperative facial morphological changes.
[0027] Machine learning method: By training machine learning models (such as support vector machines, random forests, etc.) to process large amounts of two-dimensional and three-dimensional image data, features are extracted from them and postoperative effect prediction is performed. Machine learning models can use the features in the input data for regression analysis to predict postoperative facial morphological changes.
[0028] Deep learning method: Utilize deep learning models such as convolutional neural network (CNN) and generative adversarial network (GAN) to analyze and predict the patient's facial 3D data, including:
[0029] (1) Convolutional Neural Network (CNN): CNN is used to process 3D image data and analyze and predict the patient’s facial 3D data. CNN can automatically identify and learn complex features in image data, significantly improving the accuracy of prediction.
[0030] (2) Generative Adversarial Network (GAN): GAN is used to generate prediction results of postoperative facial morphology. GAN can generate high-quality prediction images through adversarial training of the generator and the discriminator.
[0031] This type of method can automatically identify and learn complex nonlinear relationships, significantly improving the accuracy and efficiency of predictions. It mainly includes the following steps:
[0032] 1) Collect a large amount of patients’ pre- and post-operative 3D images and facial shape data to establish a training data set.
[0033] 2) Train machine learning or deep learning models to automatically learn the correspondence between pre- and post-operative images and facial shape.
[0034] 3) Use the trained model to predict facial shape, input the patient's preoperative 3D image and facial shape, and output the predicted postoperative facial shape.
[0035] AI-based forecasting methods have the following advantages:
[0036] 1) Ability to process large amounts of complex image data and improve prediction accuracy.
[0037] 2) Once trained, the model has fast prediction speed and can achieve real-time prediction.
[0038] 3) The model learns automatically, reducing dependence on doctor experience.
[0039] Although AI-based prediction methods have shown great potential in orthognathic surgery, there are still some obvious deficiencies and challenges: First, the prediction accuracy of AI models depends on a large amount of high-quality training data. Insufficient data or low data quality will affect the prediction effect. Second, the training and reasoning process of deep learning models is complex and requires a lot of computing resources, resulting in poor real-time performance. Finally, due to the "black box" characteristics of AI models, especially deep learning models, it is difficult for doctors to understand and interpret the prediction results, which affects clinical decision-making.
[0040] Therefore, existing methods have various limitations. Specifically, the limitations of prediction methods based on two-dimensional images: 1) Insufficient accuracy: Two-dimensional images cannot accurately reflect the relationship between bones and soft tissues in three-dimensional space, resulting in low accuracy of prediction results; 2) Dependence on doctor experience: The prediction results are highly dependent on the doctor's experience and technical level, are highly subjective, and lack standardization. Limitations of prediction methods based on three-dimensional images: 1) Computational complexity: The processing of three-dimensional image data and the construction of three-dimensional models are computationally complex and time-consuming, affecting the convenience of clinical application; 2) Insufficient soft tissue prediction accuracy: Existing three-dimensional image prediction methods still have large errors in predicting the morphological changes of soft tissues, and cannot fully and accurately simulate postoperative facial morphology; 3) High cost: The cost of related software is high, which limits its popularization and application in some medical institutions. Limitations of AI-based methods: 1) Strong data dependence: Data quality and quantity: The prediction accuracy of AI models is highly dependent on a large amount of high-quality training data; if the data is insufficient or the data quality is not high, it will significantly affect the prediction effect; Complex data annotation: The annotation process of 3D medical imaging data is complex and time-consuming, requiring a lot of expertise and human resources; 2) Model complexity: High computing resource requirements: Deep learning model training and reasoning processes require a lot of computing resources, resulting in poor real-time performance in medical environments; Difficulty in tuning model parameters: Deep learning models have a large number of parameters, and the tuning process is complex, requiring a lot of experiments and debugging.
[0041] In summary, the existing technical solutions have made some progress in predicting the facial shape of orthognathic surgery or the effect of cosmetic surgery, but there are still many shortcomings. It is urgent to develop a solution that can improve prediction accuracy, shorten processing time, reduce data dependence, simplify operation procedures and is economically feasible. Summary of the invention
[0042] To solve the existing problems, the present invention provides a facial deformation prediction method and device, a terminal and a storage medium.
[0043] The present disclosure adopts the following technical solutions.
[0044] An embodiment of the present disclosure provides a facial deformation prediction method, which includes: acquiring three-dimensional facial data and a three-dimensional facial data set; using the three-dimensional facial data set to construct a three-dimensional shape prediction model; using the three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face; using a facial texture correction model to correct the facial texture based on the three-dimensional facial data to obtain a corrected facial texture; and obtaining a predicted facial morphology based on the three-dimensional facial shape and the corrected facial texture.
[0045] Another embodiment of the present disclosure provides a facial deformation prediction device, which includes: a data acquisition module, configured to acquire three-dimensional facial data and a three-dimensional facial data set; a model construction module, configured to use the three-dimensional facial data set to construct a three-dimensional shape prediction model; a shape prediction module, configured to use the three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face; a texture correction module, configured to use the facial texture correction model to perform facial texture correction based on the three-dimensional facial data to obtain a facial correction texture; and a rendering module, configured to obtain a predicted facial morphology based on the three-dimensional facial shape and the facial correction texture.
[0046] Existing artificial intelligence face shape prediction methods rely on a large amount of high-quality training data. For many medical institutions, it is very difficult and time-consuming to obtain and annotate a large amount of three-dimensional medical imaging data, which limits the widespread application of these technologies. The present invention is based on a three-dimensional variable model (3DMM) combined with a neural network algorithm, which can achieve high-precision face shape prediction under small sample conditions, effectively solving the problem of strong data dependence.
[0047] Existing facial shape prediction methods lack consideration of the individual surgical styles and experiences of different surgeons, and the prediction results are relatively uniform and lack personalization. The present disclosure can achieve personalized predictions for different surgeons. By studying small sample patient data of different surgeons, a personalized prediction model is established to make the prediction results more consistent with the actual operation style and experience of specific doctors, improve the personalization and practicality of the prediction, and provide technical support for patients to choose doctors with "consistent aesthetics", which helps to improve patients' satisfaction with their appearance after orthognathic surgery.
[0048] Traditional prediction methods mainly infer the changes of postoperative soft tissue based on the bone tissue movement scheme. This method has limitations when dealing with complex soft tissue morphological changes, resulting in large prediction errors. The present invention adopts the idea of learning the changes of soft tissue before and after surgery, and directly predicts based on the changes of soft tissue, thereby significantly improving the accuracy of the prediction and simplifying the prediction process.
[0049] Traditional two-dimensional and three-dimensional image prediction methods have deficiencies in accuracy and processing speed. It is difficult to balance prediction accuracy and prediction efficiency. Most of them do not carry skin texture, lack simulation, and have limited applications. The present disclosure combines 3DMM and neural network algorithms to more accurately simulate soft tissue changes in orthognathic surgery, and performs facial texture correction through CNN, significantly improving prediction accuracy and prediction effect. At the same time, the efficient computing power of the neural network algorithm greatly improves the prediction speed, making real-time prediction and rapid response possible.
[0050] In summary, the present invention aims to solve the problems existing in the existing orthognathic facial shape prediction technology, such as high data dependence, insufficient prediction personalization, limitations of prediction methods based on bone tissue movement, and insufficient prediction accuracy and efficiency through the above-mentioned technical solutions, and to provide an efficient, accurate, personalized soft tissue-oriented facial shape prediction method for orthognathic surgery or other cosmetic surgeries, thereby improving the success rate of the operation and patient satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.
[0052] Figure 1 4 is a flowchart of a facial deformation prediction method according to an embodiment of the present disclosure.
[0053] Figure 2 The overall architecture of the prediction model of some embodiments of the present disclosure is shown.
[0054] Figure 3 An architectural diagram of a three-dimensional shape prediction model according to some embodiments of the present disclosure is shown.
[0055] Figure 4 An architectural diagram of a facial texture correction model according to some embodiments of the present disclosure is shown.
[0056] Figure 5 Partial modules of a facial deformation prediction device according to another embodiment of the present disclosure are shown. DETAILED DESCRIPTION
[0057] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0058] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in sequence and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0059] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0060] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0061] It should be noted that the modification of “one” mentioned in the present disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as “one or more” unless otherwise clearly indicated in the context.
[0062] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0063] Figure 1 A flowchart of a facial deformation prediction method according to an embodiment of the present disclosure is provided. Figure 1 and Figure 2 As shown, the present disclosure uses a three-dimensional shape prediction model to predict the postoperative facial shape, and uses a facial texture correction model to remove the impact of lighting to make its brightness consistent with the geometric structure.
[0064] The facial deformation prediction method disclosed herein may include step S101, obtaining three-dimensional facial data and a three-dimensional facial data set. In some embodiments, the three-dimensional facial data may be obtained in various ways, for example, by taking a two-dimensional face image and generating the three-dimensional facial data through image processing. In some embodiments, the three-dimensional facial data set may be obtained from an existing three-dimensional deformable model 3DMM.
[0065] In some embodiments, the method of the present disclosure may further include step S102, using a three-dimensional facial data set to construct a three-dimensional shape prediction model. The present disclosure uses an existing three-dimensional facial data set to construct a three-dimensional shape prediction model for shape prediction. In some embodiments, using the three-dimensional facial data set to construct a three-dimensional shape prediction model includes: detecting facial key points based on the three-dimensional facial data set; matching facial key points between faces in the three-dimensional facial data set; using non-rigid iterative closest points to perform dense point-to-point matching of the facial model, and representing each vertex that has been matched by the model as a shape vector; based on the mean of each shape vector corresponding to the three-dimensional facial data set and the orthogonal basis obtained by applying principal component analysis, generating a facial three-dimensional shape by weighted summation as a decoder of the three-dimensional shape prediction model; based on the mean of each shape vector corresponding to the three-dimensional facial data set and the orthogonal basis obtained by applying principal component analysis, solving the weighted sum of the orthogonal basis to fit the sum weight, rotation amount and translation amount corresponding to the three-dimensional facial data, as an encoder of the three-dimensional shape prediction model.
[0066] In some embodiments, the method of the present disclosure may further include step S103, using a 3D shape prediction model to predict the 3D shape of the face based on the 3D facial data to obtain the 3D shape of the face. In some embodiments, using a 3D shape prediction model to predict the 3D shape of the face based on the 3D facial data to obtain the 3D shape of the face includes: encoding the shape vector corresponding to the 3D facial data by an encoder; predicting the postoperative 3D facial data encoding from the encoding result of the 3D facial data; and decoding the predicted postoperative 3D facial data encoding into a shape vector by a decoder.
[0067] In some embodiments, in order to predict the postoperative three-dimensional shape of the face under the premise of a small sample, the present disclosure uses a three-dimensional deformable model (3DMM) established on a larger-scale ordinary person three-dimensional face dataset as an encoder and decoder. First, a face feature extractor is used to detect facial key points, such as the corners of the eyes and the tip of the nose, and they are matched between the faces in the dataset to provide initial values for registration. Next, a non-rigid iterative closest point (ICP) is used to achieve dense point-to-point matching of the facial model, and the vertices of each model that have been matched are matched. Represented as a vector of the following shape:
[0068] S=(x1,y1,z1,...,x n ,y n , z n ) T (1)
[0069] The mean of each shape vector corresponding to the data set and the N orthogonal bases obtained by applying principal component analysis (PCA) That is, as a decoder, it uses a weighted sum to obtain a set of weights as a low-dimensional latent code Generate a new 3D face shape:
[0070]
[0071] To realize any point cloud The encoding of this disclosure is for the energy function:
[0072] E(α, R, t) = Σε(p i (α), M(p i (α)), R, t) (3)
[0073] Optimize. i (α) is the i-th point in the point cloud corresponding to S(α), q=M(p i (α)) is the corresponding point matched in P. Since there is a pose difference between the 3DMM shape vector and the fitting object, the rotation R and translation t are also solved.
[0074] In some embodiments, the method disclosed herein may further include step S104, using a facial texture correction model to perform facial texture correction based on three-dimensional facial data to obtain facial correction texture. Facial texture correction uses a convolutional neural network (CNN) to eliminate the effects of illumination and shadow, thereby avoiding the inconsistency between illumination and the predicted three-dimensional shape. This technical solution significantly improves the accuracy of face shape prediction, reduces prediction errors, and makes the prediction results more realistic and reliable.
[0075] In some embodiments, the method of the present disclosure may further include step S105, rendering to obtain a predicted facial morphology based on the facial three-dimensional shape and the facial correction texture. In some embodiments, after obtaining the facial three-dimensional shape through shape prediction and obtaining the facial correction texture through facial texture correction, the predicted facial morphology may be obtained by rendering.
[0076] The present disclosure predicts the three-dimensional shape of the face based on the three-dimensional facial data by using a three-dimensional shape prediction model to obtain the three-dimensional shape of the face; and corrects the facial texture based on the three-dimensional facial data by using a facial texture correction model to obtain the corrected facial texture; the facial texture correction will eliminate the influence of illumination and shadow, thereby avoiding the inconsistency between illumination and the predicted three-dimensional shape. The technical solution of the present disclosure significantly improves the accuracy of face shape prediction, reduces the prediction error, and makes the prediction result more realistic and reliable. Traditional two-dimensional and three-dimensional image prediction methods have deficiencies in accuracy and processing speed, and it is difficult to take into account both the accuracy and efficiency of prediction at the same time. The present disclosure combines an efficient neural network algorithm, has powerful computing power and efficient learning ability, and can quickly process a large amount of complex data. The present disclosure not only improves the accuracy of face shape prediction, but also greatly improves the prediction speed, making real-time prediction and rapid response possible, and improving the efficiency and effect of clinical application.
[0077] In some embodiments, using a 3D shape prediction model to predict the 3D shape of a face based on 3D facial data, obtaining the 3D shape of the face further includes: weight prediction and basis vector correction of the encoding result of the 3D facial data, wherein when correcting the basis vector, the weight is set as the correction amount according to the Euclidean distance between the subset corresponding to each vertex of the model in the basis vector and the same vertex in the shape vector mean. Figure 3 As shown, the present invention uses the established 3DMM as the encoder and decoder, uses a small-scale multi-layer perceptron (MLP) to transform the potential code, and the basis vector of the decoder 3DMM Make corrections. Figure 3 As shown, the network will use two branches of MLP to perform weight prediction and basis vector correction respectively. Attention mask will be based on V decode The subset corresponding to each vertex of the model With shape vector mean Same Vertex The Euclidean distance between them sets the weight for its correction, so that for each basis vector v i The deformation area caused by the correction is constrained to v i The deformation area generated by the model itself is avoided to avoid excessive correlation between the weights of 3DMM.
[0078] In some embodiments, the 3D shape prediction model is trained by prediction loss, smoothing loss, and regularization loss, wherein the prediction loss is the error between the shape vector obtained by the decoder and the shape vector corresponding to the reference true value, that is, the shape vector S obtained by the decoder is calculated. pred The shape vector S corresponding to the ground truth after surgery g The error between:
[0079] L pred =∑ε p (S pred , S g , Φ) (4)
[0080] Where Φ represents different facial regions, such as the paranasal area, lips, chin, etc., which are affected by the surgery and the unaffected areas. The smoothness loss constrains the smoothness of the additional deformation caused by the basis vector correction in its neighborhood to avoid sudden changes in facial shape caused by this branch:
[0081]
[0082] Where Δv i，j is the basis vector v iThe correction amount corresponding to the upper vertex j, κ(i, j) is the neighborhood index set of vertex j. The regularization loss uses the L2 norm regularization L reg , to avoid overfitting.
[0083] In some embodiments, the three-dimensional facial data includes facial texture data. Figure 4 As shown, facial texture correction is performed based on three-dimensional facial data using a facial texture correction model to obtain facial correction texture, including: inputting the facial texture data and normal vector after posture normalization into a facial texture correction model (e.g., a convolutional neural network CNN) to obtain facial correction texture. The present disclosure maps facial texture to a two-dimensional UV space, thereby using CNN to correct the effects of illumination and three-dimensional geometry on the light and shade of the collected patient's facial texture. Figure 4 As shown in Figure 1, the model input is the facial texture data after pose normalization and the UV image corresponding to the normal vector. An encoder-decoder structure is used, and the adversarial loss is obtained by the CNN discriminator D.
[0084] In some embodiments, the facial texture correction model is trained by correction loss and adversarial loss, where the correction loss is the error between the facial correction texture and the reference true value without illumination influence, and the adversarial loss is obtained by a convolutional neural network discriminator based on the facial correction texture and the reference true value without illumination influence. The correction loss is a loss function based on L1 and L2 losses:
[0085] L corr =ε corr (I corr , I g ) (6)
[0086] Calculate the corrected texture I for output corr and the ground truth without illumination effect I g The error between them is used as the basic constraint.
[0087] Since only applying the loss function based on L1 and L2 losses may cause the network to tend to learn the mean of the distribution, causing the output texture image to lack high-frequency information, this disclosure uses a CNN discriminator D to introduce adversarial loss:
[0088]
[0089] In some embodiments, the training data set of the facial texture correction model is obtained by the following steps: the color of each vertex of the matched three-dimensional face is represented by a texture vector; an orthogonal basis is obtained from the texture vector corresponding to the training data set, the influence of lighting is simulated by a lighting model, and then coloring is performed to generate a training data set in which the texture brightness changes with the light source. In view of the current lack of three-dimensional facial data sets under different lighting conditions, especially the lack of data sets with reference texture information that removes the influence of lighting, the present disclosure uses the generated three-dimensional face and uses the lighting model to simulate different conditions to create a data set for training the required model. Among them, the three-dimensional shape and texture of the face are generated using a 3DMM with a texture basis vector. That is, the color of each vertex of the face model that has been densely matched point to point is Also expressed as a texture vector:
[0090] T=(r1,g1,b1,...,r n , g n , b n ) T (8) And get N orthogonal basis from the texture vector corresponding to the data set The influence of lighting is simulated using, for example, the Blinn-Phong reflection model, and then shaded to generate a data set in which the brightness of the texture changes with the light source.
[0091] Traditional face shape prediction methods rely on a large amount of high-quality training data, which is difficult to achieve in many medical institutions. The present disclosure is based on a three-dimensional variable model (3DMM) combined with a neural network algorithm, which can achieve high-precision face shape prediction under small sample conditions. By reducing the reliance on a large amount of data, the present disclosure significantly improves the applicability of face shape prediction, allowing more medical institutions to apply the technology, thereby expanding the popularity of face shape prediction technology.
[0092] Existing face shape prediction methods lack consideration of the individual surgical styles and experiences of different surgeons, and the prediction results lack personalization. The present invention can learn small sample patient data of different surgeons through the algorithm conversion module, establish a personalized prediction model, and make the prediction results more consistent with the actual operation style and experience of specific doctors. This personalized prediction increases the possibility of patients choosing doctors with "consistent aesthetics", helps improve patients' postoperative satisfaction, and increases the predictability of surgical results.
[0093] Traditional prediction methods mainly infer the changes of postoperative soft tissue based on the bone tissue movement scheme. This method has limitations when dealing with complex soft tissue morphological changes, resulting in large prediction errors. The facial three-dimensional shape prediction step disclosed in the present invention only involves the facial three-dimensional shape (i.e., the changes in soft tissue), so that predictions are made directly based on soft tissue changes, which can more accurately reflect the morphological changes of postoperative facial soft tissue and avoid the limitations of the bone tissue movement scheme when dealing with complex soft tissue changes. At the same time, in order to make the prediction results more realistic, facial texture correction will use CNN to eliminate the influence of lighting and shadows, thereby avoiding the inconsistency between lighting and the predicted three-dimensional shape. This technical solution significantly improves the accuracy of facial shape prediction, reduces prediction errors, and makes the prediction results more realistic and reliable.
[0094] Traditional two-dimensional and three-dimensional image prediction methods have deficiencies in accuracy and processing speed, and it is difficult to take into account both the accuracy and efficiency of prediction. The present disclosure combines an efficient neural network algorithm with powerful computing power and efficient learning ability, and can quickly process large amounts of complex data. The present disclosure not only improves the accuracy of facial shape prediction, but also greatly improves the prediction speed, making real-time prediction and rapid response possible, and improving the efficiency and effectiveness of clinical applications.
[0095] It should be understood that all steps of the method disclosed in the present invention are performed by a computer. The present invention combines the small-scale characteristics of the corresponding data set before and after the patient's surgery to establish a neural network model for predicting the three-dimensional shape of the patient's postoperative face. A three-dimensional variable model (3DMM) is used as the encoder and decoder of the patient's three-dimensional face, and a neural network is used to predict the changes in the potential code obtained by the 3DMM after surgery, so as to obtain the expected three-dimensional shape after surgery. In addition, the present invention can learn small sample patient data from different surgeons and establish a personalized prediction model so that the prediction results are more in line with the surgical style and experience of the specific doctor. The present invention directly learns and predicts the changes in soft tissue before and after surgery, improves the accuracy of prediction, and avoids the limitations of traditional bone tissue movement schemes. In addition, the present invention maps the preoperative texture to a two-dimensional UV space, and uses CNN to correct the influence of illumination, so that it is consistent with the geometric shape, improves the accuracy of migration, and avoids the "misalignment" phenomenon between the facial structure.
[0096] The embodiment of the present disclosure also provides a facial deformation prediction device 400 . Figure 5A facial deformation prediction device 400 according to some embodiments is shown. The facial deformation prediction device 400 includes a data acquisition module 401, a shape prediction module 402, a texture correction module 403, and a rendering module 404. In some embodiments, the data acquisition module 401 is configured to acquire three-dimensional facial data and a three-dimensional facial data set. In some embodiments, the model construction module 402 is configured to construct a three-dimensional shape prediction model using the three-dimensional facial data set. In some embodiments, the shape prediction module 403 is configured to perform facial three-dimensional shape prediction based on the three-dimensional facial data using the three-dimensional shape prediction model to obtain a facial three-dimensional shape. In some embodiments, the texture correction module 404 is configured to perform facial texture correction based on the three-dimensional facial data using the facial texture correction model to obtain a facial correction texture. In some embodiments, the rendering module 405 is configured to obtain a predicted facial morphology based on the facial three-dimensional shape and the facial correction texture.
[0097] It should be understood that the contents described about the facial deformation prediction method are also applicable to the facial deformation prediction device 400 herein, and for the purpose of simplicity, a detailed description is not given here.
[0098] In some embodiments, constructing a 3D shape prediction model using a 3D facial dataset includes: detecting facial key points based on the 3D facial dataset; matching facial key points between faces in the 3D facial dataset; performing dense point-to-point matching of the facial model using non-rigid iterative closest point, and representing each vertex that has been matched by the model as a shape vector; generating a facial 3D shape by weighted summation based on the mean of each shape vector corresponding to the 3D facial dataset and the orthogonal basis obtained by applying principal component analysis, as a decoder of the 3D shape prediction model; solving the weighted sum of the orthogonal basis fitting to the sum weight, rotation amount and translation amount corresponding to the 3D facial data based on the mean of each shape vector corresponding to the 3D facial dataset and the orthogonal basis obtained by applying principal component analysis, as an encoder of the 3D shape prediction model. In some embodiments, using a 3D shape prediction model to predict the facial 3D shape based on the 3D facial data to obtain the facial 3D shape includes: encoding the shape vector corresponding to the 3D facial data by an encoder; predicting the encoding of the 3D facial data after the encoding result of the 3D facial data; and decoding the 3D facial data after the prediction into a shape vector by a decoder. In some embodiments, 3DMM is used as an encoder and decoder, and the sum weight of the 3DMM orthogonal basis is solved by iterative optimization to perform encoding of the shape vector, and the weighted sum of the 3DMM orthogonal basis is used to perform decoding from the weight vector to the shape vector. In some embodiments, using a three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data, and obtaining the three-dimensional shape of the face also includes: weight prediction and basis vector correction of the encoding result of the three-dimensional facial data, wherein, when correcting the basis vector, the weight is set as the correction amount according to the Euclidean distance between the subset corresponding to each vertex of the model in the basis vector and the same vertex in the mean of the shape vector. In some embodiments, the three-dimensional shape prediction model is trained by prediction loss, smoothing loss and regularization loss, wherein the prediction loss is the error between the shape vector obtained by the decoder and the shape vector corresponding to the reference true value. In some embodiments, the three-dimensional facial data includes facial texture data, and using a facial texture correction model to correct the facial texture based on the three-dimensional facial data, and obtaining the facial correction texture includes: inputting the facial texture data and the normal vector after posture normalization into the facial texture correction model to obtain the facial correction texture. In some embodiments, the facial texture correction model is trained by correction loss and adversarial loss, the correction loss is the error between the facial correction texture and the reference true value without the influence of lighting, and the adversarial loss is obtained by the convolutional neural network discriminator based on the facial correction texture and the reference true value without the influence of lighting. In some embodiments, the training data set of the facial texture correction model is obtained by the following steps: the color of each vertex of the matched three-dimensional face is represented by a texture vector; an orthogonal basis is obtained from the texture vector corresponding to the training data set, the influence of lighting is simulated by a lighting model, and then coloring is performed to generate a training data set whose texture brightness changes with the light source.
[0099] According to one or more embodiments of the present disclosure, a facial deformation prediction method is provided, which includes: acquiring three-dimensional facial data and a three-dimensional facial data set; constructing a three-dimensional shape prediction model using the three-dimensional facial data set, and using the three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face; using a facial texture correction model to correct the facial texture based on the three-dimensional facial data to obtain a corrected facial texture; and obtaining a predicted facial morphology based on the three-dimensional facial shape and the corrected facial texture.
[0100] According to one or more embodiments of the present disclosure, constructing a three-dimensional shape prediction model using the three-dimensional facial dataset includes: matching the facial key points between the faces of the three-dimensional facial dataset; using a non-rigid iterative nearest point to perform dense point-to-point matching of the facial model, and representing each matched vertex of the model with a shape vector; based on the mean of each shape vector corresponding to the dataset and an orthogonal basis obtained by applying principal component analysis, a facial three-dimensional shape can be generated by weighted summation as a decoder of the three-dimensional shape prediction model; based on the mean of each shape vector corresponding to the dataset and an orthogonal basis obtained by applying principal component analysis, a solution is obtained to fit the weighted sum of the orthogonal basis to the sum weights, rotation amounts, and translation amounts corresponding to the three-dimensional facial data, as an encoder of the three-dimensional shape prediction model.
[0101] According to one or more embodiments of the present disclosure, a three-dimensional shape prediction model is used to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face, including: an encoder encoding a shape vector corresponding to the three-dimensional facial data; predicting a post-operative three-dimensional facial data code from the encoding result of the three-dimensional facial data; and a decoder decoding the predicted post-operative three-dimensional facial data code into a shape vector.
[0102] According to one or more embodiments of the present disclosure, the three-dimensional shape prediction model is trained by prediction loss, smoothing loss and regularization loss, wherein the prediction loss is the error between the shape vector obtained by calculating the decoder and the shape vector corresponding to the reference true value.
[0103] According to one or more embodiments of the present disclosure, the three-dimensional facial data includes facial texture data, and facial texture correction is performed based on the three-dimensional facial data using a facial texture correction model to obtain the facial correction texture, including: inputting the facial texture data and normal vector after pose normalization into the facial texture correction model to obtain the facial correction texture.
[0104] According to one or more embodiments of the present disclosure, the facial texture correction model is trained by correction loss and adversarial loss, wherein the correction loss is the error between the facial correction texture and the reference true value without illumination influence, and the adversarial loss is obtained by a convolutional neural network discriminator based on the facial correction texture and the reference true value without illumination influence.
[0105] According to one or more embodiments of the present disclosure, the training data set of the facial texture correction model is obtained by the following steps: the color of each vertex of the matched three-dimensional face is represented by a texture vector; an orthogonal basis is obtained from the texture vector corresponding to the training data set, the influence of lighting is simulated using a lighting model, and then coloring is performed to generate a training data set in which the texture brightness changes with the light source.
[0106] According to one or more embodiments of the present disclosure, a facial deformation prediction device is provided, which includes: a data acquisition module, configured to acquire three-dimensional facial data; a shape prediction module, configured to use a three-dimensional shape prediction model to perform facial three-dimensional shape prediction based on the three-dimensional facial data to obtain a facial three-dimensional shape; a texture correction module, configured to use a facial texture correction model to perform facial texture correction based on the three-dimensional facial data to obtain a facial correction texture; and a rendering module, configured to obtain a predicted facial morphology based on the facial three-dimensional shape and the facial correction texture.
[0107] According to one or more embodiments of the present disclosure, a terminal is provided, comprising: at least one memory and at least one processor; wherein the at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute any one of the methods described above.
[0108] According to one or more embodiments of the present disclosure, a storage medium is provided, wherein the storage medium is used to store program code, and the program code is used to execute the above method.
[0109] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0110] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0111] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A facial deformation prediction method, characterized in that: The facial deformation prediction method comprises: Acquire three-dimensional facial data and a three-dimensional facial dataset; constructing a three-dimensional shape prediction model using the three-dimensional facial data set; Using the three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face; Using a facial texture correction model to perform facial texture correction based on the three-dimensional facial data to obtain a facial correction texture; Based on the facial three-dimensional shape and the facial correction texture, a predicted facial morphology is obtained.
2. The facial deformation prediction method according to claim 1, characterized in that: Using the three-dimensional facial data set to construct a three-dimensional shape prediction model includes: Detecting facial key points based on the three-dimensional facial dataset; Matching the facial key points between faces in the three-dimensional facial data set; Perform dense point-to-point matching of the facial model using non-rigid iterative closest point, representing each matched vertex of the model as a shape vector; Based on the mean of each shape vector corresponding to the three-dimensional facial data set and the orthogonal basis obtained by applying principal component analysis, a facial three-dimensional shape is generated by weighted summation as a decoder of the three-dimensional shape prediction model; Based on the mean of each shape vector corresponding to the three-dimensional facial data set and the orthogonal basis obtained by applying principal component analysis, a weighted sum of the orthogonal basis is fitted to the sum weight, rotation amount and translation amount corresponding to the three-dimensional facial data as an encoder of the three-dimensional shape prediction model.
3. The facial deformation prediction method according to claim 2, characterized in that: The three-dimensional shape prediction model is used to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face, including: The encoder encodes the shape vector corresponding to the three-dimensional facial data; predicting the postoperative three-dimensional facial data encoding from the encoding result of the three-dimensional facial data; The decoder encodes and decodes the predicted post-operative three-dimensional facial data into a shape vector.
4. The facial deformation prediction method according to claim 3, characterized in that: Using a three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face also includes: The encoding result of the three-dimensional facial data is weighted predicted and basis vector corrected, wherein when correcting the basis vector, a weight is set as a correction amount according to the Euclidean distance between a subset corresponding to each vertex of the model in the basis vector and the same vertex in the shape vector mean.
5. The facial deformation prediction method according to claim 1, characterized in that: The three-dimensional shape prediction model is trained by prediction loss, smoothing loss and regularization loss, wherein the prediction loss is the error between the shape vector obtained by the decoder and the shape vector corresponding to the reference true value.
6. The facial deformation prediction method according to claim 1, characterized in that: The three-dimensional facial data includes facial texture data, Performing facial texture correction based on the three-dimensional facial data using a facial texture correction model to obtain facial correction texture includes: The facial texture data and normal vector after posture normalization are input into the facial texture correction model to obtain the facial correction texture.
7. The facial deformation prediction method according to claim 1, characterized in that: The facial texture correction model is trained by correction loss and adversarial loss, wherein the correction loss is the error between the facial correction texture and the reference true value without illumination influence, and the adversarial loss is obtained by a convolutional neural network discriminator based on the facial correction texture and the reference true value without illumination influence.
8. The facial deformation prediction method according to claim 1, characterized in that: The training data set of the facial texture correction model is obtained by the following steps: The color of each vertex of the matched three-dimensional face is represented by a texture vector; An orthogonal basis is obtained from the texture vectors corresponding to the training data set, and the influence of lighting is simulated using a lighting model, followed by shading to generate a training data set in which the texture brightness changes with the light source.
9. A facial deformation prediction device, characterized in that: The facial deformation prediction device comprises: A data acquisition module configured to acquire three-dimensional facial data and a three-dimensional facial data set; A model building module, configured to build a three-dimensional shape prediction model using the three-dimensional facial data set; a shape prediction module, configured to use the three-dimensional shape prediction model to predict the three-dimensional shape of the face based on the three-dimensional facial data to obtain the three-dimensional shape of the face; A texture correction module is configured to perform facial texture correction based on the three-dimensional facial data using a facial texture correction model to obtain a facial correction texture; The rendering module is configured to obtain a predicted facial morphology based on the facial three-dimensional shape and the facial correction texture.