Human face type prediction method, display method, electronic equipment and medium

Abstract

The embodiment of the invention relates to the technical field of medical instrument digital design, and discloses a human face type prediction method, a display method, electronic equipment and a medium, and the human face type prediction method comprises the steps: obtaining a human face composite model of a patient in a first state, the human face composite model comprising a tooth jaw part and a soft tissue part; identifying pre-selected tooth-jaw feature points on the tooth-jaw part of the face composite model, and obtaining first positions of the tooth-jaw feature points; based on the correction scheme, obtaining a second position of a tooth jaw feature point of the patient in a second state in the face composite model; according to the first position and the second position of the tooth jaw feature point, a deformation method is adopted to calculate the change relation between the movement of the tooth jaw feature point and the grid vertex in the face composite model; and giving a target position of a tooth jaw part in the face composite model, and obtaining a new position of a grid vertex of a soft tissue part in the face composite model based on the change relation. The change of the human face shape after oral correction can be accurately predicted.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of digital design of medical devices, and particularly to a method for predicting human face shapes, a display method, an electronic device, and a medium. Background Technique

[0002] Orthodontics is a comprehensive treatment method for tooth and oral and maxillofacial deformities, mainly aiming at correcting tooth and oral and maxillofacial deformities. Its main purpose is to treat abnormal conditions such as irregular teeth, tooth gaps, protruding teeth, and jaw development. Through orthodontic treatment, teeth can become complete and beautiful, achieving the purpose of coordination and balance of the dentoalveolar system. In theory, the effect of orthodontics is significant. It can not only restore chewing function, improve oral hygiene, restore facial beauty, but also correct problems such as unclear pronunciation caused by abnormal development, and improve learning and work efficiency. Due to the long cycle, high cost, and high medical professionalism requirements of orthodontics, it is very difficult for patients to judge whether the treatment plan is appropriate in the early stage, or whether the treatment effect meets their expectations. As a result, some users cannot obtain satisfactory treatment effects after spending time and money, and there is a large sense of difference in expectations.

[0003] Among the above factors affecting patients' expectations, the aesthetics after orthodontics is a highly subjective consideration parameter. At present, the prediction of the dental and maxillofacial part in this factor has been carried out through technologies such as three-dimensional model reconstruction and digital design of treatment plans, and can be visually displayed through models, providing reliable reference bases for patients, doctors, or professionals. However, the appearance aesthetics is mainly based on soft tissues, and users cannot perceive the appearance changes only through dental and maxillofacial models. Therefore, we need a method that can accurately predict the changes in human face shapes to provide richer reference bases for patients, doctors, or other professionals.

[0004] In the present application, "jaw" in "malocclusion", "open bite", "overbite", "occlusal plane", etc. is "occlusion" (hé). This character is a rare character, and since it is not in the general input method character library, it is often written as "jaw" in daily life and on the Internet. For the convenience of reading in this article, it is also written as "jaw". Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method for predicting human face shapes, a display method, an electronic device, and a medium, which can accurately predict the changes in human face shapes, with a fast prediction method, reliable results, and strong generalizability.

[0006] To address the aforementioned technical problems, embodiments of this application provide a facial shape prediction method, comprising: acquiring a composite facial model of a patient in a first state, the composite facial model comprising: a dental and jaw portion and a soft tissue portion; identifying pre-selected dental and jaw feature points on the dental and jaw portion of the composite facial model, and acquiring the first position of the dental and jaw feature points; acquiring the second position of the dental and jaw feature points in the composite facial model in a second state of the patient based on a treatment plan; calculating the relationship between the movement of the dental and jaw feature points and the change of mesh vertices in the composite facial model using a deformation method based on the first and second positions of the dental and jaw feature points; and, given a target position of the dental and jaw portion in the composite facial model, obtaining the new position of the mesh vertices of the soft tissue portion in the composite facial model based on the change relationship.

[0007] An embodiment of this application also provides a method for displaying facial features, including: obtaining a facial orthodontic prediction model based on the above-described facial feature prediction method; and displaying the facial orthodontic prediction model on a display device.

[0008] Embodiments of this application also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described face shape prediction method, or to perform the face shape display method as described above.

[0009] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described face shape prediction method or is capable of executing the above-described face shape display method.

[0010] The facial shape prediction method in this application obtains a composite facial model including the dental and jaw parts and soft tissue parts of the associated face, determines the correlation between the dental and jaw parts and soft tissue parts, and then determines the change relationship of the facial shape by using the known changes of the dental and jaw parts in the orthodontic plan. Thus, the positional changes of other parts are predicted based on the change relationship. Since the prediction can be made using only the patient's own data, there is no need to combine a large amount of historical data to infer the change relationship. Therefore, the facial shape prediction method in this application involves a small amount of data in the prediction calculation and has high accuracy.

[0011] In some embodiments, obtaining the composite face model of the patient in the first state includes: obtaining a registration image; wherein the registration image is a composite image formed by fusing a lateral view of the patient's face and a profile photograph, or a lateral view of the patient's face; obtaining a face mesh model and a dental model respectively; and registering and fusing the face mesh model and the dental model based on the registration image to form the composite face model. This embodiment forms a composite face model by registering and fusing a face mesh model and a dental model, resulting in a composite face model that possesses associated dental and jaw parts and soft tissue parts representing appearance. Since the methods for obtaining the face mesh model and the dental model are relatively simple, the generation of the composite face model in this embodiment is reliable and easily promoted, while also reducing the difficulty of data collection at the terminal. This embodiment further uses a lateral view of the face that simultaneously possesses dental and jaw features and some facial contour features, and a profile photograph that possesses facial contour features, for registration and fusing to form a composite image including rich dental and jaw features and facial contour features. This composite image is used as a registration reference for the three-dimensional model, allowing for a wide range of selectable parameters during registration, resulting in more accurate registration results. In addition, this embodiment can directly use lateral facial images for model registration, which is simple and fast.

[0012] In some embodiments, fusing a patient's lateral facial radiograph and profile view photograph to form a composite image includes: identifying pre-selected facial contour feature points on the acquired lateral facial radiograph and profile view photograph; and performing registration and fusion based on the positions of the facial contour feature points to form the composite image. In this embodiment, registration is performed by acquiring facial contour feature points from various types of images. Since the lateral facial radiograph simultaneously contains the jawbone and part of the facial contour, the fusion is highly operable, and the resulting composite image has rich feature information and more selectable feature points, facilitating subsequent 3D model registration.

[0013] In some embodiments, the pre-selected facial contour feature points include: the tip of the nose and the protrusions of the upper and lower lips. In this embodiment, the tip of the nose and the protrusions of the upper and lower lips are used as facial contour feature points for registration. These points have obvious features and are more accurately identified during acquisition, so as to ensure the accuracy of registration in subsequent registration.

[0014] In some embodiments, the registration and fusion of the face mesh model and the dental model based on the registration image to form the face composite model includes: identifying pre-selected facial contour feature points on the face mesh model; identifying dental feature points on the dental model; and registering and fusing the face mesh model and the dental model according to the relationship between the pre-selected dental feature points, the facial contour feature points, and the registration image to form the face composite model.

[0015] In some embodiments, the face mesh model is obtained by: stitching and fusing multi-angle facial photos of the patient; or by scanning the patient's face. This embodiment clarifies multiple methods for obtaining the face mesh model, allowing for the selection of different methods to meet varying needs and expanding the application scenarios of this application.

[0016] In some embodiments, the step of obtaining the face model by stitching and fusing multi-angle facial photos of the patient includes: removing the background from the multi-angle facial photos; performing checkerboard pattern processing on the multi-angle facial photos to achieve calibration and stereo correction, and obtaining camera intrinsic parameters; using the parallax information of the multi-angle facial photos and the camera intrinsic parameters to calculate the depth and three-dimensional coordinates of the face in each facial photo; selecting two facial photos taken from different angles as a group for registration, registering at least two different groups of facial photos; fusing the registered groups of facial photos to form a three-dimensional point cloud model; and processing the three-dimensional point cloud model into the facial mesh model. This embodiment clarifies the specific method of obtaining a facial mesh model by stitching and fusing multi-angle facial photos, using checkerboard pattern processing for positioning and correction, making the positioning and correction accurate and feasible, so as to obtain an accurate three-dimensional mesh model through registration.

[0017] In some embodiments, the deformation method includes: calculating using an interpolation deformation function. This embodiment utilizes an interpolation deformation function for deformation calculation, which is more suitable for non-rigid deformations, resulting in more accurate predictions of soft tissue deformation.

[0018] In some embodiments, the interpolation deformation function is a thin-plate spline interpolation function. In this embodiment, a thin-plate spline interpolation function is used as the deformation function, and a Gaussian kernel function is introduced to participate in the calculation, making the deformation conform to the deformation law of soft tissue. The closer to the moving tooth, the greater the change; the farther away, the smaller the change, making the prediction results more accurate.

[0019] In some embodiments, the dental and jaw feature points include: the occlusal contact points of the maxillary first molar and the mandibular first molar, and the occlusal contact points of the maxillary central incisor and the mandibular central incisor. In this embodiment, occlusal contact points are used as the dental and jaw feature points used in registration. The occlusal contact points have obvious features and are accurately identified, which ensures the accuracy and stability of the subsequent registration results.

[0020] In some embodiments, the prediction method further includes: after obtaining the new positions of the mesh vertices of the dental and soft tissue portions in the composite face model, establishing a facial orthodontic prediction model. In this embodiment, a three-dimensional facial orthodontic prediction model is established using the new positions of the mesh vertices, facilitating viewing, display, comparison, etc., and providing users with an intuitive understanding.

[0021] In some embodiments, the orthodontic treatment plan includes multiple sequentially executed orthodontic steps, each corresponding to a target position of the dentition; repeatedly executing the steps of determining the target position of the dentition in the given facial composite model, obtaining new positions of the mesh vertices of the soft tissue portion in the facial composite model based on the changes, and establishing a facial orthodontic prediction model, thereby establishing multiple facial orthodontic prediction models; wherein each model is established based on one of the multiple sequentially executed orthodontic steps in the orthodontic treatment plan. In this embodiment, the target positions of multiple orthodontic steps can be predicted separately, thereby establishing multiple facial orthodontic prediction models, so that when provided to the user, it is convenient for the user to compare the intermediate changes.

[0022] In some embodiments, establishing a facial orthodontic prediction model includes: adding texture to the established facial orthodontic prediction model based on the texture features of the facial composite model. In this embodiment, adding texture to the prediction model makes the model's appearance closer to a real face. Furthermore, the model can be updated based on the original texture to make the model more realistic. Attached Figure Description

[0023] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0024] Figure 1 This is a flowchart of a face shape prediction method according to one embodiment of this application;

[0025] Figure 2 This is a flowchart of a method for forming a facial composite model in a facial shape prediction method according to one embodiment of this application;

[0026] Figure 3 This is a flowchart of a method for obtaining a face mesh model in a face shape prediction method according to one embodiment of this application;

[0027] Figure 4a This is a schematic diagram of the dental portion of a facial composite model in a facial shape prediction method according to one embodiment of this application;

[0028] Figure 4b This is a schematic diagram of the soft tissue portion of a facial composite model in a facial shape prediction method according to one embodiment of this application;

[0029] Figure 5a This is a rendering of a lateral view of a face used in a face shape prediction method according to one embodiment of this application;

[0030] Figure 5b yes Figure 5a The black and white version;

[0031] Figure 6 This is a rendering of a composite face model in a face shape prediction method according to one embodiment of this application;

[0032] Figure 7 This is a diagram showing the alignment of a lateral view with the midline of a composite face model in a face shape prediction method according to one embodiment of this application.

[0033] Figure 8a This is a schematic diagram of a composite face model before deformation in a face shape prediction method according to one embodiment of this application;

[0034] Figure 8b This is a schematic diagram of a deformed composite face model in a face shape prediction method according to one embodiment of this application;

[0035] Figure 9 This is a schematic diagram of a composite face model with added texture features after deformation in a face shape prediction method according to one embodiment of this application;

[0036] Figure 10 This is a schematic diagram of an electronic device according to another embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0038] The terms "anterior region" and "posterior region" mentioned in the various embodiments of this application are defined according to the classification of teeth in the 2nd edition of *Introduction to Stomatology*, published by Peking University Medical Press, pages 36-38. The posterior region includes premolars and molars, teeth marked as 4-8 using the FDI notation. The anterior region includes teeth marked as 1-3 using the FDI notation, and the teeth in the anterior region include central incisors, lateral incisors, and canines.

[0039] The “occlusal plane” mentioned in the various embodiments of this application is obtained according to the definition and confirmation method on page 83 of the 6th edition of Orthodontics. One method is to connect the occlusal midpoint of the first permanent molar with the midpoint between the upper and lower central incisors (at the 1 / 2 point of overbite or open bite); the other method is to divide the occlusal contact points of the posterior teeth equally, and the occlusal contact points of the first permanent molar and the first deciduous molar or the first premolar are often used.

[0040] One embodiment of this application relates to a method for predicting facial features. The specific process of the facial feature prediction method in this embodiment can be as follows: Figure 1 As shown, it includes:

[0041] Step 101: Obtain the composite face model of the patient in the first state, wherein the composite face model includes: the dental and jaw part and the soft tissue part.

[0042] Step 102: Identify pre-selected dental and jaw feature points on the dental and jaw portion of the face composite model and obtain the first position of the dental and jaw feature points.

[0043] Step 103: Based on the treatment plan, obtain the second position of the dental and jaw feature points in the second state of the patient in the face composite model.

[0044] Step 104: Based on the first and second positions of the dental and jaw feature points, use the deformation method to calculate the relationship between the movement of the dental and jaw feature points and the changes in the mesh vertices in the face composite model.

[0045] Step 105: Given the target position of the jaw portion in the face composite model, obtain the new position of the mesh vertices of the soft tissue portion in the face composite model based on the change relationship.

[0046] The following is a detailed description of the implementation details of the face shape prediction method in this embodiment. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0047] It should be noted that the face shape prediction method in this embodiment can be implemented through hardware or a combination of computer software and hardware. For hardware implementation, the face shape prediction method can be implemented through one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic devices for implementing the face shape prediction function, or a selection and combination of the above devices.

[0048] In step 101, the resulting facial composite model is a three-dimensional model that includes the associated dental and jaw components and soft tissue components. The dental and jaw model may only include the upper and lower jaw dentition, such as... Figure 4aAs shown, this method is applied to cases involving only tooth movement, resulting in simplified data for subsequent analysis. It can also include parts of the jawbone and skull in addition to teeth and dentition, making it suitable for cases with both tooth and jaw movement, providing more complete data for subsequent analysis. In some embodiments, the soft tissue portion mainly includes the external appearance, i.e., the outer contour of the soft tissue, such as... Figure 4b As shown above. Figure 4a and Figure 4b The images are presented in black and white, but in some embodiments, color images may also be used, which will not be listed here.

[0049] In some embodiments, the method for forming the composite face model in obtaining the patient's face in the first state is as follows: Figure 2 As shown, it specifically includes:

[0050] Step 201: Acquire lateral facial radiographs and side profile photos of the patient, register and fuse them to form a composite image.

[0051] In some embodiments, the lateral facial radiograph and profile photograph can utilize pre-orthodontic data currently collected by the doctor. For example, the lateral facial radiograph is a projection of the skull at a 90-degree angle to the head, and is the most commonly used X-ray for orthodontic measurements. The original profile photograph is a two-dimensional photograph taken from a side view (left or right), and can be in color or black and white. Based on the profile photograph and the lateral facial radiograph, image fusion and feature recognition methods can be used to register and align the lateral radiograph and profile photograph, and obtain the marked feature points on the teeth (i.e., dentofacial feature points), such as... Figure 5a and Figure 5b Feature points on the dental arch (such as those indicated by label A), or such as Figure 5a and Figure 5b Feature points on the midjaw (as indicated by label B).

[0052] Specifically, this step involves identifying pre-selected facial contour feature points on the acquired lateral facial radiographs and profile photos of the patient; and then registering and fusing these facial contour feature points to form the composite image. In one embodiment, the facial contour feature points on the lateral facial radiographs can be as follows: Figure 5a and Figure 5b The label C indicates this. In this embodiment, facial contour feature points are acquired from various images for registration. Since lateral facial images simultaneously contain both the jaw and part of the facial contour, the fusion process is highly feasible, resulting in a composite image with rich feature information and more selectable feature points, facilitating subsequent 3D model registration. Specifically, the pre-selected facial contour feature points include the tip of the nose and the protrusions of the upper and lower lips. In this embodiment, the tip of the nose and the protrusions of the upper and lower lips are used as facial contour feature points for registration. These points have distinct features and are more accurately identified during acquisition, ensuring registration accuracy in subsequent registration.

[0053] Step 202: Obtain the face mesh model and the dental model respectively. Based on the composite image, register and fuse the face mesh model and the dental model to form the composite face model. This embodiment forms a composite face model by registering and fusing the face mesh model and the dental model, resulting in a composite face model that possesses associated dental and jaw components and soft tissue components representing appearance. Since the methods for obtaining the face mesh model and the dental model are relatively simple, the generation of the composite face model in this embodiment is reliable and easily promoted, while also reducing the difficulty of data collection at the terminal. This embodiment further uses a lateral facial image with both dental and jaw features and some facial contour features, along with a profile photo with facial contour features, for registration and fusion to form a composite image including rich dental and jaw features and facial contour features. This composite image is used as a registration reference for the 3D model. The registration process offers a variety of selectable parameters, making the registration results more accurate. Figure 6 The image shown is a rendering of a composite face model formed by fusing a face mesh model 10 with a jaw and dentition model 20 that includes the teeth.

[0054] In some embodiments, the obtained facial mesh model can be obtained by stitching and fusing multi-angle facial photos of the patient. The stitching and fusing method can be as follows: Figure 3 As shown, the details are as follows:

[0055] Step 301: Remove the background from face photos taken from multiple angles. In some embodiments, the K-means++ algorithm can be used to remove the background of images taken from different angles. In addition to the K-means++ algorithm mentioned above, other existing tools can also be used for background removal, which will not be listed here.

[0056] Step 302 involves performing checkerboard pattern processing on the multi-angle captured face photos to achieve calibration and stereo correction, thereby obtaining camera intrinsic parameters. In the field of computer vision, camera calibration refers to calculating the camera's intrinsic parameters using computer vision techniques to facilitate subsequent applications such as 3D reconstruction, target tracking, and image processing. The principle of checkerboard calibration utilizes the regular characteristics of a checkerboard pattern in a 3D coordinate system. This step can be divided into two parts: checkerboard image extraction and camera parameter calculation.

[0057] Regarding the chessboard image extraction, some embodiments can be performed as follows: First, prepare a chessboard image, then take a picture of this image with a camera and save it. Next, proceed to the first step—chessboard image extraction. It's important to note that the purpose of chessboard image extraction is to obtain the corner coordinates of the chessboard in the image. This process requires image processing techniques to extract the corner coordinates, including grayscale transformation, image smoothing, and edge detection, which can be determined based on the actual shape and quality of the image. In some embodiments, the image can be converted to grayscale to simplify subsequent image processing operations. Next, the image needs to be smoothed to remove noise. Common smoothing methods include Gaussian filtering and median filtering. Then, edge detection is performed to find the corner coordinates. Classic edge detection algorithms such as the Sobel operator and the Canny operator can be used in this process. Once the corner coordinates are extracted, the second step—camera parameter calculation—can begin.

[0058] Regarding camera parameter calculation, the camera's intrinsic parameters can be calculated using the extracted corner coordinates. Intrinsic parameters include the camera's focal length and optical center. Calculating these intrinsic parameters requires utilizing the coordinate information of the checkerboard grid in a 3D coordinate system. By measuring the side length and number of squares of the checkerboard, its coordinates in the 3D coordinate system can be calculated. Then, using the images captured by the camera and the extracted corner coordinates, a correspondence between pixel coordinates and 3D coordinates can be established, thereby solving for the camera's intrinsic parameters. In some embodiments, the calculation of camera intrinsic parameters can also be combined with the calculation of camera extrinsic parameters, including the camera's rotation and translation matrices. Calculating these extrinsic parameters requires utilizing the coordinate information of the checkerboard grid in a 3D coordinate system and the corner coordinates in the images captured by the camera. By solving for the correspondence between pixel coordinates and 3D coordinates, the camera's rotation and translation matrices, i.e., the camera's extrinsic parameters, can be obtained. Therefore, the calculation of these camera parameters provides fundamental support for subsequent computer vision applications, and calculating both intrinsic and extrinsic parameters can improve the accuracy of subsequent analysis.

[0059] Step 303: Using the disparity information and camera intrinsic parameters of multi-angle face photos, calculate the depth and 3D coordinates of the face in each face photo. Specifically, stereo matching is performed on images from different angles, and the disparity is calculated.

[0060] In some embodiments, after step 303, a point cloud denoising method can also be used to improve the quality of the point cloud data.

[0061] Step 304: Select two face photos taken from different angles as a group for registration, and register at least two different groups of face photos. In some embodiments, the ICP (Iterative Closest Point) method is used to register the two groups of point clouds (e.g., center-left, center-right).

[0062] Step 305: The registered groups of face photos are stitched together to form a 3D point cloud model. In some embodiments, the ICP-merging method is used to stitch together the two groups of point cloud data to form a more comprehensive and consistent point cloud model. Since the data source is face photos, the pixels in the photos can be used as point data to form the point cloud model.

[0063] Step 306: Process the above 3D point cloud model into a face mesh model.

[0064] As can be seen, steps 301-306 above clearly define the specific method of stitching and fusing multi-angle facial photos into a facial mesh model, using checkerboard pattern processing for positioning and correction, making the positioning and correction accurate and feasible, so as to obtain an accurate three-dimensional mesh model through registration.

[0065] Regarding the methods for obtaining the face mesh model, in addition to generating it using the multi-angle face photo stitching and fusion method described above, some embodiments can also directly obtain it by scanning the patient's face. Specifically, existing third-party scanning equipment can be used for the scanning process, which will not be elaborated further here. It is evident that the above embodiments clearly define multiple methods for obtaining the face mesh model, allowing for the selection of different methods to meet different needs and expanding the application scenarios of this application.

[0066] Regarding the acquisition of dental models, if the dental model only includes a portion of the model in the embodiments, it can be obtained by directly scanning the patient's teeth and jaw, or by taking an impression of the patient's teeth and jaw, creating a plaster model, and then scanning the plaster model. If the dental model also includes parts such as the jawbone, it can be obtained by CBCT scanning reconstruction. It is evident that different acquisition methods can be selected based on the actual application scenario, which will not be elaborated further here.

[0067] In some embodiments, this step specifically includes: identifying pre-selected facial contour feature points on the face mesh model; identifying dental and jaw feature points on the dental and jaw model; and registering and fusing the face mesh model and the dental and jaw model to form a composite face model based on the relationship between the pre-selected dental and jaw feature points, the facial contour feature points, and the composite image. The identification methods for dental and jaw feature points and facial contour feature points can employ existing methods obtained from lateral radiographs, such as the method for obtaining landmark points from cephalometric radiographs submitted by ZhengYa Company on March 21, 2021 (application number: CN202110345400.6), or other feature point identification methods, which will not be listed here. In one embodiment, when acquiring landmark points, the type of point can be pre-selected, such as setting pre-selected occlusal feature points, GO points, etc., or the landmark points can be identified first, and then some points can be selected from the identified landmark points as dental and jaw feature points. In one embodiment, when registering the lateral view and the composite face model, the lateral view 30 and the composite face model 10 can be aligned along the midline (e.g., ...). Figure 7 As shown, the positional relationship between the lateral view and the composite face model is relatively clear, so direct midline alignment is fast and effective. Regarding the identification of pre-selected facial contour feature points on the face mesh model, in one embodiment, the face mesh model is an outer contour model, so feature points representing contour changes, such as lip protrusion points and nose tip points, can be selected from the face mesh model. Those skilled in the art can select according to their needs, and they will not be listed one by one here.

[0068] It's worth noting that since data acquired through different methods may differ in size, image data can be calibrated before registration. Specifically, this can be done as follows: Since point cloud data consists of the pixels in the aforementioned photos, the distance between pixels can be calculated using a set scale length. The scale length refers to the actual length of a straight line segment measured at a specific pixel in the registered and fused image, used to calibrate the actual distance between pixels. The scaling ratio is calculated based on the actual length of the same object in both types of image data. By scaling and other methods, the distance between identical feature points in different image data is unified, ensuring consistent image size after calibration and avoiding registration errors.

[0069] Although the above-mentioned composite image formed by registering and fusing lateral facial images and profile photos is used to register and fuse the facial mesh model and the dental model, in other embodiments, lateral facial images (such as...) can also be used directly. Figure 5a and Figure 5bThe image shown is a registration image. The face mesh model and the dental model are registered and fused. Since there are dental and facial contour feature points on the lateral face image, dental and facial contour feature points can be identified on the lateral face image. Then, the face mesh model and the dental model are registered and fused based on these two types of feature points. The registration and fusion process is roughly the same as the above embodiment, and will not be described in detail here.

[0070] Steps 201-202 above are used to form a composite facial model of the patient in a first state, including the associated dental and jaw portions and soft tissue portions. It is understood that in other embodiments, an integrated oral and facial scanning method can be used to directly obtain the fused composite facial model, requiring third-party equipment, but the model acquisition speed is faster.

[0071] In one embodiment, the first state can be the patient's initial state before orthodontics or a certain state during the orthodontic process. The specific corresponding stage can be selected according to the actual application scenario, and will not be listed one by one here.

[0072] In step 102, pre-selected dental and jaw feature points are identified on the dental and jaw portion of the facial composite model, and the first position of the dental and jaw feature points is obtained. These dental and jaw feature points may include points characterizing features such as the shape and position of the teeth and jaws. Several dental and jaw feature points are pre-selected from these points. In one embodiment, the pre-selected dental and jaw feature points may include the occlusal contact points of the maxillary first molar and the mandibular first molar, such as... Figure 5a and Figure 5b U6 and L6, as well as the occlusal contact points of the maxillary and mandibular central incisors, such as Figure 5a and Figure 5b L1 in the model. In this embodiment, the occlusal contact point is used as the dental and jaw feature point used in registration. The occlusal contact point has obvious features and is accurately identified, so as to ensure the accuracy and stability of the subsequent registration results. Regarding the method of identifying dental and jaw feature points on the face composite model, you can refer to various existing recognition methods, such as the method for obtaining landmark points in cephalometric radiographs submitted by ZhengYa Company on March 21, 2021 (application number: CN202110345400.6), and the recognition network based on machine learning method for recognizing face composite models, which will not be listed here.

[0073] In step 103, based on the treatment plan, the second position of the dental and jaw feature points in the second state of the patient in the facial composite model is obtained. The treatment plan may include the position of the teeth and jaws in the digital orthodontic design, such as the target dental and jaw state, which is the positional state of the teeth and jaws after orthodontic treatment in the digital design, or multiple successive intermediate states, i.e., intermediate states of the teeth and jaws when reaching the target dental and jaw state. In this embodiment, the target dental and jaw state is taken as the second state. Correspondingly, the second position of the dental and jaw feature points is the spatial position of each identified dental and jaw feature point when the teeth and jaws are in the target state. Its specific expression can be represented by positional coordinates or by the distance difference (equivalent to the amount of movement) between the feature points in the first state and the corresponding feature points in the first state.

[0074] In step 104, based on the first and second positions of the dentofacial feature points, a deformation method is used to calculate the relationship between the movement of the dentofacial feature points and the changes in the mesh vertices in the face composite model. In some embodiments, the deformation method can be obtained using an interpolation deformation function. In some embodiments, the interpolation deformation function is specifically a thin-plate spline interpolation function. Using an interpolation deformation function for deformation calculation is more suitable for non-rigid deformation, making the predicted soft tissue deformation results more accurate. Moreover, using a thin-plate spline interpolation function as the deformation function and introducing a Gaussian kernel function into the calculation makes the deformation more consistent with the deformation law of soft tissue; the closer to the moving tooth, the greater the change, and the farther away, the smaller the change, making the prediction results more accurate. Combined with the face composite model, the basic principle of the thin-plate spline interpolation algorithm is as follows:

[0075] Step S1: Establish the mathematical function relationship between control points and prediction points.

[0076] by Figure 8a Taking the positions of control points and prediction points as an example, the control point corresponds to the first position of the dentofacial feature point, and the prediction point corresponds to the second position of the dentofacial feature point. Figure 8b To determine the deformed model shape, the cusp of the maxillary central incisor is selected as the control point. The coordinates of the maxillary central incisor control point are (x0, y0, z0). It is assumed that the movement of the maxillary central incisor is a buccal translation with a translation amount of d. x Then the amount of tooth movement can be expressed as (d x Given that the coordinates of the predicted point are (x1, y1, z1), then x1 = x0 + d. x y1=y0,z1=z0. Based on the interpolation function, a functional relationship between the control points and the prediction points can be established. The interpolation function can be polynomial interpolation, spline function interpolation, etc. In this example, the thin plate spline interpolation function is selected, and its mathematical expression includes (1)-(3), specifically expressed as:

[0077]

[0078]

[0079]

[0080] In the formula, the left side of the equation represents the position coordinates of the predicted point (x1, y1, z1), and the right side of the equation represents a1, a2, a3, a4, a5, a6, a7, a8, a9, a1, a1, a1, a1, a2 ...1, a2, a1, a1 x a y a z and w i These are the coefficients of the interpolation function, N is the total number of control points requiring deformation, and i is the sequence number of all control points requiring deformation. U(||(x) i ,y i ,z i The smoothing kernel function is constructed by )-(x0,y0,z0)||) to determine the distance between the control points and the prediction points. The smoothing kernel function can typically be a spline kernel function or a Gaussian kernel function; here, a Gaussian kernel function is chosen, and its mathematical form is: Where r is the distance between the control point and the prediction point, and σ is the radius of the Gaussian kernel function.

[0081] Step S2: Based on the functional relationship between the control points and prediction points established in Step S1, solve the linear equation system to obtain the coefficients of the interpolation function:

[0082] Substitute the coordinates of the control points and predicted points into the interpolation mathematical relationship established in step S1, and also substitute the Gaussian kernel function calculated based on the control points and predicted points. Use the Gauss-Seidel method to solve the linear equation system. In addition to the equations determined by the interpolation mathematical relationship in step S1, the linear equation system also introduces four constraint equations for the interpolation coefficients, the expressions of which are as follows:

[0083]

[0084] Thus, a system of linear equations determined by four equations is obtained. The coefficients {a1, a2, ..., a3} of the interpolation function are then calculated using a computer program. x a y a z and w i}, where N is the total number of control points that need to be deformed, and i is the sequence number of the control point that needs to be deformed.

[0085] In step 105, given the target position of the dental and jaw portion in the face composite model, new positions of the mesh vertices of the soft tissue portion in the face composite model are obtained based on the change relationship. In one embodiment, the given target position can be the second position obtained in step 103, or it can be the target position of the dental and jaw feature points in state A when it is necessary to predict the facial shape change in state A. In one embodiment, the above target position can be represented by the amount of movement relative to the initial state position, such as marking the value of the movement on the corresponding control point, or it can be represented by the spatial coordinates of the target position, such as marking the corresponding position in the image.

[0086] In one embodiment, the coefficients obtained in step S2 are substituted into the expression (1-3) to perform TPS transformation on all pixels of the profile image, resulting in a new profile image. At this point, the prediction of the corrected facial profile is completed. The coefficients of the interpolation function calculated in step S2 are substituted into the interpolation function expression (1-3). Based on the target position of the state to be predicted, all pixels in the profile image are interpolated, that is, all pixels in the image are used as control points. Finally, the predicted positions of all pixels after interpolation are calculated, and the predicted points of all pixels form the image of the new profile.

[0087] In some embodiments, the prediction method may further include: after obtaining the new positions of the mesh vertices of the dental and soft tissue portions in the facial composite model, establishing a facial orthodontic prediction model. Establishing a three-dimensional facial orthodontic prediction model using the new positions of the mesh vertices facilitates viewing, display, comparison, etc., providing users with an intuitive understanding. In one embodiment, when establishing the facial orthodontic prediction model, the model can be constructed based on the topological structure of the original facial composite model.

[0088] The above embodiments predict the new positions of model mesh vertices in a certain state. In other embodiments, different changes in the model in multiple states can also be predicted. For example, a treatment plan includes multiple sequentially executed treatment steps, each corresponding to a target position of the dentition; repeatedly executing the given target position of the dentition in the facial composite model, and obtaining the new positions of the mesh vertices of the soft tissue part in the facial composite model based on the change relationship, and establishing a facial orthodontic prediction model, thereby establishing multiple facial orthodontic prediction models; wherein each is established based on one of the multiple sequentially executed treatment steps in the treatment plan. In this embodiment, the target positions of multiple treatment steps can be predicted separately, so that multiple facial orthodontic prediction models are established separately, so that when provided to the user later, it is convenient for the user to compare the intermediate changes.

[0089] It is understood that although the above deformation process uses a thin-plate spline interpolation function, other deformation methods can be used in other embodiments, such as polynomial interpolation, radial basis function interpolation, Kriging interpolation, local interpolation methods, and machine learning-based methods to deform the facial composite model. These will not be listed here. The descriptions of various methods are as follows:

[0090] Polynomial interpolation: Polynomial interpolation approximates a given data point by fitting a polynomial function. Common polynomial interpolation methods include Lagrange interpolation and Newton interpolation, which are computationally simple.

[0091] Radial basis function interpolation (RBF): This method uses radial basis functions to interpolate data. Common radial basis functions include Gaussian functions and multi-aperture radial basis functions. RBF interpolation can flexibly handle irregularly distributed data and is also effective for high-dimensional problems.

[0092] Kriging interpolation: Kriging is a geostatistical interpolation method that estimates the value of unknown points by modeling spatial correlations. Kriging can effectively handle spatial correlations and variability.

[0093] Local interpolation methods: These methods interpolate the grid based on local data, such as locally weighted regression and nearest neighbor-based interpolation methods (e.g., K-nearest neighbor interpolation). These methods can provide good interpolation results within a local range.

[0094] Machine learning-based methods: Machine learning techniques, such as neural networks and support vector machines, are applied to interpolation problems. These methods can perform interpolation by learning complex nonlinear relationships between data, but typically require a large amount of training data and computational resources.

[0095] In addition to thin-plate spline interpolation, other spline interpolation methods can be used, such as natural spline interpolation and Hermite spline interpolation, which will not be listed here. Spline interpolation can effectively approximate data points while maintaining smoothness.

[0096] In some embodiments, such as Figure 9 As shown, the process of establishing a face orthodontic prediction model may further include: adding texture to the established face orthodontic prediction model based on the texture features of the face composite model. Adding texture to the prediction model makes the model's appearance closer to the effect of a real face. In addition, the model can be updated based on the original texture to make the model more realistic. The original texture can be obtained from the face composite model in the first state obtained in step 101, or from the face mesh model in the first state obtained in step 202. It should be noted that... Figure 9 The white frame covering the human eye is a privacy protection measure taken to safeguard the individual's portrait rights in the case, and is not a blank area in the generated model.

[0097] As can be seen, the facial shape prediction method in this embodiment determines the relationship between the dental and soft tissue parts by obtaining a composite model of the related dental and soft tissue parts of the face, and then determines the change relationship of the facial shape by using the known changes of the dental and soft tissue parts in the orthodontic plan. Thus, the positional changes of other parts can be predicted based on this change relationship. Since the prediction can be made using only the patient's own data, there is no need to combine a large amount of historical data to infer the change relationship. Therefore, the facial shape prediction method in this application has a small amount of data involved in the calculation during the prediction and high accuracy.

[0098] It should be noted that the examples in the above embodiments are merely illustrative for ease of understanding and do not constitute a limitation on the technical solution of the present invention.

[0099] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0100] Another embodiment of this application provides a method for displaying facial features. Specifically, a facial orthodontic prediction model can be obtained based on any of the facial feature prediction methods described in the above embodiments; then, the facial orthodontic prediction model is displayed on a display device. The display device can be any device capable of outputting images, such as a monitor, tablet PC, mobile phone screen, etc., which will not be listed here.

[0101] Another embodiment of this application relates to an electronic device, such as... Figure 10 As shown, it includes: at least one processor 1001; and a memory 1002 communicatively connected to the at least one processor 1001; wherein the memory 1002 stores instructions executable by the at least one processor 1001, the instructions being executed by the at least one processor 1001 to enable the at least one processor 1001 to perform the face shape prediction methods in the above embodiments.

[0102] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0103] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0104] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0105] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for predicting facial features, characterized in that, include: A composite facial model of the patient in a first state is obtained, the composite facial model including: dental and jaw parts and soft tissue parts; On the dental and jaw portion of the facial composite model, pre-selected dental and jaw feature points are identified, and the first position of the dental and jaw feature points is obtained; Based on the treatment plan, the second position of the dental and jaw feature points of the patient in the second state in the facial composite model is obtained; Based on the first and second positions of the dental and jaw feature points, a deformation method is used to calculate the relationship between the movement of the dental and jaw feature points and the changes in the mesh vertices of the face composite model. Given the target position of the jaw portion in the face composite model, based on the change relationship, the new positions of the mesh vertices of the soft tissue portion in the face composite model are obtained.

2. The facial shape prediction method according to claim 1, characterized in that, The method for obtaining the composite face model of the patient in the first state includes: Acquire a registration image; wherein the registration image is a composite image formed by fusing a lateral view of the patient's face and a profile photograph, or is a lateral view of the patient's face; A face mesh model and a dental model are obtained separately. Based on the registration image, the face mesh model and the dental model are registered and fused to form the face composite model.

3. The facial shape prediction method according to claim 2, characterized in that, The composite image formed by fusing the patient's lateral facial radiograph and profile photograph includes: Identify pre-selected facial contour feature points on acquired lateral radiographs and profile photos of patients; The composite image is formed by registering and fusing the position of the facial contour feature points.

4. The facial shape prediction method according to claim 3, characterized in that, The pre-selected facial contour feature points include: the tip of the nose and the protrusions of the upper and lower lips.

5. The facial shape prediction method according to claim 3, characterized in that, The process of registering and fusing the face mesh model and the dental model based on the registered image to form the composite face model includes: Identify pre-selected facial contour feature points on the facial mesh model; Identify dental and jaw feature points on the dental and jaw model; Based on the relationship between the pre-selected dental and jaw feature points and the facial contour feature points and the registered image, the facial mesh model and the dental and jaw model are registered and fused to form the facial composite model.

6. The facial shape prediction method according to claim 2, characterized in that, The face mesh model is obtained through the following methods: Obtained by stitching and fusion of multiple facial photos of the patient from different angles; or... Obtained by scanning the patient's face.

7. The facial shape prediction method according to claim 6, characterized in that, The data obtained by stitching and fusing multi-angle facial photos of the patient includes: Remove the background from photos of faces taken from multiple angles; The camera internal parameters are obtained by performing checkerboard pattern processing on face photos taken from multiple angles to achieve calibration and stereo correction. Using the parallax information of the multi-angle face photos and the camera intrinsic parameters, the depth and three-dimensional coordinates of the face in each face photo are calculated; Select two facial photos taken from different angles as a group for registration, and register at least two different groups of facial photos; The registered groups of face photos are stitched together to form a three-dimensional point cloud model; The 3D point cloud model is processed into the face mesh model.

8. The facial shape prediction method according to any one of claims 1-7, characterized in that, The deformation methods include: using interpolation deformation functions, polynomial interpolation, radial basis function interpolation, Kriging interpolation, local interpolation methods, and calculations based on machine learning methods.

9. The facial shape prediction method according to claim 8, characterized in that, The interpolation deformation function is a thin plate spline interpolation function.

10. The facial shape prediction method according to any one of claims 1-7, characterized in that, The dental and jaw feature points include: the occlusal contact points of the maxillary first molar and the mandibular first molar, as well as the occlusal contact points of the maxillary central incisor and the mandibular central incisor.

11. The facial shape prediction method according to any one of claims 1-7, characterized in that, The prediction method further includes: after obtaining the new positions of the mesh vertices of the dental and soft tissue parts in the facial composite model, establishing a facial orthodontic prediction model.

12. The facial shape prediction method according to claim 11, characterized in that, The orthodontic treatment plan includes multiple sequential orthodontic steps, each corresponding to a target location in the jaw area. Repeatedly execute the steps of setting the target position of the dental and jaw portion in the given facial composite model, obtaining the new position of the mesh vertices of the soft tissue portion in the facial composite model based on the change relationship, and establishing a facial orthodontic prediction model, thereby establishing multiple facial orthodontic prediction models; wherein each model is established based on one of the multiple sequentially executed orthodontic steps in the orthodontic scheme.

13. The facial shape prediction method according to claim 11, characterized in that, The step of establishing a facial orthodontic prediction model includes: adding texture to the established facial orthodontic prediction model based on the texture features of the facial composite model.

14. A method for displaying facial features, characterized in that, include: The facial shape prediction method according to any one of claims 11-13 obtains a facial orthodontic prediction model; The facial orthodontic prediction model is displayed on a display device.

15. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the face shape prediction method as described in any one of claims 1 to 13, or to perform the face shape display method as described in claim 14.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the facial shape prediction method according to any one of claims 1 to 13, or is capable of executing the facial shape display method according to claim 14.

Images