Method, System and Device for Predicting Facial Changes after Edentulous Jaw Implantation Based on Point Cloud

Abstract

The present invention provides a method, system and device for predicting facial changes after edentulous jaw implantation. The method comprises: obtaining facial scan data before and after edentulous jaw implantation and performing key point detection, followed by rigid body registration and obtaining facial shape data; constructing a bidirectional point cloud displacement network to establish a bidirectional spatial geometric transformation from the front implantation shape data to the back implantation shape data; based on the trained bidirectional point cloud displacement network, predicting the front implantation shape data to be predicted, and obtaining the predicted back implantation shape data. This scheme can realize accurate and rapid prediction of facial changes after edentulous jaw implantation, and is convenient for use in clinical practice.

Description

Technical Field

[0001] The present invention relates to the fields of image data processing, machine learning and dental implant-related technologies, and in particular to a point cloud-based method, system and device for predicting facial changes after edentulous jaw implantation, which can be used for predicting facial changes after edentulous jaw implantation, aiming to achieve efficient and accurate facial change simulation. Background Art

[0002] At present, an important challenge faced in edentulous jaw implants is the inability to effectively judge the changes in facial appearance after implantation. Tooth loss not only affects the patient's chewing and pronunciation functions, but also seriously damages the facial aesthetics, causing psychological disorders in patients and affecting their normal social activities. Edentulous patients usually have problems such as facial aging, inaccurate pronunciation, difficulty chewing, indigestion, and may even cause systemic diseases. Therefore, restorative treatment of edentulous jaws is particularly important. However, traditional edentulous jaw implantation methods mainly rely on the doctor's experience and the patient's oral condition, and it is difficult to accurately predict facial changes after implantation. This is mainly because the oral anatomical structure of edentulous patients is complex, involving multiple factors of hard tissue and soft tissue, such as alveolar bone absorption, changes in jaw position, lip fullness, occlusal plane, etc. Changes in these factors will have a direct impact on the face.

[0003] In order to solve the problem of facial prediction after edentulous implantation, researchers began to explore new technologies and methods. Among them, 3D facial scanning and simulation technology is considered to be a potential solution. Through 3D facial scanning, the patient's three-dimensional facial data can be obtained, including facial contours, soft tissue distribution, etc. Then, computer technology is used to simulate and analyze these data to predict the changes in the face after implantation. In addition, some researchers have proposed prediction models based on artificial intelligence and machine learning. These models can learn the laws and characteristics of facial changes by analyzing a large number of edentulous implant cases and data, thereby achieving accurate prediction of the face after implantation. However, these models still need further verification and optimization before they can be widely used in actual clinical practice.

[0004] In summary, facial prediction is still a challenge in edentulous jaw implantation. Summary of the invention

[0005] The purpose of the present invention is to provide a prediction scheme for the face after edentulous jaw implantation based on point cloud data and deep learning, which is used for judging the changes and restoration of the face after edentulous jaw implantation, aiming to achieve efficient and accurate simulation of facial changes.

[0006] Specifically, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for predicting facial changes after implantation of an edentulous jaw based on point cloud, the method comprising:

[0008] S1, obtaining facial scan data of the edentulous jaw before and after implantation and performing key point detection to obtain facial key points; performing rigid body registration on the facial scan data before and after implantation based on the facial key points; intercepting the mandibular area of ​​the facial scan data after rigid body registration, and performing maximum connected component processing to obtain anterior implantation shape data and posterior implantation shape data;

[0009] S2. constructing a bidirectional point cloud displacement network and performing training to establish a bidirectional spatial geometric transformation from the front-end implantation data to the back-end implantation data; the bidirectional point cloud displacement network includes two point displacement sub-networks in opposite directions;

[0010] S3. Based on the trained bidirectional point cloud displacement network, the front implantation shape data to be predicted is predicted to obtain the predicted back implantation shape data; the front implantation shape data to be predicted is obtained by the front implantation face scanning data to be predicted through key point detection and maximum connected component processing.

[0011] Preferably, in S1, the rigid body registration processing method is: selecting key points in a T-shaped area among the facial key points, and aligning the key point set based on the key points in the T-shaped area;

[0012] The key point set alignment aims to minimize the Frobenius norm of the difference between a key point set to be registered and another key point set after being processed by the rigid body transformation matrix, and calculate the optimal rigid body transformation matrix;

[0013] The optimal rigid body transformation matrix is ​​used as the rigid body transformation matrix from the facial scan data before implantation to the facial scan data after implantation.

[0014] Preferably, the method of calculating the optimal rigid body transformation matrix is ​​to solve it through the following rigid body transformation relationship:

[0015] ;

[0016] in, represents the Frobenius norm, A represents the rigid body transformation matrix, P and Q represent two sets of key points to be aligned, , , Represents the key point coordinate vector in the key point set P The homogeneous coordinates of , represents the set of real numbers, Denotes assignment, D denotes the dimension of the key point, and N denotes the number of points in the point set.

[0017] Preferably, the optimal rigid body transformation matrix is: .

[0018] Preferably, S1 further comprises, after the rigid body registration process, aligning the aligned facial scan data before and after implantation into a target public data space. The target public data space is a public data space of a preset value, for example, the data space where the first facial scan data of the first patient is located can be used as the target public data space.

[0019] Preferably, in S2, the two point shift sub-networks in opposite directions are a first point shift sub-network and a second point shift sub-network;

[0020] The first point shift subnetwork receives the front implant shape data , output the corresponding displacement vector , to predict the post-implantation morphological data ;

[0021] The second point displacement sub-network receives the implant posterior shape data , output the corresponding displacement vector , to predict the anterior implant morphology ;

[0022] The goal of the training is to and As much as possible and resemblance.

[0023] Preferably, in S2, the training of the bidirectional point cloud displacement network is based on the geometric loss function to determine the front face data of the implant Predicted pre-implantation morphological data , Post-implantation data Predicted post-implantation morphology data The geometric differences between

[0024] The geometric loss function includes a shape loss function and a point density loss function.

[0025] Preferably, the displacement vector is determined by the deformation consistency loss function and The consistency between them.

[0026] Preferably, the geometric loss function is:

[0027] ;

[0028] in, is the shape loss, is the point density loss, To adjust the parameters.

[0029] Preferably, the shape loss for:

[0030] ;

[0031] in, Indicate point and The Euclidean distance between Indicates the number of points in the prediction point set.

[0032] Preferably, the point density loss for:

[0033]

[0034] in, express Arrival Set The k nearest neighbors of Points, express Arrival Set The k nearest neighbors of points.

[0035] Preferably, the total loss function of the bidirectional point cloud displacement network is set to:

[0036] ;

[0037] in, To adjust the parameters.

[0038] Preferably, the deformation consistency loss function for:

[0039] ;

[0040] ;

[0041] in, Represents the Euclidean distance.

[0042] Preferably, the point shift subnetwork structure comprises four feature decoding modules, four feature encoding modules and a set of fully connected layers connected in sequence;

[0043] The feature decoding module is composed of a sampling layer, a grouping layer and a PointNet layer connected in sequence; the feature decoding module is composed of a point set interpolation layer and a PointNet layer connected in sequence.

[0044] In a second aspect, the present invention further provides a point cloud-based system for predicting facial changes after implantation of edentulous jaws, the system being applied to the method as described above, the system comprising:

[0045] A facial scanning module, used to obtain facial scanning data before and / or after the edentulous jaw is implanted, and send the data to the facial shape data calculation module;

[0046] The facial shape data calculation module detects key points of the facial scan data before and after implantation to obtain facial key points; performs rigid body registration on the facial scan data before and after implantation based on the facial key points; intercepts the mandibular area of ​​the facial scan data after rigid body registration, and performs maximum connected component processing to obtain the facial shape data before implantation and the facial shape data after implantation;

[0047] A prediction model module is used to construct a bidirectional point cloud displacement network and store the trained bidirectional point cloud displacement network to establish a bidirectional spatial geometric transformation from the front implantation shape data to the back implantation shape data; the bidirectional point cloud displacement network includes two point displacement sub-networks in opposite directions; and based on the trained bidirectional point cloud displacement network, the front implantation shape data to be predicted is predicted to obtain the predicted back implantation shape data; the front implantation shape data to be predicted is obtained by the front implantation face scan data to be predicted through key point detection and maximum connected component processing;

[0048] A model training module, used for training the bidirectional point cloud displacement network;

[0049] The output module is used to output the predicted post-implantation shape data.

[0050] In a third aspect, the present invention also provides a point cloud-based device for predicting facial changes after edentulous jaw implantation, the device comprising at least a processor and a memory, the processor calling computer instructions in the memory to execute the method for predicting facial changes after edentulous jaw implantation as described above.

[0051] Compared with the existing technology, this solution has achieved significant improvements in accuracy and prediction speed, and particularly emphasizes its convenience and practicality in actual clinical practice. The following are at least several major beneficial effects of this method invention:

[0052] Accurate prediction: This solution uses advanced 3D facial scanning technology and deep learning algorithms to capture subtle changes in facial soft and hard tissues, thereby achieving accurate simulation and prediction of facial changes after edentulous implant placement.

[0053] Fast prediction: With the help of an efficient computing platform and optimized algorithm design, this solution can complete the prediction process of facial changes in a short time, significantly shortening the waiting time for processing results. This fast response capability enables users to adjust treatment plans based on prediction results more promptly, improving work efficiency.

[0054] Convenient for clinical practice: The system and equipment of this solution are simple and easy to use, and can be used without complicated training. At the same time, it is easy to integrate with the existing medical information system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of facial scan data preprocessing provided by an embodiment of the present invention;

[0057] Figure 2 A structural diagram of a point cloud network (PointNet++) provided in an embodiment of the present invention;

[0058] Figure 3 A structural diagram of a feature encoding module and a feature decoding module provided in an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of a bidirectional point cloud displacement network architecture provided by an embodiment of the present invention;

[0060] Figure 5 The following are the facial effects of edentulous jaw after implantation generated based on the model architecture proposed in the embodiment of the present invention, where (a) is the lower part before the operation, (b) is the lower part after the operation, and (c) is the lower part of the face implantation effect image generated by the BPD-Net architecture;

[0061] Figure 6 A flow chart of a facial change prediction method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0064] The present invention is further explained below in conjunction with specific implementation methods. The scheme of the present application combines deep learning technology for the prediction of the face after edentulous jaw implantation, aiming to achieve efficient and accurate simulation of facial changes. First, the scheme proposes a preprocessing method for facial scan data. Through the facial key point detection method (Deep-MVLM), multiple key points of each individual's preoperative and postoperative facial scan data are calculated (for example, 84 key points are used in this embodiment), and the individual's preoperative and postoperative data are rigidly aligned based on multiple key points in the facial T-shaped area (for example, 49 key points are used in this embodiment), so as to obtain a rigid transformation matrix from postoperative to preoperative. Use this matrix to align the postoperative facial scan data and key points to the preoperative space, so that the preoperative and postoperative data of all individuals are in the same coordinate system. Then, the lower area of ​​the face is cropped according to the preoperative and postoperative key points, and the maximum connected component of the cropped facial data is found by the maximum connected component detection method, etc., to retain the completely connected facial scan grid data. Finally, a bidirectional neural network architecture based on point cloud was used to construct a bidirectional spatial geometric transformation from preoperative shape data to postoperative shape data, so that the network can accurately predict postoperative data based on preoperative data. Figure 1-5 And specific embodiments, this solution is described in detail.

[0065] Before introducing the technical solution of the present application, the relevant terms involved in the embodiments of the present application are first introduced. Hereinafter, several basic definitions involved in the embodiments are first explained as follows:

[0066] 1. Point cloud data: In this embodiment, point cloud data is defined as a set of points defined in three-dimensional space, each point usually contains spatial coordinates (such as x, y, z) and possible additional information (such as color, intensity, etc.).

[0067] 2. Loss function: a function used in machine learning to measure the difference between the model output and the true label. By minimizing the loss function, the model can continuously optimize parameters to make the prediction results more accurate. In this embodiment, two loss functions are used: geometric loss function and consistent transformation loss function.

[0068] 3. Rigid body transformation matrix: a mathematical matrix used to describe and calculate the movement and rotation of an object.

[0069] 4. Deformation field: It can be expressed in the form of a scalar field or a vector field. The scalar field is used to indicate the degree of deformation of each point, while the vector field is used to indicate the specific displacement of each point.

[0070] The facial change prediction method of this solution is described in detail below in conjunction with specific embodiments. Figure 6 As shown, the method involved in this embodiment mainly includes the following steps:

[0071] S1. Scan the face of the edentulous jaw before and after implantation, and pre-process the facial scan data.

[0072] Combination Figure 1 As shown, in this embodiment, the preprocessing of facial scan data mainly includes the following steps:

[0073] S11. Calculate 84 key points of the pre-operative / post-operative facial scan data using a facial key point detection algorithm. In this embodiment, 84 key points are used as an example for explanation, and the number of key points here should not be interpreted as a limitation on the scope of this solution.

[0074] The detection of facial key points aims to identify and locate key feature points in a face image, such as the contours of the eyes / nose / mouth, the edges of the eyebrows / ears, and other facial features. In this embodiment, an existing facial key point detection algorithm is used to calculate the 84 key points of each individual's preoperative and postoperative facial scans during the preprocessing of the facial scan data. This type of algorithm can be implemented using, for example, the Deep-MVLM method in GitHub, which will not be described in detail here.

[0075] S12, rigidly registering the individual's preoperative and postoperative facial scan data based on the 49 key points in the facial T-shaped region, thereby obtaining a rigid transformation matrix from postoperative to preoperative. It is understandable that the T-shaped region is illustrated by taking 49 key points as an example, and the number of key points here should not be interpreted as limiting the scope of this solution.

[0076] Rigid body registration is to align two or more three-dimensional point clouds or images so that they overlap in space. The characteristic of rigid body registration is that only rotation and translation are allowed, and no scaling or deformation is involved. In this embodiment, taking the 49 key points in the T-shaped area as an example, the facial scan data preprocessing performs rigid body registration on the individual's preoperative and postoperative data based on the key points in the facial T-shaped area.

[0077] In this embodiment, the facial T-zone refers to a part of the face, usually a "T"-shaped area, which usually contains important facial features, such as eyebrows, nose and mouth, and is a key area for facial expression and recognition. In the embodiment, the 49 key points of the T-zone are used to perform rigid body registration on the pre-operative and post-operative data of the individual, thereby obtaining a rigid body transformation matrix from post-operative to pre-operative.

[0078] In a more preferred embodiment, rigid body registration can be achieved by the following improved method:

[0079] Set the key point to a D-dimensional coordinate vector p, that is , represents the set of real numbers, Represents a D-dimensional vector. In this embodiment, Take 3 as an example. express The homogeneous coordinates of Then for two sets of key points and Then there is, , ,in, Represents the assignment, and N represents the number of points in the point set. Similar alignment of secondary coordinate key points and Then there is ,as well as The key point-based rigid body registration algorithm hopes to find a rigid body transformation matrix , so that the key point set With key point set Fully aligned, that is:

[0080] ;

[0081] ;

[0082] in, represents the Frobenius norm. right Taking the derivative and setting it to 0 gives the optimal ,Right now:

[0083] ;

[0084] Then we can get the optimal rigid body transformation matrix :

[0085] .

[0086] The above Frobenius norm is a matrix norm used to measure the size of the matrix. The specific formula is:

[0087] ;

[0088] Where G represents an m×n matrix, are the elements in the matrix.

[0089] It is understandable that the specific algorithm of rigid body registration can adopt the improved rigid body registration method proposed in this solution as above, or can adopt the existing algorithm. The introduction of the existing method may lead to certain inaccuracies in the algorithm results or degradation of the effect, but it can also achieve the basic functions of this solution.

[0090] S13, using the obtained rigid transformation matrix to align the postoperative facial scan data and facial key points to the preoperative facial scan data space. After all corresponding preoperative and postoperative facial scan data are rigidly aligned, the data space where the preoperative facial scan data of one patient is located is selected from the facial scan data set as the target common data space, and then the rigid registration algorithm described in S12 is used to align the aligned preoperative and postoperative facial scan data to the target common data space, so that the data of all individuals are located in the common coordinate space. In this embodiment, the facial scan data space of the first patient in the data set is used as the target common data space.

[0091] The alignment of facial scan data to the target public data space can be achieved by using the rigid body registration method in S12 or the space conversion method in the prior art, which will not be described in detail here.

[0092] S14. After the facial scan data is aligned, it is further preferred to intercept the face and mandible area for the aligned facial data, that is, to crop the face and mandible area of ​​the pre-operative / post-operative facial scan data according to the pre-operative / post-operative key points, and then find the maximum connected component of the cropped facial data through the maximum connected component detection method to retain the completely connected facial scan grid data. The purpose of the maximum connected component method is to remove some scattered areas that are not connected to the main area of ​​interest for the cropped face and mandible area point cloud data. It is further explained here that when predicting the actual data to be predicted after the model training is completed, the pre-operative facial data obtained by the scan also needs to be rigidly registered and aligned to the target common data space, and then the face and mandible area is cropped and enters the subsequent processing flow. In addition, the cropping of the face and mandible area can adopt the conventional area cropping method in this field, which belongs to the well-known method in this field and will not be repeated here.

[0093] In this embodiment, the maximum connected component refers to the one with the largest number of nodes among all connected components. The maximum connected component detection method can be used to find the maximum connected component of the cropped facial data, remove some redundant facial mesh structures, and retain the complete connected facial scan mesh data.

[0094] The maximum connected component method is an algorithm used to identify connected areas in the graph field. Its goal is to find the largest connected subset of nodes or pixels in the network, where each pair of nodes or pixels is connected by a path. Since the three-dimensional face scan data is composed of point clouds and edges between points, the maximum connected component detection algorithm can be used in this embodiment to find the maximum connected component of the cropped facial data (i.e., the cropped point cloud data) to retain the fully connected facial scan mesh data and remove discrete invalid three-dimensional face patches. The maximum connected component algorithm can use, for example, the connected_components function in the networkx python software package (https: / / networkx.org / ).

[0095] After the above processing, the cropped partial facial data before and after the operation, namely the facial shape data, can be obtained. For example, it mainly includes the facial image corresponding to the dental and maxillary parts, such as the mouth, mandible, etc., see Figure 1 or Figure 5 shown.

[0096] It can be understood that the detection and preprocessing of the preoperative and postoperative facial key points in step S1 are based on the training of the model. When the trained model is put into actual use, only the preoperative facial scan of the individual to be predicted can be performed, so as to perform predictive calculations through the model to obtain the facial image of the individual after implantation. That is, in actual use, the prediction and image reconstruction of the postoperative face can be achieved directly based on the preoperative facial key points.

[0097] S2. Construct a bidirectional point cloud displacement network, establish a bidirectional spatial geometric transformation from preoperative facial shape data to postoperative facial shape data, and through model training, enable the bidirectional point cloud displacement network to accurately predict postoperative facial shape data based on preoperative facial shape data.

[0098] Combination Figure 4 As shown, the bidirectional point cloud displacement network (BPD-Net) architecture design in this embodiment is as follows:

[0099] The model architecture uses a bidirectional architecture that can learn the bidirectional spatial transformation between the point clouds in the given pre-operative scan data X and the post-operative scan data Y. The bidirectional point displacement network in this embodiment uses the same network structure. The first point displacement subnetwork receives the Point Cloud (i.e. the pre-implantation profile data obtained after interception and processing in this embodiment), output the corresponding displacement vector To predict the postoperative scan point cloud ,Right now:

[0100] ;

[0101] Similarly, the second point shift subnetwork (i.e., the point shift subnetwork in the other direction) receives data from the postoperative scans. Point Cloud (i.e. the posterior implantation data), output the corresponding displacement vector Predicting preoperative point cloud ,Right now:

[0102] ;

[0103] Based on the pre-processed pre-operative and post-operative scan data (i.e., the pre-implantation and post-implantation shape data), BPD-Net is trained to predict the point cloud and Can be compared with the target point cloud and Specifically, this embodiment uses the geometric loss function To measure the prediction point set and the target point set In this embodiment, it is more preferred that the geometric loss is set to two items: shape loss and point-density loss. More preferably, the shape loss The calculation is as follows:

[0104] ;

[0105] in, Indicate point and The Euclidean distance between Indicates the number of points in the prediction point set.

[0106] The above point density loss Used to measure the prediction point set and the target point set The density similarity between them is calculated in the following way in this embodiment:

[0107] ;

[0108] in, express Arrival Set The k nearest neighbors of Points, express Arrival Set The k nearest neighbors of points.

[0109] From the above, the geometric loss function in this embodiment is defined as:

[0110] ;

[0111] in, is a tuning parameter used to control the contribution of density loss.

[0112] It should be noted here that the above-mentioned geometric loss function, and the shape loss function and point density loss function contained therein, are the preferred improved implementation modes proposed in this scheme. Those skilled in the art may know that under the technical inspiration of judging the prediction effect by the geometric loss function proposed in this scheme to train and adjust the model, those skilled in the art may adopt other currently known geometric loss functions to adjust and train the network model. These functions only need to be able to judge the geometric difference between the true value data matrix (or point cloud matrix) and the predicted value data matrix (or point cloud matrix), for example, only through one or several conventional shape loss functions, such as the mean square error function, etc. Such commonly used existing loss functions will not be described in detail.

[0113] Further preferably, in addition to the geometric loss function, this embodiment may also add a deformation consistency loss function (consistent transformation loss): , to facilitate the deformation vector (i.e. displacement vector) and The deformation consistency loss function in this embodiment is calculated as follows:

[0114] ;

[0115] ;

[0116] in, Represents the Euclidean distance between two points. Then, in this more preferred embodiment, the total loss of BPD-Net can be adjusted to:

[0117] ;

[0118] in, The adjustment parameters control the contribution of deformation consistency loss. The setting of the values ​​of the above adjustment parameters can be adjusted based on experience, actual contribution or prediction accuracy requirements.

[0119] It should be noted here that the above-mentioned deformation consistency loss function is the preferred improved implementation mode proposed in this scheme. This scheme can only use the geometric loss function to train the network model, or only use the above-mentioned deformation consistency loss function to train the network model, or combine the above-mentioned deformation consistency loss function with other known or existing geometric loss functions to train the network model. The above-mentioned training methods can be applied to the scheme, even if their training effects and prediction effects may be inferior to the preferred implementation mode of the present invention. Therefore, the combination of the deformation consistency loss function and the geometric loss function proposed in this scheme cannot be regarded as a prerequisite for the implementation or execution of this scheme.

[0120] In addition, under the technical inspiration of the present scheme for judging the consistency of deformation vectors through deformation consistency loss function to further train and adjust the model, technicians in this field can adopt other currently known consistency loss functions to adjust and train the network model. These functions only need to be able to judge the consistency between the two deformation matrices. For example, only one or several conventional shape loss functions are used, such as the minimum absolute value deviation function, etc. Such commonly used existing loss functions will not be elaborated here.

[0121] Furthermore, combined with Figure 2 , Figure 3 As shown in FIG. 1 , the structure of a single point cloud network in the above-mentioned bidirectional point cloud displacement network is introduced. A single point cloud network includes 4 feature encoding modules, 4 feature decoding modules, and a set of fully-connected layers for outputting point cloud displacement vectors.

[0122] A single feature encoding module consists of a sampling layer, a grouping layer, and a PointNet layer in sequence. When the point cloud features are input to the feature encoding module, a set of center points are first selected from the input point cloud features using the farthest point sampling method in the sampling layer. The number of center points is usually less than the number of original point cloud points to reduce the amount of calculation and focus on key areas. For each center point, a spherical query is used to find its neighborhood points within a fixed radius. The neighborhood points of each center point are grouped to form a local "point group". Each of these point groups surrounds a center point to form a small sub-point cloud. The point coordinates of each group are standardized. For example, the coordinates of each point can be subtracted from the coordinates of the center point for standardization. The standardized coordinates can capture the local geometry and remove the absolute position of the center point. These grouped points are input into a PointNet subnetwork for feature extraction. Each point group outputs a local feature vector representing the geometric features around the center point.

[0123] Recombination Figure 2 , a total of 4 feature encoding modules are set in the point cloud network. We take the point cloud data with an input data size of N×3 as an example. When the input point cloud data is imported into the first feature encoding module, it jumps to the first feature decoding module; after being processed by the first feature encoding module, the first point cloud data is obtained. The size of the first point cloud data is , the first point cloud data obtained continues to be input into the second feature encoding module, and at the same time jumps to the second feature decoding module; after being processed by the second feature encoding module, the second point cloud data is obtained, and its size is , the second point cloud data continues to be input into the third feature encoding module, and jumps to the third feature decoding module at the same time; the second point cloud data is processed by the third feature encoding module to obtain the third point cloud data, whose size is , the third point cloud data continues to be input into the fourth feature encoding module, and at the same time jumps to the fourth feature decoding module; after being processed by the fourth feature encoding module, the fourth point cloud data is obtained, and its size is The fourth point cloud data is subsequently transmitted to the fourth feature decoding module and stacked with the third point cloud data for decoding. , , , Indicates the number of points in the point cloud after downsampling after each level of encoder and decoder.

[0124] Recombination Figure 3 As shown in Figure 1, the feature decoding module consists of a point set interpolation layer and a PointNet layer connected in sequence. When the point cloud features are input to the feature decoding module, the point cloud features are first upsampled and stacked with the point cloud features transmitted by the jump connection (see Jump Connection Figure 2 As shown in Figure 3), and finally input into the PointNet layer for feature extraction.

[0125] After passing through a series of feature encoding modules and feature decoding modules, the point cloud data will eventually be input into a set of fully connected networks to obtain point cloud displacement data. At this time, based on the obtained point cloud displacement data, the input data (such as Figure 2 The N×3 point cloud data in the image is used for displacement deformation.

[0126] Furthermore, farthest point sampling is a commonly used sampling algorithm in point cloud data processing, which is mainly used to uniformly select subsets from the point cloud to reduce computational complexity while retaining geometric information. The main idea is to start from the initial point and select the point farthest from the sampled point set each time to cover the entire point cloud as evenly as possible.

[0127] Spherical query uses a neighborhood query method for point clouds, which is used to find the neighborhood points of a certain center point within a given radius. This query method helps the network obtain local geometric information with the center point as the core. The specific steps are: first select a center point and set a radius value; second, around the center point, with the set radius as the range, find all points whose distance from the center point is less than the set radius; finally, these points are used as neighborhood points together with the center point to form a local "spherical area".

[0128] S3. Train the bidirectional point cloud displacement network, set the training set and the test set, and set the learning rate, and perform network training until the training end conditions are met.

[0129] In this example, the model was trained and tested on a Linux workstation equipped with an Intel Xeon Gold6258R CPU and a 48 GB GTX Quadro RTX 8000 GPU. The Adam optimizer was used for model optimization, and the batch size was set to 2. The learning rate was set to 0.001 and decayed to 0.0001 in discrete intervals during the training process.

[0130] S4. After the network training is completed, the input pre-operative facial scan data is predicted to obtain the post-operative facial scan data.

[0131] Figure 5The BPD-Net architecture proposed in this embodiment generates a display diagram of the facial effect of the edentulous jaw after implantation. It can be seen that this solution can well predict the changes in the facial appearance of the edentulous jaw after implantation and obtain accurate simulation image data.

[0132] In another specific embodiment, the present solution can also be implemented by a system for predicting facial changes after edentulous jaw implantation, the system comprising:

[0133] A facial scanning module, used to obtain facial scanning data before and / or after the edentulous jaw is implanted, and send the data to the facial shape data calculation module;

[0134] The facial shape data calculation module detects key points of the facial scan data before and after implantation to obtain facial key points; performs rigid body registration on the facial scan data before and after implantation based on the facial key points; intercepts the mandibular area of ​​the facial scan data after rigid body registration, and performs maximum connected component processing to obtain the facial shape data before implantation and the facial shape data after implantation;

[0135] A prediction model module is used to construct a bidirectional point cloud displacement network and store the trained bidirectional point cloud displacement network to establish a bidirectional spatial geometric transformation from the front implantation shape data to the back implantation shape data; the bidirectional point cloud displacement network includes two point displacement sub-networks in opposite directions; and based on the trained bidirectional point cloud displacement network, the front implantation shape data to be predicted is predicted to obtain the predicted back implantation shape data; the front implantation shape data to be predicted is obtained by the front implantation face scan data to be predicted through key point detection and maximum connected component processing;

[0136] A model training module, used for training the bidirectional point cloud displacement network;

[0137] The output module is used to output the predicted post-implantation shape data.

[0138] When the system is running, the method for predicting facial changes after edentulous jaw implantation provided in the above embodiment can be executed.

[0139] In another embodiment, the solution can be implemented by means of a device, which may include a corresponding module that performs each or several steps in the above-mentioned various embodiments. Therefore, each step or several steps of the above-mentioned various embodiments can be performed by a corresponding module, and the electronic device may include one or more modules of these modules. The module can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination. The device can be implemented using a bus architecture.

[0140] Any process or method description in the flow steps of this solution or described in other ways herein can be understood as a module, fragment or part of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred implementation of this solution includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a technician in the technical field to which the implementation of this solution belongs. The processor executes the various methods and processes described above. For example, the method implementation in this solution can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0141] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A point cloud-based method for predicting facial changes after implantation in edentulous jaws, characterized in that: The method comprises: S1, obtaining facial scan data of the edentulous jaw before and after implantation and performing key point detection to obtain facial key points; performing rigid body registration on the facial scan data before and after implantation based on the facial key points; intercepting the mandibular area of ​​the facial scan data after rigid body registration, and performing maximum connected component processing to obtain anterior implantation shape data and posterior implantation shape data; S2. constructing a bidirectional point cloud displacement network and performing training to establish a bidirectional spatial geometric transformation from the front-end implantation data to the back-end implantation data; the bidirectional point cloud displacement network includes two point displacement sub-networks in opposite directions; S3. Based on the trained bidirectional point cloud displacement network, the front implantation shape data to be predicted is predicted to obtain the predicted back implantation shape data; the front implantation shape data to be predicted is obtained by the front implantation face scanning data to be predicted through key point detection and maximum connected component processing.

2. The method according to claim 1, characterized in that In S1, the rigid body registration processing method is: selecting key points in a T-shaped area among the facial key points, and aligning the key point set based on the key points in the T-shaped area; The key point set alignment aims to minimize the Frobenius norm of the difference between a key point set to be registered and another key point set after being processed by the rigid body transformation matrix, and calculate the optimal rigid body transformation matrix; The optimal rigid body transformation matrix is ​​used as the rigid body transformation matrix from the facial scan data before implantation to the facial scan data after implantation.

3. The method according to claim 2, characterized in that The method of calculating the optimal rigid body transformation matrix is ​​to solve it through the following rigid body transformation relationship: ； in, represents the Frobenius norm, A represents the rigid body transformation matrix, P and Q represent two sets of key points to be aligned, , , Represents the key point coordinate vector in the key point set P The homogeneous coordinates of , represents the set of real numbers, Denotes assignment, D denotes the dimension of the key point, and N denotes the number of points in the point set.

4. The method according to claim 3, characterized in that The optimal rigid body transformation matrix is: 。 5. The method according to claim 2, characterized in that: The S1 further includes, after the rigid body registration process, aligning the aligned facial scan data before and after implantation into the target common data space.

6. The method according to claim 1, characterized in that In S2, the two point shift sub-networks in opposite directions are a first point shift sub-network and a second point shift sub-network; The first point shift subnetwork receives the front implant shape data , output the corresponding displacement vector , to predict the post-implantation morphology ; The second point displacement sub-network receives the implant posterior shape data , output the corresponding displacement vector , to predict the anterior implant morphology ; The goal of the training is to and As much as possible and resemblance.

7. The method according to claim 1, characterized in that In S2, the training of the bidirectional point cloud displacement network is based on the geometric loss function to determine the front face data of the implant Predicted pre-implantation morphological data , Post-implantation data Predicted post-implantation morphology data The geometric differences between The geometric loss function includes a shape loss function and a point density loss function.

8. The method according to claim 6, characterized in that Determine the displacement vector through the deformation consistency loss function and The consistency between them.

9. A point cloud-based facial change prediction system for edentulous jaws after implantation, characterized in that: The system is applied to the method for predicting facial changes after implantation of edentulous jaw based on point cloud as claimed in any one of claims 1 to 8, and the system comprises: A facial scanning module is used to obtain facial scanning data before and after the edentulous jaw is implanted, and send the data to the facial shape data calculation module; The facial shape data calculation module detects key points of the facial scan data before and after implantation to obtain facial key points; performs rigid body registration on the facial scan data before and after implantation based on the facial key points; intercepts the mandibular area of ​​the facial scan data after rigid body registration, and performs maximum connected component processing to obtain the facial shape data before implantation and the facial shape data after implantation; A prediction model module is used to construct a bidirectional point cloud displacement network and store the trained bidirectional point cloud displacement network to establish a bidirectional spatial geometric transformation from the front implantation shape data to the back implantation shape data; the bidirectional point cloud displacement network includes two point displacement sub-networks in opposite directions; and based on the trained bidirectional point cloud displacement network, the front implantation shape data to be predicted is predicted to obtain the predicted back implantation shape data; the front implantation shape data to be predicted is obtained by the front implantation face scan data to be predicted through key point detection and maximum connected component processing; A model training module, used for training the bidirectional point cloud displacement network; The output module is used to output the predicted post-implantation shape data.

10. A point cloud-based device for predicting facial changes after edentulous jaw implantation, characterized in that: The device includes at least a processor and a memory, and the processor calls computer instructions in the memory to execute the point cloud-based method for predicting facial changes after edentulous jaw implantation as described in any one of claims 1-8.

Images