Orthodontic automatic tooth arrangement method and system based on deep learning of grid features

Abstract

The present invention discloses an automatic tooth arrangement method and system for orthodontics based on deep learning of mesh features. The method comprises: obtaining a three-dimensional model of a triangular mesh of teeth for tooth arrangement; inputting the triangular mesh model of teeth into a pre-trained deep learning model to obtain the predicted tooth position of the tooth arrangement result. The deep learning model comprises a first feature encoder, a second feature encoder and a feature decoder and mapper, wherein the first feature encoder uses the triangular mesh three-dimensional model of teeth as input to obtain tooth shape features, the second feature encoder uses the dentition point cloud as input to obtain dentition global features, and the feature decoder and mapper obtains the predicted tooth arrangement result based on the tooth shape features and the dentition global features. The present invention can automatically and accurately plan the ideal position of the patient's teeth.

Description

Technical Field

[0001] The present invention relates to the field of oral medicine technology, and more specifically, to an automatic orthodontic tooth arrangement method and system based on grid feature deep learning. Background Art

[0002] The incidence of malocclusion is high, and the harm to oral health cannot be ignored. Orthodontics is a discipline that specializes in the etiology, diagnosis, treatment and prevention of malocclusion. Clinical orthodontic treatment applies correction force to the teeth to move them, thereby restoring the normal arrangement and occlusion of the teeth by installing correction appliances inside and outside the mouth of patients with malocclusion.

[0003] Orthodontic tooth arrangement refers to arranging the teeth of patients with malocclusion into the expected ideal position after treatment, which serves as a reference for orthodontic treatment planning. It is a key step in digital orthodontic treatment. However, in the current clinical orthodontic treatment planning stage, doctors usually manually arrange the desired tooth position through a human-computer interaction interface. This manual tooth arrangement method generates excessively high labor and time costs.

[0004] In recent years, with the development of computer graphics and artificial intelligence technology, the academic community has launched research on artificial intelligence tooth arrangement algorithms, which are mainly divided into two categories: the early method based on dental arch curve fitting and the current method based on deep learning. The method based on deep learning regards the tooth arrangement task as a three-dimensional space six-degree-of-freedom posture regression problem, uses the dental posture data before and after treatment to train the deep neural network, learns the shape characteristics of malocclusion teeth, the spatial posture and the mapping relationship between the ideal tooth posture from a large number of cases, and then solves the movement amount from malocclusion teeth to ideal teeth.

[0005] In the prior art, computer-assisted tooth arrangement solutions mainly include tooth arrangement methods based on dental arch curve fitting and automatic tooth arrangement methods based on deep learning.

[0006] For the method based on dental arch curve fitting, it first uses the characteristic points on the teeth to fit the ideal dental arch curve, and then calculates the movement of each tooth based on the positional relationship between the current position of the tooth and the ideal dental arch curve, and then solves the tooth position after arrangement. The tooth arrangement effect of this method depends on the accuracy of the input characteristic points, which are usually manually selected, so it is difficult to achieve fully automatic tooth arrangement.

[0007] For deep learning-based methods, the original data used is non-perspective data obtained through scanning, which only has surface crown information but no root information, and cannot explicitly consider the root. Therefore, the root of the arranged teeth may be too close to or even move out of the alveolar bone, which is inconsistent with orthodontic clinical requirements. In addition, the mainstream data types currently used to describe three-dimensional models include point clouds, voxels, multi-view images, and mesh models. The mesh model not only contains the location information of the object, but also can extract the relative topological relationship between adjacent units on the surface of the object, and has a strong representation ability for the shape of the object. Correspondingly, the neural network that processes the mesh has achieved better results in object classification, segmentation and other tasks than the neural network that processes the point cloud. Since the existing deep neural network for automatic tooth arrangement uses the point cloud processing network as an encoder to extract features from the tooth point cloud, compared with the mesh processing network, its ability to capture and process fine-grained features of the tooth is weaker, thus restricting the effect of automatic tooth arrangement. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a new automatic tooth arrangement method based on deep learning of mesh features, which can learn its feature representation from the tooth triangular mesh model, thereby improving the accuracy of the automatic tooth arrangement results based on the deep learning method.

[0009] According to a first aspect of the present invention, a method for automatic tooth arrangement in orthodontics based on deep learning of grid features is provided. The method comprises the following steps:

[0010] Acquire a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model uses vertices, centroids, midlines, normal vectors and adjacent surface indices of triangular facets to represent shape information and spatial position information of the teeth;

[0011] Inputting the tooth triangular mesh model into a pre-trained deep learning model to obtain the predicted tooth position of the tooth arrangement result;

[0012] Among them, the deep learning model includes a first feature encoder, a second feature encoder and a feature decoder and mapper. The first feature encoder takes the tooth triangulated mesh three-dimensional model as input to obtain tooth shape features, and the second feature encoder takes the dentition point cloud as input to obtain dentition global features. The feature decoder and mapper obtains the predicted tooth arrangement results based on the tooth shape features and the dentition global features.

[0013] According to a second aspect of the present invention, there is provided an automatic tooth arrangement system for orthodontics based on deep learning of grid features. The system comprises:

[0014] A tooth model acquisition unit is used to acquire a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model uses vertices, centroids, midlines, normal vectors and adjacent surface indexes of triangular facets to represent shape information and spatial position information of the tooth;

[0015] Prediction unit: used for inputting the tooth triangular mesh model into the pre-trained deep learning model to obtain the predicted tooth position of the tooth arrangement result;

[0016] Among them, the deep learning model includes a first feature encoder, a second feature encoder and a feature decoder and mapper. The first feature encoder takes the tooth triangulated mesh three-dimensional model as input to obtain tooth shape features, and the second feature encoder takes the dentition point cloud as input to obtain dentition global features. The feature decoder and mapper obtains the predicted tooth arrangement results based on the tooth shape features and the dentition global features.

[0017] Compared with the existing technology, the advantage of the present invention is that it can automatically plan the ideal position of the teeth of patients with malocclusion based on the three-dimensional model of the tooth triangular mesh, saving the time spent by orthodontists in the treatment planning stage, and overcoming the shortcomings of the current automatic tooth arrangement method based on deep learning, which has a weak ability to extract fine-grained features of teeth and leads to low accuracy of results.

[0018] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a flowchart of an automatic tooth arrangement method for orthodontics based on deep learning of grid features according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of a triangular mesh model of a tooth according to an embodiment of the present invention;

[0022] Figure 3 is a structural diagram of an automatic tooth arrangement network model according to an embodiment of the present invention;

[0023] Figure 4 Schematic diagram of tooth arrangement effect according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0027] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0028] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0029] See also Figure 1 As shown, the provided automatic tooth arrangement method based on deep learning of grid features includes the following steps.

[0030] Step S110, constructing a training set using the tooth triangulated mesh three-dimensional model.

[0031] In one embodiment, the training set is constructed according to the following steps:

[0032] Step S111, segmenting and reconstructing the oral bone tissue model.

[0033] For example, a high-precision crown model and a three-dimensional model of complete teeth are obtained by segmenting and reconstructing the intraoral scan image (or laser scan image of a plaster model) and the oral CT image of a malocclusion patient, respectively, and then the root part of the high-precision crown model obtained from the dental laser scan image and the complete tooth model obtained from the oral CT image are registered and fused to obtain a three-dimensional triangular mesh model of teeth for tooth arrangement. The model is simplified so that the number of triangular facets of all teeth is unified to N. In one embodiment, N is 1000.

[0034] Step S112, constructing a tooth arrangement data set.

[0035] For example, after moving the malocclusion teeth to the desired position according to the six jaw standards, the triangular mesh models of all teeth in the malocclusion dentition and the desired dentition are saved respectively. The position of the desired dentition is the expected value of the tooth arrangement task, which is used as the supervision value of network training during the training process and as the gold standard during the test process. The data set is randomly divided into training set, validation set and test set.

[0036] In another embodiment, a three-dimensional oral cavity model established using post-treatment (teeth aligned) image data may be used as the expected value.

[0037] Step S113, enhancing the tooth arrangement data.

[0038] For example, data enhancement is divided into individual movement and overall movement. For individual movement, each tooth in the malocclusion dentition in the training set is randomly moved, and the malocclusion dentition before and after the movement corresponds to the same desired dentition. For overall movement, all malocclusion dentitions and desired dentitions in the training set are randomly moved synchronously to obtain the expanded training set. Through data enhancement, the data set can be expanded, thereby improving the accuracy of subsequent model training.

[0039] Step S114: data preprocessing.

[0040] Extract point cloud and mesh data from the above tooth model as input for the subsequent tooth arrangement network model and loss function. For example, the point cloud is the original vertices of all triangular facets, and the mesh data is calculated from the normalized tooth model, including the vertices, centroids, midlines, normal vectors, and adjacent face indices of the triangular facets, such as Figure 2 shown.

[0041] In one embodiment, normalizing the tooth model includes: scaling all teeth so that they are circumscribed to a unit sphere whose center of mass is the center of the sphere. The shape information of the tooth and its absolute spatial position information are decoupled by normalization, and only the relative position information between vertices is retained. This processing method improves the robustness of the subsequent automatic tooth arrangement network model for teeth of different shapes.

[0042] Step S120, constructing a deep learning model as a tooth arrangement network model, the model includes a dentition global feature encoder, a tooth shape feature encoder, and a feature decoder and mapper.

[0043] In this step, a deep learning model is constructed as an automatic tooth arrangement network model. In the description of this article, the deep learning model and the tooth arrangement network model or the automatic tooth arrangement network model have the same meaning unless otherwise specified according to the context.

[0044] See also Figure 3 As shown, the deep learning model includes a dentition global feature encoder, a tooth shape feature encoder, and a feature decoder and mapper.

[0045] (1) Dental global feature encoder

[0046] In one embodiment, the dentition global feature encoder may include multiple convolutional layers, each of which is sequentially connected to batch normalization and ReLU activation function.

[0047] The input of the dentition global feature encoder is the point cloud of the original malocclusion teeth. After convolution, the malocclusion features with global information are output.

[0048] (2) Tooth shape feature encoder

[0049] As an exception, referring to the automatic tooth arrangement network design of TANet (Wei G, Cui Z, Liu Y, et al. TANet: Towards Fully Automatic Tooth Arrangement [C] / / European Conference on Computer Vision. Springer, Cham, 2020: 481-497.) and Li et al. (Li X, Bi L, Kim J, et al. Malocclusion treatment planning via pointnet based spatial transformation network [C] / / International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2020: 105-114), the tooth shape feature encoder, for example, includes a series of identical parallel connected tooth shape feature extraction modules, each module corresponding to a single tooth one-to-one, so as to independently learn its shape representation from the data of a single tooth. Unlike the existing network, the tooth shape feature extraction module of the embodiment of the present invention learns feature representation from the triangular mesh model of the tooth to improve the ability to extract local fine-grained features of the tooth. The tooth shape feature extraction module includes a space description module, a structure description module and a grid convolution module.

[0050] It should be noted that the existing automatic tooth arrangement methods based on deep learning all use point cloud data and networks. The embodiment of the present invention proposes for the first time to use a mesh feature learning network to extract features from the triangular mesh model of teeth. MeshNet or other deep learning networks that process triangular mesh models can be used to achieve tooth-level feature extraction. By designing a tooth shape feature encoder based on mesh feature learning, the ability to extract fine-grained tooth shape features is improved.

[0051] In one embodiment, the spatial description module uses two one-dimensional convolution layers with a kernel size of 1 to map the centroid of the tooth in the local coordinate system to local spatial features, where the first convolution layer is used to extract local spatial features with a receptive field size of 1 from the local spatial position of the triangular facet. The second convolution layer is used to learn the transformation of features to align the feature vectors of triangular facets with similar shapes but different postures, and learn more robust local spatial features, denoted as

[0052] In one embodiment, the structure description module includes triangular patch rotation convolution and triangular patch kernel correlation. The triangular patch rotation convolution can be described using a symmetric function f, expressed as:

[0053]

[0054] Where h1(·,·): h2(·,·): There are two one-dimensional convolution layers respectively, h1 shares the weights of the combination of the three groups of midlines of the input, oa represents the three-dimensional vector formed by the centroid of the triangle and vertex A, ob represents the three-dimensional vector formed by the centroid of the triangle and vertex B, and oc represents the three-dimensional vector formed by the centroid of the triangle and vertex C. In this embodiment, K1=32, K2=64. The triangular patch rotation convolution operation has rotation invariance to eliminate the influence of the disorder of the midline on the extraction of unified triangular patch shape features.

[0055] Triangle patch kernel correlation is used to expand the receptive field of the structural feature vector to the adjacent patches of the triangle patch. Its input is the normal vector of the triangle patch and the adjacent face index. First, the normal vector of each face is combined with the normal vectors of the corresponding three adjacent faces through the adjacent face index to obtain the aggregated normal vector. Then, a kernel with learnable parameters is set to perform correlation operations with the aggregated normal vector.

[0056] Furthermore, the results of the triangle patch rotation convolution operation and the triangle patch kernel correlation operation are spliced, and then two layers of one-dimensional convolution with channels of 131 are used to map the spliced ​​vectors into structural features, which are recorded as

[0057] In order to increase the receptive field of local spatial features and structural features, a grid convolution module is used to reorganize the outputs of the spatial description module and the structural description module at the tooth level. and To extract deeper features.

[0058] For the local spatial features, they are concatenated with the corresponding structural feature vectors and then convolved to obtain the local spatial features that incorporate structural information, which can be expressed as:

[0059]

[0060] in Represents the concatenation of vectors, W mc1 It is a one-dimensional convolution with a convolution kernel size of 1 and 256 channels.

[0061] For structural features, similar to the normal vector aggregation operation of triangular facets in the structural description module, the structural feature vector of each triangular facet is first aggregated with the structural feature vector of the adjacent triangular facets through the adjacent face index, and then the aggregated structural feature vector is convolved and max pooled to obtain the aggregated structural features, which can be expressed as:

[0062]

[0063] Where gather represents the aggregation of structural feature vectors of adjacent triangular facets, W mc2 It is a two-dimensional convolution with a convolution kernel size of 1×1 and 256 channels.

[0064] Next, at the tooth level and Splice to get the tooth shape characteristics It can be expressed as:

[0065]

[0066] (3) Feature Decoder and Mapper

[0067] In the feature decoder and mapper, the global features of the dentition with the spatial position information of the teeth and the shape features of all teeth are fused and reduced in dimension, and then mapped into the movement of the teeth as the output of the tooth arrangement network.

[0068] exist Figure 3 In the decoder, the input dentition global feature F dentition and the shape characteristics of all teeth F shape ×N for splicing, and use two one-dimensional convolutional layers with 28 and 64 channels respectively and one fully connected layer to decode the spliced ​​features to obtain the movement to be transformed Then, the mobile representation transformation is performed, which may include two steps: constraint transformation and representation transformation.

[0069] For constrained transformation, in order to make the axis vector satisfy the unit vector and ensure the numerical stability of the transformation process, the softmax function is used to transform the axis vector L A axis =( L A0, L A1, LA2) Mapping to obtain a new axis vector L A axis ′:

[0070]

[0071] Where sign(·) represents the sign function. Considering that the movement of teeth before and after treatment does not exceed 20 mm in actual situations, the tanh function can be used to calculate the translation vector L t=( L A4, L A5, L A6) Mapping is performed to obtain the limited translation vector Lt ′.

[0072] For the representation transformation, the rotation and translation are transformed into a homogeneous transformation matrix in the world coordinate system, and the predicted tooth position is calculated. First, the rotation axis vector is converted to L A axis ′ is converted to a rotation matrix

[0073] L R=I+sin(θ) L A axis +(1-cos(θ)) L A axis (6)

[0074] Where θ = L A3, is the axis vector L A axis ′ is the antisymmetric matrix (Skew-symmetric matrix). The rotation matrix L R and translation vector L t′ is combined into the homogeneous transformation matrix in the tooth local coordinate system

[0075]

[0076] Then, L T is converted to the homogeneous transformation matrix in the world coordinate system W T, for the original malocclusion teeth Move the tooth to get the automatic tooth arrangement result, that is, the predicted tooth position, recorded as

[0077]

[0078] In the above-mentioned feature decoder and mapper, the rotation amount is constrained by using the softmax function, which ensures that the output movement amount is a rigid body transformation matrix and at the same time can meet the numerical stability of the calculation process.

[0079] S130, training a deep learning model based on a set loss function.

[0080] For example, the loss function is set using the Euclidean distance measurement network output of the tooth predicted position and the distance between the corresponding points of the expected position, expressed as:

[0081]

[0082] in, represents the expected tooth point cloud position, and avg represents the average.

[0083] It should be understood that in addition to directly using the average value of the Euclidean distance deviation between the predicted position and the expected position of the tooth point cloud after arrangement as the loss function, the maximum value of the Euclidean distance deviation can also be weighted and used as a separate item of the loss function. Alternatively, the cosine similarity error between the predicted transformation matrix and the expected transformation matrix can be used as the loss function.

[0084] Furthermore, the constructed deep learning model was trained using the training set. During the training process, the Adam optimizer was used to optimize the model parameters by minimizing the loss function value, and a series of network models with different parameters were obtained. The trained model was then verified using the validation set, and the network model with the best effect was saved as the final tooth arrangement network model, which was used to perform tooth arrangement operations on malocclusion samples that had not been tooth arranged.

[0085] Step S140, using the trained deep learning model to obtain the tooth arrangement result of the target patient.

[0086] After completing the model training and verification, the final tooth arrangement network model can be used for actual tooth arrangement prediction, that is, the triangular mesh three-dimensional model of the target patient's teeth to be arranged is input into the trained tooth arrangement network model to obtain the predicted tooth movement as the tooth arrangement result.

[0087] It should be noted that, without violating the spirit and scope of the present invention, those skilled in the art may make appropriate changes or modifications to the above embodiments. For example, in the feature decoder and mapper, in addition to using the softmax function to constrain the rotation amount, other methods may also be used to ensure that the three-dimensional axis vector used to represent the rotation is always a unit vector and that numerical stability is satisfied during calculation. For example:

[0088]

[0089] in LA axis is the axis vector in the local coordinate system, k=0, 1, 2, respectively representing the three-dimensional components of the axis vector.

[0090] Accordingly, the present invention also provides an automatic tooth arrangement system for orthodontics based on deep learning of mesh features. The system comprises: a tooth model acquisition unit, which is used to acquire a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model uses the vertices, centroids, midlines, normal vectors and adjacent face indexes of triangular facets to characterize the shape information and spatial position information of the teeth; a prediction unit, which is used to input the tooth triangular mesh model into a pre-trained deep learning model to obtain the predicted tooth position of the tooth arrangement result, wherein the deep learning model comprises a dentition global feature encoder, a tooth shape feature encoder and a feature decoder and mapper, wherein the tooth shape feature encoder uses the tooth triangular mesh three-dimensional model as input to obtain tooth shape features, the dentition global feature encoder uses the dentition point cloud as input to obtain dentition global features, and the feature decoder and mapper obtains the predicted tooth arrangement result based on the tooth shape features and the dentition global features. Each unit in the system can be implemented using DSP, FPGA, processor or dedicated hardware.

[0091] To further verify the effect of the present invention, a comparative experiment was conducted on the same data set between the method proposed in the present invention and the existing similar method based on deep learning, TANet. The results show that the automatic tooth arrangement method of the present invention can learn the mapping relationship between malocclusion and ideal tooth position from the data before and after tooth arrangement, and the accuracy of the automatic tooth arrangement result is higher than that of the similar method. Figure 4 As shown, Figure 4 (a) Corresponding to malocclusion, Figure 4 (b) is the tooth arrangement result of TANet, Figure 4 (c) is the tooth arrangement result of the present invention, Figure 4 (d) is the desired dentition, Figure 4 Each row in represents a sample. Figure 4 It can be seen that for the first sample, the positions of the central incisor and lateral incisor are in line with expectations, but the premolar gap in the TANet output result is too large; for the second sample, the crowding of the right maxillary premolars is improved; for the third sample, the dentition is obviously more orderly after tooth arrangement, but compared with the expected dentition obtained by artificial tooth arrangement, the tooth gap is too small or even penetrates the mold.

[0092] The model training process involved in the present invention can be performed offline on a server or in the cloud, and the trained model can be embedded in an electronic device to achieve real-time automatic tooth arrangement result generation. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a car computer, an intelligent wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be an independent server, a cluster server or a cloud server. In addition to being applied to automatic tooth arrangement in the field of orthodontics, the present invention can also be extended to the task of automatic denture arrangement in the field of oral prosthodontics.

[0093] In summary, the present invention designs a mesh feature learning network to extract shape features from the triangular mesh model of teeth, and on this basis learns the shape features and the underlying rules of global spatial features and the expected positions of teeth to arrange teeth. Compared with the prior art, the present invention has the following advantages:

[0094] 1) The existing deep learning-based automatic tooth arrangement method uses scanned images of the crown as input, which only contains crown information, but not the root and alveolar bone information inside the gums. Therefore, it is impossible to explicitly consider the root posture. In clinical use, the root may be too close to or even move out of the alveolar bone, resulting in root absorption, bone fenestration and bone cracking, which does not meet clinical requirements. The present invention considers the root posture, inputs the complete tooth 3D model data containing the root into the dentition global feature encoder and the tooth shape feature encoder, and the loss function also calculates the distance error of the tooth arrangement result containing the root accordingly, and then realizes the introduction of the root posture into the automatic tooth arrangement algorithm by optimizing the network model parameters.

[0095] 2) The existing deep learning-based automatic tooth arrangement method uses a point cloud processing network as a tooth point cloud feature encoder, which has limited ability to extract fine-grained features. The present invention uses a mesh processing network to extract tooth shape features, which can more fully extract features from triangular mesh data with object surface topological relationships, and designs a tooth shape encoder based on mesh feature learning to extract fine-grained shape features of the tooth triangular mesh model, thereby improving the accuracy of deep learning network automatic tooth arrangement.

[0096] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0097] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0098] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0099] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0100] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0101] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0102] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0103] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0104] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An automatic tooth arrangement method for orthodontics based on deep learning of grid features. The following steps are involved: Acquire a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model uses vertices, centroids, midlines, normal vectors and adjacent surface indices of triangular facets to represent shape information and spatial position information of the teeth; Inputting the tooth triangular mesh three-dimensional model into a pre-trained deep learning model to obtain the predicted tooth position of the tooth arrangement result; The deep learning model includes a first feature encoder, a second feature encoder and a feature decoder and mapper, the first feature encoder uses the tooth triangulated mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder uses the dentition point cloud as input to obtain dentition global features, and the feature decoder and mapper obtains the predicted tooth arrangement result based on the tooth shape features and the dentition global features; The first feature encoder includes a plurality of tooth shape feature extraction modules connected in parallel, each tooth shape feature extraction module corresponds to a single tooth one-to-one, and is used to independently learn the shape representation of a single tooth from data of the single tooth; The tooth shape feature extraction module includes a space description module, a structure description module and a grid convolution module. The space description module takes the centroid of the tooth as input, and is used to map the centroid in the local coordinate system of the tooth to a local space feature and learn the transformation between local space features; the structure description module takes the midline, normal vector and adjacent surface index of the tooth as input, and obtains the structure feature through rotation convolution and kernel-related operations; the grid convolution module obtains the tooth shape feature by fusing the local space feature and the structure feature; Wherein, the tooth shape feature is expressed as: in: in, Represents the concatenation of vectors, W mc1 is a one-dimensional convolution, gather represents the aggregation of structural feature vectors of adjacent triangular faces, and W mc2 is a two-dimensional convolution, is the output of the spatial description module, and is the output of the structure description module; Wherein, the feature decoding and mapping performs: The global features of the dentition and the shape features of all teeth are spliced, and the spliced ​​features are decoded to obtain the movement amount to be transformed; For the movement amount to be transformed, a softmax function is used to map the corresponding rotation axis vector to obtain a new rotation axis vector; The rotation and translation are transformed into a homogeneous transformation matrix in the world coordinate system, and the predicted tooth position is calculated.

2. The method according to claim 1, It is characterized in that The tooth triangulated mesh three-dimensional model is obtained according to the following steps: The crown model and the complete three-dimensional tooth model are obtained by segmenting and reconstructing the target intraoral scan image and the oral cavity image; The crown model and the root part of the three-dimensional tooth model are registered and fused to obtain the three-dimensional tooth triangulated mesh model for tooth arrangement.

3. The method according to claim 1, It is characterized in that The loss function for training the deep learning model is expressed as: in, Indicates the predicted position of the tooth point cloud after arrangement, represents the expected tooth point cloud position, and avg represents the average.

4. The method according to claim 1, It is characterized in that The training set for training the deep learning model is obtained according to the following steps: Collect multiple case samples to construct a tooth arrangement data set, each data in the tooth arrangement data set reflects the corresponding relationship between the tooth triangular mesh three-dimensional model of the tooth arrangement and the expected tooth triangular mesh three-dimensional model; Performing data enhancement on the tooth arrangement data set to obtain an expanded tooth arrangement data set, wherein the data enhancement includes: Randomly moving each tooth in the malocclusion dentition in the tooth arrangement data set, so that the malocclusion dentition before and after the movement corresponds to the same desired dentition; All malocclusion dentition and desired dentition in the tooth arrangement data set are subjected to synchronous random movement.

5. An automatic orthodontic tooth arrangement system based on deep learning of grid features. include: A tooth model acquisition unit is used to acquire a tooth triangular mesh three-dimensional model for tooth arrangement, wherein the tooth triangular mesh three-dimensional model uses vertices, centroids, midlines, normal vectors and adjacent surface indexes of triangular facets to represent shape information and spatial position information of the tooth; Prediction unit: used for inputting the tooth triangular mesh three-dimensional model into the pre-trained deep learning model to obtain the tooth prediction position of the tooth arrangement result; The deep learning model includes a first feature encoder, a second feature encoder and a feature decoder and mapper, the first feature encoder uses the tooth triangulated mesh three-dimensional model as input to obtain tooth shape features, the second feature encoder uses the dentition point cloud as input to obtain dentition global features, and the feature decoder and mapper obtains the predicted tooth arrangement result based on the tooth shape features and the dentition global features; The first feature encoder includes a plurality of tooth shape feature extraction modules connected in parallel, each tooth shape feature extraction module corresponds to a single tooth one-to-one, and is used to independently learn the shape representation of a single tooth from data of the single tooth; The tooth shape feature extraction module includes a space description module, a structure description module and a grid convolution module. The space description module takes the centroid of the tooth as input, and is used to map the centroid in the local coordinate system of the tooth to a local space feature and learn the transformation between local space features; the structure description module takes the midline, normal vector and adjacent surface index of the tooth as input, and obtains the structure feature through rotation convolution and kernel-related operations; the grid convolution module obtains the tooth shape feature by fusing the local space feature and the structure feature; Wherein, the tooth shape feature is expressed as: in: in, Represents the concatenation of vectors, W mc1 is a one-dimensional convolution, gather represents the aggregation of structural feature vectors of adjacent triangular faces, and W mc2 is a two-dimensional convolution, is the output of the spatial description module, and is the output of the structure description module; Wherein, the feature decoding and mapping performs: The global features of the dentition and the shape features of all teeth are spliced, and the spliced ​​features are decoded to obtain the movement amount to be transformed; For the movement amount to be transformed, a softmax function is used to map the corresponding rotation axis vector to obtain a new rotation axis vector; The rotation and translation are transformed into a homogeneous transformation matrix in the world coordinate system, and the predicted tooth position is calculated.

6. A computer-readable storage medium having a computer program stored thereon, in, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images