A 3D tooth segmentation method based on multi-branch fusion learning

Abstract

The present invention belongs to the field of medical image processing and computer vision technology, and particularly relates to a three-dimensional tooth segmentation method based on multi-branch fusion learning, including data downsampling and feature extraction of the original three-dimensional tooth model; feature information separation of the extracted feature information to obtain coordinate feature information, vector feature information and curvature feature information; using the K nearest neighbor algorithm to construct a local area of ​​the grid, that is, taking the center point of the grid as the base point, finding the K grids closest to the grid, and the grid and the K grids closest to the grid together constitute the range of the local area; for the coordinate feature information in the local area, using the graph convolution based on the attention mechanism to learn the fine-grained geometric features of the teeth; for the vector feature information in the local area, using the graph convolution based on the maximum pooling to learn the significant local detail features of the teeth; for the curvature feature information in the global area, using the maximum pooling and average pooling based on the offset attention mechanism to learn the global boundary constraint features of the teeth. The fine-grained geometric features, the significant local detail features and the global boundary constraint information are fused to obtain a feature description of the teeth; the present invention effectively improves the segmentation accuracy of the three-dimensional tooth model.

Description

Technical Field

[0001] The present invention belongs to the technical fields of medical image processing and computer vision, and in particular relates to a three-dimensional tooth segmentation method based on multi-branch fusion learning. Background Art

[0002] Oral health is an essential component of public health. With socioeconomic development and improved quality of life, people are paying more attention to their dental health. Currently, there is a significant shortage of dental professionals in my country. However, the application of digital dental technology will provide a breakthrough in addressing the shortage of dentists facing the traditional dental industry.

[0003] Thanks to the rapid development of artificial intelligence, digital dental care is gradually breaking down the barriers of traditional dental care. Computer-assisted treatment systems are particularly widely used in the dental field, significantly improving the efficiency of dental treatment. Accurately segmenting teeth from oral scans is a key task for computer-assisted treatment systems. The segmentation results can be applied to restoration, correction, realignment, and implant placement, assisting dentists in developing patient treatment plans.

[0004] Due to the disorder and unstructured nature of the three-dimensional data structure; the complex dental conditions of patients, including tooth misalignment, missing teeth, and teeth with brackets; and the large differences in tooth distribution and arrangement between patients. The above difficulties pose huge challenges to digital three-dimensional tooth segmentation. Traditional digital three-dimensional tooth segmentation methods rely on a large amount of prior knowledge and require a certain amount of manual intervention, and cannot achieve fully automatic and intelligent tooth segmentation. Existing digital three-dimensional tooth segmentation methods based on deep learning have poor results in tooth disorder and tooth-gum boundary segmentation, and often suffer from over-segmentation or under-segmentation, as well as unclear boundary segmentation. Summary of the Invention

[0005] In order to effectively solve the problem of inaccurate segmentation of complex teeth and inaccurate segmentation of teeth and gum boundaries in 3D tooth segmentation, the present invention proposes a 3D tooth segmentation method based on multi-branch fusion learning, which specifically includes the following steps:

[0006] S1. Downsample the original 3D tooth model and perform feature extraction;

[0007] S2. Separate the extracted feature information to obtain coordinate feature information, vector feature information, and curvature feature information;

[0008] S3. Use the K-nearest neighbor algorithm to construct a local area of ​​the grid. That is, using the center point of the grid as the base point, find the K grids closest to the grid. The grid and the K nearest neighboring grids together constitute the scope of the local area.

[0009] S4. For the coordinate feature information in the local area, the graph convolution based on the attention mechanism is used to learn the fine-grained geometric features of the teeth; for the vector feature information in the local area, the graph convolution based on the maximum pooling is used to learn the salient local detail features of the teeth; for the curvature feature information in the global area, the maximum pooling and average pooling based on the offset self-attention mechanism are used to learn the global boundary constraint features of the teeth.

[0010] S5. Fusing fine-grained geometric features, significant local detail features, and global boundary constraint features to obtain a feature description of the tooth;

[0011] S6. Input the feature description of the teeth into the multi-layer perceptron to obtain the tooth segmentation prediction.

[0012] The present invention adopts a method based on multi-branch fusion learning to segment the three-dimensional tooth model, extracts features of different attribute features respectively, obtains fine-grained geometric features of the teeth, significant local detail features and global boundary constraint features, and explores the shared feature information and unique feature information between features of different attributes through multiple views to obtain a more complete tooth feature description, effectively improving the segmentation accuracy of the three-dimensional tooth model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the overall process of a three-dimensional tooth segmentation method based on multi-branch fusion learning of the present invention;

[0014] Figure 2 Schematic diagram of the three-dimensional tooth segmentation network structure based on multi-branch fusion learning of the present invention;

[0015] Figure 3 Schematic diagram of the coordinate information branch structure of the present invention;

[0016] Figure 4 Schematic diagram of the vector information branch structure of the present invention;

[0017] Figure 5 Schematic diagram of the curvature information branch structure of the present invention;

[0018] Figure 6 The three-dimensional tooth model image of the present invention contains different attribute geometric features. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] This paper proposes a three-dimensional tooth segmentation method based on multi-branch fusion learning. Figure 1 , specifically including the following steps:

[0021] S1. Downsample the original 3D tooth model and perform feature extraction;

[0022] S2. Separate the extracted feature information to obtain coordinate feature information, vector feature information, and curvature characteristic information;

[0023] S3. Use the K-nearest neighbor algorithm to construct a local area of ​​a grid, that is, the K grids closest to the current grid are used as the local area range of the current grid;

[0024] S3. For the coordinate feature information in the local area, the graph convolution based on the attention mechanism is used to learn the fine-grained geometric features of the teeth; for the vector feature information in the local area, the graph convolution based on the maximum pooling is used to learn the salient local detail features of the teeth; for the curvature feature information in the global area, the maximum pooling and average pooling based on the offset attention mechanism are used to learn the global boundary constraint features of the teeth.

[0025] S4, fusing fine-grained geometric features, significant local detail features, and global boundary constraint features to obtain a feature description of the tooth;

[0026] S5. Input the feature description of the teeth into the multi-layer perceptron to obtain the tooth segmentation prediction.

[0027] In this embodiment, the vertex coordinates in the digital three-dimensional tooth model are used to calculate the center point coordinates of the grid and the normal vector of the grid surface, and the features of the coordinate feature information and the vector feature information are obtained. Specifically, for the grid m i For example, the coordinate values ​​of its three vertices are v i,1 =[x1,y1,z1],v i,2 =[x2,y2,z2],v i,3 =[x3,y3,z3], the normal vector values ​​of the three vertices are divided into n i,1 =[o1,p1,q1],n i,2 =[o2,p2,q2],n i,3 =[o3,p3,q3]. For grid m i , the center point coordinates m i,c The calculation formula is as follows:

[0028]

[0029] Its surface normal vector m i,n The calculation formula is as follows:

[0030]

[0031] Its eigenvector F i,total :

[0032] F i,total =[v i,1 ,v i,2 ,v i,3 ,m i,c ,n i,1 ,n i,2 ,n i,3 ,m i,n ,r i,1 ,r i,2 ,r i,3 ]

[0033] The overall segmentation network structure diagram of the present invention is as follows Figure 2 As shown in the figure, the network separates the features obtained from the downsampled digitized 3D tooth data into coordinate features, vector features, and curvature features:

[0034] F coor =[v i,1 ,v i,2 ,v i,3 ,m i,c ]

[0035] F nor =[n i,1 ,n i,2 ,n i,3 ,m i,n ]

[0036] F cur =[r i,1 ,r i,2 ,r i,3 ]

[0037] The separated coordinate features, vector features and curvature features are sent to three parallel branches for feature extraction. The two parallel branches of coordinate branch and vector branch share the local structure graph constructed by each grid point using K nearest neighbor algorithm. i , searching for the K grid center coordinates closest to it in Euclidean distance to construct the local region. This local region composition consists of K+1 grid center points. This local region composition contains both the absolute and relative geometric spatial information of the grid, shared by the coordinate information branch and the vector information branch, and updated iteratively in different graph convolutional layers. In the curvature branch, an offset attention mechanism is used to exchange information, and global boundary constraint features are extracted through maximum and average pooling.

[0038] The coordinate branch has three consecutive graph convolution layers based on the attention mechanism to extract the fine-grained geometric characteristics of the teeth; the vector branch has three consecutive graph convolution layers based on maximum pooling to extract the local characteristics of the teeth; the curvature branch has maximum pooling and average pooling layers based on the offset attention mechanism to extract the global boundary constraint characteristics of the teeth. Then, the coordinate branch and the vector branch respectively perform multi-scale feature fusion on the features generated by the graph convolution layer. The results of the multi-scale feature fusion are as follows:

[0039]

[0040] Then, multi-view learning is used to explore the shared and unique features of the features generated by the three branches, and multi-feature fusion is performed to achieve complementarity between features. The feature fusion method is as follows:

[0041]

[0042] Finally, a multi-layer perceptron is used to predict the segmentation of teeth. The expression is:

[0043] Y = Softmax(MLP(I))

[0044] Furthermore, the loss function used by the segmentation network is the cross entropy loss function (CrossEntropy Loss), and its mathematical expression is as follows:

[0045]

[0046] Where N is the total number of meshes in the 3D tooth model, y i is the grid m i Ground Trur (GT) segmentation label, p(x i ) is the predicted segmentation label of the network model.

[0047] The trained 3D tooth segmentation network based on multi-branch fusion learning will classify and judge any triangular mesh in the 3D tooth model. The segmentation network will eventually output a classification prediction vector for each triangular mesh. The dimension of the classification prediction vector is 8, corresponding to the 8 possible classification situations in the 3D tooth model, including gums and 7 tooth types. i The network will select the category corresponding to the dimension with the largest probability value in the classification prediction vector as the classification result of the grid. When all the triangulated meshes are classified, the network can use the classification results to accurately segment the three-dimensional tooth model.

[0048] The schematic diagram of the coordinate information branch structure of the present invention is as follows Figure 3As shown in the figure, the coordinate information branch consists of three graph convolutional layers based on the attention mechanism. Each graph convolutional layer contains three parts shared with the vector information branch: K-nearest neighbor composition, multi-layer perceptron, and attention layer. Specifically, it includes the following steps:

[0049] Based on the K-nearest neighbor algorithm, the absolute and relative geometric spatial information of each grid is used as input, and the local information of each grid is aggregated using a multi-layer perceptron to obtain richer spatial feature information;

[0050] All grids within the local region composition are weighted by a lightweight multi-layer perceptron and feature aggregated to obtain fine-grained geometric features;

[0051] The multi-scale features extracted in the coordinate branch are: and

[0052] The extracted multi-scale features are spliced ​​and fused to obtain the multi-scale fusion features of the coordinate branch

[0053]

[0054] The schematic diagram of the vector information branch structure is as follows: Figure 4 As shown in the figure, the vector information branch consists of three graph convolutional layers based on maximum pooling. Each graph convolutional layer includes three parts shared with the coordinate information branch: K-nearest neighbor mapping, multi-layer perceptron, and maximum pooling layer. Specifically, the following steps are included:

[0055] A multi-layer perceptron is used to aggregate the local information of each grid to obtain richer semantic feature information. The feature information is then passed through a maximum pooling layer to obtain fine-grained geometric features.

[0056] The multi-scale features extracted in the vector branch are: and

[0057] The extracted multi-scale features are spliced ​​and fused to obtain the multi-scale fusion features of the vector branch

[0058]

[0059] The schematic diagram of the curvature information branch structure is as follows: Figure 5 As shown in the figure, the curvature information branch consists of an offset attention layer, a maximum pooling layer, an average pooling layer, and a multi-layer perceptron, which specifically includes the following steps:

[0060] Use the offset attention layer to obtain more prominent boundary feature information, and use maximum pooling and average pooling to extract global boundary constraint features.

[0061] The three-dimensional tooth model image obtained after segmentation of the present invention is as follows Figure 6 As shown, the image includes fine-grained tooth set features, significant local detail features and global boundary constraint features. This embodiment explores the shared feature information and unique feature information of features between different attributes through multiple views to obtain a more complete tooth feature description, effectively improving the segmentation accuracy of the three-dimensional tooth model.

[0062] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A three-dimensional tooth segmentation method based on multi-branch fusion learning, characterized in that: The specific steps include: S1. Downsampling the original three-dimensional tooth model and extracting features, specifically including the following steps: Down-sampling the number of meshes in the original three-dimensional tooth model to M; The coordinate value of the center point of the grid and the normal vector value of the grid surface are obtained according to the coordinate values ​​and normal vector values ​​of the three vertices. That is, if the network m i The coordinate values ​​of the three vertices are v i,1 =[x1,y1,z1],v i,2 =[x2,y2,z2],v i,3 =[x3,y3,z3], the normal vector values ​​of the three vertices are divided into n i,1 =[o1,p1,q1],n i,2 =[o2,p2,q2],n i,3 =[o3,p3,q3], then the grid m i The center point coordinates m i,c It is expressed as: Grid m i The surface normal vector is expressed as: Among them, i represents the unit vector in the positive direction of the x-axis in the three-dimensional space, j represents the unit vector in the positive direction of the y-axis in the three-dimensional space, and k represents the unit vector in the positive direction of the z-axis in the three-dimensional space; Using the Normal Cycles theory to calculate the grid m i The curvature information of each vertex in the graph includes the maximum curvature, minimum curvature, average curvature and Gaussian curvature. The curvature information of the three vertices is represented by r i,1 =[b1,s1,a1,g1],r i,2 =[b2,s2,a2,g2],r i,3 =[b3,s3,a3,g3]; S2. Separate the extracted feature information to obtain coordinate feature information, vector feature information and curvature feature information, where the coordinate feature information is expressed as: F coor =[v i,1 ,v i,2 ,v i,3 ,m i,c ] The vector feature information is expressed as: F nor =[n i,1 ,n i,2 ,n i,3 ,m i,n ] The curvature feature information is expressed as: F cur ＝[r i,1 ,r i,2 ,r i,3 ]； Among them, v i,1 、v i,2 、v i,3 Respectively represented as grid m i The three vertex coordinates, n i,1 、n i,2 、n i,3 They are v i,1 、v i,2 、v i,3 Normal vectors of three vertices; m i,c v i,1 、v i,2 、v i,3 The center of the three vertices; m i,n v i,1 、v i,2 、v i,3 A mesh consisting of three vertices m i The surface vector of i,1 、r i,2 、r i,3 They are the curvature information of the three vertices respectively; S3, using the K nearest neighbor algorithm to construct the local area of ​​the grid, that is, taking the center point of the grid as the base point, finding the K grids closest to the grid, and the grid and the K nearest adjacent grids together constitute the scope of the local area; S4. For the coordinate feature information in the local area, the graph convolution based on the attention mechanism is used to learn the fine-grained geometric features of the teeth; for the vector feature information in the local area, the graph convolution based on the maximum pooling is used to learn the salient local detail features of the teeth; for the curvature feature information in the global area, the maximum pooling and average pooling based on the offset attention mechanism are used to learn the global boundary constraint features of the teeth; S5. Fusing fine-grained geometric features, significant local detail features, and global boundary constraint features to obtain a feature description of the teeth, including: Where I represents the characteristic description of the teeth obtained by fusion; α1, α2, α3 and β are hyperparameters and α1+α2+α3+β=1; represents the fine-grained geometric features obtained based on the coordinate feature information, Represents the significant local detail features obtained based on the vector feature information, represents the global boundary constraint information obtained according to the curvature eigenvector; F s Represents the shared information of the two branches, and The sum and average of represents the output of the i-th graph convolution layer in the graph convolution based on the attention mechanism, where i∈{1,2,3}; Represents the output of the i-th graph convolution layer based on maximum pooling; Indicates the splicing operation of feature information in channel form; S6. Input the feature description of the teeth into a multi-layer perceptron to obtain tooth segmentation prediction.

2. A three-dimensional tooth segmentation method based on multi-branch fusion learning according to claim 1, characterized in that: In step S1, feature extraction is performed on the downsampled data, and the extracted features are expressed as: F i,total =[v i,1 ,v i,2 ,v i,3 ,m i,c ,n i,1 ,n i,2 ,n i,3 ,m i,n ,r i,1 ,r i,2 ,r i,3 ]。 3. A three-dimensional tooth segmentation method based on multi-branch fusion learning according to claim 1, characterized in that: The calculation of the distance between two grids in step S3 includes: Among them, d i,j Represents the grid m i With grid m j The distance between i,c,n Represents the coordinates (x, y, z) in three-dimensional space of the grid with grid number i and grid center point c.

4. The three-dimensional tooth segmentation method based on multi-branch fusion learning according to claim 1, characterized in that: The graph convolution based on the attention mechanism includes a four-layer structure, including three cascaded first graph convolution layers and a multi-scale feature fusion layer, where the output of each first graph convolution layer is input to the multi-scale feature fusion layer for fusion, and each first graph attention layer is composed of a multi-layer perceptron and an attention mechanism.

5. The three-dimensional tooth segmentation method based on multi-branch fusion learning according to claim 1, characterized in that: The graph convolution based on maximum pooling includes a four-layer structure, including three cascaded second graph convolution layers and a multi-scale feature fusion layer, where the output of each second graph convolution layer is input to the multi-scale feature fusion layer for fusion, and each second graph attention layer is composed of a multi-layer perceptron and a maximum pooling layer.

6. The three-dimensional tooth segmentation method based on multi-branch fusion learning according to claim 1, characterized in that: Global boundary constrained feature extraction based on maximum pooling and average pooling of shifted attention mechanism, which consists of multi-layer perceptron, shifted attention layer, maximum pooling layer and average pooling layer.

Images