High-precision tooth segmentation method based on double-flow network and terminal

Abstract

The invention discloses a high-precision tooth segmentation method based on a double-flow network and a terminal, which are used for simplifying acquired tooth grid training data to obtain standard grid training data, so that the data volume of the tooth grid data is reduced, and the data processing efficiency of a subsequent double-flow network learning model is improved. Inputting the standard grid training data obtained through simplification into a double-flow network learning model, and extracting a position feature flow and a shape feature flow from the standard grid training data through a position feature extraction unit and a shape feature extraction unit so as to extract geometric information with high distinction degree; according to the method, the dependency between the patch units in the grid data is reduced, so that the tooth segmentation model obtained through training can better identify detail features, the generalization ability of the model is improved, and the tooth segmentation model can accurately segment the high-precision tooth grid training data.

Description

Technical Field

[0001] The present invention relates to the technical field of tooth segmentation, and in particular to a high-precision tooth segmentation method and terminal based on a dual-stream network. Background Art

[0002] With the development of artificial intelligence technology, segmenting tooth models through deep learning models has become a trend. At present, the image segmentation method based on deep learning models has basically matured, but because the triangular mesh is more complex than the digital image in data structure, the mesh segmentation theory and technical model based on deep learning models are still in the stage of discussion and exploration.

[0003] 2D (two-dimensional) / 3D (three-dimensional) digital images are composed of regularly distributed pixels / voxels with fixed adjacency relationships between each other, while triangular meshes are three-dimensional mesh structures composed of different primitives (i.e. points, lines, and surfaces), and the number of neighbors of primitives is often not fixed. Therefore, the convolution defined in the image segmentation method cannot be directly used in the mesh segmentation method.

[0004] In order to achieve deep learning on the mesh, the early idea is to convert it into a series of 2D projection images or 3D volume data, so as to follow the existing mature method system. However, errors will be introduced during the conversion process, resulting in performance bottlenecks in the model. Therefore, point cloud segmentation and patch segmentation methods based on mesh data are proposed. The point cloud segmentation method usually uses the PointNet model to extract vertices (or patch center points) from the triangular mesh as point cloud data, and then directly applies it to mesh segmentation. The cutting-edge methods based on point clouds also include RandLA-Net, FCPN, KPConv, PointCNN, PointConv, etc. These methods all take into account the rotation invariance and disorder of point clouds. No unnecessary conversion is required, and point clouds can be directly processed to segment three-dimensional data. However, the richness of the original features of point cloud data is insufficient, so the segmentation accuracy is not high. Patch segmentation has more advantages than point cloud segmentation. Therefore, the current tooth segmentation methods are mainly based on patches, and its typical representatives include GACNet, SGTNet, TSGCNet, etc. Although many feasible patch segmentation algorithms have emerged, there are some problems with the evaluation methods and generalization capabilities of these algorithms. In terms of evaluation methods, the effectiveness of face segmentation methods is evaluated on simplified and distorted data (distortion includes boundary disappearance), and the methods evaluated in this way are inaccurate. In terms of generalization ability, since the tooth segmentation model is trained on low-precision data (simplified and distorted data), when the above tooth segmentation model is applied to high-precision data, many faces are misidentified.

[0005] In summary, current tooth segmentation methods cannot accurately segment high-precision tooth data. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a high-precision tooth segmentation method and terminal based on a dual-stream network, which can accurately segment high-precision tooth data.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: A high-precision tooth segmentation method based on a two-stream network, comprising: Acquire tooth mesh training data and facet label data of the tooth mesh training data; Simplifying the tooth grid training data to obtain standard grid training data; Inputting the standard grid training data into a preset dual-stream network learning model, wherein the dual-stream network learning model includes a position feature extraction unit and a shape feature extraction unit; Extracting a position feature stream from the standard grid training data by the position feature extraction unit; Extracting a shape feature flow from the standard grid training data by the shape feature extraction unit; The position feature stream and the shape feature stream are fused to obtain a target feature stream; Segmenting the standard grid training data according to the target feature flow to obtain predicted tooth data; The dual-stream network learning model is iteratively trained according to the patch label data and the predicted tooth data to obtain a tooth segmentation model, and the target tooth mesh training data is segmented by the tooth segmentation model.

[0008] In order to solve the above technical problems, another technical solution adopted by the present invention is: A high-precision tooth segmentation terminal based on a dual-stream network includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, each step of the high-precision tooth segmentation method based on the dual-stream network is implemented.

[0009] The beneficial effect of the present invention is that the acquired tooth mesh training data is simplified to obtain standard mesh training data, thereby reducing the data volume of the tooth mesh data and improving the data processing efficiency of the subsequent dual-stream network learning model. The simplified standard mesh training data is input into the dual-stream network learning model, and the position feature stream and the shape feature stream are extracted from the standard mesh training data by the position feature extraction unit and the shape feature extraction unit respectively, so as to extract geometric information with high discrimination, reduce the dependency between each patch unit in the mesh data, so that the trained tooth segmentation model can better identify detail features, thereby improving the generalization ability of the model, so that the tooth segmentation model can accurately segment the high-precision tooth mesh training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a high-precision tooth segmentation method based on a dual-stream network provided by an embodiment of the present invention; Figure 2 A comparison diagram of the effects of a mesh simplification method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the architecture of a dual-stream network learning model provided by an embodiment of the present invention; Figure 4 An error value of a tooth segmentation boundary under different grid accuracies provided by an embodiment of the present invention; Figure 5 A tooth segmentation model provided in an embodiment of the present invention and a segmentation effect diagram of other tooth segmentation models; Figure 6 A schematic diagram of the architecture of a position feature extraction unit provided in an embodiment of the present invention; Figure 7 A schematic diagram of the architecture of a multi-feature fusion module provided in an embodiment of the present invention; Figure 8 A schematic diagram of the architecture of an attention gating module provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the architecture of a channel attention module provided by an embodiment of the present invention; Fig.10 A schematic diagram of the architecture of a spatial attention module provided by an embodiment of the present invention; Fig.11 A schematic diagram of the architecture of a shape feature extraction unit provided in an embodiment of the present invention; Fig.12 A schematic diagram of the structure of a high-precision tooth segmentation terminal based on a dual-stream network provided by an embodiment of the present invention; Description of labels: 100. A high-precision tooth segmentation terminal based on a dual-stream network; 101. Memory; 102. Processor. DETAILED DESCRIPTION

[0011] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0012] An embodiment of the present invention provides a high-precision tooth segmentation method based on a dual-stream network, comprising: Acquire tooth mesh training data and facet label data of the tooth mesh training data; Simplifying the tooth grid training data to obtain standard grid training data; Inputting the standard grid training data into a preset dual-stream network learning model, wherein the dual-stream network learning model includes a position feature extraction unit and a shape feature extraction unit; Extracting a position feature stream from the standard grid training data by the position feature extraction unit; Extracting a shape feature flow from the standard grid training data by the shape feature extraction unit; The position feature stream and the shape feature stream are fused to obtain a target feature stream; Segmenting the standard grid training data according to the target feature flow to obtain predicted tooth data; The dual-stream network learning model is iteratively trained according to the patch label data and the predicted tooth data to obtain a tooth segmentation model, and the target tooth mesh training data is segmented by the tooth segmentation model.

[0013] From the above description, it can be seen that the beneficial effect of the present invention is that the acquired tooth mesh training data is simplified to obtain standard mesh training data, thereby reducing the data volume of the tooth mesh data and improving the data processing efficiency of the subsequent dual-stream network learning model. The simplified standard mesh training data is input into the dual-stream network learning model, and the position feature stream and shape feature stream are extracted from the standard mesh training data by the position feature extraction unit and the shape feature extraction unit respectively, so as to extract geometric information with high discrimination, reduce the dependency between each patch unit in the mesh data, so that the trained tooth segmentation model can better identify detail features, thereby improving the generalization ability of the model, so that the tooth segmentation model can accurately segment the high-precision tooth mesh training data.

[0014] Further, the tooth mesh training data includes vertex coordinate data; The simplifying process of the tooth mesh training data to obtain standard mesh training data comprises: Convert the face label data into RGB color data; Constructing an error matrix according to the vertex coordinate data and the RGB color data; Calculating distance metric values ​​between vertex pairs using a quadratic error metric according to the error matrix; Vertex pairs in the vertex coordinate data are merged according to the RGB color data and the distance metric value to obtain standard grid training data.

[0015] From the above description, it can be seen that the vertex coordinate data is the vertex coordinate of each facet in the tooth mesh training data, and the facet label data is the tooth classification label corresponding to each facet in the tooth mesh training data, where the tooth classification label is represented by different colors. Since the mesh simplification method based on quadratic error metric inputs vertex coordinate data and facet label data when constructing the error matrix, when the mesh is simplified to a new coordinate point, its facet label data may produce errors. In order to eliminate this error, the facet label data is converted into continuous, multi-dimensional RGB color data, so as to more finely represent the attribute characteristics of the facet.

[0016] Further, the merging of vertex pairs in the vertex coordinate data according to the RGB color data and the distance metric value to obtain standard grid training data includes: Calculate the Euclidean distance between each pair of vertices according to the RGB color data; Detecting whether the Euclidean distance is greater than a preset distance threshold, and if so, setting the distance metric value of the vertex pair corresponding to the Euclidean distance to the maximum value of the distance metric values ​​in the vertex coordinate data; The vertex pair with the smallest distance metric value is obtained from the vertex coordinate data and merged to obtain new vertex coordinate data, and the step of constructing an error matrix according to the vertex coordinate data and the RGB color data is returned to execute until the tooth mesh training data reaches the set accuracy to obtain standard mesh training data.

[0017] From the above description, it can be seen that the current mesh simplification method based on quadratic error metric will cause the tooth boundary to disappear as the degree of simplification increases, resulting in the model being unable to accurately extract tooth boundary information for segmentation. Therefore, the Euclidean distance between vertex pairs is calculated using the RGB color data corresponding to the patch label data. When the Euclidean distance between vertex pairs exceeds the preset distance threshold, the vertex pair is considered to be at the tooth boundary, and the distance metric value at the tooth boundary is set to the maximum, thereby limiting the merging of vertex pairs at the boundary to prevent the problem of tooth boundary disappearance due to deepening simplification.

[0018] Furthermore, the position feature extraction unit includes a multi-layer feature inactivation layer, a point cloud data module, a feature fusion module, a dense connection module and a multi-layer feature restoration layer; The extracting of the position feature stream from the standard grid training data by the position feature extraction unit comprises: Performing random feature extraction on the standard grid training data through the multi-layer feature inactivation layer and the point cloud data module to obtain a first patch position feature; Performing feature restoration on the first patch position feature through the multi-layer feature restoration layer to obtain a second patch position feature; When performing the feature restoration, the first patch position feature and the second patch position feature are fused through the feature fusion module and the dense connection module to obtain a position feature stream.

[0019] From the above description, it can be seen that the first patch position feature is obtained by randomly extracting features from the standard grid training data through the multi-layer feature inactivation layer and the point cloud data module, so as to randomly inactivate the patches in the standard grid training data, thereby reducing the dependency between patches. The features restored through the multi-layer feature restoration layer are then fused with the first patch position feature, thereby avoiding the problem of model recognition errors caused by the introduction of a large number of new features and improving the accuracy of model segmentation.

[0020] Furthermore, the standard grid training data includes center point coordinate data of the facet; The randomly extracting the first patch position feature in the standard grid training data through the multi-layer feature inactivation layer and the point cloud data module comprises: Downsampling the center point coordinate data respectively through the multiple feature deactivation layers to obtain the patch position data corresponding to each feature deactivation layer; Extract the features of each patch position data through a multi-layer perceptron and maximum pooling to obtain the initial patch position features corresponding to each feature inactivation layer; Each of the initial patch position features is subjected to feature extraction through the point cloud data module to obtain a first patch position feature corresponding to each of the feature inactivation layers.

[0021] From the above description, we can know that the coordinates of the center points of the patches are grouped and downsampled through multiple layers of feature inactivation layers to achieve the location data of the patches, and then the grouped sampled patch data is subjected to feature extraction through a multi-layer perceptron and maximum pooling to clarify the location attributes of the patches. After each feature extraction, a deeper feature extraction is performed through the point cloud data module to improve the model's ability to analyze the grid position.

[0022] Furthermore, the feature restoration layer includes a first feature restoration layer and a second feature restoration layer, and the feature semantic level of the first feature restoration layer is higher than the feature semantic level of the second feature restoration layer; The fusing the first patch position feature with the second patch position feature through the feature fusion module and the dense connection module to obtain a position feature stream includes: The first patch position feature is fused with the second patch position feature corresponding to the first feature restoration layer through the feature fusion module and the dense connection module, and the first patch position feature is fused with the second patch position feature corresponding to the second feature restoration layer through the dense connection module to obtain a position feature stream.

[0023] From the above description, it can be seen that the deep reuse and fusion of features between the feature inactivation layer and the feature restoration layer is achieved through the dense connection module, which can effectively avoid the problem of model recognition errors caused by the introduction of a large number of new features. Since the feature semantic level of the first feature restoration layer is higher than that of the second feature restoration layer, the feature fusion process of the first feature restoration layer is more complicated. Therefore, feature fusion is achieved in the first feature restoration layer through the feature fusion module and the dense connection module to improve the feature fusion efficiency.

[0024] Further, the shape feature extraction unit includes a first shape feature module and a second shape feature module; The extracting of the shape feature flow from the standard grid training data by the shape feature extraction unit comprises: Extracting features from the standard grid training data using the first shape feature module to obtain a first patch shape feature; Extracting the first patch shape feature by the second shape feature module to obtain a second patch shape feature; The first patch shape feature and the second patch shape feature are concatenated to obtain a shape feature stream.

[0025] From the above description, it can be seen that by extracting features from the standard grid training data step by step through different shape feature modules, fine-grained detail features can be effectively extracted to improve the accuracy of high-precision tooth data segmentation.

[0026] Furthermore, the first shape feature module includes a graph feature layer, a shared convolution layer, and a maximum pooling layer; The extracting features of the standard grid training data by the first shape feature module to obtain the first patch shape feature comprises: Constructing a data graph structure of the standard grid training data through the graph feature layer based on a K-nearest neighbor algorithm; Extracting features from the data graph structure through the shared convolution layer to obtain initial patch shape features; The maximum pooling layer is used to extract the initial patch shape feature to obtain a first patch shape feature.

[0027] From the above description, we can see that the K-nearest neighbor algorithm is used to construct a data graph structure to characterize the shape attributes of the patch, and then the initial patch shape features of the patch shape attributes are learned through the shared convolution layer. Finally, the maximum pooling layer is used to enhance the initial patch shape features, effectively capturing the local detail features of the patch.

[0028] Furthermore, the second shape feature module includes a graph feature layer and a maximum pooling layer; The extracting the first patch shape feature by the second shape feature module to obtain the second patch shape feature comprises: Constructing a feature graph structure of the first patch shape feature through the graph feature layer based on a K-nearest neighbor algorithm; The feature map structure is subjected to feature extraction by the maximum pooling layer to obtain a second patch shape feature.

[0029] From the above description, it can be seen that since the first shape feature module has extracted the first patch shape feature, based on the first patch shape feature, feature extraction is performed again through the second shape feature module to capture features from different levels and optimize their corresponding features, thereby improving the accuracy of model segmentation.

[0030] Another embodiment of the present invention provides a high-precision tooth segmentation terminal based on a dual-stream network, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, each step of the above-mentioned high-precision tooth segmentation method based on a dual-stream network is implemented.

[0031] From the above description, it can be seen that the beneficial effect of the present invention is that the acquired tooth mesh training data is simplified to obtain standard mesh training data, thereby reducing the data volume of the tooth mesh data and improving the data processing efficiency of the subsequent dual-stream network learning model. The simplified standard mesh training data is input into the dual-stream network learning model, and the position feature stream and shape feature stream are extracted from the standard mesh training data by the position feature extraction unit and the shape feature extraction unit respectively, so as to extract geometric information with high discrimination, reduce the dependency between each patch unit in the mesh data, so that the trained tooth segmentation model can better identify detail features, thereby improving the generalization ability of the model, so that the tooth segmentation model can accurately segment the high-precision tooth mesh training data.

[0032] The embodiment of the present invention provides a high-precision tooth segmentation method and terminal based on a dual-stream network, which can be applied to computer-assisted stomatology fields such as dental orthodontics. It can accurately segment high-precision tooth data through a tooth segmentation model trained with low-precision tooth data. The following is an explanation through a specific embodiment: Please refer to Figures 1 to 5 , Embodiment 1 of the present invention is: A high-precision tooth segmentation method based on a two-stream network, comprising: S1. Acquire tooth mesh training data and patch label data of the tooth mesh training data.

[0033] S2. Simplify the tooth grid training data to obtain standard grid training data.

[0034] Specifically, the tooth mesh training data includes vertex coordinate data, and step S2 includes: S21, converting the patch label data into RGB color data.

[0035] In some embodiments, the patch label data characterizes the tooth category to which each patch unit in the tooth grid training data belongs. For example, if the patch label data of the symmetrical central incisor is 1, then the patch units whose patch label data is set to 1 in the tooth grid training data all belong to the patch units of the symmetrical central incisor. The RGB color data converted for different patch label data are all different. For example, if the patch label data of the symmetrical central incisor is 1, the RGB color data converted is (255, 255, 0), that is, yellow, because in the RGB color mode, the value ranges of the three colors red (R), green (G), and blue (B) are all 0-255. When the values ​​of red and green both reach the maximum value of 255, and the value of blue is 0, the mixed color is yellow; at this time, the patch units whose RGB color data is yellow all belong to the patch units of the symmetrical central incisor. Therefore, the distinguishing annotations of the patch units of different categories of teeth in the tooth grid training data are converted from the patch label data to the RGB color data.

[0036] S22. Construct an error matrix according to the vertex coordinate data and the RGB color data.

[0037] S23. Calculate the distance metric values ​​between vertex pairs using a quadratic error metric according to the error matrix.

[0038] In some embodiments, step S22 and step S23 are performed based on an error matrix construction method and a measurement method in a QEM algorithm (Quadric Error Metrics).

[0039] S24. Merge the vertex pairs in the vertex coordinate data according to the RGB color data and the distance metric value to obtain standard grid training data.

[0040] Specifically, the step S24 includes: S241. Calculate the Euclidean distance between each pair of vertices according to the RGB color data.

[0041] In some embodiments, the Euclidean distance between each vertex pair is calculated by RGB color data to determine the color similarity between the vertex pairs. The smaller the Euclidean distance between the vertex pairs, the closer the positions of the vertex pairs in the RGB color space are, and thus the higher the similarity is. The larger the Euclidean distance between the vertex pairs, the farther the positions of the vertex pairs in the RGB color space are, and thus the lower the similarity is.

[0042] In some embodiments, the Euclidean distance of each vertex is calculated based on the RGB color data of the patch unit to which it belongs.

[0043] S242, detecting whether the Euclidean distance is greater than a preset distance threshold, and if so, setting the distance metric value of the vertex pair corresponding to the Euclidean distance to the maximum value of the distance metric values ​​in the vertex coordinate data.

[0044] In some embodiments, since the face units of different types of teeth are marked and distinguished with different colors, when the Euclidean distance between the vertex pairs is greater than the preset distance threshold, it indicates that the color similarity between the vertex pairs is too low, that is, the face units to which the two vertices belong are marked with different colors. In this way, it is determined that the vertex pair belongs to the boundary data of the tooth model.

[0045] S243, obtaining the vertex pair with the smallest distance metric value from the vertex coordinate data and merging them to obtain new vertex coordinate data, and returning to execute the step of constructing an error matrix according to the vertex coordinate data and the RGB color data until the tooth mesh training data reaches the set accuracy to obtain standard mesh training data.

[0046] In some embodiments, since the distance measurement values ​​of the pairs of vertices located at the tooth boundary are set to the maximum values ​​in their vertex coordinate data, the boundary data will not be simplified first each time the mesh is simplified and updated, thereby implementing simplified constraints on the tooth boundary data, retaining the tooth boundary data to the greatest extent, avoiding the problem of disappearance of tooth boundaries due to increasing simplification, and improving the accuracy of subsequent tooth segmentation model training.

[0047] In this embodiment, the set accuracy achieved by the tooth grid training data is 1W.

[0048] like Figure 2 As shown, the mesh simplification method provided by the present invention can better preserve the tooth boundary after mesh simplification compared with the existing QEM algorithm.

[0049] S3. Inputting the standard grid training data into a preset dual-stream network learning model, wherein the dual-stream network learning model includes a position feature extraction unit and a shape feature extraction unit.

[0050] In some embodiments, Figure 3 As shown in the figure, the dual-stream network learning model (hereinafter referred to as PSNet) is a model trained based on the patch segmentation method. Its dual streams refer to the P feature stream and the S feature stream. The P feature stream refers to the position features of the patch units extracted from the tooth mesh training data (extracted by the position feature extraction unit), and the S feature stream refers to the shape features of the patch units extracted from the tooth mesh training data (extracted by the shape feature extraction unit). In this way, the geometric information with high discrimination of the patch units is extracted, and finally the two feature streams are fused to output the final tooth segmentation prediction result. The tooth segmentation prediction result output by PSNet is an M×C matrix, where M represents the number of patch units and C represents the number of prediction categories; that is, in the M×C matrix, each row represents a patch unit, and each column represents the probability distribution of the patch unit corresponding to the row belonging to the category of the column.

[0051] S4. Extracting a position feature stream from the standard grid training data through the position feature extraction unit. In some embodiments, the standard grid training data input to the position feature extraction unit includes center point coordinate data and vertex coordinate data of each patch unit. In this embodiment, the patch unit is in the shape of a triangle, so its input data includes a center point coordinate data and three vertex coordinate data, and the matrix dimension of the input position feature extraction unit is M×12, that is, in the M×12 matrix, each row of data includes center point coordinate data and three vertex coordinate data, and the coordinate data of each point includes xyz axis coordinate values.

[0052] Specifically, the position feature extraction unit includes a multi-layer feature inactivation layer, a point cloud data module, a feature fusion module, a dense connection module and a multi-layer feature restoration layer, and the step S4 includes: S41, performing random feature extraction on the standard grid training data through the multi-layer feature inactivation layer and the point cloud data module to obtain a first patch position feature.

[0053] In some embodiments, the feature inactivation layer randomly inactivates some facet units by sampling and extracts the position information of the facet units, thereby reducing the dependence on the overall inter-facet relationship and making the model have a certain generalization ability. The point cloud data module specifically refers to the Pnp3d module, a plug-and-play 3D point cloud module that can be integrated into the existing point cloud processing network without changing the original network architecture, significantly improving performance. Since the data of the input position feature extraction unit are all coordinate data, similar to point cloud data, deep feature extraction can be performed through the point cloud data module.

[0054] S42. Perform feature restoration on the first face patch position feature through the multi-layer feature restoration layer to obtain a second face patch position feature.

[0055] S43. When performing the feature restoration, the first patch position feature and the second patch position feature are fused through the feature fusion module and the dense connection module to obtain a position feature stream.

[0056] In some embodiments, the feature restoration layer gradually restores the first facet position feature, and when the feature restoration layer performs feature restoration, the feature inactivation layer and the feature restoration layer fuse the features through a dense connection module and a feature fusion module to obtain a position feature flow.

[0057] S5. Extracting a shape feature stream from the standard grid training data through the shape feature extraction unit. In some embodiments, the standard grid training data input to the shape feature extraction unit includes the specific shape of each patch unit. In this embodiment, the shape of the patch unit is a triangle, and the matrix dimension of the input shape feature extraction unit is M×17, which represents the 17-dimensional features of M patch units, wherein the 17-dimensional features include the three vertex coordinates of the triangular patch unit, the normal vector of the center point, the curvature between the three vertices, and the area and perimeter of the triangular patch unit.

[0058] Specifically, the shape feature extraction unit includes a first shape feature module and a second shape feature module, and step S5 includes: S51. Extract features from the standard grid training data using the first shape feature module to obtain a first patch shape feature.

[0059] In some embodiments, the number of the first shape feature modules is two, and step S51 is specifically: firstly, a first shape feature module is used to extract features of the standard grid training data to obtain an initial patch shape feature. , and then another first shape feature module extracts the initial patch shape feature to obtain the first patch shape feature .

[0060] S52: Extract the first patch shape feature by using the second shape feature module to obtain a second patch shape feature.

[0061] In some embodiments, the second shape feature module performs a function on the first patch feature module. Perform feature extraction to obtain the second patch shape feature .

[0062] S53: Concatenate the first patch shape feature and the second patch shape feature to obtain a shape feature stream.

[0063] In some embodiments, the initial patch shape feature is first , the first patch shape feature And the second patch shape feature Splice to get , and then use the multi-layer perceptron MLP to obtain the features from the splicing Feature extraction is performed to obtain the final S feature flow .

[0064] S6. Fusing the position feature flow with the shape feature flow to obtain a target feature flow.

[0065] S7. Segment the standard grid training data according to the target feature flow to obtain predicted tooth data.

[0066] S8. Iteratively train the dual-stream network learning model according to the face label data and the predicted tooth data to obtain a tooth segmentation model, and segment the target tooth mesh training data using the tooth segmentation model.

[0067] like Figure 4 As shown in the figure, when the tooth mesh data is simplified to different mesh accuracies, the error value between the tooth boundary and the original boundary increases continuously, wherein the error value between the tooth boundary and the original boundary is determined by the DCD value (Discrete Cosine Distance). Figure 4 It can be seen that the error value of the tooth boundary simplified by the QEM algorithm is smaller than the error value of other mesh simplification algorithms, and the error value of the QEM algorithm begins to increase significantly after the mesh accuracy is 2.4W. Therefore, the simplified accuracy of the tooth mesh data can find a balance between simplification efficiency and simplification quality when it is 2.4W. Therefore, it is determined that the most suitable grid accuracy for evaluating the performance of the model is 2.4W. The present invention simplifies the acquired tooth mesh training data to a grid accuracy of 2.4W, and divides the tooth mesh training data under the 2.4W grid accuracy into a training set and a test set, wherein the proportion of the training set and the test set are 80% and 20% respectively. The above steps S2 to S8 are executed by the training set.

[0068] The present invention trains PointNet++, GACNet, TSGCNet and SGTNet deep learning models based on the same training set to obtain corresponding tooth segmentation models. Based on the above test set, the PointNet++ model, GACNet model, TSGCNet model, SGTNet model and PSNet model provided by the present invention are evaluated respectively. The evaluation data results of the above models are shown in Tables 1 and 2. The tooth segmentation effects of the above models are shown in Tables 1 and 2. Figure 5 As shown in Figure 2, ground truth represents the real segmentation data of the test set.

[0069] Table 1 Performance data evaluation results

[0070] It can be seen from Table 1 that the PSNet model trained by the present invention has better effects on OA (overall accuracy) and Miou (average intersection-over-union ratio) than other models, and among the two-stream network models (GACNet, TSGCNet and SGTNet), the PSNet model occupies the least image processing memory.

[0071] Table 2 Intersection-over-union of tooth prediction segmentation for each category

[0072] As can be seen from Table 2, the PSNet model trained by the present invention has a better intersection-over-union ratio in the predicted segmentation results of tooth categories T1, T4, T5, T6, T7 and BG than other models.

[0073] Please refer to Figures 6 to 10 , Embodiment 2 of the present invention is: A high-precision tooth segmentation method based on a dual-stream network is different from the first embodiment in that the specific implementation methods of step S41 and step S43 are limited.

[0074] Specifically, the standard grid training data includes the center point coordinate data of the patch, and the step S41 includes: S411, down-sampling the center point coordinate data respectively through the multiple feature deactivation layers to obtain the patch position data corresponding to each feature deactivation layer.

[0075] In some embodiments, the multi-layer feature deactivation layer uses the group sampling method in the point cloud data segmentation model PointNet++ to achieve random feature deactivation. The center point coordinates of the patch are grouped and sampled through different feature deactivation layers to achieve the extraction of the position features of the patch.

[0076] S412, extracting features of each patch position data through a multi-layer perceptron and maximum pooling to obtain initial patch position features corresponding to each feature inactivation layer.

[0077] In some embodiments, after the patch position data is grouped and sampled through multiple feature inactivation layers, it forms a sparse graph G. Then step S412 can be specifically expressed as: ,in, represents the extracted initial patch position feature, Max represents the maximum pooling function, MLP Represents a multilayer perceptron function.

[0078] S413, extracting each of the initial patch position features through the point cloud data module to obtain a first patch position feature corresponding to each of the feature inactivation layers.

[0079] In some embodiments, in order to further enhance the position feature extraction unit's ability to analyze position data, a point cloud data module is set to extract position data in greater depth. and the position features represented by the patch units Therefore, the point cloud data module is used to extract more in-depth grid position features from the position features of these two representation levels, and the first patch position features obtained are used as the input features of the next feature inactivation layer. Specifically, the first patch position features ,in, Represents the feature extraction function of the point cloud data module. ,in, Represents the input features of the next feature dropout layer.

[0080] In some embodiments, the characteristic deactivation layer includes a first characteristic deactivation layer D1, a second characteristic deactivation layer D2, a third characteristic deactivation layer D3, a fourth characteristic deactivation layer D4, and a fifth characteristic deactivation layer D5, the first characteristic reduction layer includes a first characteristic reduction sublayer R1, a second characteristic reduction sublayer R2, and the second characteristic reduction layer includes a third characteristic reduction sublayer R3 and a fourth characteristic reduction sublayer R4. Among them, the first characteristic deactivation layer D1 corresponds to the fourth characteristic reduction sublayer R4, the second characteristic deactivation layer D2 corresponds to the third characteristic reduction sublayer R3, the third characteristic deactivation layer D3 corresponds to the second characteristic reduction sublayer R2, and the fourth characteristic deactivation layer D4 corresponds to the first characteristic reduction sublayer R1.

[0081] In some embodiments, the first patch position feature extracted by the first feature deactivation layer D1 is used as the input feature of the second feature deactivation layer D2, and the second feature deactivation layer D2 sequentially performs downsampling, feature extraction based on multi-layer perceptron and maximum pooling, and feature extraction based on point cloud data module on the first patch position feature extracted by the first feature deactivation layer D1 to obtain the first patch position feature extracted by the second feature deactivation layer D2, and so on to obtain the first patch position feature extracted by the fifth feature deactivation layer D5. The first patch position feature extracted by the fifth feature deactivation layer D5 is directly input into the feature restoration layer for feature restoration.

[0082] Specifically, the feature restoration layer includes a first feature restoration layer and a second feature restoration layer, and the feature semantic level of the first feature restoration layer is higher than the feature semantic level of the second feature restoration layer. Then, step S43 includes: S431. Fusing the first patch position feature with the second patch position feature corresponding to the first feature restoration layer through the feature fusion module and the dense connection module, and fusing the first patch position feature with the second patch position feature corresponding to the second feature restoration layer through the dense connection module to obtain a position feature stream.

[0083] In some embodiments, Figure 6 As shown, the features extracted by the first feature deactivation layer D1 and the fourth feature restoration sublayer R4 are directly fused through the dense connection module to obtain the position feature f1. The features extracted by the second feature deactivation layer R2 and the third feature restoration sublayer R3 are directly fused through the dense connection module to obtain the position feature f2. The features extracted by the first feature deactivation layer D1 and the third feature deactivation layer D3 are first spliced ​​through the dense connection module, and then fused with the features extracted by the second feature restoration sublayer R2 through the feature fusion module to obtain the position feature f3. The features extracted by the first feature deactivation layer D1, the second feature deactivation layer D2 and the fourth feature deactivation layer D3 are first spliced ​​through the dense connection module, and then fused with the features extracted by the first feature restoration sublayer R1 through the feature fusion module to obtain the position feature f4, that is, the position feature flow. It should be noted that Figure 6 The connecting lines extending to the right from the feature inactivation layer represent densely connected modules.

[0084] In some embodiments, Figure 6As shown, the features extracted by the fifth feature deactivation layer D5 are input into the first feature restoration sublayer R1 for feature restoration and fused with the first face patch position feature to obtain position feature f4. f4 is input into the second feature restoration sublayer R2 for feature restoration and fused with the first face patch position feature to obtain position feature f3. f3 is input into the third feature restoration sublayer R3 for feature restoration and fused with the first face patch position feature to obtain position feature f2. f2 is input into the fourth feature restoration sublayer R4 for feature restoration and fused with the first face patch position feature to obtain position feature f1.

[0085] In some embodiments, the feature fusion module is an MFF module, which is a multi-feature fusion module. The MFF module includes an attention gating module, a channel attention module, and a spatial attention module. Figure 7 As shown, x 1 represents the first patch position feature corresponding to the feature inactivation layer, x 2 represents the second patch position feature corresponding to the feature restoration layer, and the first patch position feature x 1 and 2nd patch position features x 2 After being processed by the attention gating module, feature A1 is obtained and feature A1 is combined with the first patch position feature x 1 is concatenated to obtain feature A2, and then feature A2 is passed through the channel attention module to obtain the channel attention weight a 1. The channel attention weight a 1 is multiplied by feature A2 to obtain feature A3. Finally, feature A3 is passed through the spatial attention module to obtain the spatial attention weight a 2. The spatial attention weight a 2 is multiplied by feature A to obtain feature A4. Specifically, Figure 8 As shown, in the attention gating module, the features are first x 1 and x 2. Perform group convolution feature extraction, add the results of convolution feature extraction and input them into the gating unit to generate a set of weight data, and finally combine the weight data with the feature x 2 to filter out redundant information, and then add it to the original feature x 2 and output feature A1. Output feature A1 and the first patch position feature x 1 is concatenated to obtain feature A2. Fig. 9 As shown in the channel attention module, feature A2 is first averaged and then convolutional features are extracted from the pooled features. Finally, the channel attention weight is obtained through the SigMoid activation function. a 1. Set channel attention weight a 1 is multiplied by feature A2 to get feature A3. Fig.10As shown in the figure, in the spatial attention module, feature A3 is first average pooled and max pooled and then concatenated, and then the concatenated features are convolutionally extracted, and finally the spatial attention weight is obtained through the SigMoid activation function. a 2. Set the spatial attention weights a 2 is multiplied by feature A3 to obtain feature A4, which is the first patch position feature x 1 and 2nd patch position features x 2. Position feature flow after fusion.

[0086] Please refer to Fig.11 , Embodiment 3 of the present invention is: A high-precision tooth segmentation method based on a dual-stream network is different from the second embodiment in that the specific implementation methods of step S51 and step S52 are limited.

[0087] Specifically, the first shape feature module includes a graph feature layer, a shared convolution layer, and a maximum pooling layer, and the step S51 includes: S511 . Constructing a data graph structure of the standard grid training data through the graph feature layer based on a K-nearest neighbor algorithm.

[0088] In some embodiments, for each patch unit, the K nearest patch units with the smallest Euclidean distance are found in the feature space by using the K-nearest neighbor algorithm. Specifically, the resulting data graph structure DG can be expressed as DG (V, E) ,in, V represents a collection of patch units, , E is the edge set, that is, E represents the set of K nearest patch units with the smallest Euclidean distance.

[0089] S512: Extract features from the data graph structure through the shared convolutional layer to obtain initial patch shape features.

[0090] In some embodiments, the embedded features of the K nearest patch units corresponding to each patch unit are learned separately by a multi-layer perceptron in a shared convolutional layer.

[0091] S513 . Extract the initial patch shape feature through the maximum pooling layer to obtain a first patch shape feature.

[0092] In some embodiments, the input data graph structure DG (V, E) The feature F of the graph extracted by MLP is expressed as , and finally the extracted feature F is highlighted through the pooling layer, specifically: ; in, l Indicates the number of layers, MaxPoolIndicates maximum pooling. That is, the data graph structure DG of the first two layers is obtained through MLP and MaxPool To obtain the feature F input to the next layer, the third layer data graph structure DG directly pools the data graph structure DG.

[0093] Specifically, the second shape feature module includes a graph feature layer and a maximum pooling layer, and the step S52 includes: S521. Construct a feature graph structure of the first patch shape feature through the graph feature layer based on a K-nearest neighbor algorithm.

[0094] S522. Perform feature extraction on the feature graph structure through the maximum pooling layer to obtain a second patch shape feature.

[0095] In some embodiments, step S521 is implemented in the same manner as step S511, and step S522 is implemented in the same manner as step S513.

[0096] Please refer to Fig.12 , Embodiment 4 of the present invention is: A high-precision tooth segmentation terminal 100 based on a dual-stream network includes a memory 101, a processor 102, and a computer program stored in the memory 101 and running on the processor 102. When the processor 102 executes the computer program, each step of the high-precision tooth segmentation method based on a dual-stream network in the above-mentioned embodiments 1 to 3 is implemented.

[0097] In summary, the present invention provides a high-precision tooth segmentation method and terminal based on a dual-stream network. The tooth mesh training data is mesh-simplified by using patch label data, which can avoid simplifying vertex pairs located at the boundary during the simplification process, so as to limit the simplification at the boundary and avoid the disappearance of the boundary due to the deepening of the simplification. The simplified standard mesh training data is input into the dual-stream network learning model, and the patch units in the standard mesh training data are randomly inactivated by the multi-layer feature inactivation layers of the position feature extraction unit to reduce the dependency between each patch unit in the mesh data; at the same time, the patch units in the standard mesh training data are fine-grained feature extracted by the shape feature extraction unit, so that the trained tooth segmentation model can better identify detail features. Based on the position feature extraction unit and the shape feature extraction unit, the position feature stream and the shape feature stream are extracted from the standard mesh training data at the same time, which can accurately extract geometric information with high discrimination, thereby improving the generalization ability of the model, so that the tooth segmentation model trained with low-precision mesh data can accurately segment high-precision tooth mesh training data.

[0098] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A high-precision tooth segmentation method based on a two-stream network, characterized in that: include: Acquire tooth mesh training data and facet label data of the tooth mesh training data; Simplifying the tooth grid training data to obtain standard grid training data; Inputting the standard grid training data into a preset dual-stream network learning model, wherein the dual-stream network learning model includes a position feature extraction unit and a shape feature extraction unit; Extracting a position feature stream from the standard grid training data by the position feature extraction unit; Extracting a shape feature flow from the standard grid training data by the shape feature extraction unit; The position feature stream and the shape feature stream are fused to obtain a target feature stream; Segmenting the standard grid training data according to the target feature flow to obtain predicted tooth data; Iteratively training the dual-stream network learning model according to the face patch label data and the predicted tooth data to obtain a tooth segmentation model, and segmenting the target tooth mesh training data using the tooth segmentation model; The tooth mesh training data includes vertex coordinate data; The simplifying process of the tooth mesh training data to obtain standard mesh training data comprises: Convert the face label data into RGB color data; Constructing an error matrix according to the vertex coordinate data and the RGB color data; Calculating distance metric values ​​between vertex pairs using a quadratic error metric according to the error matrix; Vertex pairs in the vertex coordinate data are merged according to the RGB color data and the distance metric value to obtain standard grid training data.

2. The high-precision tooth segmentation method based on a dual-stream network according to claim 1, characterized in that: The merging of the vertex pairs in the vertex coordinate data according to the RGB color data and the distance metric value to obtain standard grid training data comprises: Calculate the Euclidean distance between each pair of vertices according to the RGB color data; Detecting whether the Euclidean distance is greater than a preset distance threshold, and if so, setting the distance metric value of the vertex pair corresponding to the Euclidean distance to the maximum value of the distance metric values ​​in the vertex coordinate data; The vertex pair with the smallest distance metric value is obtained from the vertex coordinate data and merged to obtain new vertex coordinate data, and the step of constructing an error matrix according to the vertex coordinate data and the RGB color data is returned to execute until the tooth mesh training data reaches the set accuracy to obtain standard mesh training data.

3. The high-precision tooth segmentation method based on a dual-stream network according to claim 1, characterized in that: The position feature extraction unit includes a multi-layer feature inactivation layer, a point cloud data module, a feature fusion module, a dense connection module and a multi-layer feature restoration layer; The extracting of the position feature stream from the standard grid training data by the position feature extraction unit comprises: Performing random feature extraction on the standard grid training data through the multi-layer feature inactivation layer and the point cloud data module to obtain a first patch position feature; Performing feature restoration on the first patch position feature through the multi-layer feature restoration layer to obtain a second patch position feature; When performing the feature restoration, the first patch position feature and the second patch position feature are fused through the feature fusion module and the dense connection module to obtain a position feature stream.

4. The high-precision tooth segmentation method based on a two-stream network according to claim 3, characterized in that: The standard grid training data includes center point coordinate data of the facet; The randomly extracting the first patch position feature in the standard grid training data through the multi-layer feature inactivation layer and the point cloud data module comprises: Downsampling the center point coordinate data respectively through the multiple feature deactivation layers to obtain the patch position data corresponding to each feature deactivation layer; Extract the features of each patch position data through a multi-layer perceptron and maximum pooling to obtain the initial patch position features corresponding to each feature inactivation layer; Each of the initial patch position features is subjected to feature extraction through the point cloud data module to obtain a first patch position feature corresponding to each of the feature inactivation layers.

5. The high-precision tooth segmentation method based on a dual-stream network according to claim 3, characterized in that: The feature restoration layer includes a first feature restoration layer and a second feature restoration layer, and the feature semantic level of the first feature restoration layer is higher than the feature semantic level of the second feature restoration layer; The fusing the first patch position feature with the second patch position feature through the feature fusion module and the dense connection module to obtain a position feature stream includes: The first patch position feature is fused with the second patch position feature corresponding to the first feature restoration layer through the feature fusion module and the dense connection module, and the first patch position feature is fused with the second patch position feature corresponding to the second feature restoration layer through the dense connection module to obtain a position feature stream.

6. The high-precision tooth segmentation method based on a dual-stream network according to claim 1, characterized in that: The shape feature extraction unit includes a first shape feature module and a second shape feature module; The extracting of the shape feature flow from the standard grid training data by the shape feature extraction unit comprises: Extracting features from the standard grid training data using the first shape feature module to obtain a first patch shape feature; Extracting the first patch shape feature by the second shape feature module to obtain a second patch shape feature; The first patch shape feature and the second patch shape feature are concatenated to obtain a shape feature stream.

7. The high-precision tooth segmentation method based on a two-stream network according to claim 6, characterized in that: The first shape feature module includes a graph feature layer, a shared convolution layer and a maximum pooling layer; The extracting features of the standard grid training data by the first shape feature module to obtain the first patch shape feature comprises: Constructing a data graph structure of the standard grid training data through the graph feature layer based on a K-nearest neighbor algorithm; Extracting features from the data graph structure through the shared convolution layer to obtain initial patch shape features; The maximum pooling layer is used to extract the initial patch shape feature to obtain a first patch shape feature.

8. The high-precision tooth segmentation method based on a dual-stream network according to claim 6, characterized in that: The second shape feature module includes a graph feature layer and a maximum pooling layer; The extracting the first patch shape feature by the second shape feature module to obtain the second patch shape feature comprises: Constructing a feature graph structure of the first patch shape feature through the graph feature layer based on a K-nearest neighbor algorithm; The feature map structure is subjected to feature extraction by the maximum pooling layer to obtain a second patch shape feature.

9. A high-precision tooth segmentation terminal based on a dual-stream network, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, each step of the high-precision tooth segmentation method based on a dual-stream network described in any one of claims 1 to 8 is implemented.

Images