A tooth segmentation method for intraoral scanned images based on dual-space representation based on semantic coordination

Abstract

This invention discloses a tooth segmentation method for intraoral scan images based on semantic coordination and dual-space representation, relating to the fields of medical image processing and computer vision. The method includes: downsampling the original intraoral scan image; adaptively semantically coordinating the geometric properties of the Euclidean space of the intraoral scan image, focusing on the relationship between spatial and semantic features in the real spatial dimension; learning the structural features and rotation invariance of the intraoral scan image through a spatial-semantic graph learning module; and inputting the learned representation into segmentation for prediction, segmenting individual teeth and gingiva in the intraoral scan image. This invention effectively addresses the rotation invariance of intraoral scan images and improves segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and computer vision technology, and in particular to a method for tooth segmentation of an intraoral scanned image based on dual-space representation of semantic coordination. Background Art

[0002] With the rapid development of artificial intelligence technology, computer-aided technology is widely used in the oral field. Intraoral scan images have the advantages of high resolution, low radiation dose and fast imaging, and are important modality images for oral diagnosis and treatment. Accurate segmentation of intraoral scan images can greatly improve the efficiency of orthodontic treatment and is of great significance to computer-assisted digital orthodontics. However, due to the imaging process and data composition of intraoral scan images, their segmentation has the following difficulties:

[0003] (1) Differences in imaging devices: Intraoral scan images from different imaging devices have center heterogeneity, and the existing segmentation model is not robust to multi-center data;

[0004] (2) The images of different patients have large differences in tooth morphology, number of teeth, and dental arch morphology, which leads to under-segmentation and over-segmentation in the segmentation model;

[0005] (3) During the imaging process of intraoral scanning images, there may be noise such as soft tissue, which may affect the segmentation performance of the segmentation model.

[0006] Traditional intraoral scan image segmentation methods rely on a lot of prior knowledge or require a lot of manual intervention, and cannot achieve fully automatic and robust segmentation performance. Existing intraoral scan image segmentation methods based on deep learning often make segmentation errors when dealing with difficult samples (missing teeth, disordered teeth, and uneven dentition, etc.) and multi-center samples. Therefore, a new method is urgently needed to robustly segment difficult samples and multi-center samples. Summary of the invention

[0007] In order to solve the above problems existing in the prior art and the difficulty of intraoral scan image segmentation, the present invention proposes a method for tooth segmentation of intraoral scan images based on dual-space representation of semantic coordination, the method comprising:

[0008] S1. Acquire an original intra-oral scan image, downsample the original intra-oral scan image, maintain the topological structure of the original intra-oral scan image during the downsampling process, and obtain the geometric attributes of each vertex in the down-sampled intra-oral scan image;

[0009] S2, obtaining semantic information of the downsampled intra-oral scan image based on the geometric attributes of each vertex in the downsampled intra-oral scan image, and performing semantic coordination processing on the semantic information to obtain semantic coordinated semantic information;

[0010] S3. In the coordinate space, using the K nearest neighbor strategy, take each vertex in the sampled intraoral scan image as the center, sort the distances of other vertices from the center from small to large, and obtain the spatial information distance matrix between any two vertices in N vertices. Select the first K vertices in the sorting to jointly construct a first local area, calculate and obtain polar coordinate information of the first local area in the first local area, update the polar coordinate information to obtain updated polar coordinate information, and obtain rotation invariance information based on the updated polar coordinate information;

[0011] S4, based on each vertex p i The semantic information of the semantic coordination is used to obtain the semantic Euclidean distance between vertices; the semantic information distance between vertices is obtained based on the semantic Euclidean distance between vertices; for each vertex p i , with vertex p i Center, sort the semantic information distances of other vertices from the center from small to large, and obtain the semantic information distance matrix between any two vertices among N vertices Select the first K vertices in the sorting to jointly construct the second local area; based on each vertex p i The semantic information of the semantic coordination and vertex p i The semantic information set composed of the neighboring points of is obtained by aggregating local features in the semantic space to obtain the first local aggregation feature Based on spatial information distance matrix K indices in the vertex p i The semantic information of the semantic coordination and the distance vertex p i The semantic information of the first K vertices is sorted from small to large by distance, and the second local aggregation feature is obtained by local feature aggregation in the semantic space The first local aggregation feature and the second local aggregate feature Perform channel-level splicing to obtain aggregated local semantic features For each vertex p i , with vertex p i Center, sort the spatial distances of other vertices from the center from small to large, and select the top K vertices to jointly construct the third local area; based on each vertex p i The coordinate information and vertex p i The coordinate information set composed of the neighboring points of is used to obtain the third local aggregation feature by local feature aggregation in the coordinate space. Based on semantic information distance matrix K indices in the vertex p i The spatial information and distance vertex p iSort the spatial information of the first K vertices from small to large, and aggregate the local features in the coordinate space to obtain the fourth local aggregation feature The third local aggregation feature and the fourth local aggregation feature Perform channel-level splicing to obtain aggregated local spatial features

[0012] S5. Combine rotation invariance information and local spatial features Perform channel-level splicing to obtain dual-space expression features Dual-space expression features Convolution is performed through the convolutional layers respectively, and the outputs of the two convolutional layers are spliced ​​at the channel level to obtain the spliced ​​dual-space expression features. Dual-space expression features Perform feature screening to obtain dual-space expression features Select the dual space expression feature Dual-space expression features Perform channel-level splicing to obtain dual-space feature representation

[0013] S6. Dual-space feature representation and local semantic features The spatial-semantic graph learning module learns the structure and rotation invariance of teeth and obtains the output of each layer of the spatial-semantic graph learning module;

[0014] S7, based on the output of each layer of the spatial-semantic graph learning module, the multi-scale tooth features are spliced ​​to obtain the final tooth feature description;

[0015] S8, inputting the final tooth feature description into the segmentation network for prediction to obtain the final segmentation result;

[0016] S9. Calculate the overall loss function of the segmentation network according to the segmentation results and accuracy annotations, update the parameters of the segmentation network through gradient back propagation, and complete the iterative training when the loss function converges to the minimum value to obtain a trained intraoral scan image segmentation model;

[0017] S10, obtaining an intraoral scan image to be processed, inputting the intraoral scan image to be processed into a trained intraoral scan image segmentation model, and obtaining a tooth segmentation result in the intraoral scan image to be processed.

[0018] The method includes data downsampling of the original intraoral scan image; adaptive semantic coordination of the geometric attributes of the Euclidean space of the intraoral scan image, focusing on the relationship between spatial and semantic features in the real space dimension; learning the structural features and rotation invariance of the intraoral scan image through a spatial-semantic graph learning module; sending the learned representation to the segmentation for prediction, and segmenting the single tooth and gum in the intraoral scan image. The present invention can effectively deal with the rotation invariance of the intraoral scan image and improve the segmentation accuracy.

[0019] Preferably, S1 specifically includes:

[0020] The original intraoral scan image is obtained, and the original intraoral scan image is downsampled to N vertices. During the downsampling process, the topological structure of the original intraoral scan image is maintained and denoising is performed to obtain the geometric properties of each vertex in the downsampled intraoral scan image, wherein the geometric properties of the vertex include coordinate information and vector information of each vertex, wherein the coordinate information and the vector information constitute semantic information, and N is an integer greater than 1.

[0021] Among them, in order to maintain the topological structure of the original intraoral scan image, training can be performed. The original data is too large to be trained, so downsampling is performed while retaining the topological structure.

[0022] Preferably, the purpose of S2 is to semantically coordinate the semantic information of the intraoral scanned image to cope with the similarity between points in the image and avoid confusion caused by the subsequent feature extraction stage. S2 specifically includes:

[0023] S21, based on the geometric attributes of each vertex in the downsampled intra-oral scan image, obtain the semantic information f∈R of the downsampled intra-oral scan image N×6 , semantic information f∈R N×6 Contains spatial coordinates x, y, z and vectors fx, fy, fz. The matrix containing semantic information is multiplied with its transposed matrix to obtain the interaction information f between the semantic information. cross ∈R 6×6 , where N is the number of vertices of the downsampled intraoral scan image, 6 represents the length of the vector corresponding to the semantic information, and R is the vector space;

[0024] S22, the semantic information f∈R of the downsampled intraoral scan image N×6 The mutual information between the semantic information and cross ∈R 6×6 Perform matrix multiplication to obtain enhanced semantic information f cross ∈R N×6 , enhancing the interaction between semantic information while restoring it to its original dimension;

[0025] S23, the enhanced semantic information f cross ∈R N×6 and the semantic information of the downsampled intraoral scan image f∈R N ×6 Perform channel-level splicing and extract features through 1×1 convolution to obtain the semantic information f after feature extraction aug ∈R N×6 ;

[0026] S24, extract the semantic information f aug ∈R N×6 and the semantic information of the downsampled intraoral scan image f∈R N×6 Perform element-level addition to obtain semantically coordinated semantic information f sem ∈R N×6 .

[0027] Preferably, S3 constructs a local area with K+1 vertices centered on each vertex through a K-nearest neighbor strategy, calculates polar coordinate information (polar diameter, polar angle, and azimuth) of the area, and obtains rotation invariance information. S3 specifically includes:

[0028] S31, based on the geometric attributes of each vertex in the downsampled intra-oral scan image, obtain the spatial coordinate information p∈R of the downsampled intra-oral scan image N×3 , N is the number of vertices of the downsampled intraoral scan image, 3 represents the length of the spatial coordinate information vector, and the spatial coordinate information p∈R N×3 Contains spatial coordinates x, y, z. Based on spatial coordinate information p∈R N ×3 Calculate the Euclidean distance between any two vertices among N vertices in, represents the Euclidean distance between vertex i and vertex j;

[0029] The calculation formula of Euclidean distance is:

[0030]

[0031] in, represents the Euclidean distance between vertex i and vertex j. The coordinates of vertex i are (x i ,y i ,z i ), the coordinates of vertex j are (x j ,y j ,z j ), obtain the spatial information distance matrix between any two vertices among N vertices

[0032] S32. For each vertex pi , the set of its neighbor points is Take vertex p i As the center, sort the distances of other vertices from the center from small to large, select the first K vertices to jointly construct the first local area, and the local area of ​​N vertices is expressed as F local ∈R N×3×K ;

[0033] S33, calculating and obtaining polar coordinate information of the first local area in the first local area, including: polar diameter , polar angle and azimuth The calculation methods are:

[0034]

[0035]

[0036]

[0037] Among them, i represents the central vertex p i The index of k represents the center vertex p i The index of the local area is k's nearest neighbor point The polar radius of the local area of ​​N vertices is expressed as F dis ∈R N×1×K , the polar angle is expressed as F modulus ∈R N×1×K , the azimuth angle is expressed as F azimuth ∈R N×1×K ;

[0038] S34, based on neighboring points The coordinates of the first local area are calculated in the first local area. i,g The coordinate information of the first local area is i,g The coordinate information of the centroid point p is calculated i,g The polar angle θ i,g and azimuth φ i,g , based on the centroid point p i,g The polar angle θ i,g and azimuth φ i,g , polar angle and azimuth Update and obtain the updated polar angle and updated azimuth Based on the polar diameter Updated polar angle and the updated azimuth Perform channel-level splicing to obtain rotational invariance information, that is, the geometric features of the relative relationship between teeth after rotation will not be changed by polar coordinate construction. Since the teeth of different patients have a certain rotation (different angles), this rotation difference will cause the network to learn inconsistent features (increasing the difficulty of the network to find patterns). Therefore, it is hoped that the network can learn information that will not affect the judgment (identification of teeth) after rotation, that is, rotational invariance information.

[0039] Preferably, S4 uses a K nearest neighbor strategy to construct a local area with K+1 vertices as the center, and performs bidirectional local feature aggregation in the coordinate space and the feature space. S4 specifically includes:

[0040] S41. For each vertex p i The semantic information f∈R N×C , where N is the number of vertices in the downsampled intraoral scan image, C represents the length of the vector corresponding to the semantic information, and the semantic Euclidean distance between any two vertices in N vertices is calculated using the semantic information. The calculation formula is:

[0041]

[0042] in, Represents the Euclidean distance of the semantic information between vertex i and vertex j, and obtains the semantic information distance matrix between any two vertices among N vertices

[0043] S42. For each vertex p i , with vertex p i Center, sort the semantic information distances of other vertices from the center from small to large, and select the top K vertices to jointly construct the second local area; i The semantic information of i ∈R N×C , p i The semantic information set of the neighboring points is {f i 1 ,f i 2 ,...,f i k ,...f i K}, based on each vertex p i The semantic information of semantic coordination sem ∈R N×6 and vertex p i The semantic information set composed of the neighboring points of is obtained by aggregating local features in the semantic space to obtain the first local aggregation feature The local feature aggregation method of the semantic space is:

[0044]

[0045] Where l represents the number of layers of the spatial-semantic graph learning module, [·,·] represents the channel-level concatenation operation, and cos(·,·) represents the cosine similarity calculation;

[0046] Based on spatial information distance matrix The K indexes in the semantic space are aggregated to obtain the second local aggregate feature The first local aggregation feature and the second local aggregate feature Perform channel-level splicing to obtain aggregated local semantic features

[0047] S43. For each vertex p i , with vertex p i Center, sort the spatial distances of other vertices from the center from small to large, and select the top K vertices to jointly construct the third local area; based on each vertex p i The coordinate information and vertex p i The coordinate information set composed of the neighboring points of is used to obtain the third local aggregation feature by local feature aggregation in the coordinate space. ; Vertex p i The set of neighbor points of , the local feature aggregation method of the coordinate space is:

[0048]

[0049] Based on semantic information distance matrix The K indexes in the coordinate space are aggregated by local features to obtain the fourth local aggregation feature The third local aggregation feature and the fourth local aggregation feature Perform channel-level splicing to obtain aggregated local spatial features

[0050] Preferably, S5 reconstructs dual-space features for local space information of the Euclidean coordinate space and rotation invariance information of the polar coordinate space, controls the diversity and redundancy of dual-space features, and obtains robust rotation invariance perception. S5 specifically includes:

[0051] S51, the rotation invariance information F polar ∈R N×3×K and local spatial features Perform channel-level splicing to obtain dual-space expression features

[0052] S52, dual-space expression features The dual-space feature representation is learned through two 1×1 convolutions, and the outputs of the two convolutional layers are spliced ​​at the channel level to obtain the spliced ​​dual-space expression features. Among them, C' is the feature dimension after splicing;

[0053] S53. In order to learn the significant rotation invariance information in the dual-space representation, the concatenated dual-space expression features Perform feature screening to obtain dual-space expression features The screening method is:

[0054]

[0055] Among them, α is the weight learned by the convolutional layer, ⊙ represents the Hadamard product, Reshape(·) represents matrix reorganization, and Max(·) represents the maximum pooling operation;

[0056] S54, select the dual space expression feature Dual-space expression features Perform channel-level splicing to obtain dual-space feature representation In order to effectively control the diversity and redundancy of dual-space representations and facilitate subsequent graph convolution learning, the dual-space feature representation is reconstructed through feature reconstruction loss. For supervision, feature reconstruction loss The calculation method is:

[0057]

[0058] Among them, e is the natural index and C' is the characteristic dimension.

[0059] Preferably, in order to enable the spatial-semantic graph learning module to learn the association between the coordinate information and the semantic information, S6 specifically includes:

[0060] S61. The input of each layer of spatial-semantic graph learning module is a dual-space feature representation and local semantic features

[0061] S62, dual-space feature representation Feature learning is performed through a feedforward neural network, and local semantic features are matched through a 1×1 convolutional layer. The characteristic dimension of

[0062] S63. After feature dimension matching, dual-space feature representation and local semantic features Interactive learning is performed to obtain the output F of the single-layer space-semantic graph learning module l , the output F of the single-layer spatial-semantic graph learning module l The calculation method is:

[0063]

[0064] Among them, Max(·) represents the maximum pooling operation, Φ(·) is the multi-layer perceptron, and σ(·) is the convolutional layer.

[0065] Preferably, S7 aggregates the multi-scale feature values ​​of the outputs of the stacked spatial-semantic graph learning modules to obtain the final feature representation F seg For prediction, S7 specifically includes:

[0066] Based on the output of each layer of spatial-semantic graph learning module, multi-scale tooth features are spliced ​​to obtain the final tooth feature description F seg , the final tooth feature description F seg The calculation method is:

[0067] F seg =Φ([F 1 ,F 2 ,...,F l ]);

[0068] Where l is the number of layers in the spatial-semantic graph learning module, Φ(·) is a multilayer perceptron, and [·,·] represents the channel-level concatenation operation.

[0069] Preferably, the segmentation result predicted by S8 is Y, and Y is calculated as follows:

[0070] Y = Softmax(Φ(F seg ));

[0071] Among them, Y represents the probability of the tooth category corresponding to each point in the intraoral scan image, Softmax(·) is the normalization function, and Φ(·) is the multi-layer perceptron.

[0072] Preferably, S9 jointly supervises the training of the space-semantic graph learning module through cross entropy loss and overall feature reconstruction loss, wherein the dual space feature representation is reconstructed through the overall feature reconstruction loss. For supervision, overall feature reconstruction loss The calculation method is:

[0073]

[0074] Among them, e is the natural index, C' is the feature dimension, and L represents the total number of dual-space feature representations;

[0075] Cross Entropy Loss The calculation method is:

[0076]

[0077] Where N is the number of vertices of the intraoral scan image, y i and x i are the true value and predicted value of the i-th vertex respectively;

[0078] The loss function of the spatial-semantic graph learning module is:

[0079]

[0080] Among them, λ FR The balanced hyperparameters are manually set.

[0081] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0082] 1. The present invention utilizes low-dimensional semantic coordination and spatial semantic bidirectional aggregation strategies, which can effectively utilize the geometric attributes of intraoral scanned images, use spatial coordinates to guide semantic learning of tooth structure, avoid feature confusion caused by attribute conflicts, and improve the segmentation accuracy at the boundaries between teeth and gums and between teeth.

[0083] 2. The present invention uses bidirectional enhanced graph convolution based on dual-space representation reconstruction to learn the teeth and rotation invariance of intraoral scan images, which can guide the network to learn the tooth structure characteristics of heterogeneous intraoral scan images. At the same time, the introduction of polar coordinate expression in the Euclidean coordinate expression can enable the network to learn the distribution differences of heterogeneous data and improve the segmentation robustness of the segmentation model to cope with multi-center intraoral scan images. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;

[0085] Figure 1 A schematic diagram of the intraoral scan image segmentation network structure implemented in the present invention;

[0086] Figure 2 It is a schematic diagram of the overall flow of the method of an embodiment of the present invention;

[0087] Figure 3 A schematic diagram of semantic coordination according to an embodiment of the present invention;

[0088] Figure 4 It is a schematic diagram of a dynamic bidirectional enhanced graph convolution module according to an embodiment of the present invention;

[0089] Figure 5A schematic diagram of a dual-space representation reconstruction module according to an embodiment of the present invention;

[0090] Figure 6 Schematic diagram of a spatial-semantic graph learning module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0092] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0093] Embodiment 1

[0094] The present invention proposes a method for tooth segmentation of intraoral scanned images based on dual-space representation of semantic coordination, and its network structure diagram is shown in the figure. Figure 1 As shown in the figure, after downsampling the original intraoral scan image and acquiring geometric information, the geometric attributes of the Euclidean space of the intraoral scan image are adaptively semantically coordinated, focusing on the relationship between spatial and semantic features in the real space dimension. Then, the structural features and rotation invariance of the intraoral scan image are learned through stacked bidirectional enhanced graph convolution modules based on dual-space representation reconstruction. Finally, the multi-scale tooth feature description is aggregated and sent to the segmentation network to obtain the prediction result.

[0095] like Figure 1 As shown, the segmentation network in the present invention includes three dynamic bidirectional enhanced graph convolution modules, the first dynamic bidirectional enhanced graph convolution module input is the semantically coordinated semantic information f sem ∈R N×6 and spatial coordinate information, the input of the second dynamic bidirectional enhanced graph convolution module is the output and spatial coordinate information of the first dynamic bidirectional enhanced graph convolution module, and the input of the third dynamic bidirectional enhanced graph convolution module is the output and spatial coordinate information of the second dynamic bidirectional enhanced graph convolution module. Then, the multi-scale tooth features of the outputs of the three dynamic bidirectional enhanced graph convolution modules are spliced, and the splicing results are sent to the segmentation network to obtain the prediction results.

[0096] like Figure 2As shown in the figure, each module based on dynamic bidirectional enhanced graph convolution includes: a spatial-semantic bidirectional aggregation module, a dual-space representation reconstruction module and a spatial-semantic graph learning module. The spatial-semantic bidirectional aggregation module is used to construct the spatial local features and semantic local features of the teeth aggregated in the local area to obtain the local description of the teeth; the dual-space representation reconstruction module is used to measure the importance of Euclidean space information and polar coordinate space information, and make full use of the diversity of the two spatial information; the spatial-semantic graph learning module uses spatial information to guide semantic information to learn the overall geometric structure of teeth.

[0097] The embodiment of the present invention provides a method for tooth segmentation of intraoral scanned images based on dual-space representation of semantic coordination, the process is as follows: Figure 2 As shown, the following steps are included:

[0098] S1. Downsample the original intraoral scan image, maintain the original topological structure, and obtain the geometric attributes of each point, including spatial coordinates and spatial vectors;

[0099] S2. The spatial coordinates and spatial vectors constitute the semantic information of the intraoral scanned image, which will be semantically coordinated.

[0100] For the semantic information of the original dimension f∈R N×6 , perform matrix multiplication on the matrix containing semantic information and its transposed matrix to obtain the interaction information f between the semantic information cross ∈R 6×6 ; Semantic information of the original dimension f∈R N×6 With the interaction information f cross ∈R 6×6 Perform matrix multiplication to restore the original dimension while strengthening the interaction between semantic information, i.e., f cross ∈R N×6 ; Enhanced semantic information f cross ∈R N×6 The semantic information f∈R of the original dimension N×6 Perform channel-level splicing and extract features through 1×1 convolution to obtain the semantic information f after feature extraction aug ∈R N×6 ; Semantic information after feature extraction f aug ∈R N×6 The semantic information f∈R of the original dimension N×6 Perform element-level addition to obtain semantically coordinated semantic information f sem ∈R N×6 .

[0101] S3. In the coordinate space, using the K nearest neighbor strategy, select K nearest vertices with each vertex as the center to jointly construct the first local area, calculate the polar coordinate information (polar diameter, azimuth and polar angle) of the area in the first local area, and update the azimuth and polar angle to obtain rotation invariance information, such as Figure 3 shown.

[0102] For the spatial coordinate information p∈R of the original dimension N×3 , calculate the pairwise Euclidean distance between N vertices, the calculation formula is as follows:

[0103]

[0104] in, represents the Euclidean distance between vertex i and vertex j. The coordinates of vertex i are (x i ,y i ,z i ), the coordinates of vertex j are (x j ,y j ,z j ). Obtain the spatial information distance matrix between N vertices. For each vertex p i , retrieve the K points closest to it as the center, build the first local area, for vertex p i , the set of its neighbor points is The local area of ​​N vertices is expressed as F local ∈R N×3×K ; Calculate the polar coordinate information of the area in the first local area, including the polar diameter Polar Angle and azimuth The calculation method is:

[0105]

[0106]

[0107]

[0108] Among them, i represents the central vertex p i The index of k represents the center vertex p i The index of the local area is k's nearest neighbor point The polar radius of the local area of ​​N vertices is expressed as F dis ∈R N×1×K , the polar angle is expressed as F modulus ∈R N×1×K , the azimuth angle is expressed as F azimuth ∈R N×1×K ; Calculate the centroid point p of the area in the first local area i,g, calculated as:

[0109]

[0110] Among them, x k ,y k and z k The nearest neighbor The coordinates of the centroid p are calculated by following the calculation steps of S33 i,g The polar angle θ i,g and azimuth φ i,g , and the polar angle F modulus ∈R N×1×K and azimuth F azimuth ∈R N×1×K Update and obtain the updated polar angle and updated azimuth The update method is as follows:

[0111] Then the polar radius, updated polar angle and azimuth are concatenated at the channel level to obtain the rotation invariant information F polar ∈R N×3×K .

[0112] S4, spatial-semantic bidirectional aggregation module and spatial-semantic bidirectional aggregation strategy Figure 4 As shown, it is carried out in a dynamic bidirectional enhanced graph convolution module, which includes a space-semantic graph learning module. The number of layers of the dynamic bidirectional enhanced graph convolution module is the same as that of the space-semantic graph learning module. The dynamic bidirectional enhanced graph convolution module first performs bidirectional aggregation of space and semantics, and then reconstructs dual-space features. Then, the space-semantic graph learning module is used for learning. The K nearest neighbor strategy is used to select K nearest vertices with each vertex as the center in the coordinate space and the semantic space to jointly construct the second local area, aggregate local features, and at the same time, the coordinate space and the semantic space exchange the indexes of the K nearest vertices, aggregate local features, and obtain local spatial information and local semantic information respectively.

[0113] The local feature aggregation in the present invention can be understood as the aggregation of the features (semantic or spatial information) of the center point and the features (semantic or spatial information) of the K points closest to the center point (the aggregation operation includes the center point information, the neighboring point information, the difference information between points, the similarity between points, and the spatial and semantic aggregation methods are different.

[0114] For the semantic information of spatial features f∈R N×C , calculate the semantic Euclidean distance between N vertices, the calculation formula is as follows:

[0115]

[0116] in Represents the Euclidean distance of the semantic information between vertex i and vertex j. Obtain the semantic information distance matrix between N vertices For each vertex p i , taking it as the center to retrieve the K points with the closest semantic information and construct the second local area, p i The semantic information of i ∈R N×C , the semantic information set of its neighboring points is {f i 1 ,f i 2 ,...,f i k ,...f i K}, the local feature aggregation method of the semantic space is as follows:

[0117]

[0118] Where l represents the number of layers in the spatial-semantic graph learning module, and also represents the number of layers based on the dynamic bidirectional enhanced graph convolution module, [·,·] represents the channel-level concatenation operation, and cos(·,·) represents the cosine similarity calculation. The spatial information distance matrix obtained by S31 The K indexes in N are aggregated again in the semantic space to obtain Will and Perform channel-level splicing to obtain aggregated local semantic features For each vertex p i , retrieve the K points closest to it as the center, construct the third local area, for vertex p i ∈R N×3 , the set of its neighbor points is The local features of the coordinate space are aggregated as follows:

[0119]

[0120] Semantic information distance matrix obtained by S41 The K indices in , again perform local feature aggregation in the coordinate space, and obtain Will and Perform channel-level splicing to obtain aggregated local spatial features

[0121] S5, dual-space representation reconstruction module Figure 5As shown, the local spatial information and the rotation invariant information are feature reconstructed to obtain the local spatial information with rotation invariance;

[0122] Rotation invariance information F polar ∈R N×3×K and local spatial features Perform channel-level stitching to obtain dual-space expression Dual space expression The dual-space feature representation is learned through two 1×1 convolutions, and the outputs of the two convolutional layers are concatenated at the channel level to obtain Where C' is the concatenated feature dimension. In order to learn the significant rotation invariance information in the dual-space feature representation, Perform feature screening to obtain The calculation formula is as follows:

[0123]

[0124] Where α is the weight learned by the convolutional layer, ⊙ represents the Hadamard product, Reshape(·) represents matrix reorganization, and Max(·) represents the maximum pooling operation. and Perform channel-level splicing to obtain the final dual-space feature representation In order to effectively control the diversity and redundancy of dual-space feature representations and facilitate subsequent graph convolution learning, supervision is performed through reconstruction loss. The supervision method is as follows:

[0125]

[0126] Where e is the natural index, C' is the feature dimension, and Max(·) represents the maximum pooling operation.

[0127] S6, spatial-semantic learning module Figure 6 As shown, the structure and rotation invariance of teeth are learned through the local spatial information of rotation invariance and the local semantic information.

[0128] The input of the spatial-semantic graph learning module is the dual-space feature representation and semantic features Feature learning is performed through a feed-forward neural network and matched through a 1×1 convolutional layer feature dimension; after the feature dimension is matched, and Interactive learning is performed to obtain the output F of the spatial-semantic graph learning module of this layer l ∈R N×C”×K , expressed as follows:

[0129]

[0130] Among them, Max(·) represents the maximum pooling operation, Φ(·) is the multi-layer perceptron, and σ(·) is the convolutional layer.

[0131] S7, such as Figure 1 As shown, through the stacked spatial-semantic graph learning modules, multi-scale tooth features are spliced ​​together to obtain the final tooth feature description.

[0132] The output of the stacked spatial-semantic graph learning module is aggregated into multi-scale feature values ​​to obtain the final feature representation F seg For prediction, the aggregation is described as follows:

[0133] F seg =Φ([F 1 ,F 2 ,...,F l ]),

[0134] Where l is the number of layers of the spatial-semantic graph learning module, which also represents the number of layers based on the dynamic bidirectional enhanced graph convolution module, Φ(·) is a multi-layer perceptron, and [·,·] represents the channel-level concatenation operation.

[0135] S8. Input the final feature description of the teeth into the segmentation network for prediction to obtain the final segmentation result.

[0136] The segmentation results are shown as follows:

[0137] Y = Softmax(Φ(F seg )),

[0138] where Y∈R N×15 It represents the probability of the tooth category corresponding to each point in the intraoral scan image, Softmax(·) is the normalization function, and Φ(·) is the multi-layer perceptron.

[0139] S9. Calculate the loss function of the model according to the segmentation results, update the model parameters through gradient back propagation, and complete the iterative training of the model when the loss function converges to the minimum value to obtain a trained intraoral scan image segmentation model.

[0140] Reconstruction loss through cross entropy loss and overall feature Jointly supervised model training, cross entropy loss It is expressed as follows:

[0141]

[0142] Where N is the number of vertices of the intraoral scan image, y i and x iare the true value and predicted value of the i-th vertex respectively. Overall feature reconstruction loss It is expressed as follows:

[0143]

[0144] Where e is the natural index, C' is the feature dimension, and Max(·) represents the maximum pooling operation.

[0145] The final loss function is expressed as follows:

[0146]

[0147] Among them, λ FR The balanced hyperparameters are manually set.

[0148] S10, obtaining an intraoral scan image to be processed, inputting the intraoral scan image to be processed into a trained intraoral scan image segmentation model, and obtaining a tooth segmentation result in the intraoral scan image to be processed.

[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A tooth segmentation method for intraoral scanned images based on dual-space representation of semantic coordination, characterized in that: The method comprises: S1, obtaining an original intra-oral scan image, down-sampling the original intra-oral scan image, maintaining the topological structure of the original intra-oral scan image during the down-sampling process, and obtaining the geometric attributes of each vertex in the down-sampled intra-oral scan image; S2, obtaining semantic information of the downsampled intra-oral scan image based on the geometric attributes of each vertex in the downsampled intra-oral scan image, and performing semantic coordination processing on the semantic information to obtain semantic coordinated semantic information; S3. In the coordinate space, using the K nearest neighbor strategy, take each vertex in the sampled intraoral scan image as the center, sort the distances of other vertices from the center from small to large, and obtain the spatial information distance matrix between any two vertices in N vertices. Select the first K vertices in the sorting to jointly construct a first local area, calculate and obtain polar coordinate information of the first local area in the first local area, update the polar coordinate information to obtain updated polar coordinate information, and obtain rotation invariance information based on the updated polar coordinate information; S4, based on each vertex p i The semantic information of the semantic coordination is used to obtain the semantic Euclidean distance between vertices; the semantic information distance between vertices is obtained based on the semantic Euclidean distance between vertices; for each vertex p i , with vertex p i As the center, sort the semantic information distances of other vertices from the center from small to large, and obtain the semantic information distance matrix between any two vertices among N vertices. Select the first K vertices in the sorting to jointly construct the second local area; based on each vertex p i The semantic information of the semantic coordination and vertex p i The semantic information set composed of the neighboring points of is obtained by aggregating local features in the semantic space to obtain the first local aggregation feature Based on spatial information distance matrix K indices in the vertex p i The semantic information of the semantic coordination and the distance vertex p i The semantic information of the first K vertices is sorted from small to large by distance, and the second local aggregation feature is obtained by local feature aggregation in the semantic space The first local aggregation feature and the second local aggregate feature Perform channel-level splicing to obtain aggregated local semantic features For each vertex p i , with vertex p i As the center, sort the spatial distances of other vertices from the center from small to large, and select the top K vertices to jointly construct the third local area; based on each vertex p i The coordinate information and vertex p i The coordinate information set composed of the neighboring points of is used to obtain the third local aggregation feature by local feature aggregation in the coordinate space. Based on semantic information distance matrix K indices in the vertex p i The spatial information and distance vertex p i Sort the spatial information of the first K vertices from small to large, and aggregate the local features in the coordinate space to obtain the fourth local aggregation feature The third local aggregation feature and the fourth local aggregation feature Perform channel-level splicing to obtain aggregated local spatial features S5. Combine rotation invariance information and local spatial features Perform channel-level splicing to obtain dual-space expression features Dual-space expression features Convolution is performed through the convolutional layers respectively, and the outputs of the two convolutional layers are spliced ​​at the channel level to obtain the spliced ​​dual-space expression features. Dual-space expression features Perform feature screening to obtain dual-space expression features The dual-space expression features will be screened Dual-space expression features Perform channel-level splicing to obtain dual-space feature representation S6. Dual-space feature representation and local semantic features The spatial-semantic graph learning module learns the structure and rotation invariance of teeth and obtains the output of each layer of the spatial-semantic graph learning module; S7, based on the output of each layer of the spatial-semantic graph learning module, the multi-scale tooth features are spliced ​​to obtain the final tooth feature description; S8, inputting the final tooth feature description into the segmentation network for prediction to obtain the final segmentation result; S9, calculating the overall loss function of the segmentation network according to the segmentation results and the accuracy annotation, updating the parameters of the segmentation network by gradient back propagation, completing the iterative training when the loss function converges to the minimum value, and obtaining a trained intraoral scan image segmentation model; S10, obtaining an intraoral scan image to be processed, inputting the intraoral scan image to be processed into a trained intraoral scan image segmentation model, and obtaining a tooth segmentation result in the intraoral scan image to be processed; S2 specifically includes: S21, based on the geometric attributes of each vertex in the downsampled intra-oral scan image, obtain the semantic information of the downsampled intra-oral scan image, based on the semantic information f∈R of the downsampled intra-oral scan image N×6 Obtain the interactive information between semantic information cross ∈R 6×6 , where N is the number of vertices of the downsampled intraoral scan image, 6 represents the length of the vector corresponding to the semantic information, and R is the vector space; S22, the semantic information f∈R of the downsampled intraoral scan image N×6 The mutual information between the semantic information and cross ∈R 6×6 Perform matrix multiplication to obtain enhanced semantic information f cross ∈R N×6 ; S23, the enhanced semantic information f cross ∈R N×6 and the semantic information of the downsampled intraoral scan image f∈R N×6 Perform channel-level splicing and extract features through convolution to obtain the semantic information f after feature extraction aug ∈R N×6 ; S24, the semantic information after feature extraction f aug ∈R N×6 and the semantic information of the downsampled intraoral scan image f∈R N ×6 Perform element-level addition to obtain semantically coordinated semantic information f sem ∈R N×6 .

2. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S1 specifically includes: The original intraoral scan image is obtained, and the original intraoral scan image is downsampled to N vertices. During the downsampling process, the topological structure of the original intraoral scan image is maintained and denoising is performed to obtain the geometric properties of each vertex in the downsampled intraoral scan image, wherein the geometric properties of the vertex include coordinate information and vector information of each vertex, wherein the coordinate information and the vector information constitute semantic information, and N is an integer greater than 1.

3. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S3 specifically includes: S31, based on the geometric attributes of each vertex in the downsampled intra-oral scan image, obtain the spatial coordinate information p∈R of the downsampled intra-oral scan image N×3 , N is the number of vertices of the downsampled intraoral scan image, 3 represents the length of the spatial coordinate information vector, based on the spatial coordinate information p∈R N×3 Calculate the Euclidean distance between any two vertices among N vertices in, represents the Euclidean distance between vertex i and vertex j; S32. For each vertex p i , the set of its neighbor points is Take vertex p i As the center, sort the distances of other vertices from the center from small to large, select the first K vertices to jointly construct the first local area, and the local area of ​​N vertices is expressed as F local ∈R N×3×K ; S33, calculating and obtaining polar coordinate information of the first local area in the first local area, including: polar diameter Polar Angle and azimuth S34, based on neighboring points The coordinates of the first local area are calculated in the first local area. i,g The coordinate information of the first local area is: i,g The coordinate information of the centroid point p is calculated i,g The polar angle θ i,g and azimuth φ i,g , based on the centroid point p i,g The polar angle θ i,g and azimuth φ i,g , polar angle and azimuth Update and obtain the updated polar angle and updated azimuth Based on the polar diameter Updated polar angle and the updated azimuth Perform channel-level splicing to obtain rotation invariance information.

4. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S4 specifically includes: S41. For each vertex p i The semantic information f∈R N×C , where N is the number of vertices in the downsampled intraoral scan image, C represents the length of the vector corresponding to the semantic information, and the semantic Euclidean distance between any two vertices in N vertices is calculated using the semantic information. The calculation formula is: in, Represents the Euclidean distance of the semantic information between vertex i and vertex j, and obtains the semantic information distance matrix between any two vertices among N vertices S42. For each vertex p i , with vertex p i As the center, sort the semantic information distances of other vertices from the center from small to large, and select the top K vertices to jointly construct the second local area; i The semantic information of i ∈R N×C , p i The semantic information set composed of the neighboring points is For each vertex p i The semantic information of semantic coordination sem ∈R N×6 and vertex p i The semantic information set composed of the neighboring points of is obtained by aggregating local features in the semantic space to obtain the first local aggregation feature The local feature aggregation method of the semantic space is: Where l represents the number of layers of the spatial-semantic graph learning module, [·,·] represents the channel-level concatenation operation, and cos(·,·) represents the cosine similarity calculation; Based on spatial information distance matrix The K indexes in the semantic space are aggregated to obtain the second local aggregate feature The first local aggregation feature and the second local aggregate feature Perform channel-level splicing to obtain aggregated local semantic features S43. For each vertex p i , with vertex p i As the center, sort the spatial distances of other vertices from the center from small to large, and select the top K vertices to jointly construct the third local area; based on each vertex p i The coordinate information and vertex p i The coordinate information set composed of the neighboring points of is used to obtain the third local aggregation feature by local feature aggregation in the coordinate space. Vertex p i The set of neighbor points of The local feature aggregation method of the coordinate space is: Based on semantic information distance matrix The K indexes in the coordinate space are aggregated by local features to obtain the fourth local aggregation feature The third local aggregation feature and the fourth local aggregation feature Perform channel-level splicing to obtain aggregated local spatial features 5. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S5 specifically includes: S51, the rotation invariance information F polar and local spatial features Perform channel-level splicing to obtain dual-space expression features S52, dual-space expression features The dual-space feature representation is learned through two 1×1 convolutions, and the outputs of the two convolutional layers are spliced ​​at the channel level to obtain the spliced ​​dual-space expression features. S53, double-space expression features of splicing Perform feature screening to obtain dual-space expression features The screening method is: Among them, α is the weight learned by the convolutional layer, ⊙ represents the Hadamard product, Reshape(·) represents matrix reorganization, and max_pool represents the maximum pooling operation; S54, screening dual-space expression features Dual-space expression features Perform channel-level splicing to obtain dual-space feature representation Dual-space feature representation via feature reconstruction loss For supervision, feature reconstruction loss The calculation method is: Among them, e is the natural index and C' is the characteristic dimension.

6. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S6 specifically includes: S61. The input of each layer of spatial-semantic graph learning module is a dual-space feature representation and local semantic features S62, dual-space feature representation Feature learning is performed through a feedforward neural network, and local semantic features are matched through a 1×1 convolutional layer. The characteristic dimension of S63. After feature dimension matching, dual-space feature representation and local semantic features Interactive learning is performed to obtain the output F of the single-layer space-semantic graph learning module l , the output F of the single-layer spatial-semantic graph learning module l The calculation method is: Among them, max_pool represents the maximum pooling operation, Φ(·) is the multi-layer perceptron, and σ(·) is the convolutional layer.

7. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S7 specifically includes: Based on the output of each layer of spatial-semantic graph learning module, multi-scale tooth features are spliced ​​to obtain the final tooth feature description F seg , the final tooth feature description F seg The calculation method is: F seg =Φ([F 1 ,F 2 ,...,F l ]); Where l is the number of layers in the spatial-semantic graph learning module, Φ(·) is a multilayer perceptron, and [·,·] represents the channel-level concatenation operation.

8. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: The segmentation result predicted by S8 is Y, which is calculated as follows: Y=Softmax(Φ(F seg )); Among them, Y represents the probability of the tooth category corresponding to each point in the intraoral scan image, Softmax(·) is the normalization function, and Φ(·) is the multi-layer perceptron.

9. The method for tooth segmentation of intraoral scanned images based on dual-space representation based on semantic coordination according to claim 1, characterized in that: S9 jointly supervises the training of the space-semantic graph learning module through cross entropy loss and overall feature reconstruction loss, in which the dual-space feature representation is reconstructed through the overall feature reconstruction loss. For supervision, overall feature reconstruction loss The calculation method is: Among them, e is the natural index, C' is the feature dimension, and L represents the number of dual-space feature representations; Cross Entropy Loss The calculation method is: Where N is the number of vertices of the intraoral scan image, y i and x i are the true value and predicted value of the i-th vertex respectively; The loss function of the spatial-semantic graph learning module is: Among them, λ FR The balanced hyperparameters are manually set.

Images