A flexible registration based RGBD input and digitalization system for the oral cavity

Abstract

The application relates to a flexible registration oral digitalization system based on RGBD input, and belongs to the technical field of oral medical digitalization. RGBD images are used as input, a non-rigid scene reconstruction algorithm based on surface detail migration technology is used, and the albedo texture of the surface is reconstructed while rigid parts, non-rigid parts and semi-rigid parts in the oral cavity of a person are reconstructed. Further improvement of the technical scheme of the application is that: in the process, first, texture information or high-dimensional features extracted from the texture information are given to the point cloud, and then the point cloud is segmented to extract the region of interest; in the point cloud registration part, different smooth constraint coefficients are given to different parts according to the actual physical properties of different parts. The application mainly makes full use of the input texture and depth information, improves the modeling accuracy and texture authenticity, reduces the scanning reconstruction time, and improves the use comfort and convenience.

Description

TECHNICAL FIELD

[0001] The present application relates to a flexible registration oral digital system based on RGBD input, and belongs to the technical field of oral medical digitalization. BACKGROUND

[0002] Taking dental cast is an important way for diagnosis and treatment confirmation in oral medical treatment, and the dental cast is an important information medium of the oral environment of the patient, especially in tooth shape correction and denture production, the doctor usually needs to produce several pairs of dental casts at different periods for the patient to diagnose and plan or observe the treatment effect, so as to improve the pertinence of diagnosis and treatment plan.

[0003] In clinical practice, the three-dimensional data collection of the teeth in the oral cavity is divided into two ways of oral scanning and intraoral scanning, the oral scanning is to scan the dental cast of the patient by using the scanning device to obtain the digital three-dimensional, and the intraoral scanning is to directly scan and measure the teeth and related hard and soft tissues by extending the scanning device into the patient's mouth to obtain the digital three-dimensional in real time, compared with the oral scanning, the intraoral scanning realizes the true sense of moldless and digitization, is convenient and efficient, and has obvious advantages.

[0004] At present, the main problem of intraoral scanning is that the scanning device has high precision requirement, and the powder spraying method is used before scanning, the scanning device is usually large in size, and it is difficult to operate in the limited oral environment, which brings discomfort experience to the patient, and causes high equipment cost, which brings great pressure to the promotion of the equipment purchasing party and the using party, so there is a deficiency in the scanning process experience or market competitiveness. SUMMARY

[0005] The purpose of the present application is to provide a flexible registration oral digital system based on RGBD input.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0007] A flexible registration oral digital system based on RGBD input, which uses RGBD image as input, uses a non-rigid scene reconstruction algorithm based on surface detail transfer technology to reconstruct the albedo texture of the surface while reconstructing the rigid, non-rigid and semi-rigid parts in the human oral cavity.

[0008] Further improvement of the technical scheme of the present application is that in the process, first, the texture information or the high-dimensional features extracted from the texture information are given to the point cloud, and then the interested region is extracted by segmenting the point cloud; in the point cloud registration part, different smooth constraint coefficients are given to different parts according to the actual physical properties of different parts.

[0009] Further improvement of the technical scheme of the application is that, in order to improve the overall accuracy of the nearest neighbor tracking algorithm of adjacent frames, all vertices are used as optimization targets, all vertices are directly optimized, and a new calculation idea of two mesh nearest points is used.

[0010] Further improvement of the technical scheme of the application is that, the specific processing steps are as follows,

[0011] 1), RGB image processing

[0012] The input RGB image is used to extract the texture information or high-dimensional information corresponding to the point cloud, and the RGB image is subjected to the light removal processing to remove the influence of the light source and extract the texture of the oral cavity region itself.

[0013] 2), point cloud segmentation

[0014] The information extracted by the RGB image and the three-dimensional coordinates of the point cloud are combined to segment and extract the region of interest on the point cloud.

[0015] 3), point cloud preprocessing

[0016] The point cloud processing part combines the results of the previous step, uses a deep neural network to segment and smooth the depth map, removes the flying points and less noise, and at the same time tries to maintain the details; the network uses the inverse depth map as one of the input data, and the inverse depth map actually stores the reciprocal of the depth of the point; the network structure is an Image To Image network including but not limited to an Hourglass network structure, and in addition, the input channel can also contain an aligned RGB color picture to utilize the prior knowledge in the RGB color picture.

[0017] 4), point cloud uniform non-rigid registration

[0018] A non-rigid registration method based on dense point cloud is used, and the specific implementation of the method is as follows,

[0019] First, assume that a part of the reconstructed scene point cloud is Vtgt, and the newly added point cloud is Vsrc, then Vsrc can be transformed to a similar degree with Vtgt through non-rigid transformation, and a triangle is quickly established by three adjacent points in the point cloud, and the normal vector of a point in the triangle and the direction from the point to the other two points constitute a local space.

[0020] After the initial nearest points are determined by the nearest neighbor, the two meshes can be pulled closer, and all the optimization variables are all the vertices, without extracting key control points for sparse DQB (dual quaternion interpolation) control, and without interpolation operation.

[0021] 5), surface reconstruction

[0022] Surface is recovered from point cloud using methods including but not limited to Per-son Reconstruct and Marching Cube;

[0023] 6) Geometry post-processing

[0024] It includes removing duplicate points, filling holes, non-manifold surface removal and point surface processing.

[0025] The hole filling process is as follows,

[0026] First, find the most edge area of the geometry according to the half-edge structure, find the edge N edge, sort according to the vertex number of the edge, and the smaller edge of each mesh module can be considered as a hole, which needs to be triangulated, and some points are added according to the uniform area, then the triangulation process including but not limited to the Delaunay method is carried out, and then the smoothing is carried out after the triangulation is completed.

[0027] The duplicate point removal process is as follows,

[0028] Delete unreferenced vertices and duplicate vertices, delete duplicate triangular faces, separate the mesh according to the connected regions to obtain multiple different blocks, then sort according to the surface area or vertex number of the block, find the upper and lower two largest connected regions, and delete all the small area connected regions.

[0029] The non-manifold surface removal process is as follows,

[0030] Because the surface geometry of the object existing in the actual should not exist non-manifold, therefore need to carry out non-manifold processing to the geometry: first, according to the repetition degree of the edge, if the repetition degree is greater than 2, it is considered as a non-manifold area, then the edges with repetition degree greater than 2 are selected and deleted from the triangular mesh, then all the connected regions are recalculated, the smaller regions are deleted according to the connected regions, and the deleted edges are reconnected, thus completing the processing of non-manifold.

[0031] Further improvement of the technical scheme of the application is that in step 4, the three important penalty terms required in the iterative optimization process are as follows,

[0032] 1) Affine transformation smoothness penalty of source triangle adjacent points

[0033] The penalty ensures that the transformation of adjacent points is relatively smooth:

[0034] E S (v1...v n )=sum(||Ti-Tj||).

[0035] Wherein v1…vn is the vertex position in the source point cloud, is a variable, Ti and Tj are the affine transformation of the local coordinate system composed of the triangle in the source point cloud after the variable, ij is the numbering of the adjacent triangle set;

[0036] 2, the affine change of the source triangle minimizes the penalty

[0037] In order to ensure that the source triangle changes state through the least change, we take the following method:

[0038] E I (v1...v n )=sum(||Ti-I||).

[0039] Wherein I represents a 3*3 unit matrix, which means that this penalty term restricts all affine transformations to be equal to the unit matrix as much as possible, that is, no transformation.

[0040] 3, the minimum penalty of the position of the source vertex and the position of the target vertex

[0041] The penalty term is:

[0042] E c (v1...v n )=sum(||vi-ci||).

[0043] Wherein vi represents the vertex position in the source point cloud, and ci represents the position of the point in the target point cloud which is closest to vi in space position and has a consistent normal vector.

[0044] Due to the adoption of the above technical scheme, the technical effects obtained by the present application are:

[0045] The present application mainly makes full use of the input texture and depth information, reduces the scanning reconstruction time while improving the modeling accuracy and texture authenticity, and improves the use comfort and convenience.

[0046] The present application uses all vertices as optimization targets, directly optimizes all vertices, and proposes a new calculation idea of the two mesh nearest points, greatly improving the stability of calculation and the reconstruction accuracy of the algorithm.

[0047] The present application can improve the stability and speed of oral reconstruction, improve the accuracy of oral reconstruction, and improve the physical rationality of oral reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 It is the principle framework diagram of the whole processing process of the present application;

[0049] Figure 2 It is the principle of finding the nearest point;

[0050] Figure 3 is the principle of the reconstruction of the Perlin noise;

[0051] Figure 4 is a curvature-based smoothing processing schematic diagram;

[0052] Figure 5 is a non-manifold mesh;

[0053] Figure 6 is a local space system. DETAILED DESCRIPTION

[0054] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the present application is further described below in combination with specific embodiments.

[0055] The present application adopts RGBRGBDRGBD images as input, uses a non-rigid scene reconstruction algorithm based on surface detail migration technology, and reconstructs the albedo texture of the surface while reconstructing the rigid, non-rigid and semi-rigid parts in the human oral cavity.

[0056] After the aligned RGBDRGBRGBD images are collected, the information provided by the RGBRGB is given to the point cloud, and then the point cloud is segmented to extract semantic information. The point cloud is registered non-rigidly, and the multiple frame input point clouds are fused. Finally, the oral cavity mesh is reconstructed from the fused point cloud. And the albedo texture information is obtained by using the collected RGBRGB information, and finally the color three-dimensional model of the oral cavity is obtained.

[0057] In the process, first, the texture information or the high-dimensional features extracted from the texture information are given to the point cloud, and then the point cloud is segmented to extract the region of interest, such as dividing the point cloud into tongue / tooth / gum parts; in the point cloud registration part, according to the actual physical properties of different parts, the algorithm is given different smoothing constraint coefficients in different parts, which usually follows the following rules: tongue> interface part> gum> tooth. Under this framework, in order to improve the overall accuracy of the nearest neighbor tracking algorithm of adjacent frames, we abandon the idea of extracting sparse key points, and directly optimize all vertices by taking all vertices as optimization targets, and a new calculation idea of the nearest points of two meshes is proposed, which can greatly improve the stability of calculation and the reconstruction accuracy of the algorithm.

[0058] Specifically as Figure 1 shown.

[0059] The following is a specific embodiment

[0060] 1. RGB image processing

[0061] The input RGBRGB image is used to extract the texture information or high-dimensional information of the point cloud (such as RGBRGB after a semantic segmentation network, but not the final result, but the high-dimensional information after the backbone is assigned to the point cloud, which can reduce the impact of the semantic segmentation network error). At the same time, the RGBRGB image is processed to remove the light source effect and extract the texture of the oral cavity region itself.

[0062] 2、Point cloud segmentation

[0063] Combined with the information extracted by RGBRGB and the three-dimensional coordinates of the point cloud, the segmentation is performed on the point cloud to extract the region of interest, and the oral cavity region is divided into teeth, gums, tongue, etc.

[0064] 3、Point cloud preprocessing

[0065] The point cloud processing part combines the results of the previous step to use a deep neural network to segment and smooth the depth map, remove outliers and reduce noise, while trying to maintain details. The network uses an inverse depth map as one of the input data, and the inverse depth map is actually stored as the reciprocal of the depth of the point. The network structure is an ImageToImage network with an Hourglass network structure, and in addition, the input channel can also include an aligned RGB color image to utilize the prior knowledge in the RGB color image, such as the presence of strong ambient light shadows in tooth gaps and the presence of more outliers and noise in tooth and gum mirror reflection points.

[0066] 4、Point cloud uniform non-rigid registration

[0067] In traditional oral reconstruction, only gums and teeth can be better handled because these two parts are more likely to meet the rigid constraint, so only a few iterations of the rigid nearest neighbor algorithm are needed to register the two frames of point cloud input. In our scheme, a non-rigid registration method based on dense point cloud is used, and the specific implementation is as follows:

[0068] First, assume that a part of the reconstructed scene point cloud is V tgt And the newly added point cloud is V src Then V src can be transformed to a similar degree as V tgt We quickly establish a triangle in the point cloud by taking every three adjacent points as vertices (the method of constructing a triangle from two point clouds can use the classic Delaunay triangulation method), and the normal vector of a point in the triangle and the direction from the point to the other two points form a local space, as shown in Figure 1

[0069] where v4 is calculated as follows:​

[0070] v4 = v1 + (v2-v1) x (v3-v1) sqrt(|(v2-v1) x (v3-v1) |)

[0071] The representation of the local coordinate system can be written as:

[0072] V = [v2-v1 v3-v1 v4-v1]

[0073] Defining a spatial radiation transformation Q and a translation d, which is applied to the four vertices, we get:

[0074] Q*vi + d = v'i

[0075] The transformed coordinate system is:

[0076] V' = [v'2-v'1 v'3-v'1 v'4-v'1]

[0077] From the above, it can be seen that the affine change between the local coordinate systems can be determined and solved by the coordinates of the three points of the triangle before and after the non-rigid change:

[0078] Q = V'V -1

[0079] After determining the initial nearest neighbor points by the nearest neighbor method, the algorithm can pull the two meshes closer, where all the optimization variables are all the vertices, without extracting key control points for sparse DQB control, without interpolation operations, such as Figure 2 As shown, the following are three important penalty terms required in the iterative optimization process:

[0080] I. Affine transformation smoothness penalty of source triangle nearest neighbor points

[0081] This penalty ensures that the transformation of adjacent points is relatively smooth:

[0082] E S (v1...v n ) = sum(||Ti-Tj||).

[0083] Where v1...vn are the vertex positions in the source point cloud, T is a variable, Ti and Tj are the affine transformations of the local coordinate system composed of triangles in the source point cloud after transformation relative to the original coordinate system, and ij is the number of adjacent triangle sets.

[0084] II. Affine change minimization penalty of source triangle

[0085] In order to ensure that the source triangle changes its state through the least change, we take the following method:

[0086] E I(v1...v n ) = sum(||Ti-I||).

[0087] where I represents a 3*3 identity matrix, which means that this penalty term constrains all the affine transformation to be as close to identity as possible, i.e. no transformation.

[0088] Three, the minimization penalty of source vertex position and target vertex position

[0089] This penalty term is denoted as:

[0090] E c (v1...v n ) = sum(||vi-ci||).

[0091] where vi represents the position of a vertex in the source point cloud, and ci represents the position of a point in the target point cloud that is closest to vi in spatial position and has a consistent normal vector.

[0092] 5, Surface reconstruction

[0093] Surface can be recovered from point cloud using methods including but not limited to Possion reconstruction, marching cube, etc. This process is a general procedure, not the main protection point, only the common practice is introduced.

[0094] Possion reconstruction is a very intuitive method. Its core idea is that point cloud represents the position of the object surface, and its normal vector represents the direction of inside and outside. By implicitly fitting an indicator function derived from the object, a smooth estimate of the object surface can be given.

[0095] Given a region MM and its boundary The indicator function χM is defined as

[0096]

[0097] In this way, the problem of reconstructing is converted to the problem of reconstructing χM, as shown in Figure 3 .

[0098] 6, Geometry post-processing

[0099] Including removing duplicate points, filling holes, non-manifold surface removal and point cloud surface processing.

[0100] Curvature-based smoothing

[0101] Curvature-based smoothing is to move the vertex P along the straight line where the normal vector of P is located, from Figure 4As can be seen, the general Laplacian smoothing will move the points P to a position similar to the barycenter, while the curvature-based smoothing moves the position in the opposite direction of the normal vector, which is intended to better preserve the original shape of the model.

[0102] By properly choosing the weight ω in the Laplacian smoothing, we can make the curvature-based smoothing and the Laplacian smoothing algorithmically unified. It can be derived that ω is chosen as follows, which is equivalent to moving P as follows Figure 2 .

[0103] Thus, the position of the point P is still determined by the following formula, only the weight calculation method is modified:

[0104]

[0105] Hole filling:

[0106] First, according to the half-edge structure, find the most edge region of the geometric body, find N edges, sort the edges according to the number of vertices, and consider the smaller edges as holes, which need to be triangulated, add some points according to the uniform area, and then perform the triangulation process including but not limited to the Delaunay method. After triangulation, smoothing is performed.

[0107] Remove duplicate points:

[0108] Delete unreferenced vertices and duplicate vertices, delete duplicate triangular faces, separate the mesh according to the connected regions to obtain multiple different blocks, then sort the blocks according to the surface area or the number of vertices, find the two largest connected regions, and delete all the small area connected regions.

[0109] Non-manifold surface removal:

[0110] Since the surface geometry of the actual object should not exist non-manifold, it is necessary to process the non-manifold of the geometric body: first, according to the degree of repetition of the edge, if the degree of repetition is greater than 2, it is considered as a non-manifold region, then select the edges with a repetition degree greater than 2 and delete them from the triangular mesh, then recompute all the connected regions, delete the smaller regions, and reconnect the deleted edges, thus completing the processing of the non-manifold. As shown in Figure 5 .

[0111] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A flexible registration based RGBD input and oral digitalization system, comprising: Adopting RGBD image as input, using non-rigid scene reconstruction algorithm based on surface detail migration technology, the albedo texture of the surface is reconstructed while the rigid, non-rigid and semi-rigid parts in the mouth of the person are reconstructed; in the point cloud registration part, different smoothing constraint coefficients are given to different parts according to the actual physical properties of different parts; in order to improve the overall accuracy of the nearest neighbor tracking algorithm of adjacent frames, all vertices are used as optimization targets and all vertices are directly optimized; the specific processing steps are as follows, 1) RGB image processing The input RGB image is used to extract point cloud and its corresponding texture information or high-dimensional information, and the RGB image is processed to remove the influence of light source and extract the texture of the oral cavity region itself; 2) Point cloud segmentation The information extracted from the RGB image and the three-dimensional coordinates of the point cloud are combined to segment and extract the region of interest on the point cloud; 3) Point cloud preprocessing The depth image is segmented and smoothed using a deep neural network, and the flying points and noise are removed while the details are preserved as much as possible; the network uses the inverse depth image as one of the input data, and the value of a point in the inverse depth image is actually stored as the reciprocal of the depth of the point; the network includes an ImageToImage network with an Hourglass network structure, and the input channel also includes an aligned RGB color image to utilize the prior knowledge in the RGB color image; 4) Point cloud unified non-rigid registration A non-rigid registration method based on dense point cloud is adopted, and the specific implementation is as follows, Firstly, assume that a part of the reconstructed scene point cloud is Vtgt, and the newly added point cloud is Vsrc, then Vsrc can be transformed to a similar degree as Vtgt through non-rigid transformation; a triangle is quickly established by three adjacent points in the point cloud, and the normal vector of a point in the triangle and the direction from the point to the other two points constitute a local space; After determining the initial adjacent points through the nearest neighbor tracking algorithm, the two meshes are pulled closer, and the optimization variables are all vertices, without extracting key control points for sparse DQB control, and without interpolation operation; 5) Surface reconstruction The surface is recovered from the point cloud using methods including but not limited to Poisson reconstruction and marching cube; 6) Geometric post-processing Including removing duplicate flying points, filling holes, removing non-manifold surfaces and processing flying point surfaces.

2. The RGBD input and flexible registration based digital dentistry system according to claim 1, wherein: The hole filling process is as follows, Firstly, find the most edge region of the geometric body according to the half-edge structure, find N edges, sort the edges according to the number of vertices, and consider the smaller edges as holes except the largest edge, which needs to be triangulated; then some points are added according to the area uniformity, and then the triangulation process including but not limited to the Delaunay method is performed, and then smoothing is performed after triangulation; The duplicate flying point removal process is as follows, Delete uncited vertex and repeated vertex, delete repeated triangle, separate the mesh according to the connected region, obtain multiple different blocks, then sort according to the surface area or the number of vertexes, find the upper and lower two largest connected regions, and delete all the small area connected regions. The non-manifold surface removal process is as follows, First, according to the degree of repetition of the edge, if the degree of repetition is greater than 2, it is considered to be a non-manifold region, then the edges with the degree of repetition greater than 2 are selected and deleted from the triangle mesh, and then all the connected regions are recalculated, the smaller regions are deleted according to the connected region, and the deleted edges are connected again, thus completing the non-manifold processing.

3. The RGBD input and flexible registration based dental digitization system of claim 2, wherein: In step 4), an iterative optimization process is also included, and the three important penalty terms required in the iterative optimization process are as follows, 1. Affine transformation smoothness penalty of the source triangle adjacent point This penalty ensures that the transformation of adjacent points is relatively smooth: E S (v1...v n )=sum(||Ti −Tj||) where v1...v n are the vertex positions in the source point cloud, T is a variable, Ti and Tj are the affine transformations of the local coordinate system formed by the triangles in the source point cloud after the variable with respect to the original coordinate system, and i and j are the numbers of the adjacent triangle sets; 2. Affine transformation minimization penalty of the source triangle In order to ensure that the source triangle changes state through the least change, the following method is adopted: E I (v1...v n )=sum(||Ti −I||) Where I represents a 3*3 unit matrix; 3. Minimization penalty of the position of the source vertex and the position of the target vertex This penalty term is denoted as: E c (v1...v n )=sum(||v i −c i ||) where v i represents the position of a vertex in the source point cloud, c i represents the position of a point in the target point cloud that is closest in spatial position and consistent in normal vector to v i represents the position of a point in the target point cloud that is closest in spatial position and consistent in normal vector to v

Images