Three-dimensional dynamic tracking method, device, electronic device and storage medium

Abstract

The present disclosure relates to a three-dimensional dynamic tracking method, device, electronic device and storage medium. The tracking method includes: acquiring facial smile data, three-dimensional tooth data and facial template data; registering the acquired facial smile data, three-dimensional tooth data and facial template data in a preset coordinate system to obtain initial point cloud data; fusing the initial depth point cloud data to obtain an isosurface, and obtaining target depth point cloud data corresponding to the isosurface; back-projecting the target depth point cloud data to initial texture data, and obtaining target texture data corresponding to the back-projected facial smile data; optimizing the initial facial template point cloud data according to the target depth point cloud data to obtain target facial template point cloud data, thereby realizing tracking of facial expression changes; fitting the target facial template point cloud data and the tooth area identified by the initial texture data with the initial three-dimensional tooth data in a fitting area, thereby improving the accuracy of three-dimensional tooth tracking.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional deformation technology, and in particular to a three-dimensional dynamic tracking method, device, electronic device and storage medium. Background Art

[0002] The rapid development of data acquisition and 3D reconstruction technologies has made high-precision digitization of physical models possible. In the field of dental digitization, scanning and 3D reconstruction can produce 3D digital models of the dental jaw, enabling subsequent processing and customized production of dental products. 3D reconstruction technology allows users to visualize the effects of their corrected teeth before orthodontic treatment using a 3D dental model.

[0003] Existing techniques typically use a 2D face + 2D teeth approach, where a tooth model is manually adjusted based on a 2D image to obtain the corrected teeth. Alternatively, a 2D face + 3D teeth approach involves projecting a 3D tooth model onto the corresponding tooth area in a 2D image. Since 2D images typically show less mandibular exposure, and mandibular tracking is only possible when the teeth are fully exposed, tracking is often poor. Furthermore, 2D tooth tracking lacks facial scale information, resulting in limited visual quality. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a three-dimensional dynamic tracking method, device, electronic device and storage medium to improve the accuracy of three-dimensional tooth tracking.

[0005] In a first aspect, an embodiment of the present disclosure provides a three-dimensional dynamic tracking method, comprising:

[0006] Acquire facial smile data, three-dimensional tooth data, and facial template data, wherein the facial smile data includes texture data and depth data;

[0007] Registering the acquired facial smile data, 3D tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D tooth point cloud data, and initial facial template point cloud data;

[0008] Fusing the initial depth point cloud data to obtain an isosurface, and obtaining target depth point cloud data corresponding to the isosurface;

[0009] Back-projecting the target depth point cloud data to the initial texture data, and obtaining target texture data corresponding to the back-projected facial smile data;

[0010] Optimizing the initial face template point cloud data according to the target depth point cloud data to obtain target face template point cloud data, thereby tracking facial expression changes;

[0011] The target face template point cloud data, the tooth area identified by the initial texture data, and the initial three-dimensional tooth data are fitted in a fitting area.

[0012] Optionally, registering the acquired facial smile data, three-dimensional tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data includes:

[0013] Obtain source point cloud data of facial smile data, 3D tooth data, and facial template data;

[0014] Determine the rotation parameters, translation parameters and error threshold in the iterative closest point algorithm;

[0015] The source point cloud data is registered to the preset coordinate system according to the rotation parameters, translation parameters and error threshold to obtain the initial point cloud data.

[0016] Optionally, fusing the initial depth point cloud data to obtain an isosurface, and determining target depth point cloud data according to the isosurface includes:

[0017] Using the TSDF algorithm to obtain the truncated signed distance function value and weight value corresponding to the initial depth point cloud data;

[0018] According to the truncated signed distance function value and the weight value, a bubble animation simulation algorithm is used to obtain an isosurface corresponding to the initial depth point cloud data;

[0019] Target depth point cloud data is determined based on the point cloud data corresponding to the isosurface.

[0020] Optionally, the adopting of the TSDF algorithm to obtain the truncated signed distance function value and the weight value corresponding to the initial depth point cloud data includes:

[0021] Establish a rectangular bounding box corresponding to the initial point cloud data;

[0022] Voxelize the rectangular bounding box corresponding to the initial point cloud data;

[0023] The TSDF algorithm is used to calculate the TSDF value and weight value corresponding to the initial depth point cloud data in each voxelization.

[0024] Optionally, the step of obtaining an isosurface corresponding to the initial depth point cloud data using a bubble animation simulation algorithm according to the truncated signed distance function value and the weight value includes:

[0025] Preprocess the truncated signed distance function value and weight value and read them into a specific array;

[0026] Obtain the position information of the unit body corresponding to different voxelizations;

[0027] Determine a state table corresponding to each unit body according to the position information of each unit body;

[0028] According to the state table of each unit body, the edges of each unit body intersecting with the isosurface are obtained, and the position coordinates of the intersection of each unit body and the isosurface are calculated by linear interpolation method;

[0029] Using the central difference method, the normal vector of each vertex in each unit body is obtained, and the normal vector of each vertex of the triangle face is obtained by using the linear interpolation method;

[0030] The isosurface is determined according to the position coordinates of the intersection of each unit body and the isosurface and the vertex normal vector.

[0031] Optionally, back-projecting the target depth point cloud data to initial texture data and obtaining target texture data corresponding to the back-projected dynamic facial data includes:

[0032]

[0033] Where: (p) represents the weight value of point p projected to the i-th camera perspective, and represents the pixel value of point p projected to the i-th camera perspective.

[0034] Optionally, fitting the target face template point cloud data and the tooth region identified by the initial texture data with the initial three-dimensional tooth data in a fitting region includes:

[0035] The target face template point cloud data and the tooth area identified by the initial texture data are fitted with the initial three-dimensional tooth data in the fitting area using the Gauss-Newton method.

[0036] In a second aspect, the present disclosure further provides a three-dimensional dynamic tracking device, comprising:

[0037] A data acquisition module, configured to acquire facial smile data, three-dimensional tooth data, and facial template data, wherein the facial smile data includes texture data and depth data;

[0038] An initial cloud point data acquisition module is used to register the acquired facial smile data, 3D tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D tooth point cloud data, and initial facial template point cloud data;

[0039] A target depth point cloud data acquisition module is used to fuse the initial depth point cloud data to obtain an isosurface, and obtain target depth point cloud data corresponding to the isosurface;

[0040] A target texture data acquisition module is configured to back-project the target depth point cloud data onto the initial texture data and acquire target texture data corresponding to the back-projected facial smile data;

[0041] A face template tracking module is used to optimize the initial face template point cloud data according to the target depth point cloud data to obtain target face template point cloud data, thereby tracking facial expression changes;

[0042] The three-dimensional tooth tracking module is used to fit the target face template point cloud data and the tooth area identified by the initial texture data with the initial three-dimensional tooth data in a fitting area.

[0043] In a third aspect, an embodiment of the present disclosure further provides an electronic device, including:

[0044] one or more processors;

[0045] a storage device for storing one or more programs,

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional dynamic tracking method as described in any one of the first aspects.

[0047] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional dynamic tracking method as described in any one of the first aspects.

[0048] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art:

[0049] The three-dimensional dynamic tracking method, device, electronic device and storage medium provided by the embodiments of the present disclosure obtain initial point cloud data by registering the acquired facial smile data, three-dimensional tooth data and facial template data to preset coordinates, and then fuses the initial depth point cloud data in the acquired initial point cloud data to obtain an isosurface, and then obtains the target depth point cloud data corresponding to the isosurface, ensuring that the facial smile data has a better visual effect. The display of the facial texture is achieved by back-projecting the acquired target depth point cloud data to the initial texture data. The initial facial template point cloud data is optimized according to the target depth point cloud data to obtain the target facial template point cloud data, and the facial expression change tracking is achieved. Finally, the target facial template point cloud data is fitted with the tooth area identified by the initial texture data and displayed, thereby achieving tooth tracking and improving the accuracy of the tracking result. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 is a flow chart of a three-dimensional dynamic tracking method provided by an embodiment of the present disclosure;

[0053] Figure 2 is a flow chart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure;

[0054] Figure 3 This is a flow chart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure;

[0055] Figure 4 This is a flow chart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure;

[0056] Figure 5 is a structural diagram of a three-dimensional dynamic tracking device provided by an embodiment of the present disclosure;

[0057] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0060] Figure 1The figure is a flow chart illustrating a three-dimensional dynamic tracking method provided by an embodiment of the present disclosure. This embodiment enables real-time tracking of a 3D face and 3D teeth, as well as simultaneous tracking of the upper and lower jaws. The method of this embodiment can be executed by a three-dimensional dynamic tracking device, which can be implemented using hardware or software and can be configured in an electronic device. This device can implement the three-dimensional dynamic tracking method described in any embodiment of this application.

[0061] The 3D dynamic tracking method provided in the embodiments of the present invention may be performed by a 3D dynamic tracking device. The 3D dynamic tracking device may be a terminal device such as a mobile phone, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), smart watch, smart bracelet, or other terminal devices. The embodiments of the present invention do not limit the type of terminal device.

[0062] The prior art generally adopts 2D face + 2D teeth, that is, manually adjusting the tooth model according to the captured 2D picture to obtain the corrected teeth, or 2D face + 3D teeth, that is, projecting the 3D tooth model onto the corresponding tooth area on the 2D picture. Since the mandible is generally less exposed in 2D pictures, and the mandible can only be tracked when the posture with more teeth exposed is maintained, the tracking effect is poor. In addition, there is no facial scale information when using 2D for tooth tracking, and the visual effect is average. In order to ensure the accuracy of the 3D tooth model after being projected onto the corresponding tooth area on the picture, the embodiment of the present disclosure provides a three-dimensional dynamic tracking method.

[0063] like Figure 1 As shown, the method specifically includes the following:

[0064] S110 , acquiring facial smile data, three-dimensional tooth data, and facial template data, wherein the facial smile data includes texture data and depth data.

[0065] Specifically, the acquired facial smile data includes texture data and depth data, wherein the texture data is an RGB image, which is obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other to obtain various colors. RGB represents the colors of the three channels of red, green, and blue. The depth data is a depth image (DepthMap), which is an image or image channel in 3D computer graphics that contains information related to the distance from the surface of the scene object of the viewpoint. Among them, the depth image is similar to a grayscale image, except that each of its pixel values ​​is the actual distance between the sensor and the object. Usually, the RGB image and the depth image are aligned, so there is a one-to-one correspondence between the pixels. The three-dimensional dental data is data scanned by an intraoral scanner (it can also be designed data), and the facial template data is facial template data in a template library constructed using a bilinear model (Bilnear model).

[0066] S120: registering the acquired facial smile data, three-dimensional tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data.

[0067] Specifically, the initial point cloud data includes initial depth point cloud data, initial texture data, initial three-dimensional tooth point cloud data and initial face template point cloud data.

[0068] 3D dental data is acquired through scanning, facial smile data is captured using Kinect, and facial template data is obtained from a facial template database. Therefore, facial smile data, 3D dental data, and facial template data are point cloud data in different coordinate systems. To achieve dynamic tracking of 3D dental data, it is generally necessary to register the acquired facial smile data, 3D dental data, and facial template data to a preset coordinate system to obtain initial point cloud data. This means that the acquired initial point cloud data is in the same coordinate system.

[0069] S130 , fusing the initial depth point cloud data to obtain an isosurface, and obtaining target depth point cloud data corresponding to the isosurface.

[0070] By fusing the initial depth point cloud data to obtain an isosurface, that is, obtaining a smooth isosurface from the initial depth point cloud data, better visual effects are ensured. Specifically, in the process of fusing the initial depth point cloud data to obtain an isosurface, the TSDF algorithm is first used to obtain the truncated signed distance function value and weight value corresponding to the initial point cloud data. Based on the truncated signed distance function value corresponding to the obtained initial point cloud data, the bubble animation simulation algorithm is used to obtain the isosurface corresponding to the initial depth point cloud data, and then the target depth point cloud data is determined based on the isosurface.

[0071] S140 , back-projecting the target depth point cloud data to the initial texture data, and obtaining target texture data corresponding to the back-projected facial smile data.

[0072] After obtaining the target depth point cloud data, the target depth point cloud data is back-projected onto the initial texture data to obtain the target texture data corresponding to the back-projected facial smile point cloud data. That is, the target texture data of the target point cloud data at different perspectives is obtained to ensure that the obtained facial texture data is more realistic.

[0073] Specifically, the target texture data I(p) corresponding to each point cloud of the acquired target texture data is:

[0074]

[0075] Where: w i (p) represents the weight value of the projection of point p to the i-th camera perspective, I i (p) represents the pixel value of point p projected to the perspective of the i-th camera.

[0076] That is, the target texture data of point p in the initial texture data is related to the pixel value and weight value of point p projected to the i-th camera perspective. In addition, after obtaining the weight value of point p under each camera perspective, the obtained weight value needs to be normalized.

[0077] S150 , optimizing the initial face template point cloud data according to the target depth point cloud data to obtain target face template point cloud data, thereby tracking facial expression changes.

[0078] Specifically, the face template data adopts the face template data in the template database constructed by the bilinear model (Bilinear Model), wherein the face template data consists of 26317 points, 52 expression parameters and 938 individual face parameters. The face template tracking mainly optimizes the expression parameters in the template database constructed by the bilinear model (Bilinear Model) according to the target depth point cloud data determined in step S130, and then obtains the target face template point cloud data, thereby realizing the tracking of facial expression changes.

[0079] S160 , fitting the target face template point cloud data, the tooth region identified by the initial texture data, and the initial three-dimensional tooth data in a fitting region.

[0080] Since the acquired facial smile data changes dynamically, the position of the tooth area in the current frame and the tooth area in the next frame changes in real time. In order to ensure real-time tracking of the tooth area and the facial template data in the current frame. For example, if it is necessary to track both the upper and lower teeth, it is first necessary to obtain the position of the feature points in the target facial template point cloud data in the previous and next frames, and calculate the distance from the feature point of the previous frame to the plane where the feature point of the current frame is located. For example, when tracking the upper jaw of the teeth, the feature point is selected as the nose, and the position coordinates of the nose feature point in the current frame are obtained. When the facial smile changes dynamically, the position coordinates of the nose feature point in the next frame after the change are obtained. According to the transformation of the position coordinates of the nose feature point in different frames, the maxillary line in the tooth area identified by the initial texture data is transformed in coordinate position to achieve real-time tracking of the upper jaw. When implementing tooth and mandibular tracking, the chin is selected as the feature point. By obtaining the position coordinates of the chin feature point in the current frame, when the face smile changes dynamically, the position coordinates of the chin feature point in the next frame after the change are obtained. According to the transformation of the position coordinates of the chin feature point in different frames, the coordinate position of the mandibular line in the tooth area identified by the initial texture data is transformed to achieve real-time tracking of the mandible, and then the obtained three-dimensional tooth data is displayed in the fitting area.

[0081] The three-dimensional dynamic tracking method provided by the embodiment of the present disclosure obtains initial point cloud data by registering the acquired facial smile data, three-dimensional tooth data, and facial template data under preset coordinates, and then obtains the isosurface by fusing the initial depth point cloud data in the acquired initial point cloud data, and then obtains the target depth point cloud data corresponding to the isosurface, thereby ensuring that the facial smile data has a better visual effect. The display of the facial texture is achieved by back-projecting the acquired target depth point cloud data to the initial texture data. The target facial template point cloud data is obtained by optimizing the initial facial template point cloud data according to the target depth point cloud data, thereby achieving the tracking of facial expression changes. Finally, the target facial template point cloud data is fitted with the tooth area identified by the initial texture data and displayed, thereby achieving tooth tracking and improving the accuracy of the tracking results.

[0082] Optional, Figure 2 is a flow chart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure, such as Figure 2 As shown, one possible implementation of step S120 is as follows, including:

[0083] S121. Acquire source point cloud data of facial smile data, three-dimensional tooth data, and facial template data.

[0084] After obtaining the facial smile data, three-dimensional tooth data and facial template data, the source point cloud data corresponding to the facial smile data, the source point cloud data corresponding to the three-dimensional tooth data and the source point cloud data corresponding to the facial template data are obtained. The source point clouds corresponding to different data correspond to different point cloud collections. For example, the source point cloud data X corresponding to the facial smile data is composed of the point cloud collection xi, the source point cloud data Y corresponding to the three-dimensional tooth data is composed of the point cloud collection yi, and the source point cloud data Z corresponding to the facial template data is composed of the point cloud collection zi.

[0085] S122: Determine the rotation parameter, translation parameter, and error threshold in the iterative closest point algorithm.

[0086] After determining the source point cloud data corresponding to different data, find the corresponding point cloud set in the source point cloud, and register the source point cloud data to the preset coordinate system according to the rotation parameters, translation parameters and error threshold to obtain the initial point cloud data.

[0087] Specifically, the purpose of the Iterative Closest Point (ICP) algorithm is to find the rotation parameter R and translation parameter T between the point cloud data to be registered and the reference point cloud data so that the two point data meet the optimal match that satisfies the error threshold.

[0088] Assuming that the source point cloud data X1 corresponds to the facial smile data, and the target point cloud data X2 corresponds to the source point cloud data X1, the registration steps of the ICP method are as follows:

[0089] Calculate the corresponding nearest point of each point in X2 in the point set X1; obtain the rigid body transformation that minimizes the average distance between the above corresponding points, and obtain the translation parameter T and rotation parameter R; use the translation and rotation parameters obtained in the previous step on X2 to obtain a new transformed point set. If the average distance between the new transformed point set and the reference point set is less than the error threshold, stop the iterative calculation; otherwise, continue the iteration with the new transformed point set as the new X2 until the objective function is met.

[0090] Finding the closest point pair: Calculating corresponding points is the most time-consuming step in the registration process. Finding the closest point speeds up the search using a k-d tree. The k-d tree method establishes topological relationships between points based on the coordinate axis partitioning of a binary tree. The k-d tree construction follows the binary tree principle. First, a partitioning line is found along the X-axis. This involves calculating the average x-value of all points and dividing the space into two parts using the x-value of the point closest to this average. Then, within each of these subspaces, a partitioning line is found along the Y-axis, splitting each subspace into two parts. The partitioned subspaces are then partitioned along the X-axis, and so on, until there is only one point within the partitioned area. This partitioning process corresponds to a binary tree, with each node corresponding to a partitioning line, and each leaf node corresponding to a point. This establishes the topological relationship between the points.

[0091] S123 , registering the source point cloud data to a preset coordinate system according to the rotation parameter, the translation parameter, and the error threshold to obtain initial point cloud data.

[0092] If the source point cloud data X1 corresponding to the facial smile data includes point cloud xi, then the point cloud xi is registered in the preset coordinate system after the rotation parameter, translation parameter and error threshold to obtain the initial point cloud xi'=Rxi+T.

[0093] The three-dimensional dynamic tracking method provided by the embodiment of the present disclosure transfers facial smile data, three-dimensional tooth data and facial template data located in different coordinate systems to a preset coordinate system through a rigid registration algorithm, thereby obtaining depth point cloud data, initial texture data, initial three-dimensional tooth point cloud data and initial facial template point cloud data located in the preset coordinate system.

[0094] Optional, Figure 3 This is a flow chart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure, such as Figure 3 As shown, one possible implementation of step S130 is as follows, including:

[0095] S131. Use the TSDF algorithm to obtain the truncated signed distance function value and weight value corresponding to the initial depth point cloud data.

[0096] Specifically, the TSDF algorithm is used to obtain the truncated signed distance function value and weight value corresponding to the initial depth point cloud data, including:

[0097] Create a rectangular bounding box corresponding to the initial point cloud data.

[0098] Voxelize the rectangular bounding box corresponding to the initial point cloud data.

[0099] The TSDF algorithm is used to calculate the TSDF value and weight value corresponding to the initial depth point cloud data in each voxelization.

[0100] Exemplarily, obtaining the truncated signed distance function value and weight value corresponding to the initial depth point cloud data may include: first establishing a rectangular outer bounding box of the initial point cloud data as the cubic space where each initial point cloud data is located, and then voxelizing the rectangular bounding box corresponding to the initial point cloud data, and finally using the truncated signed distance function (TSDF) to calculate the TSDF value and weight value corresponding to the initial depth point cloud data in each voxelization. Specifically, we traverse all voxels, taking a voxel at a 3D position p in the world coordinate system as an example: Using the camera pose matrix of the depth data, we find the mapping point v of point p in the world coordinate system in the camera coordinate system. Using the camera intrinsic parameter matrix, we back-project v to find the corresponding pixel x in the depth image. The depth value of pixel x is value(x), and the distance from point v to the camera origin is distance(v). Then, the SDF value corresponding to point p is sdf(p) = value(x) - distance(v), and TSDF(p) = sdf(p) / |u|, where |u| is the corresponding preset threshold, which depends on the number and size of voxels. If TSDF(p) > 1, the TSDF value corresponding to point p is 1. If TSDF(p) < -1, the TSDF value corresponding to point p is -1. The weight w(p) is calculated as: w(p) = cos(θ), where θ is the angle between the projected ray and the surface normal.

[0101] S132. Acquire an isosurface corresponding to the initial depth point cloud data using a bubble animation simulation algorithm according to the truncated signed distance function value and the weight value.

[0102] Specifically, the isosurface corresponding to the initial depth point cloud data is obtained by using a marching cube algorithm according to the truncated signed distance function value and the weight value, including:

[0103] The truncated signed distance function values ​​and weight values ​​are preprocessed and read into a specific array.

[0104] Get the position information of the unit body corresponding to different voxelizations.

[0105] The state table corresponding to each unit body is determined according to the position information of each unit body.

[0106] According to the state table of each unit body, the edges of each unit body intersecting with the isosurface are obtained, and the position coordinates of the intersection of each unit body and the isosurface are calculated using a linear interpolation method.

[0107] The central difference method is used to obtain the normal vector of each vertex in each unit body, and the linear interpolation method is used to obtain the normal vector of each vertex of the triangle face.

[0108] The isosurface is determined according to the position coordinates of the intersection of each unit body and the isosurface and the vertex normal vector.

[0109] The key idea behind the bubble animation simulation algorithm is to approximate an isosurface through linear interpolation in a 3D discrete data field. Specifically, each grid cell in the 3D discrete data field is considered a voxel, and each vertex of the voxel has a corresponding scalar value. If the value at a voxel vertex is greater than or equal to the isosurface value, the vertex is defined as being outside the isosurface and marked as "0." If the value at a voxel vertex is less than the isosurface value, the vertex is defined as being inside the isosurface and marked as "1." Since each voxel cell has eight vertices, there are 2^8 = 256 possible scenarios.

[0110] The indexing rule for vertices and edges within each voxel unit is as follows: if the value of a vertex in a voxel is less than the isosurface value, the lines intersecting the voxel edges connecting the vertex form a triangle. The specific positions of the triangle vertices are calculated using linear interpolation. The central difference method is then used to obtain the normal vectors for each vertex in each unit cell, and linear interpolation is used to obtain the normal vectors for each vertex in the triangle. Finally, the isosurface is determined based on the position coordinates of the intersection of each unit cell with the isosurface and the vertex normal vectors.

[0111] S133 : Determine target depth point cloud data based on the point cloud data corresponding to the isosurface.

[0112] The target depth point cloud data is determined by the point cloud data corresponding to the obtained isosurface, that is, the depth data corresponding to the facial smile data of the current frame is determined, so as to ensure the smoothness of the depth data corresponding to the obtained three-dimensional facial smile data.

[0113] The three-dimensional dynamic tracking method provided by the embodiment of the present disclosure obtains the truncated signed distance function value and weight value corresponding to the initial depth point cloud data through the TSDF algorithm, and then adopts the bubble animation simulation algorithm to obtain the isosurface corresponding to the initial depth point cloud data, and then determines the corresponding target depth point cloud data based on the isosurface, so as to ensure that the facial smile data has a better visual effect.

[0114] Optional, Figure 4 This is a flow chart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure, such as Figure 4 As shown, one possible implementation of step S160 is as follows, including:

[0115] S161, fitting the target face template point cloud data and the tooth area identified by the initial texture data with the initial three-dimensional tooth data in the fitting area using the Gauss-Newton method.

[0116] Since the acquired facial smile data changes dynamically, the position of the tooth area in the current frame and the tooth area in the next frame changes in real time. In order to ensure real-time tracking of the tooth area and the facial template data in the current frame. For example, if it is necessary to track both the upper and lower teeth, it is first necessary to obtain the position of the feature points in the target facial template point cloud data in the previous and next frames, and calculate the distance from the feature point of the previous frame to the plane where the feature point of the current frame is located. For example, when tracking the upper jaw of the teeth, the feature point is selected as the nose, and the position coordinates of the nose feature point in the current frame are obtained. When the facial smile changes dynamically, the position coordinates of the nose feature point in the next frame after the change are obtained. According to the transformation of the position coordinates of the nose feature point in different frames, the maxillary line in the tooth area identified by the initial texture data is transformed to achieve real-time tracking of the upper jaw. When implementing tooth and mandibular tracking, the chin is selected as the feature point. By obtaining the position coordinates of the chin feature point in the current frame, when the face smile changes dynamically, the position coordinates of the chin feature point in the next frame after the change are obtained. According to the transformation of the position coordinates of the chin feature point in different frames, the coordinate position of the mandibular line in the tooth area identified by the initial texture data is transformed to achieve real-time tracking of the mandible, and then the obtained three-dimensional tooth data is displayed in the fitting area.

[0117] Specifically, first calculate the corresponding points on the nose of the target face template in the previous and next frames, and calculate the distance from the point in the previous frame to the plane where the corresponding point in the current frame is located.

[0118]

[0119] Among them, vd represents the coordinates of the point in the current frame, nd represents the normal phase of the point in the current frame, and vs represents the coordinates of the corresponding point in the previous frame.

[0120] Then, based on the upper jaw edge of the tooth region identified by the initial texture data, the initial 3D tooth data is back-projected onto the initial texture data to fit the upper tooth edge. And based on the lower jaw edge of the tooth region identified by the initial texture data, the initial 3D tooth data is back-projected onto the initial texture data to fit the lower tooth edge.

[0121] Tracking of the maxilla: There are two energy terms:

[0122] E=Enose+Eedgematch

[0123] Mandibular tracking: There are two energy terms:

[0124] E=Echin+Eedgematch Among them, the above energy is toward the center,

[0125]

[0126] Among them, ud represents the pixel coordinates of the tooth edge detected in the current frame, and vd represents the normal vector of the tooth edge pixel detected in the current frame.

[0127] x C =V C ·x / V C ·z

[0128] y C =V C ·y / VC·z

[0129] V C =KRTV w

[0130] Among them, Vw is the coordinate position in the world coordinate system, Vc is the coordinate in the camera coordinate system, K and RT are the internal and external parameters of the camera respectively.

[0131] Optionally, based on the above embodiment, Figure 5 A three-dimensional dynamic tracking device is provided in an embodiment of the present disclosure. Figure 5 The three-dimensional dynamic tracking device includes:

[0132] A data acquisition module 510 is configured to acquire facial smile data, three-dimensional tooth data, and facial template data, wherein the facial smile data includes texture data and depth data;

[0133] An initial cloud point data acquisition module 520 is configured to register the acquired facial smile data, 3D tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D tooth point cloud data, and initial facial template point cloud data;

[0134] The target depth point cloud data acquisition module 530 is used to fuse the initial depth point cloud data to obtain an isosurface, and obtain the target depth point cloud data corresponding to the isosurface;

[0135] The target texture data acquisition module 540 back-projects the target depth point cloud data to the initial texture data, and obtains the target texture data corresponding to the back-projected facial smile data;

[0136] The face template tracking module 550 is used to optimize the initial face template point cloud data according to the target depth point cloud data to obtain the target face template point cloud data, so as to realize the tracking of facial expression changes;

[0137] The 3D teeth tracking module 560 is used to fit the target face template point cloud data and the tooth area identified by the initial texture data with the initial 3D tooth data in the fitting area.

[0138] The three-dimensional dynamic tracking device provided by the embodiment of the present disclosure has the following features: the initial cloud point data acquisition module registers the facial smile data, three-dimensional tooth data and facial template data acquired by the data acquisition module to preset coordinates to obtain initial point cloud data; the target depth point cloud data acquisition module fuses the initial depth point cloud data acquired in the initial point cloud data to obtain an isosurface, and then acquires the target depth point cloud data corresponding to the isosurface, ensuring that the facial smile data has a better visual effect. The target texture data acquisition module realizes the display of facial texture by back-projecting the acquired target depth point cloud data to the initial texture data. The facial template tracking module optimizes the initial facial template point cloud data according to the target depth point cloud data to acquire the target facial template point cloud data, thereby realizing the tracking of facial expression changes. Finally, the three-dimensional tooth tracking module realizes tooth tracking by fitting the target facial template point cloud data with the tooth area identified by the initial texture data and displaying the three-dimensional tooth data in the fitting area, thereby improving the accuracy of the tracking results.

[0139] Figure 6 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 6 As shown, the electronic device includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of processors 610 in the electronic device can be one or more. Figure 6 6 takes a processor 610 as an example; the processor 610, memory 620, input device 630 and output device 640 in the electronic device can be connected through a bus or other means, and 6 takes the bus connection as an example.

[0140] Memory 620, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the three-dimensional dynamic tracking method in the embodiments of the present invention. Processor 610 executes the software programs, instructions, and modules stored in memory 620 to perform various functional applications and data processing of the electronic device, thereby implementing the three-dimensional dynamic tracking method provided in the embodiments of the present invention.

[0141] The memory 620 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 620 may further include memory remotely located relative to the processor 610, and these remote memories may be connected to the computer device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The input device 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 640 may include a display device such as a display screen.

[0143] The embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to implement the three-dimensional dynamic tracking method provided by the embodiment of the present invention.

[0144] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the three-dimensional dynamic tracking method provided by any embodiment of the present invention.

[0145] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0146] It is worth noting that in the embodiment of the above-mentioned tracking device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0147] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0148] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional dynamic tracking method, characterized in that: include: Acquire facial smile data, three-dimensional tooth data, and facial template data, wherein the facial smile data includes texture data and depth data; Registering the acquired facial smile data, 3D tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D tooth point cloud data, and initial facial template point cloud data; Fusing the initial depth point cloud data to obtain an isosurface, and obtaining target depth point cloud data corresponding to the isosurface; Back-projecting the target depth point cloud data onto the initial texture data, and obtaining target texture data corresponding to the back-projected facial smile data; the target texture data is target texture data of the target point cloud data at different viewing angles; Optimizing the initial face template point cloud data according to the target depth point cloud data to obtain target face template point cloud data, thereby tracking facial expression changes; The target face template point cloud data, the tooth area identified by the initial texture data and the initial three-dimensional tooth point cloud data are fitted in a fitting area to achieve tooth tracking.

2. The method according to claim 1, characterized in that Registering the acquired facial smile data, 3D tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data includes: Obtain source point cloud data of facial smile data, 3D tooth data, and facial template data; Determine the rotation parameters, translation parameters and error threshold in the iterative closest point algorithm; The source point cloud data is registered to the preset coordinate system according to the rotation parameters, translation parameters and error threshold to obtain the initial point cloud data.

3. The method according to claim 1, characterized in that The fusing the initial depth point cloud data to obtain an isosurface, and determining target depth point cloud data according to the isosurface, comprises: Using the TSDF algorithm to obtain the truncated signed distance function value and weight value corresponding to the initial depth point cloud data; According to the truncated signed distance function value and the weight value, a bubble animation simulation algorithm is used to obtain an isosurface corresponding to the initial depth point cloud data; Target depth point cloud data is determined based on the point cloud data corresponding to the isosurface.

4. The method according to claim 3, characterized in that The TSDF algorithm is used to obtain the truncated signed distance function value and the weight value corresponding to the initial depth point cloud data, including: Establish a rectangular bounding box corresponding to the initial point cloud data; Voxelize the rectangular bounding box corresponding to the initial point cloud data; The TSDF algorithm is used to calculate the TSDF value and weight value corresponding to the initial depth point cloud data in each voxelization.

5. The method according to claim 4, characterized in that The step of obtaining an isosurface corresponding to the initial depth point cloud data using a bubble animation simulation algorithm according to the truncated signed distance function value and the weight value includes: Preprocess the truncated signed distance function value and weight value and read them into a specific array; Obtain the position information of the unit body corresponding to different voxelizations; Determine a state table corresponding to each unit body according to the position information of each unit body; According to the state table of each unit body, the edges of each unit body intersecting with the isosurface are obtained, and the position coordinates of the intersection of each unit body and the isosurface are calculated by linear interpolation method; Using the central difference method, the normal vector of each vertex in each unit body is obtained, and the normal vector of each vertex of the triangle face is obtained by using the linear interpolation method; The isosurface is determined according to the position coordinates of the intersection of each unit body and the isosurface and the vertex normal vector.

6. The method according to claim 3, characterized in that The back-projecting the target depth point cloud data to the initial texture data and obtaining target texture data corresponding to the back-projected face dynamic data includes: in: (p) represents the weight value of the projection of point p to the i-th camera perspective, Represents the pixel value of point p projected to the i-th camera perspective.

7. The method according to claim 1, characterized in that The step of fitting the target face template point cloud data and the tooth region identified by the initial texture data with the initial three-dimensional tooth point cloud data in a fitting region includes: The target face template point cloud data and the tooth area identified by the initial texture data are fitted in the fitting area with the initial three-dimensional tooth point cloud data using the Gauss-Newton method.

8. A three-dimensional dynamic tracking device, characterized in that: include: A data acquisition module, configured to acquire facial smile data, three-dimensional tooth data, and facial template data, wherein the facial smile data includes texture data and depth data; An initial cloud point data acquisition module is used to register the acquired facial smile data, 3D tooth data, and facial template data into a preset coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D tooth point cloud data, and initial facial template point cloud data; A target depth point cloud data acquisition module is used to fuse the initial depth point cloud data to obtain an isosurface, and obtain target depth point cloud data corresponding to the isosurface; a target texture data acquisition module, which back-projects the target depth point cloud data onto the initial texture data and acquires target texture data corresponding to the back-projected facial smile data; the target texture data is target texture data of the target point cloud data at different viewing angles; A face template tracking module is used to optimize the initial face template point cloud data according to the target depth point cloud data to obtain target face template point cloud data, thereby tracking facial expression changes; The three-dimensional tooth tracking module is used to fit the target face template point cloud data and the tooth area identified by the initial texture data with the initial three-dimensional tooth point cloud data in a fitting area to achieve tooth tracking.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional dynamic tracking method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the three-dimensional dynamic tracking method according to any one of claims 1 to 7 is implemented.

Images