Three-dimensional dynamic tracking method, device, electronic device and storage medium

Abstract

The present disclosure relates to the field of image fusion technology, and provides a three-dimensional dynamic tracking method, device, electronic device and storage medium, in which the tracking method includes: acquiring human smile data, three-dimensional teeth data and face template data; registering the acquired human smile data, three-dimensional teeth data and face template data into a predetermined coordinate system to acquire initial point cloud data; fusing the point cloud data of the initial depth to acquire an isosurface, and acquiring point cloud data of a target depth corresponding to the isosurface; back-projecting the point cloud data of the target depth onto initial texture data, and acquiring target texture data corresponding to the human smile data after the back-projection; optimizing the point cloud data of the initial face template according to the point cloud data of the target depth to acquire point cloud data of a target face template, thereby realizing tracking of facial expression changes; and fitting a dental region identified by the point cloud data of the target face template and the initial texture data to the initial three-dimensional teeth data in a fitting area, thereby improving the accuracy of three-dimensional tooth tracking.

Description

[Technical field]

[0001] This disclosure claims priority to a Chinese patent application, application number 202110784787.5, filed with the China National Intellectual Property Office on July 12, 2021, and entitled "Three-dimensional dynamic tracking method, apparatus, electronic device and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the technical field of three-dimensional deformation, and in particular to a three-dimensional dynamic tracking method, device, electronic device and storage medium. [Background technology]

[0003] At present, the rapid development of data collection technology and 3D reconstruction technology has made it possible to digitize physical models with high accuracy. In the field of dental digitization, 3D digital models of teeth and jaws can be obtained through scanning and 3D reconstruction, so as to realize the customization of subsequent processing and manufacturing of dental products. 3D reconstruction technology allows users to see the effect of their teeth correction before and after the correction on the 3D tooth model.

[0004] In the prior art, 2D face + 2D teeth are generally used, that is, based on the captured 2D image, the teeth model is manually adjusted to obtain the corrected teeth, or 2D face + 3D teeth are used, that is, based on the captured 2D image, the 3D tooth mold is projected onto the area corresponding to the teeth in the image. In the 2D image, the exposure of the lower jaw is generally small, so the tracking of the lower jaw can only be achieved by maintaining a posture with a large exposure of the teeth, resulting in poor tracking effect. Furthermore, there is no scale information of the face when tracking teeth using 2D, and the visual effect is ordinary. Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, 2D face + 2D teeth is used, that is, based on the captured 2D image, the teeth model is manually adjusted to obtain the corrected teeth, or 2D face + 3D teeth is used, that is, based on the captured 2D image, the 3D tooth mold is projected onto the area corresponding to the teeth in the image. In a 2D image, the lower jaw is generally not exposed much, so the tracking effect is poor because the lower jaw can only be tracked while maintaining a posture with a large exposure of the teeth. Furthermore, there is no scale information of the face when tracking teeth using 2D, and the visual effect is ordinary. [Means for solving the problem]

[0006] According to various embodiments disclosed in the present disclosure, a three-dimensional dynamic tracking method, apparatus, electronic device, and storage medium are provided.

[0007] An embodiment of the present disclosure provides a three-dimensional dynamic tracking method, comprising: Dynamic Face Data , acquiring three-dimensional dental data and facial template data, Dynamic Face Data contains texture data and depth data, Obtained Dynamic Face Data , registering the three-dimensional tooth data and the face template data in a predetermined coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes point cloud data of an initial depth; Initial texture data, including initial 3D dental point cloud data and initial face template point cloud data; Fusing the point cloud data of the initial depth to obtain an isosurface, and obtaining point cloud data of a target depth corresponding to the isosurface; The point cloud data of the target depth is back-projected onto the initial texture data, and the back-projected point cloud data is Dynamic Face Data obtaining target texture data corresponding to Optimizing the point cloud data of the initial face template based on the point cloud data of the target depth to obtain point cloud data of a target face template, thereby realizing tracking of facial expression changes; and fitting a dental region identified by the point cloud data of the target face template and the initial texture data and the initial three-dimensional dental data in a fitting area.

[0008] As an alternative embodiment of the present disclosure, the obtained Dynamic Face Data Registering the three-dimensional tooth data and the face template data in a predetermined coordinate system to obtain initial point cloud data, Dynamic Face Data obtaining source point cloud data of 3D dental data and facial template data; determining rotation parameters, translation parameters and an error threshold in an iterative closest point algorithm; and registering the source point cloud data into a predetermined coordinate system based on the rotation parameters, the translation parameters and an error threshold to obtain initial point cloud data.

[0009] As an alternative embodiment of the present disclosure, fusing the point cloud data of the initial depth to obtain an isosurface and obtaining point cloud data of a target depth corresponding to the isosurface includes: Obtaining cutoff signed distance function values ​​and weight values ​​corresponding to the initial depth point cloud data using a TSDF algorithm; Obtaining an isosurface corresponding to the point cloud data of the initial depth using a marching cubes algorithm based on the cutoff signed distance function value and the weight value; determining point cloud data of a target depth based on point cloud data corresponding to the isosurface.

[0010] As an alternative embodiment of the present disclosure, obtaining cutoff signed distance function values ​​and weight values ​​corresponding to the initial depth point cloud data using the TSDF algorithm includes: Establishing a rectangular bounding box corresponding to the initial point cloud data; voxelizing a rectangular bounding box corresponding to the initial point cloud data; Calculating a TSDF value and a weight value corresponding to the point cloud data of the initial depth in each voxelization using a TSDF algorithm.

[0011] As an alternative embodiment of the present disclosure, the step of obtaining an isosurface corresponding to the point cloud data of the initial depth using a marching cubes algorithm based on the cutoff signed distance function value and the weight value includes: Preprocessing the blocked signed distance function values ​​and weight values ​​and loading them into a specific array; Obtaining position information of a unit body corresponding to different voxelizations; determining a state table corresponding to each of the unit bodies based on position information of each of the unit bodies; obtaining an edge of each of the unit bodies intersecting with the isosurface based on a state table of each of the unit bodies, and calculating position coordinates of an intersection between each of the unit bodies and the isosurface by a linear interpolation method; Using a central difference method to obtain the normal vector of each vertex in each of the unit bodies, and using a linear interpolation method to obtain the normal vector of each vertex of the triangular patch; determining an isosurface based on the position coordinates and vertex normal vectors of the intersections between each of the unit bodies and the isosurface.

[0012] As an alternative embodiment of the present disclosure, back projecting the point cloud data of the target depth onto the initial texture data to obtain target texture data corresponding to the dynamic data of the face after back projection includes: JPEG0007610759000001.jpg45170

[0013] As an alternative embodiment of the present disclosure, fitting the tooth region identified by the point cloud data of the target face template and the initial texture data and the initial three-dimensional tooth data in a fitting area includes: The method includes fitting the tooth region identified by the point cloud data of the target face template and the initial texture data to the initial three-dimensional tooth data in a fitting area using the Gauss-Newton method.

[0014] An embodiment of the present disclosure provides a three-dimensional dynamic tracking device, Dynamic Face Data a data acquisition module used to acquire three-dimensional dental data and facial template data, Dynamic Face Data contains texture data and depth data, Obtained Dynamic Face Data an initial point cloud data acquisition module used for registering the three-dimensional tooth data and the face template data in a predetermined coordinate system to acquire initial point cloud data, where the initial point cloud data includes initial depth point cloud data, initial texture data, initial three-dimensional tooth point cloud data and initial face template point cloud data; a target depth point cloud data acquisition module, which is used to fuse the point cloud data of the initial depth to obtain an isosurface and acquire point cloud data of a target depth corresponding to the isosurface; The point cloud data of the target depth is back-projected onto the initial texture data, and the back-projected point cloud data is Dynamic Face Data a target texture data acquisition module for acquiring target texture data corresponding to the target texture data; a face template tracking module, which is used for optimizing the point cloud data of the initial face template according to the point cloud data of the target depth to obtain point cloud data of a target face template, and realizing tracking of facial expression changes; a 3D tooth tracking module used for fitting a tooth region identified by the point cloud data of the target face template and the initial texture data to display 3D tooth data in a fitting area.

[0015] An embodiment of the present disclosure provides an electronic device, one or more processors; a storage device adapted to store one or more programs; The one or more programs, when executed by the one or more processors, cause the one or more processors to implement any one of the three-dimensional dynamic tracking methods.

[0016] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the program implementing any one of the above three-dimensional dynamic tracking methods when executed by a processor.

[0017] Other features and advantages of the present disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the present disclosure will be realized and obtained by the structure particularly pointed out in the written description, claims and drawings, and the following drawings and description set forth in detail one or more embodiments of the present disclosure.

[0018] In order to make the above objects, features and advantages of the present disclosure more clear and understandable, preferred embodiments will be described in detail below in conjunction with the accompanying drawings. [Brief description of the drawings]

[0019] In order to more clearly describe the technical solutions in the embodiments or prior art of the present disclosure, the following briefly describes the drawings used in the description of the embodiments or prior art, and it goes without saying that those skilled in the art can derive other drawings based on these drawings without giving any creative labor. [Figure 1] 1 is a flowchart of a three-dimensional dynamic tracking method provided by an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure. [Diagram 3] 1 is a flowchart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure. [Figure 4] 1 is a flowchart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure. [Diagram 5]FIG. 2 is a schematic diagram of a three-dimensional dynamic tracking device provided according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] In order to more clearly understand the above objectives, features and advantages of the present disclosure, the following further describes the solutions of the present disclosure. It should be noted that, unless contradictory, the features in the embodiments and embodiments of the present disclosure can be combined with each other.

[0021] In the following description, numerous details are set forth in order to provide a thorough understanding of the present disclosure; however, the present disclosure may be embodied in other forms than those described above, and it is understood that the embodiments in the specification are merely some embodiments of the present disclosure, and not all embodiments.

[0022] In the specification and claims of this disclosure, terms such as "first" and "second" are intended to distinguish different objects, and not to describe a particular order of the objects.

[0023] In the embodiments of the present disclosure, terms such as "exemplary" or "for example" are intended to be illustrative, demonstrative or explanatory. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be construed as preferred or advantageous over another embodiment or design. Rather, terms such as "exemplary" or "for example" are used to present related concepts in a concrete manner, and further, in the description of the embodiments of the present disclosure, the meaning of "plurality" refers to two or more than two, unless otherwise specified.

[0024] FIG. 1 is a flow chart of a three-dimensional dynamic tracking method provided by an embodiment of the present disclosure. This embodiment can realize the real-time tracking of 3D face + 3D teeth and simultaneous tracking of upper and lower jaws. The method of this embodiment can be performed by a three-dimensional dynamic tracking device, which can be realized in the form of hardware / or software and installed in an electronic device. The three-dimensional dynamic tracking method described in any embodiment of the present application can be realized.

[0025] The execution subject of the three-dimensional dynamic tracking method provided by the embodiment of the present invention may be a three-dimensional dynamic tracking device, which may be a terminal device such as a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart watch, a smart handheld, etc., or the terminal device may also be other types of terminal devices, and the embodiment of the present invention does not limit the type of the terminal device.

[0026] In the prior art, 2D face + 2D teeth are generally used, that is, based on the captured 2D image, the teeth after correction are obtained by manually adjusting the tooth model, or 2D face + 3D teeth are used, that is, based on the captured 2D image, the 3D tooth model is projected onto the area corresponding to the teeth in the image. In the 2D image, the exposure of the lower jaw is generally small, so the tracking of the lower jaw can only be achieved by maintaining a posture with a large exposure of the teeth, resulting in poor tracking effect. Furthermore, there is no scale information of the face when tracking teeth using 2D, and the visual effect is ordinary. In order to ensure the accuracy after the 3D tooth model is projected onto the area corresponding to the teeth in the image, the embodiment of the present disclosure provides a three-dimensional dynamic tracking method. As shown in FIG. 1 , the method specifically includes:

[0027] S110, obtaining human smile data, 3D teeth data and face template data, where the human smile data includes texture data and depth data. Furthermore, smile data of a person is included in the dynamic face data.

[0028] Specifically, the acquired human smile data includes texture data and depth data, where the texture data is an RGB image, and various colors are obtained by changing or superimposing three color channels, red (R), green (G), and blue (B), on each other, and RGB indicates the colors of the three channels, red, green, and blue. The depth data is a depth map, which is an image or image channel that contains information about the distance of the surface of a scene object from a viewpoint in 3D computer graphics. Here, the depth map is similar to a gray level image, but each pixel value is the actual distance between the sensor and the object. Generally, the RGB image and the depth map are aligned, so that there is a one-to-one correspondence between the pixel points. The three-dimensional tooth data is data (which may be designed data) scanned by an intraoral scanner, and the face template data is face template data in a template database constructed by a bilinear model (Bilnear model).

[0029] S120: The acquired smile data, three-dimensional tooth data, and face template data are registered in a predetermined coordinate system to acquire initial point cloud data.

[0030] Specifically, the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D teeth point cloud data and initial face template point cloud data.

[0031] The 3D tooth data is acquired by scanning, the human smile data is photographed by Kinect, and the face template data is acquired from the face template database. Therefore, the human smile data, the 3D tooth data, and the face template data are point cloud data in different coordinate systems, and in order to realize the dynamic tracking of the 3D tooth data, it is generally necessary to register the acquired human smile data, the 3D tooth data, and the face template data in a certain coordinate system to acquire the initial point cloud data, that is, the acquired initial point cloud data are located in the same coordinate system.

[0032] S130, the point cloud data of the initial depth is fused to obtain an isosurface, and point cloud data of the target depth corresponding to the isosurface is obtained.

[0033] The point cloud data of the initial depth is fused to obtain the isosurface, that is, a smooth isosurface is obtained from the point cloud data of the initial depth, so as to ensure a better visual effect. Specifically, in the process of fusing the point cloud data of the initial depth to obtain the isosurface, first, the cutoff signed distance function value and weight value corresponding to the initial point cloud data are obtained by using the TSDF algorithm, and the cutoff signed distance function value corresponding to the obtained initial point cloud data is used to obtain the isosurface corresponding to the point cloud data of the initial depth by using the marching cubes algorithm, and then the point cloud data of the target depth corresponding to the isosurface is obtained.

[0034] S140, the point cloud data of the target depth is back-projected onto the initial texture data to obtain target texture data corresponding to the person's smile data after back-projection.

[0035] After obtaining the point cloud data of the target depth, the obtained point cloud data of the target depth is back-projected onto the initial texture data to further obtain target texture data corresponding to the person's smile point cloud data after back-projection, that is, to obtain target texture data at different viewing angles of the target point cloud data, thereby ensuring that the obtained facial texture data is more realistic.

[0036] JPEG0007610759000002.jpg51170

[0037] In other words, the target texture data of point p in the initial texture data is related to the pixel value and weight value when point p is projected onto the i-th camera angle of view, and further, after obtaining the weight value of point p at each camera angle of view, a normalization process must be performed on the obtained weight value.

[0038] S150, optimizing the point cloud data of the initial face template according to the point cloud data of the target depth to obtain point cloud data of a target face template, thereby realizing tracking of facial expression changes.

[0039] Specifically, the facial template data uses facial template data in a template database constructed by a bilinear model, where the facial template data includes 26317 points, 52 facial expression parameters and 938 individual face parameters. The facial template tracking is mainly performed by optimizing the facial expression parameters in the template database constructed by a bilinear model based on the point cloud data of the target depth determined in step S130, and further obtaining the point cloud data of the target facial template, thereby realizing tracking of facial expression changes.

[0040] S160, the tooth region identified by the point cloud data of the target face template and the initial texture data and the initial 3D tooth data are fitted in a fitting area.

[0041] Since the acquired person's smile data changes dynamically, in order to ensure that the tooth region and the face template data in the current frame are tracked in real time, the tooth region in the current frame changes in real time from the position of the tooth region in the next frame. For example, if both the upper and lower jaws of the teeth need to be tracked, the positions of the feature points in the point cloud data of the target face template in the previous and next frames need to be obtained first, and the planar distance from the feature point of the previous frame to the feature point of the current frame is calculated. For example, when tracking the upper jaw of the teeth, the nose is selected as the feature point, and the position coordinate of the nose feature point in the current frame is obtained after the person's smile changes dynamically, and the position coordinate of the nose feature point in the next frame after the change is obtained, and according to the change in the position coordinate of the nose feature point in the different frames acquired, a coordinate position transformation is performed for the upper jaw line in the tooth region identified by the initial texture data, so as to realize the tracking of the upper jaw in real time. When tracking the lower jaw of the teeth, the jaw is selected as the feature point, and the position coordinate of the jaw feature point in the current frame is obtained. After a person's smile changes dynamically, the position coordinate of the jaw feature point in the next frame after the change is obtained. Based on the change in the position coordinate of the jaw feature point in the different frames obtained, a coordinate position transformation is performed for the jaw line in the tooth area identified by the initial texture data, so as to realize real-time tracking of the lower jaw, and the obtained three-dimensional tooth data is displayed in the fitting area.

[0042] The 3D dynamic tracking method provided by the embodiment of the present disclosure registers the acquired human smile data, 3D tooth data and face template data into a predetermined coordinate system to obtain initial point cloud data, fuses the point cloud data of the initial depth in the acquired initial point cloud data to obtain an isosurface, and then obtains the point cloud data of the target depth corresponding to the isosurface, thereby ensuring that the human smile data has a better visual effect; back-projects the acquired point cloud data of the target depth onto the initial texture data to realize the display of facial texture; optimizes the point cloud data of the initial face template according to the point cloud data of the target depth to obtain the point cloud data of the target face template, thereby realizing the tracking of facial expression changes; and finally fits and displays the tooth region identified by the point cloud data of the target face template and the initial texture data, thereby realizing the tracking of teeth and improving the accuracy of the tracking result.

[0043] Optionally, FIG. 2 is a flowchart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure, and as shown in FIG. 2, one possible embodiment of step S120 includes:

[0044] S121, source point cloud data of human smile data, 3D tooth data and face template data are obtained.

[0045] After obtaining the human smile data, the three-dimensional teeth data, and the face template data, source point cloud data corresponding to the human smile data, source point cloud data corresponding to the three-dimensional teeth data, and source point cloud data corresponding to the face template data are obtained, where the source point clouds corresponding to different data correspond to different point cloud sets, for example, the source point cloud data X corresponding to the human smile data is composed of point cloud set xi, the source point cloud data Y corresponding to the three-dimensional teeth data is composed of point cloud set yi, and the source point cloud data Z corresponding to the face template data is composed of point cloud set zi.

[0046] S122, determining rotation parameters, translation parameters and an error threshold in the iterative closest point algorithm.

[0047] After determining the source point cloud data corresponding to the different data, a corresponding point cloud set in the source point cloud is found, and the source point cloud data is registered into a predetermined coordinate system based on the rotation parameter, the translation parameter and the error threshold to obtain the initial point cloud data.

[0048] Specifically, the purpose of the Iterative Closest Point (ICP) algorithm is to find the rotation parameter R and translation parameter T between the matched point cloud data and the reference point cloud data, so that the two points satisfy the optimal matching error threshold.

[0049] Suppose the source point cloud data corresponding to the human smile data is X1, and the target point cloud data corresponding to the source point cloud data X1 is X2. The matching steps of the ICP method are as follows:

[0050] Calculate the corresponding near point in the X1 point set for each point in X2, find the rigid transformation that minimizes the average distance of the corresponding point pairs, find the translation parameter T and the rotation parameter R, and use the translation and rotation parameters found in the previous step for X2 to obtain a new set of transformed points. If the average distance between the new transformed point set and the reference point set is smaller than the error threshold, the iterative calculation is stopped; otherwise, continue to iterate with the new transformed point set as the new X2 until the requirement of the objective function is reached.

[0051] In searching for the closest point pair, the calculation of the corresponding points is the step that consumes the longest time in the whole matching process. When searching for the closest point, the search speed can be improved by using kd tree. The topological relationship of points established by the Kd tree method is a coordinate axis division based on a binary tree. The construction process of kd tree is generated according to the binary tree rule. First, a dividing line is found according to the X axis, that is, the average value of the x values ​​of all points is calculated, and the x value of the point closest to this average value is used to divide the space into two parts. Then, in the divided subspace, a dividing line is found according to the Y axis, which is divided into two parts respectively. The divided subspace is further divided according to the X axis... and so on until there is only one point in the divided area. Such a dividing process corresponds to one binary tree, the child node of the binary tree corresponds to one dividing line, and each leaf node of the binary tree corresponds to one point. In this way, the topological relationship of points is established.

[0052] S123, the source point cloud data is registered in a predetermined coordinate system based on the rotation parameters, the translation parameters and the error threshold value to obtain initial point cloud data.

[0053] If the source point cloud data X1 corresponding to human smile data includes a point cloud xi, the point cloud xi is registered in a predetermined coordinate system using a rotation parameter, a translation parameter, and an error threshold to obtain an initial point cloud xi′=Rxi+T.

[0054] The 3D dynamic tracking method provided by the embodiment of the present disclosure transfers a person's smile data, 3D teeth data and face template data located in different coordinate systems to a predetermined coordinate system through a rigid registration algorithm, and further obtains depth point cloud data, initial texture data, initial 3D teeth point cloud data and initial face template point cloud data located in the predetermined coordinate system.

[0055] Optionally, FIG. 3 is a flowchart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure, and as shown in FIG. 3, one possible embodiment of step S130 includes:

[0056] S131, obtain the cutoff signed distance function value and weight value corresponding to the point cloud data of the initial depth using the TSDF algorithm.

[0057] Specifically, obtaining the cutoff signed distance function value and weight value corresponding to the point cloud data of the initial depth using the TSDF algorithm: Establishing a rectangular bounding box corresponding to the initial point cloud data; voxelizing a rectangular bounding box corresponding to the initial point cloud data; Calculating a TSDF value and a weight value corresponding to the point cloud data of the initial depth in each voxelization using a TSDF algorithm.

[0058] For example, obtaining truncated signed distance function values ​​and weight values ​​corresponding to the point cloud data of the initial depth may include first establishing a rectangular bounding box of the initial point cloud data as a cubic space where each initial point cloud data is located, then voxelizing the rectangular bounding box corresponding to the initial point cloud data, and finally calculating a TSDF value and weight value corresponding to the point cloud data of the initial depth in each voxelization using a truncated signed distance function (TSDF). Specifically, all voxels are ergodicized, and taking a three-dimensional position point p in the world coordinate system of one voxel as an example, a mapping point v in the camera coordinate system of the point p in the world coordinate system is obtained according to the camera position and orientation matrix of the depth data, and a corresponding pixel point x in the depth map is obtained by backprojecting the point v according to the internal parameter matrix of the camera, where the depth value of the pixel point x is value(x), and the distance from the point v to the origin of the camera coordinate system is distance(v). Then the sdf value corresponding to point p is sdf(p)=value(x)-distance(v), TSDF(p)=sdf(p) / |u|, where |u| is a corresponding predetermined threshold, which is related to the number and size of voxels, if TSDF(p)>1, the TSDF value corresponding to point p is 1, and if TSDF(p)<-1, the TSDF value corresponding to point p is -1. The calculation formula for weight w(p) is: w(p)=cos(θ), where θ is the included angle between the projection ray and the surface normal vector.

[0059] S132, obtaining an isosurface corresponding to the point cloud data of the initial depth using a marching cubes algorithm based on the cutoff signed distance function value and the weight value.

[0060] Specifically, obtaining an isosurface corresponding to the point cloud data of the initial depth using a marching cube simulation algorithm based on the cutoff signed distance function value and the weight value; Pre-processing the blocked signed distance function values ​​and weight values ​​and loading them into specific arrays; Obtaining position information of a unit body corresponding to different voxelizations; determining a state table corresponding to each unit body based on the position information of each unit body; obtaining an edge of each unit body intersecting with the isosurface based on the state table of each unit body, and calculating position coordinates of an intersection between each unit body and the isosurface by a linear interpolation method; Using the central difference method to obtain the normal vector of each vertex in each unit body, and using the linear interpolation method to obtain the normal vector of each vertex of the triangular patch; determining an isosurface based on the position coordinates and vertex normal vectors of the intersections between each unit body and the isosurface.

[0061] The main idea of ​​the Marching Cubes algorithm is to approach the isosurface by linear interpolation in the 3D discrete data field, specifically, each grid unit in the 3D discrete data field is a voxel, and each vertex of the voxel has a corresponding scalar value. If the value at the voxel vertex is greater than or equal to the value on the isosurface, the vertex is defined as being outside the isosurface and is represented as "0", and if the value at the voxel vertex is less than the value on the isosurface, the vertex is defined as being on the isosurface and is represented as "1". Each voxel unit has 8 vertices, so there are a total of 2^8=256 different situations.

[0062] The index rule for the vertices and edges in each voxel unit is as follows: if the value of the vertex in the voxel is less than the value of the isosurface, the lines intersecting with the edges of the voxel connected to the vertex form a triangular patch, and the specific positions of the vertices of the triangular patch need to be calculated according to the method of linear interpolation. Then, the central difference method is used to obtain the normal vectors of each vertex in each unit body, and the method of linear interpolation is used to obtain the normal vectors of each vertex of the triangular patch. Finally, the isosurface is determined according to the position coordinates of the intersections between each unit body and the isosurface and the vertex normal vectors.

[0063] S133, point cloud data of a target depth is determined based on the point cloud data corresponding to the isosurface.

[0064] Determine the point cloud data of the target depth through the point cloud data corresponding to the acquired isosurface, i.e., determine the depth data corresponding to the human smile data of the current frame, and ensure the smoothness of the depth data corresponding to the acquired three-dimensional human smile data.

[0065] The 3D dynamic tracking method provided in the embodiments of the present disclosure obtains the blocked signed distance function value and weight value corresponding to the point cloud data of the initial depth by the TSDF algorithm, then obtains the isosurface corresponding to the point cloud data of the initial depth by the marching cubes algorithm, and further determines the corresponding point cloud data of the target depth based on the isosurface, so as to ensure that the human smile data has a better visual effect.

[0066] Optionally, FIG. 4 is a flowchart of another three-dimensional dynamic tracking method provided by an embodiment of the present disclosure, and as shown in FIG. 4, one possible embodiment of step S160 includes:

[0067] S161, the tooth region identified by the point cloud data of the target face template and the initial texture data is fitted to the initial three-dimensional tooth data in a fitting area using the Gauss-Newton method.

[0068] Since the acquired person's smile data changes dynamically, in order to ensure that the tooth region and the face template data in the current frame are tracked in real time, the tooth region in the current frame changes in real time from the position of the tooth region in the next frame. For example, if both the upper and lower jaws of the teeth need to be tracked, the positions of the feature points in the point cloud data of the target face template in the previous and next frames need to be obtained first, and the planar distance from the feature point of the previous frame to the feature point of the current frame is calculated. For example, when tracking the upper jaw of the teeth, the nose is selected as the feature point, and the position coordinate of the nose feature point in the current frame is obtained after the person's smile changes dynamically, and the position coordinate of the nose feature point in the next frame after the change is obtained, and according to the change in the position coordinate of the nose feature point in the different frames acquired, a coordinate position transformation is performed for the upper jaw line in the tooth region identified by the initial texture data, so as to realize the tracking of the upper jaw in real time. When tracking the lower jaw of the teeth, the jaw is selected as the feature point, and the position coordinate of the jaw feature point in the current frame is obtained. After a person's smile changes dynamically, the position coordinate of the jaw feature point in the next frame after the change is obtained. Based on the change in the position coordinate of the jaw feature point in the different frames obtained, a coordinate position transformation is performed for the jaw line in the tooth area identified by the initial texture data, so as to realize real-time tracking of the lower jaw, and the obtained three-dimensional tooth data is displayed in the fitting area.

[0069] Specifically, first, corresponding points on the nose of the previous and next frames of the target face template are calculated, and the distance on a plane from the point in the previous frame to the corresponding point in the current frame is calculated. JPEG0007610759000003.jpg17170

[0070] Here, vd denotes the coordinates of the point in the current frame, nd denotes the normal vector of the point in the current frame, and vs denotes the coordinates of the corresponding point in the previous frame.

[0071] Then, based on an edge of the upper jaw of the dental region identified by the initial texture data, the edges of the upper teeth are fitted by back projecting the initial three-dimensional dental data onto the initial texture data, and based on an edge of the lower jaw of the dental region identified by the initial texture data, the edges of the lower teeth are fitted by back projecting the initial three-dimensional dental data onto the initial texture data.

[0072] There are two energy terms in tracking the maxilla: JPEG0007610759000004.jpg6170There are two energy terms in tracking the mandible: JPEG0007610759000005.jpg6170Here, in the above energy term, JPEG0007610759000006.jpg16170 Here, ud indicates the coordinates of the edge pixel of the tooth detected in the current frame, and vd indicates the normal vector of the edge pixel of the tooth detected in the current frame.

[0073] JPEG0007610759000007.jpg14170 Here, Vw indicates a coordinate position in the world coordinate system, Vc indicates a coordinate in the camera coordinate system, and K and RT indicate the inside and outside parameters of the camera, respectively.

[0074] Optionally, according to the above embodiment, FIG. 5 is a three-dimensional dynamic tracking device provided by an embodiment of the present disclosure. As shown in FIG. 5, the three-dimensional dynamic tracking device includes: a data acquisition module 510 used to acquire smile data, 3D dental data and face template data of a person, where the smile data of the person includes texture data and depth data; an initial point cloud data acquisition module 520, used for registering the acquired human smile data, 3D tooth data and face template data in a predetermined coordinate system to acquire initial point cloud data, where the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D tooth point cloud data and initial face template point cloud data; a target depth point cloud data acquisition module 530, which is used to fuse the point cloud data of the initial depth to obtain an isosurface and acquire point cloud data of the target depth corresponding to the isosurface; a target texture data acquisition module 540 for back-projecting the point cloud data of the target depth onto the initial texture data and acquiring target texture data corresponding to the person's smile data after back-projection; a face template tracking module 550, which is used for optimizing the point cloud data of the initial face template according to the point cloud data of the target depth to obtain the point cloud data of the target face template, and realizing tracking of facial expression changes; and a 3D tooth tracking module 560 that is used to fit the dental regions identified by the point cloud data and initial texture data of the target face template and the initial 3D dental data in the fitting area.

[0075] In the 3D dynamic tracking device provided by the embodiment of the present disclosure, the initial point cloud data acquisition module registers the human smile data, the 3D tooth data and the face template data acquired by the data acquisition module into a predetermined coordinate system to acquire initial point cloud data; the target depth point cloud data acquisition module fuses the point cloud data of the initial depth in the acquired initial point cloud data to acquire an isosurface, and further acquires the point cloud data of the target depth corresponding to the isosurface, so as to ensure that the human smile data has a better visual effect; the target texture data acquisition module back-projects the acquired point cloud data of the target depth onto the initial texture data to realize the display of the facial texture; the face template tracking module optimizes the point cloud data of the initial face template according to the point cloud data of the target depth to obtain the point cloud data of the target face template, so as to realize the tracking of the facial expression changes; and finally, the 3D tooth tracking module fits the tooth area identified by the point cloud data of the target face template and the initial texture data, so as to display the 3D tooth data in the fitting area, so as to realize the tracking of the teeth, and improve the accuracy of the tracking result.

[0076] Fig. 6 is a block diagram of an electronic device provided by an embodiment of the present disclosure. As shown in Fig. 6, the electronic device includes a processor 610, a memory 620, an input device 630 and an output device 640. The number of processors 610 in the electronic device may be one or more. In Fig. 6, one processor 610 is taken as an example. The processor 610, the memory 620, the input device 630 and the output device 640 in the electronic device may be connected by a bus or other methods. In Fig. 6, the case of being connected by a bus is taken as an example.

[0077] The memory 620 may be used as a computer-readable storage medium to store software programs, computer-executable programs and modules, such as program instructions / modules corresponding to the three-dimensional dynamic tracking method in the embodiment of the present invention. The processor 610 executes the software programs, instructions and modules stored in the memory 620 to perform various functional applications and data processing of the electronic device, i.e., to realize the three-dimensional dynamic tracking method provided by the embodiment of the present invention.

[0078] The memory 620 may mainly include a program storage area and a data storage area, where the program storage area can store applications required for an operating system, at least one function, and the data storage area can store data established based on the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk memory, a flash memory device, or other non-volatile solid-state memory. In some embodiments, the memory 620 may further include memories located remotely to the processor 610, and these remote memories may be connected to the computer device via a network. Examples of the above network include, but are not limited to, the Internet, a corporate intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] The input device 630 may be used to receive input numeric or textual information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 640 may include a display device such as a display screen.

[0080] An embodiment of the present disclosure further provides a storage medium including computer-executable instructions, which when executed by a computer processor are used to implement the three-dimensional dynamic tracking method provided by an embodiment of the present invention.

[0081] Of course, in a storage medium containing computer-executable instructions provided by embodiments of the present invention, the computer-executable instructions are not limited to the method operations described above, and may further perform correlation operations in the three-dimensional dynamic tracking methods provided by any embodiment of the present invention.

[0082] Based on the description of the above embodiments, it should be clearly understood by those skilled in the art that the present invention can be realized by software and necessary general-purpose hardware, and of course, it can be realized by hardware, but in many cases the former is a more preferred embodiment. Based on such understanding, in the technical solution of the present invention, the part that essentially or contributes to the prior art can be expressed in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk of a computer, and can include some instructions used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the method according to each embodiment of the present invention.

[0083] It should be noted that in the above embodiment of the tracking device, each unit and module provided is merely defined according to functional logic, and it is sufficient to realize the corresponding function, and is not limited to the above definition, and the specific names of each functional unit are merely intended to distinguish them from each other, and are not intended to limit the scope of the claims of the present invention.

[0084] It should be explained that in this specification, the use of related terms such as "first" and "second" is intended to distinguish one entity or operation from another, and does not necessarily require or imply the existence of any such actual relationship or sequence between these entities or operations. And the terms "comprise", "include", "includes" or any other variations thereof cover a non-exclusive "comprise", so that a process, method, article or device that includes a set of elements not only includes those elements, but also further includes other elements not expressly listed, or further includes elements that are specific to such process, method, article or device. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of another identical element in the process, method, article or device that includes the element.

[0085] The above contents are merely embodiments of the present disclosure, by which those skilled in the art can realize or understand the present disclosure. A number of modifications to these embodiments can be easily realized by those skilled in the art, and the general principles defined in this specification can be realized in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to these embodiments described in this specification, but should be accorded the widest scope consistent with the principles and novel features disclosed in this specification.

[0086] The technical features of the above embodiments can be combined in any manner. For convenience of explanation, all possible combinations of the technical features in the above embodiments have not been described. However, as long as there is no contradiction in the combination of these technical features, it should be understood that all of them are included in the scope described in this specification.

[0087] The above examples only represent some embodiments of the present disclosure, and the description is specific and detailed, but should not be understood as limiting the scope of the invention. It should be noted that those skilled in the art may make some modifications and improvements without departing from the concept of the present disclosure, and all of these are included in the scope of the claims of the present disclosure. Therefore, the scope of the claims of the present disclosure should be subject to the scope of the accompanying claims. [Industrial Applicability]

[0088] The 3D dynamic tracking method, device, electronic device and storage medium provided by the embodiments of the present disclosure register the acquired human smile data, 3D tooth data and face template data into a predetermined coordinate system to obtain initial point cloud data, fuse the point cloud data of the initial depth in the acquired initial point cloud data to obtain an isosurface, and further obtain the point cloud data of the target depth corresponding to the isosurface, so as to ensure that the human smile data has a better visual effect; back-project the acquired point cloud data of the target depth onto the initial texture data to realize the display of the facial texture; optimize the point cloud data of the initial face template according to the point cloud data of the target depth to obtain the point cloud data of the target face template, realize the tracking of the facial expression changes; finally, fit and display the tooth region identified by the point cloud data of the target face template and the initial texture data, realize the tracking of the teeth, and improve the accuracy of the tracking result.

Claims

1. 1. A three-dimensional dynamic tracking method, comprising: acquiring facial dynamic data, three-dimensional dental data and facial template data, wherein the facial dynamic data includes texture data and depth data; registering the acquired facial dynamic data, 3D dental data and facial template data in a predetermined coordinate system to obtain initial point cloud data, wherein the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D dental point cloud data and initial facial template point cloud data; Fusing the point cloud data of the initial depth to obtain an isosurface, and obtaining point cloud data of a target depth corresponding to the isosurface; Back-projecting the point cloud data of the target depth onto the initial texture data to obtain target texture data corresponding to the dynamic data of the face after back-projection; Optimizing the point cloud data of the initial face template based on the point cloud data of the target depth to obtain point cloud data of a target face template, thereby realizing tracking of facial expression changes; and fitting a dental region identified by the point cloud data of the target face template and the initial texture data and the initial three-dimensional dental data in a fitting area.

2. The acquisition of the initial point cloud data by registering the acquired facial dynamic data, the three-dimensional dental data, and the facial template data in a predetermined coordinate system includes: Obtaining source point cloud data of facial dynamic data, 3D dental data, and facial template data; determining rotation parameters, translation parameters and an error threshold in an iterative closest point algorithm; and registering the source point cloud data to a predetermined coordinate system based on the rotation parameters, the translation parameters and an error threshold to obtain the initial point cloud data.

3. Fusing the point cloud data of the initial depth to obtain an isosurface and obtaining point cloud data of a target depth corresponding to the isosurface includes: Obtaining cutoff signed distance function values ​​and weight values ​​corresponding to the initial depth point cloud data using a TSDF algorithm; Obtaining an isosurface corresponding to the point cloud data of the initial depth using a marching cubes algorithm based on the cutoff signed distance function value and the weight value; and determining point cloud data of a target depth based on point cloud data corresponding to the isosurface.

4. Obtaining cutoff signed distance function values ​​and weight values ​​corresponding to the initial depth point cloud data using the TSDF algorithm includes: Establishing a rectangular bounding box corresponding to the initial point cloud data; voxelizing a rectangular bounding box corresponding to the initial point cloud data; and calculating TSDF values ​​and weight values ​​corresponding to the point cloud data at the initial depth for each voxelization using a TSDF algorithm.

5. obtaining an isosurface corresponding to the point cloud data of the initial depth using a marching cubes algorithm based on the cutoff signed distance function value and the weight value; Preprocessing the blocked signed distance function values ​​and weight values ​​and loading them into a specific array; Obtaining position information of a unit body corresponding to different voxelizations; determining a state table corresponding to each of the unit bodies based on position information of each of the unit bodies; obtaining an edge of each of the unit bodies intersecting with the isosurface based on a state table of each of the unit bodies, and calculating position coordinates of an intersection between each of the unit bodies and the isosurface by a linear interpolation method; Using a central difference method to obtain the normal vector of each vertex in each of the unit bodies, and using a linear interpolation method to obtain the normal vector of each vertex of the triangular patch; determining an isosurface based on position coordinates and vertex normal vectors of intersections between each of the unit bodies and the isosurface,

6. The back-projecting of the point cloud data of the target depth onto the initial texture data to obtain target texture data corresponding to the dynamic data of the face after the back-projection includes: Including,

7. The fitting of the dental region identified by the point cloud data of the target face template and the initial texture data and the initial three-dimensional dental data in a fitting area includes:

2. The method of claim 1, further comprising fitting the tooth region identified by the point cloud data and initial texture data of the target face template to the initial three-dimensional tooth data in a fitting area using a Gauss-Newton method.

8. A three-dimensional dynamic tracking device, comprising: A data acquisition module used for acquiring facial dynamic data, three-dimensional dental data and facial template data, where the facial dynamic data includes texture data and depth data; an initial point cloud data acquisition module, used for registering the acquired facial dynamic data, 3D dental data and facial template data into a predetermined coordinate system to acquire initial point cloud data, where the initial point cloud data includes initial depth point cloud data, initial texture data, initial 3D dental point cloud data and initial facial template point cloud data; a target depth point cloud data acquisition module, which is used to fuse the point cloud data of the initial depth to obtain an isosurface and acquire point cloud data of a target depth corresponding to the isosurface; a target texture data acquisition module for back-projecting the point cloud data of the target depth onto the initial texture data to acquire target texture data corresponding to the dynamic data of the face after back-projection; a face template tracking module, which is used for optimizing the point cloud data of the initial face template according to the point cloud data of the target depth to obtain point cloud data of a target face template, and realizing tracking of facial expression changes; and a three-dimensional tooth tracking module used for fitting a tooth region identified by the point cloud data of the target face template and the initial texture data to display three-dimensional tooth data in a fitting area.

9. An electronic device, one or more processors; a storage device adapted to store one or more programs; An electronic device, characterized in that, when the one or more programs are executed by the one or more processors, the one or more processors realize the three-dimensional dynamic tracking method described in any one of claims 1 to 7.

10. A computer readable storage medium having a computer program stored thereon, the program implementing the three-dimensional dynamic tracking method according to any one of claims 1 to 7 when executed by a processor.

Images