Face model reconstruction method and device, equipment and medium

Abstract

The invention provides a face model reconstruction method and device, equipment and a medium. The method comprises the following steps: acquiring multi-frame original point clouds of a user face; determining an original point cloud of a reference frame and an original point cloud of a non-reference frame in the multiple frames of original point clouds; according to the multiple frames of original point clouds, reconstructing a first face model matched with the expression state corresponding to the original point clouds of the reference frame; and reconstructing a second face model matched with the expression state corresponding to the original point cloud of each non-reference frame by aligning the first face model to the original point cloud of each non-reference frame. By using the method, the reconstruction efficiency of the dynamic face model can be improved.

Description

Technical Field

[0001] The present application belongs to the field of three-dimensional scanning and relates to a facial model reconstruction method, device, equipment and medium. Background Art

[0002] With the rapid development of 3D digital technology, its application in various fields is becoming more and more extensive. Taking the medical field as an example, the design of treatment plans and the evaluation of treatment effects are increasingly dependent on the reconstruction technology of 3D models. In order to achieve the requirements of accurate simulation and evaluation, it is often necessary to obtain the user's dynamic facial model, which includes 3D facial models in multiple different expression states.

[0003] When reconstructing the 3D facial model corresponding to each expression state, the relevant technology relies on multi-angle point cloud data under the same expression state. However, if only a single scanning device is used, it takes a certain amount of scanning time to collect point cloud data. During the scanning time, it is often difficult for users to maintain a fixed facial expression. If the facial expression undergoes non-rigid changes (such as smiling, blinking, etc.), repeated scanning will be required, resulting in poor reconstruction efficiency of the dynamic facial model. Summary of the invention

[0004] In view of the above, it is necessary to provide a facial model reconstruction method, device, equipment and medium that can solve the technical problem of poor reconstruction efficiency of dynamic facial models due to the difficulty for users to maintain fixed facial expressions in the case of a single scanning device.

[0005] On the one hand, the present application provides a facial model reconstruction method, the method comprising: obtaining multiple frames of original point clouds of a user's face, determining the original point clouds of a reference frame and the original point clouds of a non-reference frame in the multiple frames of original point clouds, and reconstructing a first facial model that matches the expression state corresponding to the original point cloud of the reference frame based on the multiple frames of original point clouds, and reconstructing a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame.

[0006] In some embodiments of the present application, the multi-frame original point cloud is obtained based on scanning the user's face with changing expressions. If the original point cloud of the reference frame and the original point cloud of the non-reference frame are determined in the process of obtaining the multi-frame original point cloud, the original point cloud of the reference frame is the first frame original point cloud obtained, or, if the original point cloud of the reference frame and the original point cloud of the non-reference frame are determined after obtaining the multi-frame original point cloud, the original point cloud of the reference frame is any frame original point cloud selected from the multi-frame original point cloud, and the original point cloud of the non-reference frame is other original point clouds in the multi-frame original point cloud except the original point cloud of the reference frame. The first facial model is reconstructed based on the original point cloud of the reference frame and the deformed point cloud corresponding to the original point cloud of each non-reference frame, wherein the multi-frame deformed point cloud corresponding to the original point cloud of the non-reference frame has the same morphology as the original point cloud of the reference frame.

[0007] In some embodiments of the present application, if the original point cloud of the reference frame is the first frame original point cloud acquired, the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame includes: aligning the original point cloud of each non-reference frame with the target point cloud corresponding to the original point cloud of each non-reference frame to obtain the deformed point cloud corresponding to the original point cloud of each non-reference frame, the target point cloud corresponding to the second frame original point cloud in the original point cloud of the non-reference frame is the first frame original point cloud, and the target point cloud corresponding to the nth frame original point cloud in the original point cloud of the non-reference frame is obtained by fusing the target point cloud corresponding to the n-1th frame original point cloud with the deformed point cloud corresponding to the n-1th original point cloud, wherein 3≤n≤m, and m represents the number of the multiple frames of original point clouds.

[0008] In some embodiments of the present application, if the original point cloud of the reference frame is any frame original point cloud selected from the multiple frames of original point clouds, the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame includes: selecting the original point cloud from the original point cloud of the non-reference frame, taking the original point cloud of the reference frame as the target point cloud corresponding to the selected original point cloud, and aligning the selected original point cloud with the corresponding target point cloud to obtain the deformed point cloud corresponding to the selected original point cloud, fusing the deformed point cloud corresponding to the selected original point cloud with the corresponding target point cloud to obtain a first fused point cloud, each first fused point cloud being used as the target point cloud corresponding to the next frame of original point cloud selected, and repeating the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame until all the original point clouds in the original point cloud of the non-reference frame are selected to obtain the deformed point cloud corresponding to the original point cloud of each non-reference frame.

[0009] In some embodiments of the present application, the original point cloud of each non-reference frame includes a plurality of first coordinate points, the target point cloud corresponding to the original point cloud of each non-reference frame includes a plurality of second coordinate points, and the alignment method between the original point cloud of each non-reference frame and the corresponding target point cloud includes: determining a deformation field corresponding to each first coordinate point, the deformation field including a plurality of deformation control points corresponding to each first coordinate point, and determining a first dense correspondence relationship between the plurality of first coordinate points and the plurality of second coordinate points by densely aligning the plurality of first coordinate points and the plurality of second coordinate points, the first dense correspondence relationship including a plurality of groups of point pairs, wherein each group of point pairs includes mutually corresponding first points. A coordinate point and a second coordinate point, determine the target deformation parameter of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and the multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determine the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of the multiple deformation control points corresponding to each first coordinate point, and determine the deformation point cloud corresponding to the original point cloud of each non-reference frame based on the multiple first coordinate points and the target coordinates of each first coordinate point.

[0010] In some embodiments of the present application, the method for determining the target deformation parameters of each deformation control point includes: executing a solution process for the initial deformation parameters of each deformation control point, including: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function, repeatedly executing the solution process, updating the initial deformation parameters of each deformation control point, until the number of updates to the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stopping the solution process, and determining the updated initial deformation parameters as the target deformation parameters of each deformation control point.

[0011] In some embodiments of the present application, reconstructing a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame includes: densely aligning the first facial model with the original point cloud of each non-reference frame according to the first dense correspondence, determining a second dense correspondence between the first facial model and the original point cloud of each non-reference frame, and aligning the first facial model with the original point cloud of each non-reference frame according to the second dense correspondence to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame.

[0012] In some embodiments of the present application, reconstructing a first facial model that matches the expression state corresponding to the original point cloud of the reference frame based on the multiple frames of original point clouds includes: fusing the multiple frames of deformed point clouds to obtain a second fused point cloud, and reconstructing the first facial model based on the second fused point cloud and the original point cloud of the reference frame using a preset reconstruction algorithm.

[0013] In some embodiments of the present application, the multi-frame original point cloud is obtained based on scanning the user's face with fixed expression and changing expression, the original point cloud of the reference frame is obtained when the expression state of the user's face remains fixed, the original point cloud of the non-reference frame is obtained when the expression state of the user's face changes dynamically, and the first facial model is reconstructed based on the original point cloud of the reference frame.

[0014] In some embodiments of the present application, the method also includes: if it is detected that the expression state of the user's face has changed, obtaining point cloud data of the user's face, and according to the point cloud data, driving the target model to change from the corresponding expression state to the expression state corresponding to the point cloud data, the target model including at least one of the first facial model, the second facial model, the model after the first facial model is aligned with the user's dental model, and the model after the second facial model is aligned with the user's dental model.

[0015] On the other hand, the present application provides a facial model reconstruction device, which includes: an acquisition module for acquiring multiple frames of original point clouds of a user's face, a determination module for determining the original point cloud of a reference frame and the original point cloud of a non-reference frame in the multiple frames of original point clouds, and a reconstruction module for reconstructing a first facial model that matches the expression state corresponding to the original point cloud of the reference frame based on the multiple frames of original point clouds. The reconstruction module is also used to reconstruct a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame.

[0016] On the other hand, the present application provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the facial model reconstruction method.

[0017] On the other hand, the present application provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and when the at least one instruction is executed by a processor in an electronic device, the facial model reconstruction method is implemented.

[0018] In the facial model reconstruction scheme of the embodiment of the present application, multiple frames of original point clouds may be from the same scanning device, and by determining the original point cloud of the reference frame and the original point cloud of the non-reference frame, a first facial model matching the expression state of the original point cloud of the reference frame can be reconstructed. Since aligning the first facial model to the original point cloud of each non-reference frame is the reverse idea of ​​reconstructing the first facial model, in the process of aligning the first facial model to the original point cloud of each non-reference frame, the partial association relationship in the process of reconstructing the first facial model can be used for alignment, so that the second facial model matching the expression state corresponding to the original point cloud of each non-reference frame can be quickly reconstructed, and then the facial model corresponding to each frame of the original point cloud of the user's face can be obtained, and the reconstruction of the dynamic facial model is completed, thereby improving the reconstruction efficiency of the dynamic facial model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a facial model reconstruction method provided in one embodiment of the present application.

[0020] Figure 2 It is a schematic diagram of the facial expression state corresponding to a frame of original point cloud provided by an embodiment of the present application.

[0021] Figure 3 It is a schematic diagram of the facial expression state corresponding to another frame of original point cloud provided by an embodiment of the present application.

[0022] Figure 4 This is a flowchart of a method for generating a deformable point cloud provided in one embodiment of the present application.

[0023] Figure 5 It is a flowchart of a method for constructing a constraint function provided in one embodiment of the present application.

[0024] Figure 6 It is a functional module diagram of a facial model reconstruction device provided in one embodiment of the present application.

[0025] Figure 7 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0027] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. The following embodiments and features in the embodiments may be combined with each other without conflict.

[0028] With the rapid development of three-dimensional digital technology, its application in various fields is becoming more and more extensive. Taking the medical field as an example, the design of treatment plans and the evaluation of treatment effects increasingly rely on the reconstruction technology of three-dimensional models. In order to achieve the requirements of accurate simulation and evaluation, it is often necessary to obtain the user's dynamic facial model, where the dynamic facial model includes three-dimensional facial models in multiple different expression states. When reconstructing the three-dimensional facial model corresponding to each expression state, the relevant technology relies on multi-angle point cloud data under the same expression state. In the relevant technology, the following two methods are mainly used to reconstruct the dynamic facial model:

[0029] The first method is to obtain multi-angle point cloud data of the user at the same time by increasing the range of the simultaneously photographed format. This method often requires setting up multiple scanning devices and using synchronization technology to control all scanning devices to simultaneously shoot faces, combining the images taken by all scanning devices at the same time, and reconstructing the three-dimensional facial model corresponding to the expression state at that moment based on the point cloud data corresponding to the combined images, and obtaining a dynamic facial model through point cloud data at multiple moments. However, this reconstruction method not only takes a lot of time to set up and synchronize the equipment, but also requires a high cost investment. These factors will lead to high cost and time-consuming reconstruction of the dynamic facial model, and poor reconstruction efficiency.

[0030] The second method is to use a single scanning device to collect multi-angle point cloud data under a fixed facial expression state, and reconstruct the three-dimensional facial model corresponding to the expression state based on the point cloud data, and obtain a dynamic facial model through the point cloud data under multiple facial expression states. However, using only a single scanning device, it takes a certain amount of scanning time to collect point cloud data. During the scanning time, it is often difficult for users to maintain a fixed facial expression. If the facial expression undergoes non-rigid changes (such as smiling, blinking, etc.), repeated scanning will be required, resulting in poor reconstruction efficiency of the dynamic facial model.

[0031] In order to solve the above technical problems, the embodiment of the present application provides a facial model reconstruction method, which uses only a single scanning device and reconstructs the user's facial model at one time when the scanning format is small, and can improve the reconstruction efficiency of dynamic facial models. The facial model reconstruction method provided in the embodiment of the present application can be applied to one or more electronic devices, and the electronic device can be a laptop computer, a mobile phone, a computer, a server and other devices. The present application does not limit the type of electronic device.

[0032] like Figure 1 FIG. 1 is a flowchart of a facial model reconstruction method provided by an embodiment of the present application. According to different requirements, the order of each step in the flowchart can be adjusted according to actual requirements, and some steps can be omitted. The method is applied to electronic devices, such as Figure 7 An electronic device 10 is shown.

[0033] S11, obtaining multiple frames of original point clouds of the user's face.

[0034] In some embodiments of the present application, the electronic device can obtain the multiple frames of original point clouds of the user's face through a variety of methods. For example, the electronic device can be connected to a scanning device for communication, and obtain the multiple frames of original point clouds obtained by scanning the user's face from the scanning device, wherein the scanning device can be a face scanner, a tablet computer, a mobile phone, or AR glasses, etc., and the present application does not limit this. Alternatively, the electronic device can read multiple frames of original point clouds from a folder under a preset path, wherein the preset path can be customized, and the present application does not limit this. The above examples of the method for obtaining the multiple frames of original point clouds are only examples and are not limited to this in actual applications.

[0035] In some embodiments of the present application, a multi-frame original point cloud can be obtained based on scanning the user's face with changing expressions. Exemplarily, the multi-frame original point cloud can be obtained by scanning the user's face multiple times or continuously capturing the user's face when the user's expression state changes dynamically, so that the multi-frame original point cloud corresponds to multiple expression states. Alternatively, the multi-frame original point cloud is obtained based on scanning the user's face with fixed expressions and changing expressions. Exemplarily, the multi-frame original point cloud can be obtained by scanning the user's face multiple times or continuously capturing the user's face when the user's expression state first remains fixed and then changes dynamically, so that the multi-frame original point cloud can include the original point cloud with the same expression state and the original point cloud with multiple expression states. Among them, the original point cloud with the same expression state in the multi-frame original point cloud can be multiple frames, and the original point cloud with multiple expression states in the multi-frame original point cloud can be one frame or multiple frames.

[0036] Take the original point cloud of multiple frames corresponding to various expression states as an example. Figure 2 As shown in FIG. 1 , it is a schematic diagram of the expression state corresponding to a frame of original point cloud provided by an embodiment of the present application. Figure 3 , which is a schematic diagram of the expression state corresponding to another frame of original point cloud provided by an embodiment of the present application. Figure 2 , the user's lips are closed. Figure 3 The user's lips are open. Figure 2 and Figure 3 The expression states are different.

[0037] S12, determining an original point cloud of a reference frame and an original point cloud of a non-reference frame in the multiple frames of original point clouds.

[0038] In some embodiments of the present application, if a multi-frame original point cloud is obtained based on scanning the user's face with changing expressions, the electronic device can determine the original point cloud of the reference frame and the original point cloud of the non-reference frame in the process of obtaining the multi-frame original point cloud, or determine the original point cloud of the reference frame and the original point cloud of the non-reference frame after obtaining the multi-frame original point cloud, wherein the multi-frame deformed point cloud corresponding to the original point cloud of the non-reference frame has the same form as the original point cloud of the reference frame. Since the determination timing of these two situations is different, the method of determining the original point cloud of the reference frame and the original point cloud of the non-reference frame is also different. The method of determining the original point cloud of the reference frame and the original point cloud of the non-reference frame will be described below for two situations respectively:

[0039] If the electronic device determines the original point cloud of the reference frame and the original point cloud of the non-reference frame in the process of acquiring multiple frames of original point cloud, since not all the original point clouds of the user's face have been acquired, the electronic device can use the first frame of original point cloud acquired as the original point cloud of the reference frame. For example, if the electronic device acquires the original point cloud from a scanning device, the electronic device can determine the first frame of original point cloud received from the scanning device as the original point cloud of the reference frame, and determine the original point cloud after the first frame of original point cloud received from the scanning device as the original point cloud of the non-reference frame.

[0040] If the electronic device determines the original point cloud of the reference frame and the original point cloud of the non-reference frame after obtaining multiple frames of original point clouds, since all the original point clouds of the user's face have been obtained, the electronic device can determine any frame of the original point cloud in the obtained multiple frames of original point clouds as the original point cloud of the reference frame, and determine the original point clouds in the multiple frames of original point clouds except the original point cloud of the reference frame as the original point cloud of the non-reference frame.

[0041] In this embodiment, by executing different strategies to determine the original point cloud of the reference frame and the original point cloud of the non-reference frame in different situations, the flexibility in determining the original point cloud of the reference frame and the original point cloud of the non-reference frame can be improved, providing a basis for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame below.

[0042] In other embodiments of the present application, if multiple frames of original point clouds are obtained by scanning the face of a user with a fixed expression and a changing expression, the original point cloud of the reference frame may be an original point cloud obtained when the expression state of the user's face remains fixed, and the original point cloud of the non-reference frame may be an original point cloud obtained when the expression state of the user's face changes dynamically. The original point cloud of the reference frame may be multiple frames.

[0043] In this embodiment, since the shape of the original point cloud obtained when the user's facial expression state remains fixed is the same, the original point cloud obtained when the user's facial expression state remains fixed can be directly determined as the original point cloud of the reference frame.

[0044] S13, reconstructing a first facial model that matches the expression state corresponding to the original point cloud of the reference frame according to the multiple frames of original point cloud.

[0045] In some embodiments of the present application, if multiple frames of original point clouds are obtained based on scanning the user's face with changing expressions, the first facial model can be reconstructed based on the original point cloud of the reference frame and the deformed point cloud corresponding to the original point cloud of each non-reference frame.

[0046] If the electronic device determines that the original point cloud of the reference frame and the original point cloud of the non-reference frame and the original point cloud of the reference frame are the first frame of original point cloud received during the process of acquiring multiple frames of original point clouds, the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame includes: the electronic device aligns the original point cloud of each non-reference frame with the target point cloud corresponding to the original point cloud of each non-reference frame to obtain the deformed point cloud corresponding to the original point cloud of each non-reference frame, the target point cloud corresponding to the second frame of the original point cloud of the non-reference frame is the first frame of the original point cloud, and the target point cloud corresponding to the nth frame of the original point cloud of the non-reference frame is obtained by fusing the target point cloud corresponding to the n-1th frame of the original point cloud with the deformed point cloud corresponding to the n-1th original point cloud, wherein 3≤n≤m, and m represents the number of the multiple frames of original point clouds.

[0047] For example, the electronic device can use the first frame of original point cloud received from the scanning device as the original point cloud of the reference frame and the target point cloud corresponding to the second frame original point cloud. When the second frame original point cloud is received from the scanning device, the second frame original point cloud is aligned to the target point cloud (first frame original point cloud) corresponding to the second frame original point cloud to obtain the deformed point cloud corresponding to the second frame original point cloud. The deformed point cloud corresponding to the second frame original point cloud is fused with the target point cloud corresponding to the second frame original point cloud to obtain the target point cloud corresponding to the original point cloud of the third frame. When the third frame original point cloud is received from the scanning device, the third frame original point cloud is aligned to the target point cloud corresponding to the third frame original point cloud to obtain the deformed point cloud corresponding to the third frame original point cloud. By analogy, the deformed point cloud corresponding to the original point cloud of each non-reference frame can be obtained.

[0048] In this embodiment, if the electronic device determines the original point cloud of the reference frame and the original point cloud of the non-reference frame in the process of acquiring multiple frames of original point cloud, the received first frame of original point cloud is determined as the original point cloud of the reference frame, so as to generate corresponding deformed point cloud for the original point cloud of the non-reference frame after the first frame of original point cloud. By using the original point cloud of the first frame as the target point cloud corresponding to the original point cloud of the second frame, and using the fused point cloud between the target point cloud corresponding to the original point cloud of each non-reference frame in the original point cloud of the non-reference frame and the corresponding deformed point cloud as the target point cloud corresponding to the original point cloud of the next non-reference frame in the original point cloud of the non-reference frame, it is possible to reduce the cumulative error, improve the smoothness and accuracy of the deformed point cloud corresponding to the original point cloud of each non-reference frame, and also make the deformed point cloud corresponding to the original point cloud of each non-reference frame have the same morphology, contour, coordinate system and other features as the original point cloud of the first frame (reference frame).

[0049] In other embodiments of the present application, if the electronic device determines that the original point cloud of the reference frame and the original point cloud of the non-reference frame and the original point cloud of the reference frame are any frame original point cloud selected from the obtained multi-frame original point cloud after obtaining the original point cloud of multiple frames. The method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame includes: the electronic device selects the original point cloud from the original point cloud of the non-reference frame, uses the original point cloud of the reference frame as the target point cloud corresponding to the selected original point cloud, and aligns the selected original point cloud with the corresponding target point cloud to obtain the deformed point cloud corresponding to the selected original point cloud, fuses the deformed point cloud corresponding to the selected original point cloud with the corresponding target point cloud to obtain the first fused point cloud, each first fused point cloud is used as the target point cloud corresponding to the next selected original point cloud, and repeatedly executes the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame until all the original point clouds in the original point cloud of the non-reference frame are selected, and the deformed point cloud corresponding to the original point cloud of each non-reference frame is obtained, wherein the original point clouds selected each time are not repeated.

[0050] For example, the electronic device can select the third frame original point cloud from the obtained multiple frames of original point clouds as the original point cloud of the reference frame and the target point cloud corresponding to the first frame original point cloud selected from the original point cloud of the non-reference frame, and determine the original point clouds other than the third frame original point cloud as the original point cloud of the non-reference frame. If the first frame original point cloud selected from the original point cloud of the non-reference frame is the seventh frame original point cloud, align the seventh frame original point cloud to the target point cloud (the third frame original point cloud) corresponding to the seventh frame original point cloud, and obtain the deformed point cloud corresponding to the seventh frame original point cloud. The deformed point cloud corresponding to the starting point cloud is fused with the target point cloud corresponding to the original point cloud of the seventh frame to obtain a first fused point cloud. The first fused point cloud is determined as the target point cloud corresponding to the original point cloud of the second frame selected from the original point cloud of the non-reference frame. The original point cloud of the second frame selected from the original point cloud of the non-reference frame is the original point cloud of the fifth frame. The original point cloud of the fifth frame is aligned to the target point cloud (first fused point cloud) corresponding to the original point cloud of the fifth frame to obtain the deformed point cloud corresponding to the original point cloud of the fifth frame. By analogy, the deformed point cloud corresponding to the original point cloud of each non-reference frame can be obtained.

[0051] In this embodiment, if the electronic device determines the original point cloud of the reference frame and the original point cloud of the non-reference frame after obtaining the original point cloud of multiple frames, the flexibility of selecting the original point cloud of the reference frame can be improved, so that the reference facial expression state can be flexibly selected from the facial expression states corresponding to the original point clouds of multiple frames. By using the first fused point cloud between the target point cloud corresponding to the original point cloud of each non-reference frame in the original point cloud of the non-reference frame and the corresponding deformed point cloud as the target point cloud corresponding to the next original point cloud of the non-reference frame selected from the original point cloud of the non-reference frame, not only can the cumulative error be reduced, the smoothness and accuracy of the deformed point cloud corresponding to the original point cloud of each non-reference frame be improved, but also the deformed point cloud corresponding to the original point cloud of each non-reference frame can be made to have the same morphology, coordinate system and other features as the original point cloud of the reference frame.

[0052] In some embodiments of the present application, the original point cloud of each non-reference frame includes a plurality of first coordinate points, and the target point cloud corresponding to the original point cloud of each non-reference frame includes a plurality of second coordinate points, the first coordinate points and the second coordinate points may be points reflecting facial features of the user, and the first coordinate points and the second coordinate points both have corresponding initial coordinates, wherein the initial coordinates may be three-dimensional coordinates. The alignment method between the original point cloud of each non-reference frame and the corresponding target point cloud includes: the electronic device may perform non-rigid deformation on the original point cloud of each non-reference frame relative to the corresponding target point cloud based on a first dense correspondence between the plurality of first coordinate points and the plurality of second coordinate points, thereby determining the target coordinates of each first coordinate point, and transforming each first coordinate point from the corresponding initial coordinates to the corresponding target coordinates, to obtain a deformed point cloud corresponding to the original point cloud of each non-reference frame.

[0053] The method for fusing the target point cloud corresponding to the original point cloud of each non-reference frame with the corresponding deformed point cloud can refer to the method for fusing multi-frame deformed point clouds described below. Based on the first dense correspondence, the original point cloud of each non-reference frame is non-rigidly deformed relative to the corresponding target point cloud to determine the target coordinates of each first coordinate point will be described below.

[0054] In this embodiment, the first dense correspondence includes aligning the original point cloud of each non-reference frame to the deformation prior information of the corresponding target point cloud. Through continuous alignment and fusion, the multi-frame deformed point cloud can have the same morphology, contour, coordinate system and other features as the original point cloud of the reference frame.

[0055] In some embodiments of the present application, the first facial model may be a three-dimensional mesh model. The electronic device reconstructs the first facial model that matches the expression state corresponding to the original point cloud of the reference frame based on the multi-frame deformed point cloud and the original point cloud of the reference frame, including: the electronic device fuses the multi-frame deformed point cloud to obtain a second fused point cloud, and reconstructs the first facial model based on the second fused point cloud and the original point cloud of the reference frame using a preset reconstruction algorithm.

[0056] Among them, the electronic device can use a variety of methods to fuse multiple frames of deformed point clouds to obtain a second fused point cloud. For example, the electronic device can use a voxel fusion algorithm to fuse multiple frames of deformed point clouds to obtain a second fused point cloud. The voxel fusion algorithm can be a truncated signed distance function (TSDF), etc., which is not limited in this application. The preset reconstruction algorithm can be a marching cubes algorithm or a Poisson reconstruction algorithm, etc., which is not limited in this application. For example, the electronic device can use the features of the original point cloud of the reference frame as a reference, and use the truncated signed distance function to represent each point in each frame of the deformed point cloud as a three-dimensional voxel, each voxel has a corresponding signed distance value, and the signed distance value represents the distance between the corresponding voxel and the user's face. The signed distance values ​​corresponding to all voxels are weighted averaged and fused to obtain a signed distance field, and the isosurface (Isosurface) is extracted from the signed distance field as the first facial model using the marching cubes algorithm.

[0057] In this embodiment, since the multi-frame deformed point clouds have the same morphology, coordinate system and other features as the original point cloud of the reference frame, the user's expression state can change. Using the features of the original point cloud of the reference frame as a reference, the multi-frame deformed point clouds are fused and reconstructed to obtain a first facial model that matches the expression state corresponding to the original point cloud of the reference frame.

[0058] In other embodiments of the present application, if multiple frames of original point clouds are obtained based on scanning the user's face with fixed expressions and with changing expressions, the first facial model can be reconstructed based on the original point cloud of the reference frame.

[0059] The original point cloud of the reference frame may be multiple frames. The method for reconstructing the first facial model matching the expression state corresponding to the original point cloud of the reference frame according to the original point cloud of the reference frame may refer to the description of the method for reconstructing the first facial model matching the expression state corresponding to the original point cloud of the reference frame according to the deformation point clouds of multiple frames and the original point cloud of the reference frame.

[0060] In this embodiment, since the original point cloud obtained while keeping the facial expression state of the user fixed can be directly determined as the original point cloud of the reference frame, the first facial model matching the expression state corresponding to the original point cloud of the reference frame can be quickly reconstructed.

[0061] S14, reconstructing a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame.

[0062] In some embodiments of the present application, if multiple frames of original point clouds are obtained based on scanning the user's face with changing expressions, the electronic device reconstructs a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame, including: the electronic device can densely align the first facial model with the original point cloud of each non-reference frame according to a first dense correspondence relationship, determine a second dense correspondence relationship between the first facial model and the original point cloud of each non-reference frame, and align the first facial model with the original point cloud of each non-reference frame according to the second dense correspondence relationship to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame.

[0063] Among them, the second dense correspondence may include multiple groups of point pairs, each group of point pairs includes a vertex of the first facial model and the coordinate point corresponding to the vertex in the original point cloud of each non-reference frame. There is overlap between the second dense correspondence and the first dense correspondence. Since aligning the first facial model to the original point cloud of each non-reference frame is the reverse idea of ​​reconstructing the first facial model, and the first dense correspondence includes deformation prior information that aligns the original point cloud of each non-reference frame to the corresponding target point cloud, the electronic device can determine the coordinate points corresponding to some vertices on the first facial model in the original point cloud of each non-reference frame according to the first dense correspondence. Therefore, when densely aligning the first facial model with the original point cloud of each non-reference frame, the partial correspondence in the first dense correspondence can be directly used.

[0064] For example, the electronic device may deform the first facial model to the original point cloud of each non-reference frame according to the second dense correspondence between the first facial model and the original point cloud of each non-reference frame to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame. The method for deforming the first facial model to the original point cloud of each non-reference frame according to the second dense correspondence to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame may refer to the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame, and the present application will not repeat the description.

[0065] In this embodiment, since the coordinate points corresponding to some vertices on the first facial model can be determined in the original point cloud of each non-reference frame according to the first dense correspondence, the second dense correspondence between the first facial model and the original point cloud of each non-reference frame can be quickly determined, so that the second facial model matching the expression state corresponding to the original point cloud of each frame can be quickly reconstructed according to the second dense correspondence, thereby improving the reconstruction efficiency of the dynamic facial model. In the facial model reconstruction scheme provided in the embodiment of the present application, only a single scanning device can be used to acquire the point cloud, and the user does not need to maintain a fixed expression, so the number of repeated scans can be reduced, which can further improve the reconstruction efficiency of the dynamic facial model.

[0066] In some other embodiments of the present application, the electronic device may not need to densely align the first facial model with the original point cloud of each non-reference frame based on the first dense correspondence relationship, and the present application does not impose any restrictions on this.

[0067] In other embodiments of the present application, if multiple frames of original point clouds are obtained based on scanning the user's face with fixed expressions and changing expressions, the electronic device can reconstruct a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame, including: densely aligning the first facial model with the original point cloud of each non-reference frame to obtain a third dense correspondence relationship, and aligning the first facial model with the original point cloud of each non-reference frame according to the third dense correspondence relationship to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame.

[0068] In this embodiment, since the first facial model can be quickly reconstructed, by aligning the first facial model to the original point cloud of each non-reference frame, a second facial model matching the expression state corresponding to the original point cloud of each frame can be quickly reconstructed, thereby improving the reconstruction efficiency of the dynamic facial model.

[0069] In other embodiments of the present application, after obtaining the first facial model and / or the second facial model, if a change in the facial expression state of the user is detected, the electronic device can obtain point cloud data of the user's face, and drive the target model to change from the corresponding expression state to the expression state corresponding to the point cloud data based on the obtained point cloud data.

[0070] If the facial expression of the user changes dynamically, the point cloud data obtained may be multiple frames.

[0071] In some embodiments, the target model may be a first facial model and / or a second facial model, and the expression state of the target model may be driven to change dynamically through the acquired point cloud data. Exemplarily, the electronic device may drive the target model to change from a corresponding expression state to an expression state corresponding to the acquired point cloud data by performing non-rigid registration of the target model with the acquired point cloud data. For example, the electronic device may perform non-rigid registration of the target model with the acquired point cloud data through an iterative closest point algorithm (ICP).

[0072] In this embodiment, by driving the target model to transform from the corresponding expression state to the expression state corresponding to the acquired point cloud data, accurate expression migration can be achieved, thereby achieving tracking of the user's facial expression.

[0073] In other embodiments, the electronic device obtains a dental model of the user, the dental model has multiple occlusion states, aligns the first facial model and / or the second facial model with the dental model of the user, and determines the model after the first facial model and the dental model are aligned and / or the model after the second facial model and the dental model are aligned as target models, wherein the target model has a corresponding relationship with the occlusion state of the dental model.

[0074] The model after the first facial model and the tooth model are aligned may include the first facial model and the tooth model, and the model after the second facial model and the tooth model are aligned may include the second facial model and the tooth model. The tooth model may be a three-dimensional model of the user's oral cavity. Exemplarily, the tooth model may be obtained by scanning the user's oral cavity using an oral scanning device, or may be obtained by modeling using a simulation algorithm. The above examples of the method for obtaining the tooth model are only examples and are not limited to this in actual applications.

[0075] When the target model is a model formed by aligning a first facial model with a dental model, the expression state corresponding to the target model may be the expression state corresponding to the original point cloud of the reference frame; when the target model is a model formed by aligning a second facial model with a dental model, the expression state corresponding to the target model may be the expression state of the original point cloud of the non-reference frame corresponding to the second facial model.

[0076] If the target model is a model after the first facial model is aligned with the tooth model and / or a model after the second facial model is aligned with the tooth model, the target model can be non-rigidly registered with the acquired point cloud data, so that the first facial model or the second facial model in the target model can be transformed from the corresponding expression state to the expression state corresponding to the acquired point cloud data and the tooth model can be transformed from the original occlusion state to the occlusion state corresponding to the acquired point cloud data, wherein the occlusion state of the tooth model can be adjusted following the change of the expression state. The expression state corresponding to the driven target model corresponds to the expression state of the acquired point cloud data, and the occlusion state of the tooth model has a corresponding relationship with the occlusion state corresponding to the point cloud data.

[0077] In this embodiment, by determining the model after the first facial model is aligned with the tooth model and / or the model after the second facial model is aligned with the tooth model as the target model, the tooth details in the target model can be enriched, thereby improving the model accuracy of the target model. By driving the target model using the acquired point cloud data, the expression state and occlusion state of the target model can be adjusted, and the migration of the expression state and tooth morphology can be achieved, thereby achieving the tracking of the user's facial expression and teeth.

[0078] In the facial model reconstruction scheme of the embodiment of the present application, the multiple frames of original point clouds can be from the same scanning device. If the multiple frames of original point clouds are obtained by scanning the user's face with changing expressions, the first facial model matching the expression state of the original point cloud of the reference frame can be accurately reconstructed according to the original point cloud of the reference frame and the multiple frames of deformed point clouds with the same form. The expression state of the user's face can change, so the number of repeated scans can be reduced. Since aligning the first facial model to the original point cloud of each non-reference frame is the reverse idea of ​​reconstructing the first facial model, if the multiple frames of original point clouds correspond to multiple expression states, in the process of aligning the first facial model to the original point cloud of each non-reference frame, the partial association relationship in the process of reconstructing the first facial model can be used for alignment, so that the second facial model matching the expression state corresponding to the original point cloud of each non-reference frame can be quickly reconstructed, and then the facial model corresponding to each frame of the original point cloud of the user's face can be obtained, completing the reconstruction of the dynamic facial model. Since the number of repeated scans can be reduced and the second facial model matching the expression state corresponding to each frame of the original point cloud can be quickly reconstructed, the reconstruction efficiency of the dynamic facial model can be improved. If multiple frames of original point clouds are obtained by scanning the faces of users with fixed and changing expressions, since the original point clouds obtained when the expression state of the user's face remains fixed have the same form, the original point cloud obtained when the expression state of the user's face remains fixed can be directly determined as the original point cloud of the reference frame, so that the first facial model matching the expression state corresponding to the original point cloud of the reference frame can be quickly reconstructed. By aligning the first facial model to the original point cloud of each non-reference frame, the second facial model matching the expression state corresponding to each frame of the original point cloud can be quickly reconstructed, thereby improving the reconstruction efficiency of the dynamic facial model.

[0079] like Figure 4 , which is a flow chart of a method for generating a deformable point cloud provided in one embodiment of the present application.

[0080] S131, determining a deformation field corresponding to each first coordinate point, where the deformation field includes a plurality of deformation control points corresponding to each first coordinate point.

[0081] In some embodiments of the present application, the electronic device can sample multiple first coordinate points as deformation control points from the first coordinate points corresponding to the original point cloud of each non-reference frame, determine the deformation control point corresponding to each first coordinate point among the multiple deformation control points according to preset rules, and establish a corresponding relationship between each first coordinate point and the corresponding deformation control point, thereby completing the determination of the deformation field corresponding to each first coordinate point.

[0082] Among them, the number of deformation control points obtained by sampling can be customized, and this application does not impose any restrictions on this. The electronic device can use various sampling methods to sample the first coordinate point corresponding to the original point cloud of each non-reference frame to obtain multiple deformation control points. For example, the electronic device can use the nearest neighbor sampling method, uniform sampling method, voxel sampling method and other methods to sample multiple first coordinate points as deformation control points from the first coordinate point corresponding to the original point cloud of each non-reference frame.

[0083] The preset rule can be customized, and the present application does not impose any restrictions on this. For example, if the preset rule is the nearest point rule, the electronic device determines the deformation control point corresponding to each of the multiple deformation control points according to the nearest point rule, including: the electronic device calculates the first distance between each deformation control point and each of the first coordinate points, and determines the deformation control point corresponding to the first distance less than the first preset value among the multiple deformation control points as the deformation control point corresponding to each of the first coordinate points. Among them, the first distance can be Euclidean distance, geodesic distance (Geodesic) or other types of distances, and the first preset value can be customized, and the present application does not impose any restrictions on this.

[0084] In this embodiment, a plurality of first coordinate points are sampled from the first coordinate points corresponding to the original point cloud of each non-reference frame as deformation control points, which can reduce the amount of calculation.

[0085] S132: Determine a first dense correspondence relationship between the plurality of first coordinate points and the plurality of second coordinate points by densely registering the plurality of first coordinate points and the plurality of second coordinate points.

[0086] In this embodiment, the first dense correspondence includes multiple groups of point pairs, each of which includes first coordinate points and second coordinate points that correspond to each other. For dense registration of multiple first coordinate points and multiple second coordinate points, reference can be made to the description of methods such as the nearest point method based on point cloud distance and the dense matching method based on RGB optical flow, and this application will not repeat the description. Among them, the first coordinate point in the multiple groups of point pairs may include the multiple first coordinate points, or may only be part of the first coordinate points in the multiple first coordinate points, and this application does not limit this.

[0087] S133, determining the target deformation parameters of each deformation control point based on the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point.

[0088] In some embodiments of the present application, the electronic device can execute a solution process for the initial deformation parameters of each deformation control point, including: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function, repeating the solution process, updating the initial deformation parameters of each deformation control point, until the number of updates to the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stopping the solution process, and determining the updated initial deformation parameters as the target deformation parameters of each deformation control point.

[0089] The initial deformation parameters include a linear transformation matrix and a translation vector. The preset number of times and the preset threshold value can be customized, and this application does not limit this. For example, the preset number of times can be 20 times. The preset threshold value can be 0.1.

[0090] The electronic device may optimize the constraint function using an optimization algorithm, for example, the optimization algorithm may be a gradient descent method and a least square method, etc. The constraint residual is the value of the constraint function, and the constraint function may include a data term constraint and a smooth term constraint, wherein the value of the data term constraint may represent the gap or difference between the target coordinates of the first coordinate point in each set of point pairs and the initial coordinates of the second coordinate point in each set of point pairs, and the value of the smooth term constraint may represent the gap or difference between the target deformation parameter of each deformation control point and the target deformation parameter of the deformation control point adjacent to each deformation control point.

[0091] The above conditions for stopping the solution process are only examples and are not limited to this in practical applications. For example, if the update time of the initial deformation parameters of each deformation control point reaches a preset time, the electronic device can stop executing the above solution process and determine the updated initial deformation parameters as the target deformation parameters of each deformation control point, wherein the preset time can be customized.

[0092] For example, if minimizing the constraint residual is the optimization goal, the calculation of the target deformation parameter of each deformation control point can refer to the following formula (1):

[0093]

[0094] Where E represents the constraint function, E d Represents data item constraints, E s represents the smoothness constraint, Tj represents the target deformation parameter of the jth deformation control point obtained after minimizing the optimization constraint residual, A j Represents the linear transformation matrix in the target deformation parameters of the j-th deformation control point.

[0095] In this embodiment, since the deformation of each first coordinate point is often affected by the adjacent deformation control points, the constraint function is determined by using the deformation control points adjacent to each deformation control point, and the constraint function is minimized, which can reduce errors and ensure the accuracy of the target deformation parameters.

[0096] S134, determining the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of the plurality of deformation control points corresponding to each first coordinate point.

[0097] In some embodiments of the present application, since the target deformation parameters of each deformation control point include a linear transformation matrix and a translation vector, the electronic device can determine the candidate coordinates of each first coordinate point based on the initial coordinates of each first coordinate point and the linear transformation matrix and translation vector of each deformation control point corresponding to each first coordinate point, thereby obtaining multiple candidate coordinates, and determine the target coordinates of each first coordinate point based on the multiple candidate coordinates.

[0098] Among them, the electronic device calculates the initial coordinates of each first coordinate point and the target deformation parameters of a deformation control point to obtain a candidate coordinate of each first coordinate point, and calculates the initial coordinates of each first coordinate point and the target deformation parameters of multiple deformation control points to obtain multiple candidate coordinates of each first coordinate point. The electronic device can determine the average value or weighted average value among the multiple candidate coordinates of each first coordinate point as the target coordinate of each first coordinate point. The weighted weight can be customized, and this application does not limit this.

[0099] For example, the calculation formula of the candidate coordinates can refer to formula (2):

[0100] Y=A i X+t i ; (2)

[0101] Wherein, Y represents the candidate coordinates of each first coordinate point for the i-th deformation control point among the corresponding multiple deformation control points, A i represents the linear transformation matrix in the target deformation parameter of the i-th deformation control point, X represents the initial coordinates in the target deformation parameter of each first coordinate point, t i The translation vector representing the linear transformation matrix of the i-th deformation control point.

[0102] S135 , determining a deformed point cloud corresponding to the original point cloud of each non-reference frame based on the multiple first coordinate points and the target coordinates of each first coordinate point.

[0103] In some embodiments of the present application, the electronic device transforms each of the first coordinate points from the corresponding initial coordinates to the corresponding target coordinates, and multiple first coordinate points at the corresponding target coordinates can constitute a deformed point cloud corresponding to the original point cloud of each non-reference frame.

[0104] like Figure 5 FIG. 1 is a flowchart of a method for constructing a constraint function provided in an embodiment of the present application, comprising the following steps:

[0105] S1331: construct an objective function of target deformation parameters of a plurality of deformation control points corresponding to the first coordinate point in each group of point pairs according to the initial coordinates of the first coordinate point in each group of point pairs.

[0106] In some embodiments of the present application, the target function corresponds to the target coordinates of the first coordinate point in each set of point pairs. For example, the target function can refer to the following formula (3):

[0107] f(v n )=∑ m [w m (A m v n +t m )]; (3)

[0108] Among them, f(v n ) represents the objective function corresponding to the target coordinates of the first coordinate point in the nth set of point pairs, A m Represents the linear transformation matrix in the target deformation parameters of the mth deformation control point corresponding to the first coordinate point in the nth point pair, t m Represents the translation vector in the target deformation parameter of the mth deformation control point corresponding to the first coordinate point in the nth point pair, w m Indicates the preset weight corresponding to the mth deformation control point corresponding to the first coordinate point in the nth point pair.

[0109] S1332: construct data item constraints according to the initial coordinates of the second coordinate point in each group of point pairs and the objective function.

[0110] For example, the construction of data item constraints can refer to the following formula (4):

[0111]

[0112] Among them, E d Represents data item constraints, vn represents the initial coordinates of the first coordinate point in the nth set of point pairs, f(v n ) represents the objective function corresponding to the target coordinates of the first coordinate point in the nth set of point pairs, p n Represents the initial coordinates of the second coordinate point in the nth point pair in the dense correspondence.

[0113] In this embodiment, since the value of the data item constraint can represent the gap or difference between the target coordinates of the first coordinate point in each group of point pairs and the initial coordinates of the second coordinate point in each group of point pairs, by optimizing the data item constraint, the gap or difference between the target coordinates of the first coordinate point in each group of point pairs and the initial coordinates of the second coordinate point in each group of point pairs can be narrowed.

[0114] S1333: Construct a smoothing term constraint on a target deformation parameter of each deformation control point and a target deformation parameter of each deformation control point adjacent to each deformation control point.

[0115] In some embodiments of the present application, the electronic device may calculate a second distance between each deformation control point and other deformation control points other than each deformation control point, and determine a deformation control point corresponding to a second distance less than a second preset value as a deformation control point adjacent to each deformation control point, wherein the second distance may be a Euclidean distance or other types of distances. The second preset value may be custom set, and the present application does not limit this.

[0116] For example, the construction of the smoothness term constraint can refer to the following formulas (5)-(7):

[0117]

[0118] T j ={A j , t j}; (6)

[0119] T k ={A k , t k}; (7)

[0120] Among them, E s represents the smoothness constraint, T j represents the target deformation parameter of the jth deformation control point, A j Represents the linear transformation matrix in the target deformation parameters of the j-th deformation control point, t j represents the translation vector in the target deformation parameter of the j-th deformation control point, N(j) represents all deformation control points adjacent to the j-th deformation control point, T krepresents the target deformation parameter of the kth deformation control point in N(j), A k Represents the linear transformation matrix in the target deformation parameters of the kth deformation control point, t k Represents the translation vector in the target deformation parameters of the k-th deformation control point.

[0121] In this embodiment, since the value of the smoothness term constraint can represent the gap or difference between the target deformation parameters of each deformation control point and the target deformation parameters of the deformation control points adjacent to each deformation control point, by optimizing the smoothness term constraint, the gap or difference between the target deformation parameters of each deformation control point and the target deformation parameters of the deformation control points adjacent to each deformation control point can be narrowed.

[0122] S1334, determine the constraint function according to the data item constraint and the smoothing item constraint.

[0123] In some embodiments of the present application, the method for constructing the constraint function may refer to the above formula (1).

[0124] In other embodiments of the present application, the electronic device may perform weighted operations on the data item constraints and the smoothing item constraints to obtain a constraint function.

[0125] like Figure 6 1 is a functional module diagram of a facial model reconstruction device provided by an embodiment of the present application. The facial model reconstruction device 11 includes an acquisition module 110, a determination module 111 and a reconstruction module 112. The module / unit referred to in the present application refers to a Figure 7 A series of computer readable instruction segments acquired by the processor 103 in the embodiment and capable of completing fixed functions, which are stored in Figure 7 In the memory 102 in the present embodiment, the functions of each module / unit will be described in detail in the subsequent embodiments.

[0126] The acquisition module 110 is used to acquire multiple frames of original point clouds of the user's face.

[0127] The determination module 111 is used to determine the original point cloud of the reference frame and the original point cloud of the non-reference frame in the multiple frames of original point cloud.

[0128] In some embodiments of the present application, a multi-frame original point cloud is obtained based on scanning the user's face with changing expressions. If the original point cloud of the reference frame and the original point cloud of the non-reference frame are determined in the process of obtaining the multi-frame original point cloud, the original point cloud of the reference frame is the first frame original point cloud obtained. Alternatively, if the original point cloud of the reference frame and the original point cloud of the non-reference frame are determined after obtaining the multi-frame original point cloud, the original point cloud of the reference frame is any frame original point cloud selected from the multi-frame original point cloud, and the original point cloud of the non-reference frame is other original point clouds in the multi-frame original point cloud except the original point cloud of the reference frame.

[0129] In some embodiments of the present application, multiple frames of original point clouds are obtained based on scanning the user's face with fixed expressions and changing expressions. The original point cloud of the reference frame is obtained when the expression state of the user's face remains fixed, and the original point cloud of the non-reference frame is obtained when the expression state of the user's face changes dynamically.

[0130] The reconstruction module 112 is used to reconstruct a first facial model that matches the expression state corresponding to the original point cloud of the reference frame based on the multiple frames of original point cloud.

[0131] In some embodiments of the present application, if multiple frames of original point clouds are obtained based on scanning the user's face with changing expressions, the first facial model is reconstructed based on the original point cloud of the reference frame and the deformed point cloud corresponding to the original point cloud of each non-reference frame, wherein the multiple frames of deformed point clouds corresponding to the original point cloud of the non-reference frame have the same morphology as the original point cloud of the reference frame.

[0132] In some embodiments of the present application, if the original point cloud of the reference frame is the first frame original point cloud acquired, the reconstruction module 112 is also used to align the original point cloud of each non-reference frame with the target point cloud corresponding to the original point cloud of each non-reference frame to obtain the deformed point cloud corresponding to the original point cloud of each non-reference frame, the target point cloud corresponding to the second frame original point cloud in the original point cloud of the non-reference frame is the first frame original point cloud, and the target point cloud corresponding to the nth frame original point cloud in the original point cloud of the non-reference frame is obtained by fusing the target point cloud corresponding to the n-1th frame original point cloud with the deformed point cloud corresponding to the n-1th original point cloud, wherein 3≤n≤m, and m represents the number of the multiple frames of original point clouds.

[0133] In other embodiments of the present application, if the original point cloud of the reference frame is any frame original point cloud selected from the multiple frames of original point clouds, the reconstruction module 112 is also used to select an original point cloud from the original point cloud of the non-reference frame, and use the original point cloud of the reference frame as the target point cloud corresponding to the selected original point cloud, and align the selected original point cloud with the corresponding target point cloud to obtain a deformed point cloud corresponding to the selected original point cloud, and fuse the deformed point cloud corresponding to the selected original point cloud with the corresponding target point cloud to obtain a first fused point cloud, and each first fused point cloud is used as the target point cloud corresponding to the next frame of original point cloud selected, and the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame is repeated until all the original point clouds in the original point cloud of the non-reference frame are selected to obtain the deformed point cloud corresponding to the original point cloud of each non-reference frame.

[0134] In some embodiments of the present application, the original point cloud of each non-reference frame includes a plurality of first coordinate points, and the target point cloud corresponding to the original point cloud of each non-reference frame includes a plurality of second coordinate points. The reconstruction module 112 is further used to determine a deformation field corresponding to each first coordinate point, and the deformation field includes a plurality of deformation control points corresponding to each first coordinate point. By densely registering the plurality of first coordinate points and the plurality of second coordinate points, a first dense correspondence relationship between the plurality of first coordinate points and the plurality of second coordinate points is determined, and the first dense correspondence relationship includes a plurality of groups of point pairs, wherein each group of point pairs includes first coordinate points and second coordinate points corresponding to each other. According to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and the multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, the target deformation parameters of each deformation control point are determined; according to the initial coordinates of each first coordinate point and the target deformation parameters of the multiple deformation control points corresponding to each first coordinate point, the target coordinates of each first coordinate point are determined; based on the multiple first coordinate points and the target coordinates of each first coordinate point, the deformed point cloud corresponding to the original point cloud of each non-reference frame is determined.

[0135] In some embodiments of the present application, the reconstruction module 112 is also used to execute a solution process for the initial deformation parameters of each deformation control point, including: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs and multiple deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and the deformation control points adjacent to each deformation control point, determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function, repeating the solution process, updating the initial deformation parameters of each deformation control point, until the number of updates to the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stopping the solution process, and determining the updated initial deformation parameters as the target deformation parameters of each deformation control point.

[0136] In other embodiments of the present application, if multiple frames of original point clouds are obtained based on scanning the face of a user with a fixed expression and a user with a changing expression, the first facial model is reconstructed based on the original point cloud of the reference frame.

[0137] The reconstruction module 112 is further configured to reconstruct a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame.

[0138] In some embodiments of the present application, the reconstruction module 112 is further used to fuse multiple frames of deformed point clouds to obtain a second fused point cloud, and reconstruct the first facial model based on the second fused point cloud and the original point cloud of the reference frame using a preset reconstruction algorithm.

[0139] The reconstruction module 112 is further configured to reconstruct a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame.

[0140] In some embodiments of the present application, the reconstruction module 112 is further used to densely align the first facial model with the original point cloud of each non-reference frame according to the first dense correspondence relationship, determine the second dense correspondence relationship between the first facial model and the original point cloud of each non-reference frame, and align the first facial model with the original point cloud of each non-reference frame according to the second dense correspondence relationship to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame.

[0141] In some embodiments of the present application, the facial model reconstruction device 11 also includes a driving module for obtaining a user's tooth model, the tooth model has multiple occlusion states, and if a change in the facial expression state of the user is detected, the point cloud data of the user's face is obtained, and according to the point cloud data, the target model is driven to change from the corresponding expression state to the expression state corresponding to the point cloud data, and the target model includes at least one of a first facial model, a second facial model, a model after the first facial model is aligned with the user's tooth model, and a model after the second facial model is aligned with the tooth model.

[0142] like Figure 7 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 10 can be a mobile phone, a tablet computer, a laptop computer, or other electronic device. The present embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0143] exist Figure 7 In the embodiment, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the input / output interface 104 through the bus 105.

[0144] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (wireless fidelity, Wi-Fi), Bluetooth (Bluetooth, BT), mobile communication network, frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR), etc.

[0145] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data. The random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0146] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 103. The non-volatile memory may include a disk storage device and a flash memory. For example, the flash memory may be a flash memory (Nand Flash).

[0147] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions, and when the multiple instructions are executed by the processor 103, the facial model reconstruction method executed on the electronic device 10 can be implemented.

[0148] In other embodiments, Figure 7 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0149] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0150] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned facial model reconstruction method.

[0151] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0152] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the input / output interface 104 in the electronic device 10 .

[0153] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0154] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0155] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0156] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0158] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0160] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present application is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0161] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in this application can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A facial model reconstruction method, characterized in that: The method comprises: Obtain multi-frame original point clouds of the user's face; Determine an original point cloud of a reference frame and an original point cloud of a non-reference frame in the multiple frames of original point clouds; Reconstructing a first facial model that matches the expression state corresponding to the original point cloud of the reference frame according to the multiple frames of original point cloud; By aligning the first facial model to the original point cloud of each non-reference frame, a second facial model matching the expression state corresponding to the original point cloud of each non-reference frame is reconstructed.

2. The facial model reconstruction method according to claim 1, characterized in that: The multi-frame original point cloud is obtained based on scanning the user's face with changing expressions. If the original point cloud of the reference frame and the original point cloud of the non-reference frame are determined in the process of obtaining the multi-frame original point cloud, the original point cloud of the reference frame is the first frame original point cloud obtained, or, if the original point cloud of the reference frame and the original point cloud of the non-reference frame are determined after obtaining the multi-frame original point cloud, the original point cloud of the reference frame is any frame original point cloud selected from the multi-frame original point cloud, and the original point cloud of the non-reference frame is other original point clouds in the multi-frame original point cloud except the original point cloud of the reference frame. The first facial model is reconstructed based on the original point cloud of the reference frame and the deformed point cloud corresponding to the original point cloud of each non-reference frame, wherein the multi-frame deformed point cloud corresponding to the original point cloud of the non-reference frame has the same shape as the original point cloud of the reference frame.

3. The facial model reconstruction method according to claim 2, characterized in that: If the original point cloud of the reference frame is the first frame of the original point cloud acquired, the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame includes: The original point cloud of each non-reference frame is aligned with the target point cloud corresponding to the original point cloud of each non-reference frame to obtain the deformed point cloud corresponding to the original point cloud of each non-reference frame, the target point cloud corresponding to the second frame original point cloud in the original point cloud of the non-reference frame is the first frame original point cloud, and the target point cloud corresponding to the n-th frame original point cloud in the original point cloud of the non-reference frame is obtained by fusing the target point cloud corresponding to the n-1-th frame original point cloud with the deformed point cloud corresponding to the n-1-th original point cloud, wherein 3≤n≤m, and m represents the number of the multiple frames of original point clouds.

4. The facial model reconstruction method according to claim 2, characterized in that: If the original point cloud of the reference frame is any frame of original point cloud selected from the multiple frames of original point clouds, the method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame includes: Selecting an original point cloud from the original point cloud of the non-reference frame, using the original point cloud of the reference frame as a target point cloud corresponding to the selected original point cloud, aligning the selected original point cloud with the corresponding target point cloud to obtain a deformed point cloud corresponding to the selected original point cloud, fusing the deformed point cloud corresponding to the selected original point cloud with the corresponding target point cloud to obtain a first fused point cloud, each first fused point cloud being used as a target point cloud corresponding to the selected next frame of original point cloud; The method for generating the deformed point cloud corresponding to the original point cloud of each non-reference frame is repeatedly executed until all the original point clouds in the original point cloud of the non-reference frame are selected, thereby obtaining the deformed point cloud corresponding to the original point cloud of each non-reference frame.

5. The facial model reconstruction method according to claim 3 or 4, characterized in that: The original point cloud of each non-reference frame includes a plurality of first coordinate points, the target point cloud corresponding to the original point cloud of each non-reference frame includes a plurality of second coordinate points, and the alignment method between the original point cloud of each non-reference frame and the corresponding target point cloud includes: Determine a deformation field corresponding to each first coordinate point, wherein the deformation field includes a plurality of deformation control points corresponding to each first coordinate point; Determine a first dense correspondence relationship between the plurality of first coordinate points and the plurality of second coordinate points by densely registering the plurality of first coordinate points and the plurality of second coordinate points, wherein the first dense correspondence relationship includes a plurality of groups of point pairs, wherein each group of point pairs includes first coordinate points and second coordinate points corresponding to each other; Determine a target deformation parameter of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and a plurality of deformation control points corresponding to the first coordinate point in each group of point pairs, as well as each deformation control point and deformation control points adjacent to each deformation control point; Determining the target coordinates of each first coordinate point according to the initial coordinates of each first coordinate point and the target deformation parameters of a plurality of deformation control points corresponding to each first coordinate point; Based on the multiple first coordinate points and the target coordinates of each of the first coordinate points, a deformed point cloud corresponding to the original point cloud of each non-reference frame is determined.

6. The facial model reconstruction method according to claim 5, characterized in that: The method for determining the target deformation parameters of each deformation control point includes: The process of solving the initial deformation parameters of each deformation control point includes: constructing a constraint function about the initial deformation parameters of each deformation control point according to the initial coordinates of the first coordinate point in each group of point pairs, the initial coordinates of the second coordinate point in each group of point pairs, and a plurality of deformation control points corresponding to the first coordinate point in each group of point pairs, and each deformation control point and deformation control points adjacent to each deformation control point, and determining the initial deformation parameters corresponding to each deformation control point by optimizing the constraint function; Repeat the solution process to update the initial deformation parameters of each deformation control point until the number of updates of the initial deformation parameters of each deformation control point reaches a preset number or the constraint residual of the constraint function is less than or equal to a preset threshold, stop executing the solution process, and determine the updated initial deformation parameters as the target deformation parameters of each deformation control point.

7. The facial model reconstruction method according to claim 5, characterized in that: The step of reconstructing a second facial model matching the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame comprises: Performing dense registration on the first facial model and the original point cloud of each non-reference frame according to the first dense correspondence relationship, and determining a second dense correspondence relationship between the first facial model and the original point cloud of each non-reference frame; According to the second dense correspondence, the first facial model is aligned with the original point cloud of each non-reference frame to obtain a second facial model that matches the expression state corresponding to the original point cloud of each non-reference frame.

8. The facial model reconstruction method according to claim 2, characterized in that: The step of reconstructing a first facial model matching the expression state corresponding to the original point cloud of the reference frame according to the multiple frames of original point cloud comprises: Fusing the multiple frames of deformed point clouds to obtain a second fused point cloud; The first facial model is reconstructed based on the second fused point cloud and the original point cloud of the reference frame using a preset reconstruction algorithm.

9. The facial model reconstruction method according to claim 1, characterized in that: The multiple frames of original point clouds are obtained based on scanning the user's face with fixed expressions and changing expressions. The original point cloud of the reference frame is obtained when the expression state of the user's face remains fixed. The original point cloud of the non-reference frame is obtained when the expression state of the user's face changes dynamically. The first facial model is reconstructed based on the original point cloud of the reference frame.

10. The facial model reconstruction method according to claim 1, characterized in that: The method further comprises: If a change in the facial expression state of the user is detected, obtaining point cloud data of the user's face; According to the point cloud data, drive the target model to change from a corresponding expression state to an expression state corresponding to the point cloud data, the target model including at least one of the first facial model, the second facial model, a model after the first facial model is aligned with a user's dental model, and a model after the second facial model is aligned with a user's dental model.

11. A facial model reconstruction device, characterized in that: The device comprises: An acquisition module is used to acquire multiple frames of original point clouds of the user's face; A determination module, used to determine the original point cloud of the reference frame and the original point cloud of the non-reference frame in the multiple frames of original point cloud; A reconstruction module, configured to reconstruct a first facial model matching the expression state corresponding to the original point cloud of the reference frame according to the multiple frames of original point cloud; The reconstruction module is further used to reconstruct a second facial model matching the expression state corresponding to the original point cloud of each non-reference frame by aligning the first facial model to the original point cloud of each non-reference frame.

12. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one instruction; and A processor executes the at least one instruction to implement the facial model reconstruction method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor in an electronic device, the facial model reconstruction method according to any one of claims 1 to 10 is implemented.

Images