Method for 3D facial reconstruction, apparatus, device and storage medium for 3D facial reconstruction

Abstract

A method, an apparatus, a device and a storage medium for 3D face reconstruction are disclosed, which includes: estimating dynamic reconstruction parameters of a current video frame in 3D face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model in response to estimating steady-state reconstruction parameters of a current video frame in 3D face reconstruction by a pre-constructed teacher network model; and reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into the pre-constructed 3D morphable model.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application with application number 202011583279.2 filed with the China Patent Office on December 28, 2020, the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD The embodiments of the present application relate to the field of image processing technology, such as a method for 3D face reconstruction, an apparatus, device and storage medium for 3D face reconstruction. [Background technology]

[0003] With the development of video technology, there is an increasing need to create realistic face models in entertainment applications such as facial animation, facial recognition, and Augmented Reality (AR), which have the need to display facial images. In a video recorded with the same face, there are usually multiple facial expressions in different video frames, so when reconstructing a 3D face, it is necessary to track the facial expressions in each video frame in order to place the corresponding facial expressions on the reconstructed 3D face and increase the realism of the 3D face reconstruction.

[0004] Regarding 3D face reconstruction in each video frame in a video recorded by a user, usually, each video frame is treated as an independent face image, and a pre-constructed neural network model is used to predict the reconstruction parameters that each video frame needs to refer to when reconstructing a 3D face, and then the reconstruction parameters are used to perform 3D face reconstruction for each video frame. In this case, due to the continuity between video frames, if the reconstruction parameters predicted under adjacent video frames become unstable, jitter will occur in the reconstructed 3D face, and since the prediction of the reconstruction parameters in each video frame is the prediction of the reconstruction parameters for the same face, the amount of unnecessary prediction calculation increases when predicting the reconstruction parameters for each video frame, which affects the efficiency of 3D face reconstruction. Summary of the Invention [Means for solving the problem]

[0005] In the embodiments of the present application, a method for 3D face reconstruction, an apparatus for 3D face reconstruction, a device and a storage medium are disclosed.

[0006] In a first aspect, an embodiment of the present application relates to a method for 3D face reconstruction, the method comprising: In response to estimating steady-state reconstruction parameters of a current video frame in 3D face reconstruction by a pre-constructed teacher network model, estimating dynamic reconstruction parameters of the current video frame in 3D face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model; reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; Includes.

[0007] In a second aspect, an embodiment of the present application relates to an apparatus for 3D face reconstruction, the apparatus comprising: a reconstruction parameter estimation module configured to, in response to estimating steady-state reconstruction parameters of a current video frame in three-dimensional face reconstruction by a pre-constructed teacher network model, estimate dynamic reconstruction parameters of the current video frame in three-dimensional face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model; a 3D face reconstruction module configured to reconstruct a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; Includes.

[0008] In a third aspect, an embodiment of the present application relates to a computing device, the computing device comprising: one or more processors; a storage device configured to store one or more programs; Including, The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for 3D face reconstruction described in any embodiment of the present application.

[0009] In a fourth aspect, an embodiment of the present application relates to a computer readable storage medium having a computer program stored thereon, the computer program implementing, when executed by a processor, a method for 3D face reconstruction as described in any embodiment of the present application.

[0010] Other characteristics, objects and advantages of the embodiments of the present application will become more apparent on reading the following description of non-limiting examples, given with reference to the drawings, in which: [Brief description of the drawings]

[0011] [Figure 1A] 1 is a flowchart of a method for 3D face reconstruction according to Example 1 of the present application. [Figure 1B] FIG. 2 is a schematic diagram of a 3D face reconstruction process according to Example 1 of the present application. [Figure 1C] FIG. 2 is a structural schematic diagram of a 3D morphable model for 3D face reconstruction in the method according to Example 1 of the present application. [Figure 1D] FIG. 2 is a schematic diagram of a bilinear face model structure in a 3D morphable model in the method according to Example 1 of the present application. [Figure 2A] 4 is a flowchart of a method for 3D face reconstruction according to Example 2 of the present application. [Figure 2B] FIG. 1 is a schematic diagram of a 3D face reconstruction process according to Example 2 of the present application. [Figure 3A]11 is a flowchart of a method for training a student network model in 3D face reconstruction according to Example 3 of the present application. [Figure 3B] FIG. 11 is a schematic diagram of a training process of a student network model according to Example 3 of the present application; [Figure 3C] FIG. 11 is a schematic distribution diagram of key feature points in the method according to Example 3 of the present application. [Figure 4] FIG. 11 is a structural schematic diagram of an apparatus for 3D face reconstruction according to Example 4 of the present application. [Diagram 5] FIG. 11 is a structural schematic diagram of a computing device according to Example 5 of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present application will be described below with reference to the drawings and examples. It is understood that the specific examples described in this specification do not limit the present application, but are used only to describe the present application. In addition, in order to facilitate the description, the drawings show only parts related to the present application, not the entire configuration. In addition, the examples in the present application and the features in the examples can be combined with each other if not inconsistent.

[0013] Example 1

[0014] 1A is a flowchart of a method for 3D face reconstruction according to a first embodiment of the present application, in which 3D face reconstruction can be performed on multiple video frames having the same face. The method for 3D face reconstruction according to the first embodiment of the present application may be performed by an apparatus for 3D face reconstruction according to an embodiment of the present application, which can be realized by software and / or hardware and integrated into a computer device that executes the method.

[0015] For example, referring to FIG. 1A, the method may include the following steps.

[0016] In S110, in response to estimating steady-state reconstruction parameters of a current video frame in 3D face reconstruction by a pre-constructed teacher network model, dynamic reconstruction parameters of a current video frame in 3D face reconstruction are estimated by inputting the current video frame into a student network model obtained after distillation on the teacher network model.

[0017] For example, in entertainment applications such as face animation, face recognition and augmented reality, which have the need for face image display, 3D face reconstruction is performed on a video frame in which a face image exists in a video recorded by a user, and an entertainment interactive function is performed under a certain face effect. In this case, for the 3D face reconstruction, at least one reconstruction parameter can control a standard face shape to perform corresponding deformation and facial expression change, thereby generating a corresponding 3D face and improving the detailed display of the light illumination situation, posture angle, etc. in the 3D face. As can be seen from the above, the accuracy of the estimation of each reconstruction parameter for each video frame to be reconstructed can directly affect the realism of the 3D face corresponding to the reconstruction of the video frame, and therefore, in order to make the reconstructed 3D face more real, it is required to accurately estimate each reconstruction parameter specified for the 3D face reconstruction from the video frame to be reconstructed. For example, the reconstruction parameters in this embodiment may include a deformation parameter representing a change in face shape, an expression parameter representing a change in facial expression, an albedo parameter representing a change in face albedo, a light illumination parameter representing a change in light illumination of the face, a position parameter representing the movement of the face, and a rotation parameter representing the posture of the face head. Here, the albedo parameters include RGB color information of the two-dimensional face image to be reconstructed.

[0018] When performing 3D facial reconstruction for each video frame in a video recorded by a user, a corresponding neural network model is usually pre-trained to accurately estimate the reconstruction parameters that each to-be-reconstructed video frame needs to refer to when reconstructing a 3D face, and then the estimated reconstruction parameters are used to perform 3D facial reconstruction for the corresponding video frame. However, in a video recorded by a user, there is usually the same face in consecutive video frames, that is, facial expressions in consecutive video frames are tracked. In this case, a pre-trained neural network model is used to estimate each reconstruction parameter of each video frame in 3D face reconstruction, which may cause the problem of jitter in the 3D face reconstructed by consecutive video frames. And even if the neural network model estimates each reconstruction parameter of each video frame, there is a large amount of computation overhead in the estimation by the model. In this case, considering that there is usually the same face in consecutive video frames, the reconstruction parameters indicating changes in facial identity and texture, etc., referenced by consecutive video frames when reconstructing a 3D face hardly change, and only the reconstruction parameters indicating changes in facial expression and light illumination, etc. change. Therefore, in this embodiment, each reconstruction parameter of each video frame in 3D face reconstruction is divided into steady-state reconstruction parameters (e.g., reconstruction parameters indicating changes in facial identity, albedo, etc.) that hardly change for the same face, and dynamic reconstruction parameters specific to each video frame (e.g., reconstruction parameters indicating changes in facial expression, light illumination, etc.).

[0019] The degree of variation of the dynamic reconstruction parameters across multiple video frames of the same face is higher than the degree of variation of the steady-state reconstruction parameters across multiple video frames of the same face.

[0020] In this case, a neural network model that can accurately estimate each reconstruction parameter of each video frame in 3D face reconstruction that has been pre-trained for consecutive video frames in a video recorded by a user can be used as a teacher network model pre-constructed in this embodiment, and the teacher network model estimates each reconstruction parameter of the first video frame of the consecutive video frames or the first few video frames in 3D face reconstruction, and among the estimated reconstruction parameters, reconstruction parameters that hardly change for the same face are saved as steady-state reconstruction parameters of the subsequent video frames of the consecutive video frames in 3D face reconstruction, so that there is no need to estimate the corresponding steady-state reconstruction parameters when the subsequent video frames reconstruct a 3D face, and it is only necessary to estimate the active reconstruction parameters in 3D face reconstruction, which greatly reduces the amount of parameter estimation when reconstructing a 3D face, thereby improving the efficiency of 3D face reconstruction.

[0021] In this embodiment, considering that the pre-constructed teacher network model can accurately estimate each reconstruction parameter of each video frame in 3D face reconstruction, in this case, as shown in FIG. 1B, in order to ensure the accuracy of the estimation of the dynamic reconstruction parameters of each video frame in 3D face reconstruction, a refinement analysis is performed on the estimation function for the dynamic reconstruction parameters in the teacher network model by knowledge distillation, and the function of accurately estimating the dynamic reconstruction parameters of each video frame in the teacher network model is transferred to the student network model, so that the student network model can be used to accurately estimate the dynamic reconstruction parameters of each video frame in 3D face reconstruction, but does not estimate the steady-state reconstruction parameters of the video frame in 3D face reconstruction, thereby reducing the parameter estimation amount of 3D face reconstruction.

[0022] In one embodiment, when performing 3D facial reconstruction on each video frame in a video recorded by a user, it can first be determined whether the steady-state reconstruction parameters of the current video frame in 3D facial reconstruction have been estimated in advance by the teacher network model. If the steady-state reconstruction parameters of the current video frame in 3D facial reconstruction have been estimated by the teacher network model, the current video frame is directly input into the student network model obtained after knowledge distillation is performed on the teacher network model, and the active reconstruction parameters of the current video frame in 3D facial reconstruction are accurately estimated by the student network model, so that each reconstruction parameter of the current video frame in 3D facial reconstruction can be obtained. Then, each obtained reconstruction parameter can be directly used to perform 3D facial reconstruction on the current video frame.

[0023] At S120, a 3D face corresponding to the current video frame is reconstructed by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model.

[0024] In one example, after the dynamic reconstruction parameters of the current video frame in 3D facial reconstruction are estimated by the student network model, the steady-state reconstruction parameters of the current video frame estimated by the teacher network model in 3D facial reconstruction and the dynamic reconstruction parameters can be directly input into a pre-constructed 3D morphable model in a batch, and the 3D morphable model uses the steady-state reconstruction parameters and the dynamic reconstruction parameters to model the 3D facial shape and appearance for the current video frame, and after modeling, the light illumination information of the corresponding facial expression is adjusted to reconstruct the 3D face corresponding to the current video frame.

[0025] Illustratively, the 3D morphable model in this embodiment is composed of a bilinear face model and a Principal Component Analysis (PCA) model, and as shown in FIG. 1C, the bilinear face model is mainly used to model changes in facial contour and expression in the 3D face reconstruction process, and the PCA model is mainly used to model changes in facial albedo in the 3D face reconstruction process.

[0026] In the bilinear face model, a core tensor of a 3D face is defined, which is composed of three dimensions: 3D face mesh vertices, face contour (which can correspond to face identity), and facial expression. In this case, as shown in FIG. 1D, the bilinear face model is

number

number

number

number

number

number

[0027] In this embodiment, in order to accurately construct the core tensor defined in the bilinear face model, 3D face scan data is collected in a multidimensional data source, and the corresponding core tensor is obtained by performing high-order singular value decomposition on the 3D face reconstructed from the 3D face scan data in the facial contour dimension and the facial expression dimension.

[0028] For example, a depth camera can be used to collect a large amount of face data of individuals of different races, ages, and genders in advance, where one expressionless face data and face data with multiple different expressions are collected for each individual as the 3D face scan data in this embodiment, and for each 3D face scan data, a standard face network including 20084 mesh vertices is used to perform a corresponding deformation, so that the deformed face network is as close as possible to the 3D face scan data, and each facial feature point in the deformed face network corresponds one-to-one to a facial feature point in the 3D face scan data, thereby to provide a standard face network for each individual. In this case, assuming that 3D face scan data of N individuals are collected, a face mesh model including N individuals and each individual including 46 different facial expressions can be obtained, and each face mesh model includes a face database of 20084 mesh vertices. After matching the face database, a 3D tensor of 60252×N×46 can be obtained. In this case, a high-order singular value decomposition operation is performed on the 3D tensor with the individual dimension (N) and the facial expression dimension (46) to obtain the corresponding core tensor.

number

[0029] In the single PCA model in this embodiment, a principal component basis representing the average face albedo and the change in face albedo is defined.

number

number

number

number

[0030] In addition, in order to guarantee the 3D detailed features during 3D face reconstruction, the 3D morphable model of this embodiment includes a lighting parameter that represents the lighting change of the face.

number

number

number

number

[0031] Therefore, in order to accurately reconstruct a 3D face, in this embodiment, each reconstruction parameter specified for 3D face reconstruction is

number

number

number

number

[0032] The number of each reconstruction parameter described in this embodiment is merely an example, and specifically can be set according to the corresponding reconstruction requirements, and this embodiment does not limit this.

[0033] For example, in this embodiment, the reconstruction process of performing 3D face reconstruction using a 3D morphable model composed of a bilinear face model and a PCA model is as follows: reconstruction parameters representing changes in facial shape among the steady-state reconstruction parameters and reconstruction parameters representing changes in facial expression among the dynamic reconstruction parameters are input into the bilinear face model, and a corresponding 3D deformed face is obtained by adjusting the core tensor; the 3D deformed face and reconstruction parameters among the steady-state reconstruction parameters and the dynamic reconstruction parameters that match the principal component basis in the PCA model are input into the PCA model; and albedo correction is performed on the 3D deformed face based on the facial average albedo to obtain a reconstructed 3D face.

[0034] For example, after the steady-state reconstruction parameters and dynamic reconstruction parameters of the current video frame in 3D face reconstruction are estimated, the steady-state reconstruction parameters and dynamic reconstruction parameters are input collectively into a 3D morphable model to perform deformation and facial expression adjustment on the 3D standard face. In this case, the 3D morphable model is composed of a bilinear face model and a PCA model, and different models have different reconstruction functions. The bilinear face model is mainly used to model changes in facial contour and facial expression in the 3D face reconstruction process, and the PCA model is mainly used to model changes in facial albedo in the 3D face reconstruction process. Therefore, the bilinear face model and the PCA model are used sequentially to perform 3D face reconstruction on the current video frame.

[0035] In this case, first, from the steady-state reconstruction parameters estimated for the current video frame, identity parameters representing the changes in facial shape defined for the bilinear face model are selected, and from the dynamic reconstruction parameters estimated for the current video frame, facial expression parameters representing the changes in facial expression defined for the bilinear face model are selected. Then, the selected identity parameters and facial expression parameters are input into the bilinear face model, and corresponding external shape changes and facial expression changes are performed on the core tensors defined therein using the model representation function of the bilinear face model to obtain a corresponding 3D deformed face. Finally, from the steady-state reconstruction parameters and dynamic reconstruction parameters estimated for the current video frame, reconstruction parameters that match the principal component basis in the PCA model are selected. Furthermore, the 3D deformed face and the selected reconstruction parameters are input into the PCA model collectively, and albedo correction is performed on the 3D deformed face on the average facial albedo using the model representation function of the PCA model to obtain a reconstructed 3D face.

[0036] In this embodiment, considering that consecutive video frames in the same video contain the same face, the teacher network model constructed in advance is used to collectively estimate the steady-state reconstruction parameters of each video frame in 3D face reconstruction, which hardly change. Then, when performing 3D face reconstruction on each current video frame, the student network model obtained after distillation on the teacher network model is directly used to estimate the dynamic reconstruction parameters of the current video frame in the 3D face reconstruction. In this case, the student network model is obtained after knowledge distillation on the teacher network model, and there is no need to individually calculate the steady-state reconstruction parameters of each video frame in 3D face reconstruction, which greatly reduces the calculation overhead of 3D face reconstruction, thereby improving the efficiency of 3D face reconstruction and ensuring the accuracy of the dynamic reconstruction parameters of the current video frame in 3D face reconstruction. After that, the steady-state reconstruction parameters and dynamic reconstruction parameters of the current video frame in the 3D face reconstruction are directly input into the pre-constructed 3D morphable model to reconstruct a 3D face corresponding to the current video frame. In this case, the steady-state reconstruction parameters of successive video frames in the 3D face reconstruction do not change, so as to avoid the jitter problem occurring in successive video frames in the 3D face reconstruction. At the same time, the dynamic reconstruction parameters of each video frame in the 3D face reconstruction are used to ensure the stability of the 3D face reconstruction of successive video frames, and then the individual reconstruction of the 3D face of each video frame is realized, thereby improving the accuracy of the 3D face reconstruction.

[0037] Example 2

[0038] FIG 2A is a flow chart of a method for 3D face reconstruction according to embodiment 2 of the present application, and FIG 2B is a schematic diagram of a 3D face reconstruction process according to embodiment 2 of the present application. This embodiment is optimized based on the above embodiment. As shown in FIG 2A, this embodiment mainly describes the estimation process of steady-state reconstruction parameters of a current video frame in 3D face reconstruction.

[0039] As shown in FIG. 2A, this embodiment may include the following steps.

[0040] In S210, by inputting related video frames having the same face as the current video frame into a teacher network model, overall reconstruction parameters of the related video frames in 3D face reconstruction are estimated, and steady-state reconstruction parameters among the overall reconstruction parameters are set as steady-state reconstruction parameters of the current video frame in 3D face reconstruction.

[0041] Illustratively, regarding the pre-estimation of the steady-state reconstruction parameters of the current video frame in 3D face reconstruction, in this embodiment, before reconstructing the 3D face of the current video frame, first determine the related video frame having the same face as the current video frame, and the related video frame in this embodiment may be the first video frame in the video in which the current video frame is located that has a face in the current video frame, or the first m video frames in the video in which the current video frame is located that have a face in the current video frame, where m represents the number of targets, that is, the related video frame may be one or more. In this case, the related video frame in the video in which the current video frame is located can be input into a teacher network model, and the teacher network model can accurately estimate the overall reconstruction parameters of the related video frame in the 3D face reconstruction. Then, the reconstruction parameters that hardly change in consecutive video frames, such as the identity parameter representing the change in face shape and the albedo parameter representing the change in face texture, among the overall reconstruction parameters, are used as the steady-state reconstruction parameters of the current video frame in the 3D face reconstruction of this embodiment.

[0042] Illustratively, m may be a positive integer greater than one.

[0043] In this case, for consecutive video frames having the same face in the video in which the current video frame is located, the first video frame of the consecutive video frames can be the related video frame, or several video frames can be the related video frames, and for 3D face reconstruction of the related video frames, each corresponding reconstruction parameter is estimated using a teacher network model, and corresponding steady-state reconstruction parameters are selected from the estimated overall reconstruction parameters as the steady-state reconstruction parameters of each video frame after the subsequent related video frame in 3D face reconstruction, that is, when other video frames other than the related video frames in the video to be reconstructed perform 3D face reconstruction as the current video frame, there are steady-state reconstruction parameters estimated by the teacher network model in 3D face reconstruction, and when the related video frame performs 3D face reconstruction as the current video frame, there are no steady-state reconstruction parameters estimated by the teacher network model in 3D face reconstruction, and each reconstruction parameter in 3D face reconstruction needs to be estimated by inputting it into the teacher network model.

[0044] For example, when the related video frames are the first m video frames having a face in the current video frame in the video where the current video frame is located, as shown in Fig. 2B, in this embodiment, the steady-state reconstruction parameters of the overall reconstruction parameters are set as the steady-state reconstruction parameters of the current video frame in the 3D face reconstruction, and may include respectively fusing the steady-state reconstruction parameters of the same type of the overall reconstruction parameters of each associated video frame to obtain the steady-state reconstruction parameters of the current video frame. That is, when there are multiple associated video frames in the current video frame, the overall reconstruction parameters of each associated video frame in the 3D face reconstruction can be estimated by the teacher network model, and then, in order to ensure the accuracy of the steady-state reconstruction parameters and avoid errors when estimating the steady-state reconstruction parameters using only one frame, the steady-state reconstruction parameters after fusion can be set as the steady-state reconstruction parameters of the current video frame by performing weighted addition on the steady-state reconstruction parameters of the same type of the overall reconstruction parameters of each associated video frame.

[0045] In S220, it is determined whether the steady-state reconstruction parameters of the current video frame in the 3D face reconstruction have been estimated by the pre-constructed teacher network model, and if the steady-state reconstruction parameters of the current video frame in the 3D face reconstruction have been estimated by the pre-constructed teacher network model, S230 is executed, and if the steady-state reconstruction parameters of the current video frame in the 3D face reconstruction have not been estimated by the pre-constructed teacher network model, S260 is executed.

[0046] In S230, the dynamic reconstruction parameters of the current video frame in the 3D face reconstruction are estimated by inputting the current video frame into the student network model obtained after distillation on the teacher network model.

[0047] In S240, a 3D face corresponding to the current video frame is reconstructed by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model.

[0048] At S250, a differentiable renderer is used to render a 3D face corresponding to the current video frame.

[0049] For example, by rendering a 3D face using a differentiable renderer, the texture and screen of the rendered 3D face can be made closer to the current video frame, improving the realism of the 3D face reconstruction.

[0050] In S260, in response to the steady-state reconstruction parameters of the current video frame in 3D face reconstruction not being estimated by the pre-constructed teacher network model, the current video frame is input into the teacher network model to estimate overall reconstruction parameters of the current video frame in 3D face reconstruction, and the overall reconstruction parameters of the current video frame in 3D face reconstruction are input into a 3D morphable model to reconstruct a 3D face corresponding to the current video frame.

[0051] For example, if the current video frame is a related video frame set in the video in which the current video frame is located, it indicates that the steady-state reconstruction parameters of the current video frame in 3D face reconstruction have not been estimated by the teacher network model, and the current video frame is directly input into the teacher network model, and the teacher network model accurately estimates each comprehensive reconstruction parameter of the current video frame in 3D face reconstruction, and the comprehensive reconstruction parameters of the current video frame in 3D face reconstruction are input into a 3D morphable model, and the 3D morphable model uses the comprehensive reconstruction parameters to perform deformation and facial expression adjustment on the standard 3D face, thereby reconstructing a 3D face corresponding to the current video frame.

[0052] In this embodiment, the teacher network model constructed in advance is used to collectively estimate the steady-state reconstruction parameters of each video frame in 3D face reconstruction, which hardly change. Then, when performing 3D face reconstruction on each current video frame, the student network model obtained after distillation on the teacher network model is directly used to estimate the dynamic reconstruction parameters of the current video frame in the 3D face reconstruction. In this case, the student network model is obtained after knowledge distillation on the teacher network model, and there is no need to individually calculate the steady-state reconstruction parameters of each video frame in 3D face reconstruction, which greatly reduces the calculation overhead of 3D face reconstruction, thereby improving the efficiency of 3D face reconstruction and ensuring the accuracy of the dynamic reconstruction parameters of the current video frame in 3D face reconstruction. After that, the steady-state reconstruction parameters and dynamic reconstruction parameters of the current video frame in the 3D face reconstruction are directly input into the pre-constructed 3D morphable model to reconstruct a 3D face corresponding to the current video frame. In this case, the steady-state reconstruction parameters of successive video frames in the 3D face reconstruction do not change, so as to avoid the jitter problem occurring in successive video frames in the 3D face reconstruction. At the same time, the dynamic reconstruction parameters of each video frame in the 3D face reconstruction are used to ensure the stability of the 3D face reconstruction of successive video frames, and then the individual reconstruction of the 3D face of each video frame is realized, thereby improving the accuracy of the 3D face reconstruction.

[0053] Example 3

[0054] FIG. 3A is a flow chart of a method for training a student network model in 3D face reconstruction according to Example 3 of the present application, and FIG. 3B is a schematic diagram of a training process of a student network model according to Example 3 of the present application. This embodiment is described based on the above embodiment. As shown in FIG. 3A, this embodiment mainly describes a training process of training a student network model by performing knowledge distillation on a teacher network model.

[0055] As shown in FIG. 3A, this embodiment may include the following steps.

[0056] In S310, a first overall reconstruction parameter of a first video frame in 3D face reconstruction and a second overall reconstruction parameter of a second video frame in 3D face reconstruction are estimated by inputting a first video frame and a second video frame in a video frame training pair having the same face in a video training set into a teacher network model.

[0057] In one example, the student network model in this embodiment is obtained after performing knowledge distillation on a teacher network model that can accurately estimate each reconstruction parameter of each video frame in pre-trained 3D face reconstruction, so when training the student network model, it is still necessary to estimate each reconstruction parameter of video frames with the same face by the teacher network model, and the estimation accuracy of the student network model is analyzed by using it as a reference.Therefore, in the pre-set video training set, the training samples are mainly video frame training pairs with the same face, and in this case, the first video frame in the video frame training pair can be the source of the steady-state reconstruction parameters estimated for the second video frame by the teacher network model.

[0058] For example, as shown in FIG. 3B, when training the student network model, the first and second video frames in each video frame training pair can be input to a teacher network model, and the teacher network model can estimate the overall reconstruction parameters of the first and second video frames in 3D face reconstruction, respectively. In this case, the second overall reconstruction parameter of the second video frame estimated by the teacher network model in 3D face reconstruction can be used later to analyze the accuracy of the target dynamic reconstruction parameters estimated for the second video frame by the student network model. The steady-state reconstruction parameter of the first overall reconstruction parameter of the first video frame estimated by the teacher network model in 3D face reconstruction can be set as a target steady-state reconstruction parameter that matches the target dynamic reconstruction parameters estimated for the second video frame by the student network model later. 3D face reconstruction is performed for the second video frame using the target dynamic reconstruction parameters estimated for the second video frame by the student network model and the target steady-state reconstruction parameter.

[0059] In S320, the second video frame in the video frame training pair is input into a student network model to estimate target dynamic reconstruction parameters of the second video frame in 3D face reconstruction, and the target steady-state reconstruction parameters and the target dynamic reconstruction parameters among the first overall reconstruction parameters are input into a 3D morphable model to reconstruct a 3D face corresponding to the second video frame.

[0060] For example, the second video frame in each video frame training pair is synchronously input into a student network model, and the student network model estimates the target dynamic reconstruction parameters of the second video frame in the 3D face reconstruction. In this case, since the student network model is in a model training process, the estimated steady-state reconstruction parameters may not be very accurate. In this case, the corresponding target steady-state reconstruction parameters are selected from the first overall reconstruction parameters estimated for the first video frame by the teacher network model as the steady-state reconstruction parameters of the second video frame in the 3D face reconstruction. Then, the target steady-state reconstruction parameters and the target dynamic reconstruction parameters estimated by the student network model are combined to obtain a target steady-state reconstruction parameter. The data is input into a 3D morphable model all at once, and the 3D morphable model uses the target steady-state reconstruction parameters and the target dynamic reconstruction parameters to perform corresponding deformation, facial expression change, albedo change, etc. on the standard 3D face, and adjusts the corresponding 3D detailed display information to reconstruct a 3D face corresponding to the second video frame.The realism of the reconstructed 3D face can then be analyzed by comparing the similarity between the second video frame and the reconstructed 3D face, and the similarity between the second overall reconstruction parameters estimated for the second video frame by the teacher network model and the target dynamic reconstruction parameters estimated for the second video frame by the student network model.

[0061] In S330, a first loss function between the second overall reconstruction parameters and the target active reconstruction parameters, and a second loss function in a preset two-dimensional supervision term between the three-dimensional face corresponding to the second video frame and the second video frame are calculated, and a weight corresponding to each loss function among the first loss function and the second loss function is adjusted.

[0062] In order to analyze the realism of the reconstructed 3D face relative to the face in the second video frame, in this embodiment, a corresponding loss function is preset, and the similarity between the reconstructed 3D face and the second video frame is compared by this loss function. The loss function in this embodiment can be divided into two types: a loss function between the second comprehensive reconstruction parameters and the target reconstruction parameters (i.e., the first loss function), and a loss function in the preset 2D superposition between the 3D face corresponding to the second video frame and the second video frame (i.e., the second loss function), so that the reconstruction error of the 3D face can be avoided as comprehensively as possible.

[0063] Illustratively, in this embodiment, the loss function between the second overall reconstruction parameter and the target reconstruction parameter may be a regularization term corresponding to the reconstruction parameter, etc.,

number

[0064] Here, the second overall reconstruction parameters estimated by the teacher network model for the second video frame are

number

number

number

[0065] At the same time, the loss function in the preset 2D supervision term in this embodiment may include an image pixel loss function, a keypoint loss function, an identity feature loss function and an albedo penalty function.

[0066] For example, after the 3D face corresponding to the second video frame is reconstructed, the reconstruction error between the reconstructed 3D face and the second video frame, i.e., the specific value of the loss function, can be calculated by the loss function between the second overall reconstruction parameters and the active reconstruction parameters, and the loss function in the preset 2D supervision term between the 3D face corresponding to the second video frame and the second video frame. Further, based on the specific value of each loss function, the realism of the reconstructed 3D face in each loss function is analyzed to judge the accuracy of the estimation of the trained student network model in each loss function, and the weight corresponding to each loss function is correspondingly adjusted to improve the estimation ability during the next training.

[0067] Illustratively, in this embodiment, by respectively calculating the image pixel loss function, keypoint loss function, identity feature loss function and albedo penalty function between the 3D face and the second video frame, and the regularization terms corresponding to the loss function between the second comprehensive reconstruction parameters and the target dynamic reconstruction parameters, the accurate reconstruction ability of the current training process for the image pixels, keypoints, identity features, albedo, and each reconstruction parameter during 3D face reconstruction is clearly determined, and the weight of each loss function is correspondingly adjusted according to the specific reconstruction ability, and training is continued, so as to constantly improve the estimation ability of the student network model to estimate the dynamic reconstruction parameters during 3D face reconstruction.

[0068] For example, in this embodiment, the image pixel loss function, keypoint loss function, identity feature loss function, and albedo penalty function included in the loss function in the preset 2D supervision term are analyzed as follows.

[0069] 1. As the process of determining the image pixel loss function, the corresponding skin mask is segmented from the second video frame, and based on the skin mask, the pixel error of the 3D face corresponding to the second video frame and the same pixel point within the face skin region in the second video frame is calculated, thereby obtaining the corresponding image pixel loss function.

[0070] For example, the mask is a binary image consisting of pixel values ​​0 and 1. In this embodiment, when setting the image pixel loss function, in order to ensure the accuracy of the image pixel loss in the training process, by applying a skin mask to the second video frame, all pixel values ​​of the facial skin area in the second video frame can be set to 1, and all pixel values ​​of the non-facial skin area can be set to 0, so that the skin segmentation algorithm can be used to accurately segment the corresponding facial skin area from the second video frame, and to avoid interference with the 3D face reconstruction caused by the pixel features in the non-facial skin area. Then, after segmenting the corresponding skin mask from the second video frame, pixel points at the same pixel positions can be found from the reconstructed 3D face and the second video frame, and then based on the segmented skin mask, it can be accurately determined whether each identical pixel point is within the facial skin area. By calculating the pixel error of each pixel point within the facial skin area in the 3D face and the second video frame, the overall pixel error of the 3D face and the second video frame within the facial skin area is analyzed, and a corresponding image pixel loss function is obtained, which only compares the pixel error in the facial skin area before and after reconstruction, and masks the pixel influence of the non-facial skin area, making the facial identity features and albedo information in the estimated reconstruction parameters more accurate.

[0071] Illustratively, the image pixel loss function in this embodiment is:

number

[0072] Where:

number

number

[0073] 2. As the process of determining the keypoint loss function, extract key feature points at preset positions from the second video frame, determine the visibility of each key feature point, and calculate the position reconstruction error of each visible key feature point between the 3D face corresponding to the second video frame and the second video frame, thereby obtaining the corresponding keypoint loss function.

[0074] For example, in order to ensure that the key feature points in the reconstructed 3D face correspond one-to-one to the key feature points in the second video frame, in this embodiment, when setting the key point loss function in the training process, the Landmark algorithm can be used to extract key feature points at preset positions in each face region from the training sample, such as 17 key feature points of the face contour, 5 key feature points of each of the left and right eyebrows, 6 key feature points of each of the left and right eyes, 9 key feature points of the nose, 20 key feature points of the mouth, etc. Exemplarily, in this embodiment, a total of 68 key feature points can be adopted, as shown in FIG. 3C. In this case, because the facial head postures are different in different training samples, the key feature points at some preset positions may be invisible, so after extracting the key feature points at each preset position from the second video frame, it can also be determined whether each key feature point is visible, and after determining each visible key feature point, the position reconstruction error of each visible key feature point before and after reconstruction is calculated by analyzing whether the pixel positions of each visible key feature point in the reconstructed 3D face and the second video frame match, and the corresponding key point loss function is obtained. In this case, for a face with a large head rotation angle in the second video frame, it is necessary to select half of the visible key feature points to calculate the corresponding key point reconstruction loss, and the invisible key feature points are not involved in the key point reconstruction loss calculation.

[0075] Illustratively, the keypoint loss function in this embodiment is:

number

[0076] Where:

number

number

[0077] 3. As a process of determining the identity feature loss function, the 3D face corresponding to the second video frame and the second video frame are input into a pre-constructed face recognition model to obtain a first identity feature of the second video frame and a second identity feature of the 3D face corresponding to the second video frame, and the corresponding identity feature loss function is calculated based on the similarity between the first identity feature and the second identity feature.

[0078] For example, the identity feature loss function is essentially used to analyze whether the identity features change before and after reconstruction. Therefore, in this embodiment, a corresponding face identification model is constructed in advance for identity feature recognition, and the identity features before and after reconstruction can be extracted by the face recognition model. First, the second video frame and the reconstructed 3D face are input into the face recognition model, and the face features in the second video frame and the 3D face are analyzed by the face recognition model to determine the first identity feature corresponding to the second video frame and the second identity feature corresponding to the 3D face, and then it is determined whether there is an error in the identity features before and after reconstruction. After obtaining the first identity feature corresponding to the second video frame and the second identity feature corresponding to the 3D face, in order to analyze whether there is an error in the identity features before and after reconstruction, it is necessary to first determine the similarity between the first identity feature and the second identity feature, and then calculate the corresponding identity feature loss function based on the similarity.

[0079] Illustratively, the identity feature loss function in this embodiment is:

number

[0080] Where:

number

number

[0081] 4. As the process of determining the albedo penalty function, calculate the albedo of each mesh vertex in the 3D face corresponding to the second video frame, and calculate the corresponding albedo penalty function based on the albedo of each mesh vertex in the 3D face and the preset albedo interval.

[0082] For example, in this embodiment, the albedo of each pixel point can be calculated by detecting information such as the color and reflected light intensity of each pixel point in the second video frame, and then the albedo of each vertex in the reconstructed 3D face can be set according to the situation of position matching between each vertex in the reconstructed 3D face and each pixel point in the second video frame, so as to ensure the consistency of the face albedo before and after reconstruction.In this case, in order to ensure that the albedo of each vertex in the reconstructed 3D face is not too dark or too bright, the albedo of each vertex in the 3D face is adjusted in this embodiment, in this case, a reasonable preset albedo interval is preset, and the preset albedo interval in this embodiment can be [0.05, 0.95], so that the albedo of each vertex in the reconstructed 3D face is all within the preset albedo interval, and therefore, by analyzing whether the albedo of each vertex in the 3D face is within the preset albedo interval, the corresponding albedo penalty function can be calculated, and the albedo of the reconstructed 3D face is constantly optimized in the training process.

[0083] Illustratively, the albedo penalty function in this embodiment is:

number

[0084] Here, T i,j is the albedo of the jth pixel point in the second video frame in the ith video frame training pair,

number

[0085] As a result, in this embodiment, the loss function in the preset two-dimensional supervision term between the three-dimensional face corresponding to the second video frame and the second video frame is:

number

[0086] Where:

number

[0087] At S340, a corresponding fitting loss function is generated based on each loss function and the corresponding weight, and the student network model is trained by inversely correcting the student network model using the fitting loss function.

[0088] For example, calculate each loss function between the 3D face and the second video frame, adjust the weight corresponding to each loss function, and then use the weight corresponding to each loss function to integrate each loss function to generate a corresponding fitting loss function, and use the fitting loss function as the loss function of the entire training process of the student network model, and then use the fitting loss function to backpropagate through the entire training process to modify the network parameters in the student network model, and then use the modified student network model according to the above steps to continue estimating the dynamic reconstruction parameters when reconstructing the 3D face using the second video frame in the next video frame training pair, thereby continuously executing the training process, constantly reverse-correcting the student network model, and further obtaining a trained student network model, in this case, the fitting loss function in this embodiment makes the reference information in the training process more comprehensive, and improves the accuracy of the estimation of the dynamic reconstruction parameters by the student network model.

[0089] Illustratively, the fitting loss function in this embodiment is

number

number

number

[0090] In this embodiment, for the student network model obtained after distillation is performed on the teacher network model, the accuracy of the dynamic reconstruction parameters estimated by the student network model for the second video frame is analyzed with reference to each comprehensive reconstruction parameter estimated by the teacher network model for the first video frame and the second video frame in each video frame training pair, and the steady-state reconstruction parameters estimated by the teacher network model for the first video frame and the dynamic reconstruction parameters estimated by the student network model for the second video frame are input into a 3D morphable model to reconstruct a 3D face corresponding to the second video frame, in this case, By analyzing the loss function in the predetermined 2D supervisory term between the 3D face corresponding to the image frame and the second video frame, and the loss function between the reconstruction parameters estimated in the teacher network model and the student network model for the second video frame, the student network model is continuously reverse-corrected, and a student network model with high estimation accuracy is trained. In this case, the reference information of the student network model in the entire training process is more comprehensive, so as to improve the accuracy of the estimation of the active reconstruction parameters by the student network model, and while ensuring the stability of the 3D face reconstruction of continuous video frames, the personalized reconstruction of the 3D face of the video frame is realized, thereby improving the accuracy of the 3D face reconstruction.

[0091] Example 4

[0092] FIG. 4 is a structural schematic diagram of a 3D face reconstruction apparatus according to embodiment 4 of the present application. Specifically, as shown in FIG. 4, the apparatus includes: a reconstruction parameter estimation module 410 configured to, in response to estimating steady-state reconstruction parameters of a current video frame in 3D face reconstruction by a pre-constructed teacher network model, estimate dynamic reconstruction parameters of the current video frame in 3D face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model; a 3D face reconstruction module 420 configured to reconstruct a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; may include.

[0093] In this embodiment, considering that consecutive video frames in the same video contain the same face, the teacher network model constructed in advance is used to collectively estimate the steady-state reconstruction parameters of each video frame in 3D face reconstruction, which hardly change. Then, when performing 3D face reconstruction on each current video frame, the student network model obtained after distillation on the teacher network model is directly used to estimate the dynamic reconstruction parameters of the current video frame in the 3D face reconstruction. In this case, the student network model is obtained after knowledge distillation on the teacher network model, and there is no need to individually calculate the steady-state reconstruction parameters of each video frame in 3D face reconstruction, which greatly reduces the calculation overhead of 3D face reconstruction, thereby improving the efficiency of 3D face reconstruction and ensuring the accuracy of the dynamic reconstruction parameters of the current video frame in 3D face reconstruction. After that, the steady-state reconstruction parameters and dynamic reconstruction parameters of the current video frame in the 3D face reconstruction are directly input into the pre-constructed 3D morphable model to reconstruct a 3D face corresponding to the current video frame. In this case, the steady-state reconstruction parameters of successive video frames in the 3D face reconstruction do not change, so as to avoid the jitter problem occurring in successive video frames in the 3D face reconstruction. At the same time, the dynamic reconstruction parameters of each video frame in the 3D face reconstruction are used to ensure the stability of the 3D face reconstruction of successive video frames, and then the individual reconstruction of the 3D face of each video frame is realized, thereby improving the accuracy of the 3D face reconstruction.

[0094] The apparatus for 3D face reconstruction according to this embodiment can be applied to the method for 3D face reconstruction according to any of the above embodiments, and has corresponding functions and beneficial effects.

[0095] Example 5

[0096] Fig. 5 is a structural schematic diagram of a computer device according to Example 5 of the present application, and as shown in Fig. 5, the device includes a processor 50, a storage device 51, and a communication device 52. The number of processors 50 included in the device may be one or more, and Fig. 5 takes one processor 50 as an example. The processor 50, the storage device 51, and the communication device 52 included in the device may be connected by a bus or in other ways, and Fig. 5 takes the connection by a bus as an example.

[0097] A computing device according to this embodiment may be configured to perform the method for 3D face reconstruction according to any of the above embodiments, with corresponding functionality and beneficial effects.

[0098] Example 6

[0099] In a sixth embodiment of the present application, a computer-readable storage medium is further provided, which stores a computer program, and when the program is executed by a processor, can realize a method for 3D face reconstruction according to any of the above embodiments. The method includes: In response to estimating steady-state reconstruction parameters of a current video frame in 3D face reconstruction by a pre-constructed teacher network model, estimating dynamic reconstruction parameters of the current video frame in 3D face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model; reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; may include.

[0100] In an embodiment of the present application, a storage medium including computer-executable instructions is further provided, the computer-executable instructions being capable of performing operations in the above-described method as well as related operations in the method for 3D face reconstruction according to any embodiment of the present application.

[0101] The storage medium may be a non-transitory storage medium.

[0102] From the above description of the embodiments, it will be obvious to those skilled in the art that the present application can be realized by software and necessary general-purpose hardware, and of course it can also be realized by hardware, but the former is a more preferred embodiment. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the related art can be embodied as a software product, and the computer software product can be stored in a computer-readable storage medium such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash (FLASH), a hard disk or an optical disk of a computer, and includes a plurality of instructions for making a computer device (which may be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.

[0103] In the above embodiment of the 3D face reconstruction device, each unit and module included is divided according to functional logic, but is not limited to the above division, and may realize the corresponding function. The names of each functional unit are given for easy distinction from each other, and do not limit the scope of protection of the present application. [Explanation of symbols]

[0104] 410 Reconstruction Parameter Estimation Module 420 3D Face Reconstruction Module 50 processors 51 Storage device 52 Communication Equipment

Claims

1. In response to estimating steady-state reconstruction parameters of a current video frame in three-dimensional face reconstruction by a pre-constructed teacher network model, estimating dynamic reconstruction parameters of the current video frame in three-dimensional face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model; reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; Including, the degree of change of the dynamic reconstruction parameters in multiple video frames of the same face is higher than the degree of change of the steady-state reconstruction parameters in multiple video frames of the same face; Methods for 3D face reconstruction.

2. before estimating dynamic reconstruction parameters of the current video frame in a three-dimensional face reconstruction by inputting the current video frame into a student network model obtained after performing distillation on the teacher network model; The method further includes estimating overall reconstruction parameters of the related video frame in three-dimensional face reconstruction by inputting related video frames having the same face as the current video frame into the teacher network model, and setting a steady-state reconstruction parameter of the overall reconstruction parameter as a steady-state reconstruction parameter of the current video frame in three-dimensional face reconstruction. The method of claim 1.

3. 3. The method of claim 2, wherein the relevant video frame is the first video frame in the video in which the current video frame is located that has a face in the current video frame, or the first m video frames in the video in which the current video frame is located that have a face in the current video frame, where m is a positive integer greater than 1.

4. The student network model is obtained after distilling the teacher network model in the following steps: inputting a first video frame and a second video frame in a video frame training pair having the same face in a video training set into the teacher network model to estimate a first overall reconstruction parameter of the first video frame in a three-dimensional face reconstruction and a second overall reconstruction parameter of the second video frame in a three-dimensional face reconstruction; inputting a second video frame in the video frame training pair into the student network model to estimate target dynamic reconstruction parameters of the second video frame in 3D face reconstruction; and inputting a target steady-state reconstruction parameter among the first overall reconstruction parameters and the target dynamic reconstruction parameter into the 3D morphable model to reconstruct a 3D face corresponding to the second video frame; Calculate a first loss function between the second overall reconstruction parameters and the target dynamic reconstruction parameters, and a second loss function in a preset two-dimensional supervision term between the three-dimensional face corresponding to the second video frame and the second video frame, and adjust a weight corresponding to each of the first loss function and the second loss function; generating a corresponding fitting loss function based on each of the loss functions and the corresponding weights, and training the student network model by inversely correcting the student network model using the fitting loss function; The method of claim 1.

5. The method of claim 4 , wherein the second loss function in the predetermined two-dimensional supervision term includes an image pixel loss function, a keypoint loss function, an identity feature loss function, and an albedo penalty function.

6. When calculating an image pixel loss function between a 3D face corresponding to the second video frame and the second video frame, Segmenting a corresponding skin mask from the second video frame; Calculating an image pixel loss function between a 3D face corresponding to the second video frame and the second video frame includes: and calculating pixel errors of identical pixel points within a facial skin region in the three-dimensional face corresponding to the second video frame and the second video frame based on the skin mask to obtain a corresponding image pixel loss function. The method according to claim 5.

7. When calculating a keypoint loss function between the 3D face corresponding to the second video frame and the second video frame, extracting key feature points at preset positions from the second video frame and determining visibility of each of the key feature points; Calculating a keypoint loss function between a 3D face corresponding to the second video frame and the second video frame includes: calculating a position reconstruction error of each visible key feature point between the 3D face corresponding to the second video frame and the second video frame to obtain a corresponding keypoint loss function; The method according to claim 5.

8. When calculating an identity feature loss function between the 3D face corresponding to the second video frame and the second video frame, and further comprising inputting each of the three-dimensional face corresponding to the second video frame and the second video frame into a pre-constructed face recognition model to obtain a first identity feature of the second video frame and a second identity feature of the three-dimensional face corresponding to the second video frame; Calculating an identity feature loss function between a 3D face corresponding to the second video frame and the second video frame includes: calculating a corresponding identity feature loss function based on a similarity between the first identity feature and the second identity feature; The method according to claim 5.

9. When calculating a 3D facial albedo penalty function corresponding to the second video frame, calculating an albedo for each mesh vertex in the 3D face corresponding to the second video frame; Calculating a 3D facial albedo penalty function corresponding to the second video frame includes: calculating a corresponding albedo penalty function based on the albedo of each mesh vertex in the 3D face and a preset albedo interval; The method according to claim 5.

10. The 3D morphable model is composed of a bilinear face model and a Principal Component Analysis (PCA) model, and a 3D face core tensor is defined in the bilinear face model, and the core tensor is composed of three dimensions, namely, a 3D face mesh vertex, a face contour, and a face expression; In the PCA model, a principal component basis representing a face mean albedo and a face albedo change is defined; reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; inputting a reconstruction parameter representing a change in face shape among the steady-state reconstruction parameters and a reconstruction parameter representing a change in face expression among the dynamic reconstruction parameters into the bilinear face model, and adjusting the core tensor to obtain a corresponding three-dimensional deformed face; inputting the three-dimensional deformed face and the steady-state reconstruction parameters and the dynamic reconstruction parameters, which correspond to a principal component basis in the PCA model, into the PCA model, and performing albedo correction on the three-dimensional deformed face based on the face average albedo, thereby obtaining a reconstructed three-dimensional face; Including, The method of claim 1.

11. prior to reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; The method further includes collecting 3D face scan data in a multi-dimensional data source, and performing a high-order singular value decomposition on the 3D face after the 3D face scan data is reconstructed in a facial contour dimension and a facial expression dimension to obtain a corresponding core tensor. The method of claim 10.

12. The method of claim 10, wherein the three-dimensional morphable model also defines lighting parameters that represent changes in lighting of the face, position parameters that represent movement of the face, and rotation parameters that represent a pose of the face-head.

13. reconstructing a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; and and rendering a three-dimensional face corresponding to the current video frame using a differentiable renderer.

13. The method according to any one of claims 1 to 12.

14. A method according to any one of claims 1 to 12, further comprising, in response to steady-state reconstruction parameters of a current video frame in 3D face reconstruction not being estimated by a pre-constructed teacher network model, estimating overall reconstruction parameters of the current video frame in 3D face reconstruction by inputting the current video frame into the teacher network model, and reconstructing a 3D face corresponding to the current video frame by inputting the overall reconstruction parameters of the current video frame in 3D face reconstruction into the 3D morphable model.

15. a reconstruction parameter estimation module configured to, in response to estimating steady-state reconstruction parameters of a current video frame in three-dimensional face reconstruction by a pre-constructed teacher network model, estimate dynamic reconstruction parameters of the current video frame in three-dimensional face reconstruction by inputting the current video frame into a student network model obtained after distillation on the teacher network model; a 3D face reconstruction module configured to reconstruct a 3D face corresponding to the current video frame by inputting the steady-state reconstruction parameters and the dynamic reconstruction parameters into a pre-constructed 3D morphable model; Including, the degree of change of the dynamic reconstruction parameters in multiple video frames of the same face is higher than the degree of change of the steady-state reconstruction parameters in multiple video frames of the same face; A device for 3D facial reconstruction.

16. one or more processors; a storage device configured to store one or more programs; Including, The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for 3D face reconstruction according to any one of claims 1 to 14. Computer device.

17. A computer readable storage medium having a computer program stored thereon which, when said computer program is executed by a processor, implements the method for three-dimensional face reconstruction according to any one of claims 1 to 14.

Images