A speech-driven face sequence generation method and system based on reconstructed human faces

Abstract

The application discloses a speech-driven face sequence generation method and system based on reconstructed human faces, and relates to the field of computer vision.The method comprises the following steps: inputting a set of multi-view face pictures to be processed and corresponding relative camera external parameters of each picture into a three-dimensional face reconstruction model to obtain a three-dimensional reconstructed face; the three-dimensional face reconstruction model comprises an optical flow feature extraction module and a face construction module; the optical flow feature extraction module is used for feature extraction to obtain predicted face optical flow features; the face construction module is used for reconstructing a face through a triangulation process to obtain a three-dimensional reconstructed face; inputting a speech segment to be processed into a speech-driven face sequence model to obtain a three-dimensional template face sequence; and constructing a three-dimensional reconstructed speech-driven face sequence based on the three-dimensional reconstructed face and the three-dimensional template face sequence.The application can quickly obtain a high-precision speech-driven sequence of a reconstructed face.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for generating speech-driven face sequences based on reconstructed faces. Background Technology

[0002] High-precision 3D face reconstruction technology mainly involves recovering 3D facial information from images, videos, and other data, and then applying it to scenarios such as virtual reality, 3D digital humans, and 3D digital games. In game and animation production, facial expressions and lip movements are crucial for players and viewers. Currently, exquisite game and animation character visuals rely on artists creating them frame by frame, consuming a significant amount of manpower and resources. Algorithms that automatically generate corresponding character movements based on speech can greatly reduce the workload of artists and significantly shorten the production time of animation and games.

[0003] Currently, high-precision face reconstruction schemes mainly include: structured light acquisition of real faces, multi-view stereo matching schemes, and deep learning-based schemes. Although structured light acquisition and stereo matching algorithms have achieved relatively high-quality face reconstruction results, their widespread application still faces many obstacles due to equipment costs and algorithm execution efficiency. With the continuous development of deep learning, more and more researchers are exploring high-precision 3D face reconstruction algorithms based on deep learning, such as statistical model 3DMM and CNN-based 3DMM parameter estimation. These algorithms generally estimate the 3DMM parameters and other parameters required for rendering from a single input image, re-render the reconstructed face into an image, and use the L2 loss between the rendered image and the input image as the network's supervision function. Statistical model-based methods often obtain relatively smooth reconstruction results but cannot recover high-frequency information such as wrinkles on the face. The driving force of 3D digital humans is also a hot research direction recently. The mainstream solution currently used in the industry is based on ArKit's BlendShape and motion capture. This solution requires artists to create a 51-dimensional BlendShape for the virtual digital human, and the expressiveness of the digital human depends on the level of detail in the BlendShape production.

[0004] In summary, existing methods mainly suffer from the following problems: high cost, low efficiency, long processing time, and limited accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for generating speech-driven face sequences based on reconstructed faces, which can quickly obtain high-precision speech-driven sequences of reconstructed faces.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A speech-driven face sequence generation method based on face reconstruction includes:

[0008] Acquire a set of data to be processed; the data to be processed includes a set of multi-view face images to be processed, a voice segment to be processed, and the relative camera extrinsic parameters corresponding to each image in the set of multi-view face images to be processed;

[0009] The set of multi-view face images to be processed and the relative camera extrinsic parameters corresponding to each image are input into a 3D face reconstruction model to obtain a 3D reconstructed face. The 3D face reconstruction model includes an optical flow feature extraction module and a face construction module. The optical flow feature extraction module extracts features from each image in the set of multi-view face images to obtain predicted face optical flow features. The face construction module reconstructs the face through a triangulation process based on the predicted face optical flow features and the relative camera extrinsic parameters corresponding to each image to obtain a 3D reconstructed face.

[0010] The speech segment to be processed is input into a speech-driven face sequence model to obtain a 3D template face sequence. The speech-driven face sequence model is obtained by training the speech-driven 3DMM template face model based on a speech sample segment set. Each sample in the speech sample segment set includes a speech sample segment and the corresponding ground truth value of the 3D template face sequence. The speech-driven 3DMM template face model includes a generator and a discriminator. The generator includes a speech feature encoding module and a face animation decoding and prediction module. Both the speech feature encoding module and the face animation decoding and prediction module are Transformer structures.

[0011] Based on the reconstructed 3D face and the 3D template face sequence, a voice-driven 3D reconstructed face sequence is constructed.

[0012] Optionally, the optical flow feature extraction module is obtained by training a preset convolutional neural network based on a multi-visual face sample set;

[0013] The construction process of the multi-vision face sample set specifically includes:

[0014] Obtain a set of raw multi-view face images;

[0015] Based on a preset topologically consistent head model and a preset 3D head model, a 3D face model is determined;

[0016] The three-dimensional face model is projected onto each image in the original multi-view face image set using a weak perspective projection method, and then segmented to obtain a segmented multi-view face image set.

[0017] Based on the weak perspective projection of the three-dimensional face model, the camera intrinsic parameters corresponding to each image in the segmented multi-view face image set are adjusted to obtain the processed multi-view face image set.

[0018] Randomly select a point on the 3D face model as a marker point;

[0019] The marker points are projected onto any two images in the processed multi-view face image set to obtain the first pixel and the second pixel.

[0020] Based on the first pixel and the second pixel, the pixel optical flow between two processed multi-view face images is determined; the segmented multi-view face image set and the pixel optical flow between any two corresponding images constitute a multi-view face sample; multiple multi-view face samples constitute a multi-view face sample set.

[0021] Optionally, determining the 3D face model based on a preset topologically consistent head model and a preset 3D head model specifically includes:

[0022] The face region is cropped from the preset topologically consistent head model to obtain a topologically consistent face model;

[0023] The iterative nearest-point algorithm is used to align and crop the topologically consistent face model with the preset 3D head model in sequence to obtain the 3D face model.

[0024] Optionally, based on the first pixel and the second pixel, the pixel optical flow between the two processed multi-view face images is determined, specifically including:

[0025] Determine whether the values ​​of the first pixel and the second pixel are both integer values;

[0026] When both the value of the first pixel and the value of the second pixel are integer values, the optical flow of the first pixel and the optical flow of the second pixel are calculated based on the values ​​of the first pixel and the second pixel, respectively.

[0027] When either the value of the first pixel or the value of the second pixel is a non-integer value, the optical flow of the first pixel and the optical flow of the second pixel are calculated using bilinear interpolation based on the values ​​of the first pixel and the second pixel, respectively.

[0028] The pixel optical flow between the two processed multi-view face images is determined based on the optical flow of the first pixel and the optical flow of the second pixel.

[0029] Optionally, the multi-view face image set includes a main visual image and multiple secondary visual images;

[0030] The preset convolutional neural network includes:

[0031] The first convolutional unit is used to perform multiple convolutions on the main visual image in sequence to obtain the main facial features;

[0032] The second convolutional unit is used to perform multiple deformable convolutions on each auxiliary visual image in sequence to obtain facial auxiliary features.

[0033] The feature fusion unit is used to fuse the main facial features and multiple auxiliary facial features based on feature correlation to obtain comprehensive facial features.

[0034] The optical flow estimation unit is used to determine the predicted facial optical flow features based on the comprehensive facial features according to the DenseNet structure.

[0035] Optionally, the training process of the voice-driven 3DMM template face model specifically includes:

[0036] The speech feature encoding module is pre-trained;

[0037] The set of speech sample segments is sequentially input into the pre-trained speech feature encoding module and the face animation decoding and prediction module for training to obtain an optimal generator. The optimal generator is used to generate a 3D face animation sequence based on the speech sample segments. During the training of the optimal generator, the loss function includes a positional loss function, a velocity loss function, and a smoothing loss function. The positional loss function is used to supervise the first-order derivative of each frame of the 3D face animation sequence; the velocity loss function is used to supervise the first-order derivative of each frame of the 3D face animation sequence; and the smoothing loss function is used to supervise the second-order derivative of each frame of the 3D face animation sequence.

[0038] The optimal generator and the discriminator are trained iteratively and alternately to obtain a speech-driven 3DMM template face model.

[0039] Optionally, based on the reconstructed 3D face and the 3D template face sequence, a voice-driven 3D reconstructed face sequence is constructed, specifically including:

[0040] Obtain the 3DMM template face corresponding to the 3D template face sequence;

[0041] Based on the NICP algorithm, the deformation parameters from the 3DMM template face to the 3D reconstructed face are calculated according to the 3D reconstructed face and the 3DMM template face.

[0042] Based on the deformation parameters and the three-dimensional template face sequence, a three-dimensional reconstructed voice-driven face sequence is obtained.

[0043] To achieve the above objectives, the present invention also provides the following technical solutions:

[0044] A speech-driven face sequence generation system based on reconstructed faces, comprising:

[0045] The data acquisition subsystem is used to acquire a set of data to be processed; the data to be processed includes a set of multi-view face images to be processed, a voice segment to be processed, and the relative camera extrinsic parameters corresponding to each image in the set of multi-view face images to be processed;

[0046] A face reconstruction subsystem is used to input the set of multi-view face images to be processed and the relative camera extrinsic parameters corresponding to each image into a 3D face reconstruction model to obtain a 3D reconstructed face. The 3D face reconstruction model includes an optical flow feature extraction module and a face construction module. The optical flow feature extraction module is used to extract features from each image in the set of multi-view face images to obtain predicted face optical flow features. The face construction module is used to reconstruct the face through a triangulation process based on the predicted face optical flow features and the relative camera extrinsic parameters corresponding to each image to obtain a 3D reconstructed face.

[0047] A template face sequence construction subsystem is used to input the speech segment to be processed into a speech-driven face sequence model to obtain a 3D template face sequence. The speech-driven face sequence model is obtained by training the speech-driven 3DMM template face model based on a speech sample segment set. Each sample in the speech sample segment set includes a speech sample segment and the corresponding ground truth value of the 3D template face sequence. The speech-driven 3DMM template face model includes a generator and a discriminator. The generator includes a speech feature encoding module and a face animation decoding and prediction module. Both the speech feature encoding module and the face animation decoding and prediction module are Transformer structures.

[0048] A face reconstruction sequence construction subsystem is used to construct a three-dimensional reconstructed voice-driven face sequence based on the three-dimensional reconstructed face and the three-dimensional template face sequence.

[0049] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0050] This invention discloses a method and system for generating speech-driven face sequences based on face reconstruction. It pre-constructs an optical flow feature extraction module, a face construction module, and a speech-driven face sequence model. Based on these three structures, and using the obtained multi-view face image set to be processed, the speech segments to be processed, and the relative camera extrinsic parameters corresponding to each image in the multi-view face image set, customized 3D faces can be quickly and conveniently acquired and driven. The use of multi-view face images for optical flow feature extraction, combined with camera extrinsic parameter reconstruction, results in higher accuracy of the obtained 3D faces. The training of the speech-driven process using a generator and discriminator makes the obtained face sequences more realistic, thereby improving the overall precision of the 3D reconstructed speech-driven face sequence. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the speech-driven face sequence generation method based on face reconstruction according to the present invention.

[0053] Figure 2 This is a schematic diagram of the pre-defined convolutional neural network structure of the present invention;

[0054] Figure 3 This is a schematic diagram of the structure of the voice-driven 3DMM template face model of the present invention;

[0055] Figure 4 This is a schematic diagram of the structure of the speech-driven face sequence generation system based on face reconstruction according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention provides a voice-driven face sequence generation method and system based on face reconstruction, which can achieve high-precision three-dimensional face reconstruction without the need for a parameterized model, and the driving of face reconstruction can be achieved without the need for BlendShape.

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Example 1

[0060] like Figure 1 As shown, this invention proposes a speech-driven face sequence generation method based on reconstructed faces, comprising:

[0061] Step 100: Obtain a set of data to be processed; the data to be processed includes a set of multi-view face images to be processed, a voice segment to be processed, and the relative camera extrinsic parameters corresponding to each image in the set of multi-view face images to be processed.

[0062] In a specific practical application, before step 100, the method further includes preprocessing the obtained original images, specifically: 1) acquiring a set of original multi-view face images; 2) determining a 3D face model based on a preset topologically consistent head model and a preset 3D head model; 3) using weak perspective projection, projecting the 3D face model onto each image in the original multi-view face image set, and segmenting it to obtain a segmented multi-view face image set, increasing the proportion of the face in the image, generally requiring the face frame to occupy one-third of the image area; the segmented multi-view face image set is the multi-view face image set to be processed in the data to be processed. Furthermore, image data enhancement can be performed on the multi-view face image set to be processed to ensure image clarity.

[0063] Step 200: Input the set of multi-view face images to be processed and the relative camera extrinsic parameters corresponding to each image into the 3D face reconstruction model to obtain a 3D reconstructed face; wherein, the 3D face reconstruction model includes an optical flow feature extraction module and a face construction module; the optical flow feature extraction module is used to extract features from each image in the set of multi-view face images to be processed to obtain predicted face optical flow features; the face construction module is used to reconstruct the face through a triangulation process based on the predicted face optical flow features and the relative camera extrinsic parameters corresponding to each image to obtain a 3D reconstructed face.

[0064] (I) Optical Flow Feature Extraction Module

[0065] The optical flow feature extraction module is obtained by training a preset convolutional neural network based on a multi-visual face sample set.

[0066] The construction process of the multi-vision face sample set specifically includes:

[0067] 1) Obtain a set of original multi-view face images; specifically, the original multi-view face image set is taken from the FaceScape dataset; the FaceScape dataset includes the original multi-view face image set, the original 3D face model, the camera parameters of the corresponding view for each view face image, and the topologically consistent model.

[0068] 2) Based on the preset topologically consistent head model and the preset 3D head model, determine the 3D face model; specifically, crop the face region of the preset topologically consistent head model to obtain the topologically consistent face model; use the Iterative Closest Point (ICP) algorithm to align and crop the topologically consistent face model and the preset 3D head model in sequence to obtain the 3D face model.

[0069] 3) Using weak perspective projection, the three-dimensional face model is projected onto each image in the original multi-view face image set and segmented to obtain the segmented multi-view face image set.

[0070] 4) Based on the weak perspective projection of the three-dimensional face model, adjust the camera intrinsic parameters corresponding to each image in the segmented multi-view face image set to obtain the processed multi-view face image set.

[0071] 5) Randomly select a point on the three-dimensional face model as a marker point.

[0072] 6) Project the marked points onto any two images in the processed multi-view face image set to obtain the first pixel and the second pixel.

[0073] 7) Determine the pixel optical flow between two processed multi-view face images based on the first pixel and the second pixel; the segmented multi-view face image set and the pixel optical flow between any two corresponding images constitute a multi-view face sample; multiple multi-view face samples constitute a multi-view face sample set.

[0074] Further, based on the first pixel and the second pixel, the pixel optical flow between the two processed multi-view face images is determined, specifically including:

[0075] 71) Determine whether the values ​​of the first pixel and the second pixel are both integer values.

[0076] 72) When both the value of the first pixel and the value of the second pixel are integer values, calculate the optical flow of the first pixel and the optical flow of the second pixel based on their respective values. Since optical flow is vector data, it has a direction. If the image containing the first pixel is taken as the optical flow direction, then the optical flow of the first pixel is: the value of the second pixel minus the value of the first pixel. If the image containing the second pixel is taken as the optical flow direction, then the optical flow of the second pixel is: the value of the first pixel minus the value of the second pixel.

[0077] 73) When either the value of the first pixel or the value of the second pixel is a non-integer value, the optical flow of the first pixel and the optical flow of the second pixel are calculated using bilinear interpolation based on the value of the first pixel and the value of the second pixel.

[0078] 74) Determine the pixel optical flow between the two processed multi-view face images based on the optical flow of the first pixel and the optical flow of the second pixel.

[0079] The multi-view face image set includes a primary visual image and multiple secondary visual images; such as Figure 2 As shown, the preset convolutional neural network includes a first convolutional unit, a second convolutional unit, a feature fusion unit, and an optical flow estimation unit.

[0080] The first convolutional unit is used to perform multiple convolutions on the main visual image to obtain the main facial features. Specifically, the input main visual image has a resolution of 256×256. After six convolutions by the first convolutional unit, five new features with resolutions ranging from 8×8 to 128×128 are obtained. High-resolution facial feature extraction requires the use of low-resolution optical flow information. Figure 2 The diagram does not show all six convolutions; only three of them are shown as examples. The convolution used in the first convolutional unit is a standard convolution.

[0081] The second convolutional unit performs multiple deformable convolutions on each auxiliary visual image to obtain facial auxiliary features. Specifically, the input auxiliary visual image has a resolution of 256×256, and after six convolutions in the second convolutional unit, five new features with resolutions ranging from 8×8 to 128×128 are obtained. The convolutions used in the second convolutional unit are deformable convolutions, and low-resolution optical flow information is used as the offset of the deformable convolutions to ensure that the physical receptive field of the features is the same after multiple convolutions of the main viewpoint and the auxiliary viewpoint. Furthermore, the offset of the deformable convolutions is obtained by upsampling the optical flow of the 128×128 resolution by a factor of 2.

[0082] The feature fusion unit is used to fuse the main facial features and multiple auxiliary facial features based on feature correlation to obtain a comprehensive facial feature. Specifically, the main facial features and multiple auxiliary facial features are fused by calculating feature correlation. For each feature point in the main view, based on the coarse matching position information provided by the low-resolution optical flow, the correlation is calculated sequentially with feature points in the 9×9 neighborhood of the coarse matching position, and 81 results are vectorized as the fused feature at this position.

[0083] Furthermore, if we know that in the primary visual image I1 corresponding to the primary facial feature, the current pixel coordinates are (x1, y1), the optical flow is (u1, v1), and the primary facial feature is f1 = FPN(I1)(x1, y1); and in the secondary visual image I2 corresponding to the secondary facial feature, the corresponding pixel coordinates are (x2, y2) = (x1 + u1, y1 + v1), and the secondary facial feature is f2 = BIL(FPN(I2); x2, y2); then the correlation between the primary and secondary viewpoints can be expressed as Corr(f1, f2) =<f1,f2> Where FPN() represents the feature extraction function, BIL() represents the bilinear interpolation function, and <> represents the vector inner product. For simplification, the above process is denoted as Corr(x,y,x2,y2;I1,I2).

[0084] In one specific embodiment, feature fusion needs to be correlated in the 9×9 neighborhood of the secondary viewpoint to obtain the fusion cost volume V at the current resolution, with a shape of H*W*81. Where V(x,y,i*9+j)=Corr(x,y,x2+i,y2+j;I1,I2), V represents the face synthesis features, x represents the horizontal coordinate variable of the pixel in the main visual image, y represents the vertical coordinate variable of the pixel in the main visual image; Corr(x,y,x2+i,y2+j;I1,I2) represents the feature correlation calculation between the main visual image I1 and the secondary visual image I2 in the i×j neighborhood, where i and j both take values ​​of [-4,-3,…,0,1,..3,4].

[0085] The optical flow estimation unit is used to determine the predicted facial optical flow features based on the comprehensive facial features according to the DenseNet structure. Specifically, the optical flow at a resolution of 256×256 from the primary and secondary viewpoints is obtained by feature fusion and optical flow estimation to obtain a 256×256 resolution optical flow result.

[0086] (II) Face Construction Module

[0087] Specifically, based on the triangulation process in Structure from Motion, the DenseNet structure is used to estimate the optical flow at the current resolution and the primary and secondary viewpoints relative to the camera extrinsic parameters to reconstruct a 3D face. Structure from Motion is a 3D reconstruction method used to achieve 3D reconstruction from motion, inferring 3D information from a time-series of 2D images.

[0088] Step 300: Input the speech segment to be processed into the speech-driven face sequence model to obtain a 3D template face sequence; the speech-driven face sequence model is obtained by training the speech-driven 3DMM template face model based on a speech sample segment set; each sample in the speech sample segment set includes a speech sample segment and the corresponding ground truth value of the 3D template face sequence; such as Figure 3 As shown, the voice-driven 3DMM template face model includes a generator and a discriminator; the generator includes a voice feature encoding module and a face animation decoding and prediction module; both the voice feature encoding module and the face animation decoding and prediction module are Transformer structures to complete the prediction from voice sequence to face animation sequence.

[0089] The training process of the voice-driven 3D deformable face model (3DMM) template face model specifically includes:

[0090] 1) The speech feature encoding module is pre-trained; specifically, it is pre-trained using large-scale unlabeled speech data, and the output of the penultimate layer of the pre-trained network is used as the speech feature, thereby achieving better results compared with features such as Mel spectrum.

[0091] 2) The speech sample fragment set is sequentially input into the pre-trained speech feature encoding module and the face animation decoding prediction module for training to obtain the optimal generator; the optimal generator is used to generate a three-dimensional face animation sequence based on the speech sample fragments; wherein, during the training process of the optimal generator, the loss function includes a position loss function, a velocity loss function and a smoothness loss function.

[0092] The positional loss function is used to supervise the first-norm of each frame of the 3D face animation sequence to ensure that each frame of the generated face animation sequence is identical to the ground truth. The velocity loss function is used to supervise the first derivative of each frame of the 3D face animation sequence to ensure that the inter-frame variation trend of the generated face animation sequence is identical to the ground truth variation trend. The smoothness loss function is used to supervise the second derivative of each frame of the 3D face animation sequence to ensure that the generated face animation sequence does not have too many abrupt changes. By using a self-supervised pre-trained speech feature extractor with a large amount of data, the model's support for different languages ​​and interjections is increased.

[0093] Specifically, the location term loss function is:

[0094]

[0095] The velocity term loss function is:

[0096]

[0097] The smoothing term loss function is:

[0098]

[0099] Among them, LapLace(y i )=y i-1 +y i+1 -2*y i L pos Let y represent the location-based loss function. i This represents the 3D face animation sequence of the i-th frame. Denotes the ground truth value of the 3D template face sequence in the i-th frame, || || 1 Let L denote the norm. vec Let y represent the velocity term loss function. i-1 This represents the (i-1)th frame of the 3D face animation sequence. L represents the ground truth value of the 3D template face sequence in the (i-1)th frame. laplace Let y represent the loss function for the smoothing term. i+1 This represents the (i+1)th frame of the 3D face animation sequence.

[0100] 3) The optimal generator and discriminator are trained iteratively and alternately to obtain a speech-driven 3DMM template face model. Specifically, a Generative Adversarial Network (LSGAN) training scheme is adopted. The LSGAN loss is added to the supervision of the face animation sequence to stabilize the output of the face lip-sync animation sequence, thereby improving the stability and realism of the face animation. The generator and discriminator are trained alternately for 200 iterations, where the loss for training the discriminator is... D for:

[0101]

[0102] Where b = 1 and a = -1.

[0103] Loss of training generator all for:

[0104] Loss all =Loss pos +0.1*Loss vec +0.05*Loss LapLace+0.1*Loss G

[0105] Among them, Loss G =E x～y (D(x)-c) 2 c = 0.

[0106] Step 400: Based on the 3D reconstructed face and the 3D template face sequence, construct a 3D reconstructed voice-driven face sequence.

[0107] Step 400 specifically includes:

[0108] 1) Obtain the 3DMM template face S corresponding to the 3D template face sequence.

[0109] 2) Based on the 3D reconstructed face T and the 3DMM template face S, calculate the deformation parameter Q from the 3DMM template face to the 3D reconstructed face; specifically, in a practical application, firstly, the 3D reconstructed face and the 3DMM template face S are redirected using the NICP (Normal Iterative ClosestPoint) algorithm; then, the calculation formula T = QS is solved based on the NICP algorithm to obtain the deformation parameter Q from the 3DMM template face to the 3D reconstructed face.

[0110] 3) Based on the deformation parameters and the three-dimensional template face sequence, a three-dimensional reconstructed voice-driven face sequence is obtained.

[0111] In summary, this invention performs high-precision 3D face reconstruction based on multi-view RGB face images. It uses multi-view face images as input to extract face features at different resolutions. After feature extraction from all perspectives, feature fusion is performed by calculating correlation. The fused result is input into an optical flow prediction module to obtain optical flow prediction, thereby obtaining the 3D reconstruction result from the main viewpoint. Then, the reconstructed 3D face is redirected with a 3DMM template face model, and voice-driven reconstruction is applied to the reconstructed face. This invention implements an end-to-end face reconstruction and driving system, enabling users to quickly perform face reconstruction and voice-driven reconstruction.

[0112] Example 2

[0113] like Figure 4 As shown, in order to execute the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, this embodiment also provides a voice-driven face sequence generation system based on reconstructed face, including:

[0114] The data acquisition subsystem 101 is used to acquire a set of data to be processed; the data to be processed includes a set of multi-view face images to be processed, a voice segment to be processed, and the relative camera extrinsic parameters corresponding to each image in the set of multi-view face images to be processed.

[0115] A face reconstruction subsystem 201 is used to input the set of multi-view face images to be processed and the relative camera extrinsic parameters corresponding to each image into a 3D face reconstruction model to obtain a 3D reconstructed face. The 3D face reconstruction model includes an optical flow feature extraction module and a face construction module. The optical flow feature extraction module is used to extract features from each image in the set of multi-view face images to obtain predicted face optical flow features. The face construction module is used to reconstruct the face through a triangulation process based on the predicted face optical flow features and the relative camera extrinsic parameters corresponding to each image to obtain a 3D reconstructed face.

[0116] A template face sequence construction subsystem 301 is used to input the speech segment to be processed into a speech-driven face sequence model to obtain a three-dimensional template face sequence. The speech-driven face sequence model is obtained by training the speech-driven 3DMM template face model based on a speech sample segment set. Each sample in the speech sample segment set includes a speech sample segment and the corresponding ground truth value of the three-dimensional template face sequence. The speech-driven 3DMM template face model includes a generator and a discriminator. The generator includes a speech feature encoding module and a face animation decoding and prediction module. Both the speech feature encoding module and the face animation decoding and prediction module are Transformer structures.

[0117] The face reconstruction sequence construction subsystem 401 is used to construct a three-dimensional reconstructed voice-driven face sequence based on the three-dimensional reconstructed face and the three-dimensional template face sequence.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0119] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A speech-driven face sequence generation method based on reconstructed face, characterized in that, The methods include: Obtain a set of data to be processed; The data to be processed includes a set of multi-view face images to be processed, a voice segment to be processed, and the relative camera extrinsic parameters corresponding to each image in the set of multi-view face images to be processed; The set of multi-view face images to be processed and the relative camera extrinsic parameters corresponding to each image are input into a 3D face reconstruction model to obtain a 3D reconstructed face. The 3D face reconstruction model includes an optical flow feature extraction module and a face construction module. The optical flow feature extraction module extracts features from each image in the set of multi-view face images to obtain predicted face optical flow features. The face construction module reconstructs the face through a triangulation process based on the predicted face optical flow features and the relative camera extrinsic parameters corresponding to each image to obtain a 3D reconstructed face. The optical flow feature extraction module is obtained by training a preset convolutional neural network based on a multi-view face sample set. The construction process of the multi-view face sample set specifically includes: acquiring a set of original multi-view face images; determining a 3D face model based on a preset topologically consistent head model and a preset 3D head model; projecting the 3D face model onto each image in the original multi-view face image set using weak perspective projection, and segmenting it to obtain a segmented multi-view face image set; adjusting the segmented multi-view face images according to the weak perspective projection of the 3D face model. The camera intrinsic parameters corresponding to each image are collected to obtain a processed multi-view face image set; a point on the 3D face model is randomly selected as a marker point; the marker point is projected onto any two images in the processed multi-view face image set to obtain a first pixel and a second pixel; the pixel optical flow between two processed multi-view face images is determined based on the first pixel and the second pixel; the segmented multi-view face image set and the pixel optical flow between any two corresponding images constitute a multi-view face sample; multiple multi-view face samples constitute a multi-view face sample set; The multi-view face image set includes a main visual image and multiple auxiliary visual images; the preset convolutional neural network includes: a first convolutional unit, used to perform multiple convolutions on the main visual image to obtain the main facial features; a second convolutional unit, used to perform multiple deformable convolutions on each auxiliary visual image to obtain auxiliary facial features; a feature fusion unit, used to fuse the main facial features and multiple auxiliary facial features based on feature correlation to obtain comprehensive facial features; and an optical flow estimation unit, used to determine the predicted facial optical flow features based on the comprehensive facial features according to the DenseNet structure. The speech segment to be processed is input into a speech-driven face sequence model to obtain a 3D template face sequence. The speech-driven face sequence model is obtained by training the speech-driven 3DMM template face model based on a speech sample segment set. Each sample in the speech sample segment set includes a speech sample segment and the corresponding ground truth value of the 3D template face sequence. The speech-driven 3DMM template face model includes a generator and a discriminator. The generator includes a speech feature encoding module and a face animation decoding and prediction module. Both the speech feature encoding module and the face animation decoding and prediction module are Transformer structures. The training process of the voice-driven 3DMM template face model specifically includes: The speech feature encoding module is pre-trained; the speech sample fragment set is sequentially input into the pre-trained speech feature encoding module and the face animation decoding and prediction module for training to obtain an optimal generator; the optimal generator is used to generate a 3D face animation sequence based on the speech sample fragments; wherein, during the training process of the optimal generator, the loss function includes a positional loss function, a velocity loss function, and a smoothing loss function; the positional loss function is used to supervise the first norm of each frame of the 3D face animation sequence; the velocity loss function is used to supervise the first derivative of each frame of the 3D face animation sequence; the smoothing loss function is used to supervise the second derivative of each frame of the 3D face animation sequence; the optimal generator and the discriminator are iteratively trained alternately to obtain a speech-driven 3DMM template face model; Based on the reconstructed 3D face and the 3D template face sequence, a voice-driven 3D reconstructed face sequence is constructed.

2. The speech-driven face sequence generation method based on reconstructed face according to claim 1, characterized in that, The process of determining the 3D face model based on a preset topologically consistent head model and a preset 3D head model specifically includes: The face region is cropped from the preset topologically consistent head model to obtain a topologically consistent face model; The iterative nearest-point algorithm is used to align and crop the topologically consistent face model with the preset 3D head model in sequence to obtain the 3D face model.

3. The speech-driven face sequence generation method based on reconstructed face according to claim 1, characterized in that, Based on the first pixel and the second pixel, the pixel optical flow between the two processed multi-view face images is determined, specifically including: Determine whether the values ​​of the first pixel and the second pixel are both integer values; When both the value of the first pixel and the value of the second pixel are integer values, the optical flow of the first pixel and the optical flow of the second pixel are calculated based on the values ​​of the first pixel and the second pixel, respectively. When either the value of the first pixel or the value of the second pixel is a non-integer value, the optical flow of the first pixel and the optical flow of the second pixel are calculated using bilinear interpolation based on the values ​​of the first pixel and the second pixel, respectively. The pixel optical flow between the two processed multi-view face images is determined based on the optical flow of the first pixel and the optical flow of the second pixel.

4. The speech-driven face sequence generation method based on reconstructed face according to claim 1, characterized in that, The feature fusion unit specifically includes: According to the formula V(x,y,i) The function 9+j)=Corr(x,y,x2+i,y2+j;I1,I2) determines the comprehensive facial features. Where V represents the comprehensive facial features, x represents the horizontal coordinate variable of the pixel in the main visual image, and y represents the vertical coordinate variable of the pixel in the main visual image; Corr(x,y,x2+i,y2+j;I1,I2) represents the feature correlation calculation between the main visual image and the secondary visual image in the i×j neighborhood, where i and j both take values ​​of [-4,-3,…,0,1,..3,4], I1 represents the main visual image, I2 represents the secondary visual image, and (x2,y2) represents the pixel coordinates in the secondary visual image.

5. The speech-driven face sequence generation method based on reconstructed face according to claim 1, characterized in that, The location term loss function is: The velocity term loss function is: The smoothing term loss function is: in, ; This represents the positional loss function. This represents the 3D face animation sequence of the i-th frame. This represents the ground truth value of the 3D template face sequence in the i-th frame. Describes the norm, Represents the loss function for the velocity term. This represents the (i-1)th frame of the 3D face animation sequence. This represents the ground truth value of the 3D template face sequence in the (i-1)th frame. Represents the loss function of the smoothing term. This represents the (i+1)th frame of the 3D face animation sequence.

6. The speech-driven face sequence generation method based on reconstructed face according to claim 1, characterized in that, Based on the reconstructed 3D face and the 3D template face sequence, a voice-driven 3D reconstructed face sequence is constructed, specifically including: Obtain the 3DMM template face corresponding to the 3D template face sequence; Based on the NICP algorithm, the deformation parameters from the 3DMM template face to the 3D reconstructed face are calculated according to the 3D reconstructed face and the 3DMM template face. Based on the deformation parameters and the three-dimensional template face sequence, a three-dimensional reconstructed voice-driven face sequence is obtained.

7. A voice-driven face sequence generation system based on reconstructed faces, characterized in that, The system employing the speech-driven face sequence generation method based on reconstructed face as described in any one of claims 1-6 comprises: The data acquisition subsystem is used to acquire a set of data to be processed; the data to be processed includes a set of multi-view face images to be processed, a voice segment to be processed, and the relative camera extrinsic parameters corresponding to each image in the set of multi-view face images to be processed; A face reconstruction subsystem is used to input the set of multi-view face images to be processed and the relative camera extrinsic parameters corresponding to each image into a 3D face reconstruction model to obtain a 3D reconstructed face. The 3D face reconstruction model includes an optical flow feature extraction module and a face construction module. The optical flow feature extraction module is used to extract features from each image in the set of multi-view face images to obtain predicted face optical flow features. The face construction module is used to reconstruct the face through a triangulation process based on the predicted face optical flow features and the relative camera extrinsic parameters corresponding to each image to obtain a 3D reconstructed face. A template face sequence construction subsystem is used to input the speech segment to be processed into a speech-driven face sequence model to obtain a 3D template face sequence. The speech-driven face sequence model is obtained by training the speech-driven 3DMM template face model based on a speech sample segment set. Each sample in the speech sample segment set includes a speech sample segment and the corresponding ground truth value of the 3D template face sequence. The speech-driven 3DMM template face model includes a generator and a discriminator. The generator includes a speech feature encoding module and a face animation decoding and prediction module. Both the speech feature encoding module and the face animation decoding and prediction module are Transformer structures. A face reconstruction sequence construction subsystem is used to construct a three-dimensional reconstructed voice-driven face sequence based on the three-dimensional reconstructed face and the three-dimensional template face sequence.

Images