A method, device and apparatus for voice-driven video face lip shape reconstruction

Through NeRF face renderer and variational autoencoder technology, a target 3D face key point sequence matching audio features is generated, solving the problem of high requirements for video materials and audio in the existing technology, and achieving natural and smooth Lip reconstruction, suitable for multifaceted video and noise audio scenes.

CN118351222BActive Publication Date: 2024-11-26CHONGQING WEIAIYUN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410487299.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-19
Publication Date
2024-11-26
Estimated Expiration
2044-04-19

AI Technical Summary

Technical Problem

The existing voice-driven lip generation technology has high requirements for video materials and audio, and the generated lip shape is poor, unnatural, and is not suitable for videos with large face movement amplitude and audio scenes containing noise.

Method used

By decomposing the original video data and audio data, use the NeRF face renderer and variational autoencoder to generate target 3D face key points sequences that match the audio features, perform lip reconstruction, reduce the requirements for audio and video data, and improve the requirements for audio and video data. The naturalness and clarity of the lip shape.

Benefits of technology

Achieve natural and smooth lip reconstruction in multi-face video scenes, reducing the requirements for audio and video data, avoiding jaw jumps and unnatural lip fits, and improving the authenticity and nature of the reconstructed video. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118351222B_ABST
    Figure CN118351222B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and equipment for voice-driven video face lip reconstruction. The method comprises: decomposing original video data to obtain original picture sequence and driving audio data; obtaining original head picture sequence and original 3D face key point sequence of selected face based on original picture sequence; training NeRF face renderer using original head picture sequence and original 3D face key point sequence of selected face; obtaining audio features based on driving audio data; generating target 3D face key point sequence matching audio features using trained variational autoencoder; inputting target 3D face key point sequence into trained NeRF face renderer to obtain generated face picture sequence; fusing original picture sequence and generated face picture sequence to obtain synthetic picture sequence; combining driving audio data and synthetic picture sequence to obtain reconstructed video. The present invention can obtain more natural and high-definition lip replacement video compared with the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of virtual digital humans, and in particular to a method, device and equipment for reconstructing the lip shape of a human face in a voice-driven video. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology and the rapid popularity of the concept of the metaverse, voice-driven face video generation technology has been used more and more widely. Especially in the field of virtual digital humans, such as digital human anchors, digital human customer service and digital human teaching, they can simulate human thinking and behavior and provide various services and entertainment for humans. Among them, voice-driven lip shape generation technology is involved. This technology allows the characters in static images or dynamic videos to perform lip shape conversion, output videos that match the target voice, and realize self-made video dubbing.

[0003] The general voice-driven lip shape generation technology has high requirements for video materials. In order to obtain better synthesis effects, stable video materials are often used. For example, the Chinese patent with patent publication number CN11 3256765A discloses a virtual human video synthesis method based on voice drive and face self-drive. The patent requires that the facial expressions of the characters in the video materials used are natural, the eyes are naturally open, and the mouth is naturally open with teeth exposed, so as to ensure the clarity of the subsequent lip and tooth features. The video materials to be driven often need to be recorded in advance, and the general use scenes are limited. It is especially not suitable for videos with large facial movements and user-level entertainment videos.

[0004] The general speech-driven lip shape generation technology has high requirements for the driving audio. The speech needs to be clear and noise-free, and there must not be long silent segments. When there is reverberation and noise in the driving audio, it often affects the lip shape generation effect, and the jitter is obvious. When the driving audio has a long silent segment, the generated lip shape is greatly affected by the lip shape of the original video, and it cannot achieve natural closed lips or slightly open lips. In addition, due to the lip shape driving model, the lip shape generation of Chinese audio is poor. Summary of the invention

[0005] The present invention aims to solve the technical problems that the existing speech-driven lip shape generation technology has high requirements on video materials and driving audio, limited application scenarios, poor clarity and unnaturalness of the generated lip shape, and provides a method, device and equipment for speech-driven video face lip shape reconstruction.

[0006] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a method for speech-driven video face lip reconstruction, including: decomposing original video data to obtain original picture sequence and driving audio data; obtaining the original head picture sequence and the original 3D face key point sequence of the selected face based on the original picture sequence; training the NeRF face renderer using the original head picture sequence and the original 3D face key point sequence of the selected face; obtaining audio features based on the driving audio data; generating a target 3D face key point sequence matching the audio features using a trained variational autoencoder; inputting the target 3D face key point sequence into the trained NeRF face renderer to obtain a generated face picture sequence; fusing the original picture sequence and the generated face picture sequence to obtain a synthesized picture sequence; and combining the driving audio data and the synthesized picture sequence to obtain a reconstructed video.

[0007] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a device for voice-driven video face lip reconstruction, which is used to implement the method for voice-driven video face lip reconstruction described in the first aspect of the present invention, including: a decomposition module, decomposing original video data to obtain original picture sequence and driving audio data; a video preprocessing module, obtaining the original head picture sequence and original 3D face key point sequence of the selected face based on the original picture sequence; an audio preprocessing module, obtaining audio features based on the driving audio data; a face lip shape generation module, executing: using the original head picture sequence and the original 3D face key point sequence of the selected face to train the NeRF face renderer; using the trained variational autoencoder to generate a target 3D face key point sequence that matches the audio features; inputting the target 3D face key point sequence into the trained NeRF face renderer to obtain a generated face picture sequence; a face lip shape fusion module, executing: fusing the original picture sequence and the generated face picture sequence to obtain a synthetic picture sequence; combining the driving audio data and the synthetic picture sequence to obtain the reconstructed video.

[0008] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for voice-driven video facial lip shape reconstruction described in the first aspect of the present invention.

[0009] In the speech-driven video face lip shape reconstruction technical solution provided by the present invention, faces can be selected in multi-face video data for lip shape reconstruction, which can be applicable to multi-face video scenes. After obtaining the audio features of the selected faces, a target 3D face key point sequence matching the audio features is first generated by a variational autoencoder, and the target 3D face key point sequence has strong coherence. The target 3D face key point sequence is then processed by a NeRF face renderer to obtain a generated face picture sequence, so that the lip shape of the generated face picture sequence is not affected by the original lip shape in the original video data. Compared with the existing technology, it is smoother and reduces the requirements for audio data. There will be no chin jumping and unnatural lip shape fitting. The pictures rendered by the NeRF face renderer retain more facial details of the characters, improve the authenticity and naturalness of lip shape generation, and also reduce the requirements for the original video data. Scenes such as face shaking in the video can be accepted. In addition, the NeRF face renderer is trained by using the original head picture sequence and the original 3D face key point sequence of the selected face, which can further improve the naturalness and clarity of the lip shape in the reconstructed video. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of a method for voice-driven video face lip shape reconstruction in a preferred embodiment of the present invention;

[0011] Figure 2 It is a flowchart of a NeRF face renderer in a preferred embodiment of the present invention;

[0012] Figure 3 It is a schematic diagram of plane movement of generating a face image in a preferred embodiment of the present invention;

[0013] Figure 4 This is an execution flow chart of a device for voice-driven video face lip shape reconstruction in a preferred embodiment of the present invention;

[0014] Figure 5 is a flow chart of a video preprocessing module in a preferred embodiment of the present invention;

[0015] Figure 6 This is a flow chart of an audio preprocessing module in a preferred embodiment of the present invention;

[0016] Figure 7 This is a flow chart of a human face lip shape generation module in a preferred embodiment of the present invention;

[0017] Figure 8 This is a flow chart of a face and lip shape fusion module in a preferred embodiment of the present invention;

[0018] Fig. 9 It is a hardware structure diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0019] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0020] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0021] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "connection" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection of two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0022] The present invention discloses a method for reconstructing the lip shape of a human face in a speech-driven video. In a preferred embodiment, Figure 1 As shown, the method includes a decomposition step, a video preprocessing step, an audio preprocessing step, a face lip shape generation step and a face lip shape fusion step, specifically:

[0023] The decomposition step includes: step S1, decomposing the original video data to obtain the original picture sequence and driving audio data. Decomposing the original picture sequence and driving audio data of the same length from the same original video data. Preferably, but not limited to, using the audio and video processing tool FFmpeg to convert, crop, decompose and other operations on the audio and video format of the original video data. For the convenience of subsequent processing, preferably, the video frame number fps of the original video data needs to be greater than 25 frames. When the fps is greater than 25 frames, use the ffmpeg audio and video processing tool to reduce the video to 25 frames, and maintain the original resolution, and split the frame-reduced video into the original picture sequence set imgs_1.

[0024] The video preprocessing steps include:

[0025] Step S2, based on the original picture sequence imgs_1, obtaining the original head picture sequence head_imgs and the original 3D face key point sequence 3d_lms1 of the selected face.

[0026] In step S2, preferably, obtaining the original head image sequence head_imgs of the selected face based on the original image sequence imgs_1 includes:

[0027] Step S201, performing face detection on the original image in the original image sequence imgs_1, can use the existing face detection method to detect whether there is a face in the original image, such as the face detection method used in the Chinese patent with publication number CN101369315A or CN101887514A, or can also use the face detection model based on the dlib library, which will not be repeated here.

[0028] Step S202, label the faces detected in the original image, and determine the selected face from the labeled faces. Perform face detection on each frame in the original image sequence imgs_1, obtain the regional coordinate set c_list of the corresponding face, and label the detected faces as n1, n2... In each frame of the image, calculate the area of ​​the labeled face region according to c_list, and obtain the face with the largest area face region by comparison as the selected face. It is also possible to track the labeled face region in the original image sequence imgs_1, and select the face that meets the preset conditions (such as speaking actions or meets the preset face identity features) as the selected face. It can be seen that the present invention can support the adaptation scene of multiple faces through multi-face comparison, thereby expanding the selection range of the driven video.

[0029] Step S203, cropping the cropped face image sequence imgs_2 of the selected face from the original image sequence imgs_1 according to a preset alignment and cropping method, and scaling the cropped face image sequence imgs_2 to obtain the face image sequence imgs_3.

[0030] Preferably, the preset alignment and cropping method includes: using a face key point detection algorithm to obtain 68 face key point coordinates for the face selected in the original picture, and calculating the cropping area according to the face key point coordinates to obtain a picture sequence aligned in FFHQ style. The FFHQ style picture is square, with the tip of the nose in the center, the middle of the neck as the lower boundary, and the top of the head slightly exceeding the hair as the upper boundary. Record the upper left and lower right coordinates of the cropping to obtain the motion area c (c_x1, c_x2, c_y1, c_y2) of the entire video face. Crop each frame of the picture according to the area, that is, obtain the original face material sequence set, that is, the face cropped picture sequence imgs_2. Using FFHQ style alignment, it is guaranteed to the greatest extent that the face is the main area of ​​the picture, and the key information of the face will not be missing, which can improve the accuracy and naturalness of subsequent training and reasoning models. The face key point detection algorithm can adopt the existing 68 face key point detection model based on the dlib library, and can also adopt the existing regression-based key point detection algorithm or template fitting key point detection algorithm.

[0031] Furthermore, in order to unify the training process and better retain the facial details of the trained faces, the face cropped image sequence imgs_2 is scaled to a preset size, and the preset size is not limited to 512*512, and a scaled face material sequence set, namely, the face image sequence imgs_3, can be obtained.

[0032] Step S204, extract the original head image sequence head_imgs including only the head partition from the face image sequence imgs_3. The area including the face and hair is used as the head partition. Specifically, each image of the face image sequence imgs_3 is partitioned using the image semantic segmentation technology, and the obtained partition map is used to extract the image sequence including only the head partition according to the color mask, that is, the original head image sequence head_imgs. The original head image sequence head_imgs will be used as the input for the subsequent NeRF face renderer training. Image semantic segmentation technology is not limited to the use of the Segmenter model based on Mediapipe, or the segmentation method disclosed in the Chinese patent with publication number CN109741338A or CN112949605A, which will not be repeated here.

[0033] Further preferably, in step S2, obtaining an original 3D face key point sequence 3d_1ms1 of the selected face based on the original image sequence imgs_1 includes:

[0034] Step 211, using a facial key point detection algorithm to obtain a 2D facial key point set 2d_1ms1 of the facial image sequence imgs_3;

[0035] Step 212, reconstruct the 3DMM model of the selected face using the 2D face key point set 2d_1ms1 and the face image sequence imgs_3, and extract the original 3D face key point sequence 3d_1ms1 from the 3DMM model. The full name of the 3DMM model is 3D MorphableFace Model, which can be a deformable face model. The face3d technology can be used to extract the three-dimensional deformable face model 3DMM of the selected face based on the 2D face key point set 2d_1ms1 and the face image sequence imgs_3. The three-dimensional deformable face model 3DMM contains a 3D face key point set, that is, the original 3D face key point sequence 3d_1ms1.

[0036] The audio preprocessing steps include:

[0037] Step S4, obtaining audio features based on the driving audio data. In order to reduce the impact of noise such as reverberation in the driving audio data on the reconstructed video and improve the naturalness and accuracy of the matching between the lip shape and the audio in the generated picture sequence, preferably, step S4 includes:

[0038] Step S401, separate the human voice audio and noise audio of the same duration from the driving audio data audio; the existing human voice accompaniment separation model (such as uvr5) can be used to separate the human voice and noise in the driving audio data audio to obtain the human voice audio and noise audio of the same duration.

[0039] Step S402, extracting audio features of human voice audio voice. Preferably, but not limited to, using the existing hubert self-supervised model or mel frequency cepstral coefficient (MFCC) method or linear prediction coefficient (LPC) method or linear prediction cepstral coefficient (LPCC) method or perceptual linear prediction (PLP) method. The advantage of self-supervised learning is that there is no label, because for speech, detailed labels will limit the expressiveness of the model, while self-supervised learning will not be affected by it, which makes the self-supervised learning model have very good generalization ability. The audio features will be used as the input of the subsequent variational autoencoder VAE model to obtain the corresponding target 3D face key point set 3d_1ms2.

[0040] The steps of generating face lip shape include:

[0041] Step S3, using the original head picture sequence head_imgs of the selected face and the original 3D face key point sequence 3d_1ms1 to train the NeRF face renderer. NeRF, or Neural Radiant Network, is a method for generating three-dimensional scenes. Its core concept is to use MLP neural networks to learn the radiation properties of light in the scene. The trained NeRF face renderer head_ckpt can reconstruct the face image by inputting 3DMM face key points, and the face image generated by the NeRF face renderer retains the texture details of the original image face, and only needs 3DMM face key points to drive it.

[0042] Step S5, using the trained variational autoencoder (VAE model) to generate a target 3D face key point sequence 3d_1ms2 that matches the audio features. The basic network structure of the VAE model consists of two parts: an encoder and a decoder. To support audio containing long silent segments as driving audio. When the silent segment is used as the driving audio, the lip key points in the target 3D face key points generated by the variational autoencoder (VAE model) of the present invention maintain a slightly open movement, which is not affected by the original driven video. After the trained NeRF face renderer is subsequently input, the NeRF face renderer outputs a generated face picture with a slightly open mouth shape corresponding to the long silent segment, which is more natural and smooth. The target 3D face key point sequence 3d_1ms2 generated by the variational autoencoder (VAE model) has a strong coherence, so that the reconstructed video is synchronized with the face movement.

[0043] Step S6, input the target 3D facial key point sequence 3d_1ms2 to the trained NeRF face renderer to obtain the generated face image sequence imgs_4.

[0044] The steps of face lip fusion include:

[0045] Step S7, fusing the original image sequence imgs_1 and the generated face image sequence imgs_4 to obtain a synthesized image sequence imgs_5;

[0046] Step S8, combining the driving audio data audio and the synthetic picture sequence imgs_5 to obtain a reconstructed video. Specifically, ffmpeg can be used to combine the fused synthetic picture sequence imgs_5 and the driving audio audio into a complete reconstructed video.

[0047] In a preferred embodiment, the existing static NeRF can be represented by the following implicit function F:

[0048] F: (x, d) → (c, σ) (1)

[0049] where x is the 3D position in space, d is the viewing direction, and c and σ represent the color and opacity in the radiation field, respectively.

[0050] The input of static NeRF is a multi-view picture to obtain a static 3D scene that can be observed from multiple perspectives. In order to represent a dynamic speaking video, the present invention uses a fixed-view video picture sequence as input, that is, the camera perspective is fixed. At the same time, in order to control the generation of the face, the 3D key points (I) at consecutive moments are used as implicit control conditions, which can be expressed by the following implicit function F:

[0051] F θ :(x,d,I)→(c,σ) (2)

[0052] In the formula I∈R T×204 , using 3D key points of T consecutive adjacent frames (i.e., face key point set I) to represent the facial shape, thereby improving the continuity between adjacent frames. T is preferably, but not limited to, 2, 3, or 4.

[0053] In order to allow the MLP network (multi-layer perceptron network) in the NeRF face renderer to learn more high-frequency information and thus enrich the rendering details, compared with the existing technology, the NeRF face renderer is equipped with a position encoding module, a facial shape data acquisition module and a connection module. For details, refer to Figure 2 , NeRF face renderer includes:

[0054] The position encoding module performs position encoding (Positional Encoding) on ​​the position coordinate x of each facial key point to obtain the first dimension coordinate information, and performs position encoding (Positional Encoding) on ​​the observation direction coordinate to obtain the second dimension coordinate information; wherein, the facial key points belong to the original 3D facial key point sequence or the target 3D facial key point sequence. Because x and d have small changes, small dimensions, and similar adjacent inputs, the details cannot be rendered after learning the MLP neural network. In order to allow MLP to learn more high-frequency information and thus enrich the details of the rendering, d and x are positionally encoded (Positional Encoding), x is encoded from 3D position to 67 dimensions, and d is encoded from 3D position to 27 dimensions. Frequency transformation is used in NeRF to implement position encoding, and the specific formula is as follows:

[0055] γ(p)=(sin(2 0 πp), cos(2 0 πp),…,sin(2 L-1 πp), cos(2 L-1 πp)) (3)

[0056] Among them, p represents the initial position information of x / d, L represents the parameter of the control encoding dimension of x / d, γ(p) represents the representation obtained by position encoding of x / d, and the dimension of γ(p) = the dimension of p*2*L. Taking the observation direction d as an example, its initial position information p is 3-dimensional, and the representation γ(p) obtained by taking L as 4 and performing position encoding is a 24-dimensional representation. If the original 3-dimensional position information is retained, the second dimensional coordinate information obtained after the position encoding process is 27-dimensional.

[0057] The facial shape data acquisition module combines the facial key points of the adjacent T consecutive frames of the i-th frame for each facial key point of the original 3D facial key point sequence / target 3D facial key point sequence to obtain a facial key point set, and inputs the facial key point set into a convolutional network encoder (Encoder) to obtain facial shape data, where i is the frame index of the original 3D facial key point sequence / target 3D facial key point sequence, and T is a positive integer greater than 2. The adjacent T consecutive frames include the i-1th frame, the i-th frame, and the i+1th frame, or the i-th frame, the i+1th frame, and the i+2th frame, or the i-2th frame, the i-1th frame, and the i-th frame.

[0058] I is input into a convolutional network encoder to regress the latent code of each frame and ensure the stability of the internal 3D key points.

[0059] The connection module connects the facial shape data of each facial key point and the first dimension coordinate information to obtain the coordinate fusion information of each facial key point. The encoded I and the encoded x are connected as the initial input of the MLP network. By training the 3D key points, the head rendering can be implicitly controlled.

[0060] The MLP network obtains the color and opacity of each facial key point in the radiation field based on the coordinate fusion information, the first dimension coordinate information and the second dimension coordinate information of each facial key point.

[0061] More preferably, if Figure 2 As shown, the MLP network includes a main branch, and a color branch and an opacity branch respectively connected to the output end of the main branch; the main branch includes 8 cascaded fully connected linear layers, the color branch includes 3 cascaded fully connected linear layers, and the opacity branch includes one fully connected linear layer; the coordinate fusion information of each facial key point is input into the first fully connected linear layer of the main branch, the first dimensional coordinate information of each facial key point is input into the seventh fully connected linear layer of the main branch, and the second dimensional coordinate information of each facial key point is input into the second fully connected linear layer of the color branch.

[0062] The main branch consists of 8 fully connected linear layers with 256-dimensional outputs. Each layer uses ReLU as the activation function. The position x (the first dimension-upgraded coordinate information) after position encoding is added again at the 7th fully connected linear layer as the input of the 7th fully connected linear layer. The opacity branch obtains the opacity σ of the facial key points in the radiation field through a fully connected linear layer with 256-dimensional output. The color branch passes through 3 fully connected linear layers with 128-dimensional outputs, and the direction d after position encoding is added to the second fully connected linear layer of the color branch to obtain the color RGB of the facial key points in the radiation field.

[0063] The rendering module generates pixel information of each facial key point in the generated face image based on the color and opacity of the facial key point in the radiation field.

[0064] According to the principle of volume rendering, to render each pixel, a camera ray is fired:

[0065] r(t)=o+t·d (4)

[0066] In the radiation field, t is the current time parameter, the camera center is o, and the viewing direction is d. Finally, the color C of the ray can be expressed by integral:

[0067]

[0068] t n and t f are the near and far boundaries of ray r, c θ and σ θ It is implicit F θ The output of l represents the information of the facial key points, where T(t) is the distance from t along the ray. n The cumulative transparency up to t is defined as:

[0069]

[0070] In this embodiment, the training process of the NeRF face renderer is to use the original head image sequence head_imgs as the real image sequence, and the original 3D face key point sequence 3d_1ms1 as the training sample set. The training samples correspond to the original 3D face key points of each frame. During the training process, each face key point of each frame of the 3D face key point sequence 3d_1ms1 is input into the above-mentioned NeRF face renderer. The NeRF face renderer outputs the pixel information of the face key point of each frame (pixel information on the rendered face image), and then obtains the rendered face image corresponding to the 3D face key point of the frame, calculates the loss function value L(θ) according to the real image and the rendered face image, and adjusts the network parameters of the MLP network according to the loss function value.

[0071]

[0072] Among them, R is the set camera light, C t is the color value of the real image, C θ (r, l) represents the color value of the rendered face image.

[0073] In a preferred embodiment, a new face offset algorithm is designed to evaluate whether the generated face image sequence has obvious offset. After the offset evaluation, the naturalness of the fusion of the new face and the original video can be improved. In step S7, the original image sequence and the generated face image sequence are fused to obtain a synthetic image sequence, including:

[0074] Step S701, calculate the coordinates of the head motion center point of the face image sequence imgs_3, and record them as the average head motion center point coordinates of the initial face.

[0075] Step S702, calculating and generating the head motion center point coordinates of the face image sequence, which are recorded as the average head motion center point coordinates of the newborn adult face.

[0076] First, use the facial key point detection technology to obtain the 2D facial key set. The 2D facial key point set of the facial image sequence is 2d_1ms1, and the 2D facial key point set of the generated facial image sequence is 2d_1ms2. For the facial image sequence / generated facial image sequence, calculate the average facial contour point of the facial image sequence / generated facial image sequence based on the 2D facial key point set, and then average it again to obtain the average head motion center point coordinate avg(x, y). The specific calculation formula is as follows:

[0077]

[0078] n represents the number of face image sequences / generated face image sequences, and m represents the number of face contour key points. Specifically, the 1st to 17th key points among the 68 face key points are face contour key points. (x ij ,y ij ) represents the image plane coordinates of the jth facial contour key point in the i-th image.

[0079] According to the above calculation formula (8), the coordinates of the average head motion center point of the initial face are expressed as avg u (x, y), the average head motion center coordinates of the newborn adult face are expressed as avg v (x, y).

[0080] Step s703, based on the average head motion center coordinates avg of the initial face u (x, y) and the average head motion center coordinates of newborn adult faces avg v(x, y) to get the average offset offset(a, b):

[0081] offset(a, b) = avg u (x, y)-avg v (x,y) (9)

[0082] Step S704: calculate the offset value offset of the x-axis of the plane coordinate system based on the average offset offset(a, b) a and the offset value of the y-axis b Calculate the offset index Index:

[0083]

[0084] It should be noted that when the value of a in the offset offset(a, b) is negative, the offset is to the right; when the value of a is positive, the offset is to the left; when the value of b is negative, the offset is downward; when the value of b is positive, the offset is upward.

[0085] Step S705: If the offset index is less than or equal to the preset index threshold, the original image sequence imgs_1 and the generated face image sequence imgs_4 are fused to obtain a synthesized image sequence; if the offset index is greater than the preset index threshold, the generated face image sequence imgs_4 is offset adjusted, and the original image sequence imgs_1 and the generated face image sequence after the offset adjustment are fused to obtain a synthesized image sequence. The preset index threshold is preferably, but not limited to, 30 to 50, preferably 40.

[0086] In this embodiment, the fusion process of fusing the original image sequence imgs_1 and the generated face image sequence (or the generated face image sequence after offset adjustment) is:

[0087] Step A, use the facexlib library to detect the face area of ​​the generated face image sequence imgs_4, and mask the detected area to obtain a mask, and blur the edge of the mask to achieve a smooth fusion effect. According to the previously obtained face motion area c(c_x1, c_x2, c_y1, c_y2), scale the mask and the generated face image sequence imgs_4 to the size of (c_x2-c_x1, c_y2-c_y1).

[0088] Step B, the original image sequence imgs_1 and the face sequence imgs_4 generated after scaling in step A are fused pixels according to the mask mask to obtain the image sequence completed by the audio driver, that is, the synthesized image sequence imgs_5.

[0089] In this embodiment, in order to improve the naturalness of the fusion of the new adult face and the original video, it is further preferred that Figure 3 As shown, the generated face picture sequence is offset adjusted, including:

[0090] In the picture size frame (such as 512*512) of the face picture sequence, the face picture is generated based on the average offset plane movement, and the blank area (i.e. Figure 3 Pixel filling is performed in the pixel filling area (in the pixel filling area), and the filling pixel color is preferably white or transparent. The entire image formed in the picture size frame after pixel filling is used as the generated face picture after offset adjustment, and all the generated face pictures after offset adjustment constitute a generated face picture sequence after offset adjustment.

[0091] The present invention also discloses a device for voice-driven video face lip reconstruction, which is used to implement the above-mentioned voice-driven video face lip reconstruction method, such as Figure 4 As shown, including:

[0092] A decomposition module decomposes the original video data to obtain the original picture sequence and driving audio data;

[0093] The video preprocessing module obtains the original head image sequence and the original 3D face key point sequence of the selected face based on the original image sequence. The specific processing flow is as follows: Figure 5 shown.

[0094] The audio preprocessing module obtains audio features based on the driving audio data. The specific processing flow is as follows: Figure 6 shown.

[0095] The face lip shape generation module performs the following steps: using the original head image sequence and the original 3D face key point sequence of the selected face to train the NeRF face renderer; using the trained variational autoencoder to generate the target 3D face key point sequence that matches the audio features; inputting the target 3D face key point sequence into the trained NeRF face renderer to obtain the generated face image sequence. The specific processing flow is as follows: Figure 7 shown.

[0096] The face and lip shape fusion module performs the following operations: fusing the original image sequence and the generated face image sequence to obtain a synthetic image sequence; combining the driving audio data and the synthetic image sequence to obtain a reconstructed video. The specific processing flow is as follows: Figure 8 shown.

[0097] The method and device for voice-driven video face lip reconstruction provided by the present invention can perform modeling training on the face part of the video, and can re-render a clearer and more natural face lip sequence according to the audio features, thereby obtaining a more natural and high-definition lip replacement video, which has the following advantages:

[0098] Applicable scenarios are diverse: Because the rendering and reconstruction of human faces is based on NeRF technology, the requirements for the original video to be driven are lower, and scenes such as shaking faces in the video can be accepted. The video driven by voice will not have chin jumping and unnatural lip shape.

[0099] Diverse driving audio: Supports audio containing long silent segments and audio containing reverberation as driving audio. When silent segments are used as driving audio, the output video lip shape will be slightly open, because for silent input, the 3D key point lips generated by the VAE used in the present invention remain slightly open and are not affected by the original video lip shape. When the driving audio contains reverberation, the normal fitting of the lip shape corresponding to the human voice is less affected by the reverberation because noise separation is done in advance.

[0100] Natural and smooth lip matching: The 3DMM key point sequence generated by VAE has strong coherence and evaluates the offset between the generated face and the original video face. Therefore, the lip shape generated by key point rendering is smoother and synchronized with the face movement when it is fitted back to the original video.

[0101] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for voice-driven video face lip shape reconstruction provided by the present invention.

[0102] like Fig. 9 , which is a schematic diagram of the structure of an electronic device for a method for voice-driven video face lip reconstruction provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a method program for voice-driven video face lip reconstruction.

[0103] Among them, the processor 10 can be composed of an integrated circuit in some embodiments, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes the program or module stored in the memory 11 (for example, a method for executing voice-driven video face lip shape reconstruction, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0104] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the method program for voice-driven video face lip shape reconstruction, etc., but also can be used to temporarily store data that has been output or is to be output.

[0105] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0106] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0107] Fig. 9 Only an electronic device with components is shown, and those skilled in the art will understand that Fig. 9 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0108] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0109] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0110] The present invention also discloses a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method for voice-driven video face lip shape reconstruction provided by the present invention are implemented. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0112] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for reconstructing lip shape from speech-driven videos. It is characterized in that include: Decomposing the original video data to obtain the original picture sequence and driving audio data; Based on the original image sequence, an original head image sequence and an original 3D face key point sequence of the selected face are obtained; The NeRF face renderer is trained using the original head image sequence and the original 3D face key point sequence of the selected face; Acquiring audio features based on driving audio data; Use the trained variational autoencoder to generate a target 3D facial key point sequence that matches the audio features; Input the target 3D face key point sequence to the trained NeRF face renderer to obtain the generated face image sequence; The original image sequence and the generated face image sequence are fused to obtain a synthetic image sequence; Combining the driving audio data and the synthesized picture sequence to obtain a reconstructed video; The NeRF face renderer includes: A position encoding module, which performs position encoding on the position coordinates of each facial key point to obtain first dimensional coordinate information, and performs position encoding on the observation direction coordinates to obtain second dimensional coordinate information; wherein the facial key point belongs to an original 3D facial key point sequence or a target 3D facial key point sequence; A facial shape data acquisition module, for each facial key point of the i-th frame in the original 3D facial key point sequence / target 3D facial key point sequence, combines the facial key points of T consecutive frames adjacent to the i-th frame to obtain a facial key point set, and inputs the facial key point set into a convolutional network encoder to obtain facial shape data, where i is the frame index of the original 3D facial key point sequence / target 3D facial key point sequence, and T is a positive integer greater than 2; A connection module connects the facial shape data of each facial key point and the first dimension coordinate information to obtain the coordinate fusion information of each facial key point; The MLP network obtains the color and opacity of each facial key point in the radiation field based on the coordinate fusion information of each facial key point, the first dimension coordinate information, and the second dimension coordinate information; The rendering module generates pixel information of each facial key point in the generated face image based on the color and opacity of the facial key point in the radiation field.

2. The method for reconstructing lip shape of human face from speech-driven video according to claim 1, It is characterized in that Based on the original image sequence, the original head image sequence of the selected face is obtained, including: Perform face detection on the original images in the original image sequence; Label the faces detected in the original image, and determine the selected face from the labeled faces; A face cropped image sequence of selected faces is cropped from the original image sequence according to a preset alignment cropping method, and the face cropped image sequence is scaled to obtain a face image sequence; An original head image sequence including only head partitions is extracted from the face image sequence.

3. The method for reconstructing lip shape of human face in speech-driven video according to claim 2, It is characterized in that The original 3D facial key point sequence of the selected face is obtained based on the original image sequence, including: Use the facial key point detection algorithm to obtain the 2D facial key point set of the facial image sequence; The 3DMM model of the selected face is reconstructed using the 2D face key point set and the face image sequence, and the original 3D face key point sequence is extracted from the 3DMM model.

4. The method for reconstructing lip shape of human face from speech-driven video as claimed in claim 1, It is characterized in that The MLP network includes a main branch, and a color branch and an opacity branch respectively connected to the output end of the main branch; The main branch includes 8 cascaded fully connected linear layers, the color branch includes 3 cascaded fully connected linear layers, and the opacity branch includes one fully connected linear layer; The coordinate fusion information of each facial key point is input into the first fully connected linear layer of the main branch, the first dimensional coordinate information of each facial key point is input into the seventh fully connected linear layer of the main branch, and the second dimensional coordinate information of each facial key point is input into the second fully connected linear layer of the color branch.

5. The method for reconstructing lip shape of human face in speech-driven video according to claim 1, 2, 3 or 4, It is characterized in that Acquire audio features based on driving audio data, including: Separate vocal audio and noise audio of the same duration from the driving audio data; Extract audio features of human voice audio.

6. The method for reconstructing lip shape of human face in speech-driven video according to claim 1, 2, 3 or 4, It is characterized in that The original image sequence and the generated face image sequence are combined to obtain a synthetic image sequence, including: Calculate the coordinates of the center point of head motion of the face image sequence, and record them as the average center point coordinates of head motion of the initial face; Calculate and generate the coordinates of the center point of head movement of the face image sequence, and record them as the average center point coordinates of head movement of newborn adult faces; The average offset is obtained based on the average head movement center point coordinates of the initial face and the average head movement center point coordinates of the newborn adult face; Calculating a shift index based on the shift value of the average shift on the x-axis and the shift value of the y-axis; If the offset index is less than or equal to the preset index threshold, the original image sequence and the generated face image sequence are fused to obtain a synthetic image sequence. If the offset index is greater than the preset index threshold, the generated face image sequence is offset adjusted, and the original image sequence and the generated face image sequence after the offset adjustment are fused to obtain a synthetic image sequence.

7. The method for reconstructing lip shape of human face from speech-driven video as claimed in claim 6, It is characterized in that Perform offset adjustment on the generated face image sequence, including: Within the picture size frame of the face picture sequence, a face picture is generated by plane movement based on an average offset, and pixels are filled in the blank area within the picture size frame after the plane movement of the generated face picture. The entire image formed within the picture size frame after the pixel filling is used as the offset-adjusted generated face picture, and all the offset-adjusted generated face pictures constitute a sequence of offset-adjusted generated face pictures.

8. A device for voice-driven video face lip reconstruction, used to implement the method for voice-driven video face lip reconstruction according to any one of claims 1 to 7, It is characterized in that include: A decomposition module decomposes the original video data to obtain the original picture sequence and driving audio data; A video preprocessing module obtains an original head image sequence and an original 3D face key point sequence of a selected face based on the original image sequence; An audio preprocessing module, which obtains audio features based on driving audio data; The face lip shape generation module performs the following steps: using the original head image sequence and the original 3D face key point sequence of the selected face to train the NeRF face renderer; using the trained variational autoencoder to generate a target 3D face key point sequence that matches the audio features; inputting the target 3D face key point sequence into the trained NeRF face renderer to obtain a generated face image sequence; The face and lip shape fusion module performs the following operations: fusing the original image sequence and the generated face image sequence to obtain a synthetic image sequence; and combining the driving audio data and the synthetic image sequence to obtain a reconstructed video.

9. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for voice-driven video face lip shape reconstruction as described in any one of claims 1 to 7.