A face driving generation method based on neural radiance field

Abstract

A face drive generation method based on neural radiance field includes the following steps: S1, extracting feature points of a face region from image frames of a video, calculating expression features and posture features of a corresponding three-dimensional face model according to the feature points, and extracting speech features from a speech part of the video; S2, for each position in a three-dimensional space where the three-dimensional face is located, predicting a mixed shape vector for controlling the three-dimensional face, a posture correction coefficient, and a linear mixed skinning weight through a neural network; S3, based on a deformation technology of the three-dimensional face model, calculating a deformed position of each current position according to the posture features and the speech features obtained in step S1 and the parameters predicted in step S2; S4, predicting a flow field according to each calculated position, aggregating feature information of adjacent time through a multi-frame set manner, and optimizing a face reconstruction loss by using time aggregation, so as to constrain smooth transformation of a dynamic neural radiance field in time. The method improves the accuracy of expression deformation, and makes the generated video more smooth and consistent.

Description

Technical Field

[0001] This invention relates to video generation technology, and in particular to a face-driven generation method based on neural radiation fields. Background Technology

[0002] In various application areas, such as digital humans, filmmaking, and video dubbing, the generation of high-fidelity video portraits based on speech audio is of paramount importance. Deep generative models have been used by numerous researchers for audio-driven facial or video portrait creation tasks. Some work relies on learning-based image reconstruction, which often only synthesizes low-resolution static images. Other methods employ explicit structured intermediate representations such as 2D landmarks or 3D facial models. Although some studies have generated high-fidelity images, errors in the predictions of structured representations (such as expression parameters in 3D deformable models (3DMMs)) lead to inaccuracies in facial deformation.

[0003] Neural radiation fields, as an implicit representation of 3D scenes, have recently offered a novel approach to generating realistic images. Compared to explicit methods, neural radiation fields provide higher image quality and more flexible viewpoint control, making them particularly suitable for video portrait generation tasks. Previous works using neural radiation fields to generate speech-driven high-fidelity video portraits essentially treated speech features as a condition, simulating expression deformation through direct conditions of the implicit field, without enforcing shared canonical geometry for different expressions. These methods are too unconstrained and cannot handle the ill-posed problem of extracting geometry from monocular videos. Furthermore, these methods often neglect the relationship between consecutive frames, training and modeling each frame separately.

[0004] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The main objective of this invention is to overcome the deficiencies of the aforementioned background technology and provide a face-driven generation method based on neural radiation fields to improve the accuracy of facial expression deformation and make the generated videos smoother and more consistent.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A face generation method based on neural radiation fields includes the following steps:

[0008] S1. Extract feature points of the face region from the image frames of the video, calculate the expression features and posture features of the corresponding three-dimensional face model based on the feature points, and extract speech features from the speech part of the video.

[0009] S2. For each location in the three-dimensional space where the three-dimensional face is located, using the expression features and posture features, a neural network is used to predict the hybrid shape vector, posture correction coefficient and linear hybrid skinning weights used to control the three-dimensional face.

[0010] S3. Based on the deformation technology of the three-dimensional face model, calculate the deformed position of each current position according to the posture features and voice features obtained in step S1 and the parameters predicted in step S2.

[0011] S4. Predict the flow field based on each calculated location, aggregate the feature information of adjacent times through a multi-frame set, and use temporal aggregation to optimize the face reconstruction loss, thereby constraining the smooth transformation of the dynamic neural radiation field over time.

[0012] Furthermore:

[0013] The 3D face model is a FLAME model.

[0014] The neural network is a multilayer perceptron (MLP), and its expression is as follows:

[0015] ρ,ε,W=MLP(x d )

[0016] Where ρ, ε, and W are the current position x predicted by the multilayer perceptron (MLP), respectively. d The mixed shape coefficient, pose correction coefficient, and linear skin weight.

[0017] In step S3, based on the deformation technology of the three-dimensional face model, the deformed position of each current position is calculated according to the following formula:

[0018] x c =LBS(x d +B p (θ;ρ)+B E (a,ε),J(a),θ,W)

[0019] Where, x c x represents the position after deformation. d The current position is J(a); J(a) represents the parameters extracted from the speech features, where a is the speech feature, θ is the pose feature, ρ, ε, and W are the mixed shape coefficient, pose correction coefficient, and linear skinning weights of the current position predicted by the neural network, respectively, and B... p (θ; ρ) represents the calculated attitude offset, B E (a,ε) represents the calculation of facial expression offset.

[0020] In step S4, feature information is aggregated using a multi-frame set according to the following formula:

[0021] f = MLP f (x d a)

[0022] g = w1 × MLP l (Deform(x d +f,a))+w2×MLP l (Deform(x d ,a))

[0023] Where, f = MLP f (x d a) represents the flow field prediction process, MLP f This refers to a flow field sensor, used to calculate the flow field, or MLP. l Let g represent the feature perceptron, g represent the aggregated features, and Deform represent the deformed displacement features. For both cases considering the flow field and cases not considering the flow field, the feature information of multiple frames is aggregated by weighted summation. w1 and w2 represent the weights.

[0024] The method further includes the following steps:

[0025] S5. Predict the density and color of the current point using features from a multi-frame set, and obtain the pixel color value through volume rendering.

[0026] In step S5, during the rendering process, the density and RGB values ​​of the light samples passing through each pixel are integrated.

[0027] In step S5, the rendered color of each pixel in the 2D image space is determined by the following formula:

[0028]

[0029] Where C(r) is the color value calculated along ray r, and N k It is the number of sampling points, T k It is the cumulative transparency from the starting point of the light ray to the k-th sampling point, σ(x) k ) is x k Volume density at point ‖x k+1 -x k ‖ represents the adjacent distance between sampling points, exp(-σ(x) k )‖x k+1 -x k ‖) represents the weight of the k-th sampling point, c(x) k ) is in x k The color value at that location.

[0030] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned face-driven generation method based on neural radiation fields.

[0031] A computer program product, which, when run by a processor, implements the aforementioned face-driven generation method based on neural radiation fields.

[0032] The present invention has the following beneficial effects:

[0033] This invention provides a face-driven generation method based on neural radiation fields. Utilizing linear skinning technology, it improves the expressive power of the model in speech-driven scenarios. Simultaneously, it aggregates feature information from adjacent time periods using multi-frame sets, allowing the model to be optimized through multi-frame splicing training. This results in smooth temporal transformations of the dynamic neural radiation field, making the entire transformation process more natural. Compared to traditional techniques, this invention solves the problems of poor face deformation quality and choppy generated videos. By combining linear skinning technology to replace the original conditional deformation module, this invention improves the accuracy of expression deformation. Furthermore, by combining multi-frame features and considering the relationship between the current frame and its preceding frames, rather than modeling them individually, the generated videos are smoother and more consistent.

[0034] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0035] Figure 1 This is a flowchart of a face-driven generation method based on neural radiation fields, according to an embodiment of the present invention.

[0036] Figure 2 This is a simplified flowchart of the algorithm according to a preferred embodiment of the present invention.

[0037] Figure 3 From left to right are a heatmap controlled by eye gaze parameters, a heatmap controlled by voice parameters, and a depth map, respectively, according to an embodiment of the present invention. Detailed Implementation

[0038] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0039] See Figure 1 This invention provides a face-driven generation method based on neural radiation fields, comprising the following steps:

[0040] S1. Extract feature points of the face region from the image frames of the video, calculate the expression features and posture features of the corresponding three-dimensional face model based on the feature points, and extract speech features from the speech part of the video.

[0041] S2. For each location in the three-dimensional space where the three-dimensional face is located, using the expression features and posture features, a neural network is used to predict the hybrid shape vector, posture correction coefficient and linear hybrid skinning weights used to control the three-dimensional face.

[0042] S3. Based on the deformation technology of the three-dimensional face model, calculate the deformed position of each current position according to the posture features and voice features obtained in step S1 and the parameters predicted in step S2.

[0043] S4. Predict the flow field based on each calculated location, aggregate the feature information of adjacent times through a multi-frame set, and use temporal aggregation to optimize the face reconstruction loss, thereby constraining the smooth transformation of the dynamic neural radiation field over time.

[0044] In a preferred embodiment, the 3D face model is a FLAME model. The neural network is a multilayer perceptron (MLP).

[0045] In a preferred embodiment, the method further includes the following steps:

[0046] S5. Predict the density and color of the current point using features from a multi-frame set, and obtain the pixel color value through volume rendering.

[0047] The following describes specific embodiments of the present invention.

[0048] For a video of approximately 5 minutes, it is segmented into a series of images. For the entire image set, a face tracking algorithm is used to obtain the expression and pose coefficients of the corresponding FLAME model. For the video's audio, the audio features are obtained through an audio feature processing module. For each location in 3D space, a neural network predicts the corresponding FLAME model's blended shape vector, pose correction coefficient, and linear blending skinning weights. These parameters, along with the audio features, are used to perform matrix calculations to determine the 3D location's position in standard space. Using this standard space position, the density and color corresponding to the current location in standard space can be calculated. Finally, volume rendering is used to calculate the RGB color of the corresponding pixel in the image.

[0049] Figure 2 The algorithm flow of a preferred embodiment of the present invention is illustrated. Specifically, the operation of the preferred embodiment of the present invention includes:

[0050] First, this invention utilizes linear skinning technology to replace the conditional generation module in traditional methods. Traditional methods directly extract features from speech information and use them as conditions to control the color and density generation at a specific location in the neural radiation field. In contrast, this invention utilizes the FLAME model. Based on the hybrid shape coefficient ρ, pose correction coefficient ε, and linear skinning weights W predicted by the neural network, the deformation network can calculate the deformed position using the deformation location, pose, and expression parameters as conditions, based on the deformation formula of the FLAME model, and can also perform reverse deformation.

[0051] The neural network used is a multilayer perceptron (MLP), and its expression is as follows:

[0052] ρ,ε,W=MLP(x d )

[0053] Where ρ, ε, and W are the current position x predicted by the multilayer perceptron (MLP), respectively. d The mixed shape coefficient, pose correction coefficient, and linear skin weight.

[0054] Based on the deformation technique of the aforementioned 3D face model, the deformed position of each current location is calculated according to the following formula:

[0055] x c =LBS(x d +B p (θ;ρ)+B E (a,ε),J(a),θ,W)

[0056] Where, x c x represents the position after deformation. d The current position is J(a); J(a) represents the parameters extracted from the speech features, where a is the speech feature, θ is the pose feature, ρ, ε, and W are the mixed shape coefficient, pose correction coefficient, and linear skinning weights of the current position predicted by the neural network, respectively, and B... p (θ; ρ) represents the calculated attitude offset, B E (a,ε) represents the calculation of facial expression offset.

[0057] The overall transformation formula is marked as:

[0058] x c =Deform(x d ,a).

[0059] Furthermore, the flow field is predicted simultaneously during optimization. By aggregating feature information from adjacent time periods, the smooth transformation of the dynamic neural radiation field over time is forced, making the entire transformation process more natural and the overall three-dimensional structure more standardized.

[0060] Feature information is aggregated from multiple frames using the following formula:

[0061] f = MLP f (x d a)

[0062] g = w1 × MLP l (Deform(x d +f,a))+w2×MLP l (Deform(x d ,a))

[0063] Where, f = MLP f (x d a) represents the flow field prediction process, MLP f This refers to a flow field sensor, used to calculate the flow field, or MLP. l This represents a feature perceptron, where g is the aggregated feature and Deform is the deformed displacement feature. For both cases considering and not considering the flow field, feature information from multiple frames is aggregated through a weighted sum. w1 and w2 represent the weights. As an example:

[0064] g = 0.25 × MLP l (Deform(x d +f,a))+0.75×MLP l (Deform(x d ,a)).

[0065] The above operations do not use any explicit flow supervision to guide the flow estimation process. Instead, this capability comes from the temporal aggregation step, which simultaneously optimizes the face reconstruction loss. The assumption is that only temporally consistent features benefit from multi-frame feature ensemble, which indirectly drives the scene flow field toward the optimal solution—predicting the correct flow at all points.

[0066] Then, the density and color of the current point are predicted using the features from this multi-frame set:

[0067] c,σ=MLP c (g)

[0068] The final pixel color value is obtained through volume rendering. During the rendering process, the density and RGB values ​​of the d-sampled rays passing through each pixel are integrated. The rendered color of each pixel in 2D image space is given by the following formula:

[0069]

[0070] Where C(r) is the color value calculated along ray r, and N k It is the number of sampling points, T kIt is the cumulative transparency from the starting point of the light ray to the k-th sampling point, σ(x) k ) is x k Volume density at point ‖x k+1 -x k ‖ represents the adjacent distance between sampling points, exp(-σ(x) k )‖x k+1 -x k ‖) represents the weight of the k-th sampling point, c(x) k ) is in x k The color value at that location.

[0071] This invention utilizes speech features to guide the deformation of neural radiation fields. DeepSpeech can be used to process speech features, and other speech feature processing modules can achieve similar purposes.

[0072] The model in this embodiment of the invention is mainly used to generate high-fidelity video portraits based on speech audio, using speech as the control condition. However, the model can be modified to accept other inputs besides speech, such as the facial expression coefficients of a three-dimensional deformable model as the control condition.

[0073] Performance testing

[0074] By comparing the voice-driven, driverable face model constructed by this invention with previous benchmark models, it can be found that the method of this invention effectively improves the accuracy of the model in predicting corresponding expressions and improves image quality in multiple face datasets.

[0075] Figure 3 From left to right are the heatmap controlled by eye gaze parameters, the heatmap controlled by speech parameters, and the depth map. The model of this invention can generate high-quality rendered images, and it correctly distinguishes between eye gaze parameter control and speech parameter control. Speech parameters mainly affect the lips and their surrounding area. Simultaneously, the model can also generate good depth and a well-defined 3D structure.

[0076] As shown in Table 1, in the quantitative comparison with other models, the method of this invention also leads in all numerical values.

[0077] Table 1

[0078]

[0079]

[0080] Meanwhile, the actual effect of the model constructed by this invention is closer to the original GT data in terms of lip shape and more detailed in terms of hair and other details compared to the current best method.

[0081] Compared to traditional techniques, this invention solves the problems of poor facial deformation quality and choppy video generation. By combining linear skinning technology with the original conditional deformation module, this invention improves the accuracy of facial expression deformation. Through multi-frame feature combination, considering the relationship between the current frame and its preceding and succeeding frames, rather than modeling them individually, the generated video is smoother and more consistent.

[0082] Specific application scenarios of this invention include, but are not limited to: driving game characters based on voice in games to reduce labor costs; reproducing realistic human images in online video conferencing scenarios; providing post-production assistance such as face replacement in film and television creation scenarios; and generating avatars that replace human faces in AR and VR products.

[0083] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0084] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0085] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0086] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0087] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0090] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0092] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0093] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0094] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0095] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A face-driven generation method based on neural radiation fields, characterized in that, Includes the following steps: S1. Extract feature points of the face region from the image frames of the video, calculate the expression features and posture features of the corresponding three-dimensional face model based on the feature points, and extract speech features from the speech part of the video. S2. For each location in the three-dimensional space where the three-dimensional face is located, using the expression features and posture features, a neural network is used to predict the hybrid shape vector, posture correction coefficient and linear hybrid skinning weights used to control the three-dimensional face. S3. Based on the deformation technology of the three-dimensional face model, calculate the deformed position of each current position according to the posture features and voice features obtained in step S1 and the parameters predicted in step S2. S4. Predict the flow field based on each calculated location, aggregate the feature information of adjacent times through a multi-frame set, and use temporal aggregation to optimize the face reconstruction loss, thereby constraining the smooth transformation of the dynamic neural radiation field over time.

2. The face generation method based on neural radiation field as described in claim 1, characterized in that, The 3D face model is a FLAME model.

3. The face-driven generation method based on neural radiation fields as described in claim 1 or 2, characterized in that, In step S2, the neural network is a multilayer perceptron (MLP), and its expression is as follows: in, The current position is predicted using a multilayer perceptron (MLP). The mixed shape coefficient, pose correction coefficient, and linear skin weight.

4. The face-driven generation method based on neural radiation fields as described in claim 3, characterized in that, In step S3, based on the deformation technology of the three-dimensional face model, the deformed position of each current position is calculated according to the following formula: in, This is the position after deformation. Current position; These are parameters extracted from speech features, where 'a' represents the speech feature. It is a posture feature. These are the hybrid shape coefficients, pose correction coefficients, and linear skinning weights at the current position, predicted through a neural network. This indicates the calculation of the posture offset. This indicates the calculation of facial expression offset.

5. The face generation method based on neural radiation field as described in claim 4, characterized in that, In step S4, feature information is aggregated using a multi-frame set according to the following formula: in, This represents the process of predicting the flow field. This refers to a flow field sensor, used to calculate the flow field. Represents a feature-based perceptron. As a characteristic of aggregation, To determine the displacement characteristics after deformation, feature information from multiple frames is aggregated using a weighted sum method, considering both cases with and without considering the flow field. and Indicates the weight.

6. The face-driven generation method based on neural radiation fields as described in any one of claims 1 to 2, characterized in that, It also includes the following steps: S5. Predict the density and color of the current point using features from a multi-frame set, and obtain the pixel color value through volume rendering.

7. The face generation method based on neural radiation field as described in claim 6, characterized in that, In step S5, during the rendering process, the density and RGB values ​​of the light samples passing through each pixel are integrated.

8. The face generation method based on neural radiation field as described in claim 7, characterized in that, In step S5, the rendered color of each pixel in the 2D image space is determined by the following formula: in, It is the color value calculated along the ray r. It is the number of sampling points. It is the cumulative transparency from the starting point of the light ray to the k-th sampling point. yes Bulk density at point Indicates the adjacent distance between sampling points. This represents the weight of the k-th sampling point. Is The color value at that location.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the face-driven generation method based on neural radiation fields as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product, when run by a processor, implements the face-driven generation method based on neural radiation fields as described in any one of claims 1 to 8.

Images