Moving face model modeling method and device and computer equipment

Abstract

The invention discloses a moving face model modeling method and device and computer equipment. The method comprises the steps that an original video of a target object is collected, the original video is split into an image frame sequence, and the original video is a single-view-angle video; acquiring a face parameter of each image frame in the image frame sequence, and determining a camera parameter based on each image frame; applying the human face parameters to the camera parameters to obtain converted parameters, and performing three-dimensional reconstruction on the human face in the image frame based on the converted parameters and Gaussian points to obtain a human face model; and driving the face model to generate a driven face video frame. The method provided by the invention at least solves the technical problem that the construction process of the moving face model is relatively complex in the related art.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a method, apparatus, and computer device for modeling a moving face model. Background Art

[0002] Creating an animated human head avatar has been a long-standing problem in computer vision and graphics. By using a video of a person's head moving while speaking, a highly realistic 3D digital portrait can be created. This motion can be driven by audio or video, and finally, rendered to produce a video of the virtual speaking human. This application can provide a more realistic and immersive interactive experience for AR / VR applications and be used in scenarios such as remote video conferencing. Currently, mainstream technologies for achieving this effect rely on neural radiance fields. However, using neural radiance fields to drive implicit 3D models presents challenges, as training is slow and computationally expensive. Furthermore, rendering is slow, preventing real-time performance and hindering practical application. Alternatively, 3D face reconstruction based on 3D Gaussian splattering can improve facial detail and driving quality. However, this requires reconstruction from multi-view videos, which places high demands on the photographer's equipment and makes data acquisition difficult. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, and computer device for modeling a motion face model, so as to at least solve the technical problem of the relatively complex motion face model construction process in the related art.

[0004] According to one aspect of an embodiment of the present application, a method for modeling a motion face model is provided, comprising: capturing an original video of a target object, and splitting the original video into an image frame sequence, wherein the original video is a single-view video; obtaining face parameters of each image frame in the image frame sequence, and determining camera parameters based on each image frame; applying the face parameters to the camera parameters to obtain converted parameters, and performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a face model; and driving the face model to generate a driven face video frame.

[0005] Optionally, obtaining facial parameters of each image frame in the image frame sequence includes: extracting a facial area image from each image frame to obtain a facial area image corresponding to each image frame; using a three-dimensional face detection model to identify each facial area image to obtain facial parameters corresponding to each image frame, wherein the facial parameters include at least one of the following: a position parameter for characterizing the position of the head, a posture parameter for characterizing the posture of the head, a shape parameter for characterizing the shape of the head, and an expression parameter for characterizing the facial expression.

[0006] Optionally, determining the camera parameters based on each image frame includes: obtaining the coordinates of multiple feature points in the face area image corresponding to each image frame, and obtaining a standard three-dimensional face model; obtaining the position coordinates of the multiple feature points in the standard three-dimensional face model, and determining the camera parameters according to the coordinates of the multiple feature points and the position coordinates of the multiple feature points in the standard three-dimensional face model.

[0007] Optionally, the facial parameters are inversely applied to the camera parameters to obtain converted parameters, including: determining an inverse transformation matrix based on the external parameters in the camera parameters; and using the inverse transformation matrix to transform the facial parameters from the camera coordinate system to the world coordinate system to obtain the converted parameters.

[0008] Optionally, a three-dimensional reconstruction of the face in the image frame is performed based on the converted parameters and Gaussian points to obtain a face model, including: binding each Gaussian point of a plurality of Gaussian points to each face mesh patch in the initial face model one by one, wherein the attributes of the Gaussian point include at least one of the following: the center coordinates of the Gaussian point, the rotation quaternion parameter of the Gaussian point, the scale coefficient of the Gaussian point, the transparency parameter of the Gaussian point, the color coefficient of the Gaussian point, the Gaussian point patch weight coefficient and the binding patch index of the Gaussian point; determining the movement parameters of the face mesh patch in each image frame according to the face parameters; determining the movement parameters of each face mesh patch and the movement parameters of the face mesh patches adjacent to each face mesh patch from the movement parameters of the face mesh patches in each image frame; determining the attributes of each Gaussian point after movement according to the movement parameters of each face mesh patch and the movement parameters of the face mesh patches adjacent to each face mesh patch; and constructing the face model according to the attributes of each Gaussian point and the attributes of each Gaussian point after movement.

[0009] Optionally, the method further includes: determining a rendering equation according to the attributes of each Gaussian point and the attributes of each Gaussian point after movement; and determining the face model according to the rendering equation.

[0010] Optionally, the face model is driven to generate a driven face video frame, including: receiving video data of the driving video, the video data including: face parameters and camera parameters of each frame image; determining parameters after the position of the Gaussian points in the face model is changed according to the video data; and generating a driven face video frame based on the parameters after the position of the Gaussian points in the face model is changed.

[0011] According to another aspect of an embodiment of the present application, a motion face modeling device is also provided, including: an acquisition module for acquiring an original video of a target object and splitting the original video into an image frame sequence, wherein the original video is a single-view video; a parameter module for obtaining face parameters of each image frame in the image frame sequence and determining camera parameters based on each image frame; a reconstruction module for applying the face parameters to the camera parameters to obtain converted parameters, and performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a face model; and a driving module for driving the face model to generate a driven face video frame.

[0012] According to another aspect of the embodiment of the present application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned motion face modeling method.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned motion face modeling method by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which implement the above-mentioned motion face modeling method when executed by a processor.

[0015] In an embodiment of the present application, an original video of a target object is collected and split into an image frame sequence, wherein the original video is a single-view video; facial parameters of each image frame in the image frame sequence are obtained, and camera parameters are determined based on each image frame; the facial parameters are applied to the camera parameters to obtain converted parameters, and the face in the image frame is three-dimensionally reconstructed based on the converted parameters and Gaussian points to obtain a face model; the face model is driven to generate a driven face video frame, thereby achieving the purpose of constructing a face model only through a single-view video, thereby realizing the technical effect of simplifying the construction process of the motion face model, and further solving the technical problem that the construction process of the motion face model in the related art is relatively complicated. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1This is a hardware structure block diagram of a computer terminal for implementing a method for building a motion face model according to an embodiment of the present application;

[0018] Figure 2 is a flow chart of a method for modeling a motion face model according to an embodiment of the present application;

[0019] Figure 3 is a flowchart of a method for converting facial parameters according to an embodiment of the present application;

[0020] Figure 4 is a flow chart of a method for driving a motion face model according to an embodiment of the present application;

[0021] Figure 5 This is a structural diagram of a motion face modeling device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] 3D Gaussian Splatting: 3D Gaussian Splatting is a method that uses a large number of Gaussian distributions distributed in three-dimensional space to represent three-dimensional objects. It has the characteristics of simple representation, fast reconstruction speed and efficient rendering, and is relatively suitable for the representation of dynamic scenes.

[0026] Triangle mesh: A triangle mesh is a classic 3D scene representation method. A triangle mesh consists of a set of triangular facets, each of which is represented by three vertices and a normal vector.

[0027] FLAME model: The FLAME model (Faces Learned with an Articulated Model and Expressions) is a triangular mesh-based 3D facial modeling framework designed to generate highly controllable and realistic 3D face models. It combines parametric control of geometry, expression, and pose, and is primarily used in fields such as computer vision, graphics, and animation.

[0028] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject the automated decision results; if the user chooses to reject, the expert decision-making process will be entered.

[0029] In order to solve the problems existing in the related art, the embodiment of the present application provides a method for modeling a motion face model. Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0030] The embodiment of the method for building a motion face model provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for modeling a motion face model. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the motion face modeling method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned motion face modeling method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0035] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0036] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for modeling a motion face model. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 2 is a flow chart of a method for modeling a motion face model according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0038] Step S202: capturing an original video of the target object and splitting the original video into a sequence of image frames, wherein the original video is a single-view video;

[0039] Step S204, obtaining facial parameters of each image frame in the image frame sequence, and determining camera parameters based on each image frame;

[0040] Step S206, applying the facial parameters to the camera parameters to obtain converted parameters, and performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a facial model;

[0041] In step S206, Gaussian points are used to represent the elements of the 3D face model. During the reconstruction process, these Gaussian points are bound to the mesh patches of the FLAME model. As the shape of the model changes (for example, with a change in expression or head rotation), the position and properties of the Gaussian points change accordingly. By optimizing the properties of these Gaussian points, it is possible to accurately reconstruct facial details, including high-frequency details and boundary conditions, which is difficult to achieve with traditional triangular mesh models or neural radiation fields.

[0042] Step S208: driving the face model to generate a driven face video frame.

[0043] Through steps S202 to S208, the original video (source video) of the target object is captured and split into a sequence of image frames, where the original video is a single-view video. Facial parameters of each image frame in the sequence are obtained, and camera parameters are determined based on each image frame. The facial parameters are applied to the camera parameters to obtain converted parameters, and the face in the image frame is three-dimensionally reconstructed based on the converted parameters and Gaussian points to obtain a facial model. The facial model is driven to generate driven facial video frames, thereby achieving the purpose of constructing a facial model using only a single-view video, thereby simplifying the process of constructing a motion facial model and solving the technical problem of the relatively complex process of constructing a motion facial model in related technologies. Detailed description is given below.

[0044] In some embodiments of the present application, the specific process of obtaining the facial parameters of each image frame in the image frame sequence is as follows: extracting a facial area image from each image frame to obtain a facial area image corresponding to each image frame; using a three-dimensional face detection model to identify each facial area image to obtain facial parameters corresponding to each image frame, wherein the facial parameters include at least one of the following: a position parameter for characterizing the head position, a posture parameter for characterizing the head posture, a shape parameter for characterizing the head shape, and an expression parameter for characterizing the facial expression.

[0045] Specifically, the input original video (2D video) is split into a sequence of multiple image frames, and preprocessing steps such as image quality detection are performed. If the video does not completely correspond to the face area, face detection and cropping are applied to each frame to obtain the face area.

[0046] For each frame in the video, a three-dimensional face detection model is applied to identify its corresponding FLAME parameters (face parameters), including the parameter vectors such as position, rotation, expression and shape corresponding to the FLAME model.

[0047] It is understandable that FLAME parameters include but are not limited to: position parameters for characterizing the position of the head; posture parameters for characterizing the head posture, such as: global rotation of the head and neck posture, for simulating the movement of the head; shape parameters for characterizing the shape of the head, such as: for adjusting the basic geometric structures of the head and face, such as face shape, nose, eyes, etc.; expression parameter shape parameters for characterizing facial expressions, for controlling changes in expression, such as smiling, frowning, blinking, etc.

[0048] It's important to note that shape parameters describe the geometry of the FLAME model, including but not limited to face shape, nose bridge width, and eye and mouth position. These parameters are typically linear combinations of basic shape vectors derived from principal component analysis (PCA) and are used to tailor the model to the facial features of a specific individual. For example, by adjusting the shape parameters, the model can be rendered as a round or long face, with large or small eyes. Expression parameters control the FLAME model's expressiveness and are also linear combinations of basic expression vectors derived from PCA. These parameters can be used to simulate a variety of facial expressions, such as smiles, blinks, and surprise. By varying the values ​​of the expression parameters, the model can be driven to produce complex emotional expressions and lip movements, which are crucial for facial dynamics during animation and reconstruction. Position parameters refer to the position of the FLAME model in three-dimensional space. These parameters are typically expressed as a three-dimensional vector (x, y, z) that controls the model's displacement along the x, y, and z axes. This ensures that the model is correctly positioned in the reconstructed scene, consistent with the face's position in the actual image. Rotation parameters control the orientation and direction of the FLAME model, typically expressed as rotation quaternions or Euler angles. These parameters allow the model to rotate based on head poses (e.g., nodding or shaking the head), ensuring that the reconstructed 3D face matches the pose of the face in the actual video frame. Neck pose parameters provide finer control over the relative position of the head and neck, which is important for simulating natural human motion.

[0049] Local pose parameters control local movements of the face, such as eye rotation or mouth opening, which enhance the realism of expressions.

[0050] In some embodiments of the present application, the specific steps for determining the camera parameters based on each image frame are as follows: obtaining the coordinates of multiple feature points in the face area image corresponding to each image frame, and obtaining a standard three-dimensional face model; obtaining the position coordinates of the multiple feature points in the standard three-dimensional face model, and determining the camera parameters based on the coordinates of the multiple feature points and the position coordinates of the multiple feature points in the standard three-dimensional face model.

[0051] Specifically, the camera's intrinsic and extrinsic parameters are identified by combining a 3D face detection model with video frames. Since the face is approximately a rigid body, a standard 3D face shape is assumed, and the focal length is solved using known 2D key points (feature points). The extrinsic parameters are calculated using the PnP algorithm using the known 3D face model (standard face key points) and the detected 2D key points.

[0052] For example, face detection models typically output 2D key points, such as the locations of eyes, nose, and mouth. Based on the coordinates of the key points on the 3D model, the coordinates of the corresponding key points in the 2D image are found, and methods such as triangulation are applied to solve the focal length (camera intrinsics).

[0053] According to the coordinates of the key points on the 3D model, find the coordinates of the key points in the corresponding 2D image, and use the PnP algorithm to solve the rotation and translation matrix (camera extrinsic parameters) of the camera from the world coordinate system to the camera coordinate system.

[0054] After acquiring the facial and camera parameters, the identified FLAME and camera parameters are smoothed to ensure the continuity of the generated results. It should be noted that to ensure the accuracy of 3D face reconstruction, parameter smoothing is not applied to the preprocessed data for reconstruction; only to the preprocessed data of the driving video.

[0055] In some embodiments of the present application, Figure 3 As shown, the specific steps of applying the facial parameters to the camera parameters to obtain the converted parameters are as follows: determining an inverse transformation matrix based on the extrinsic parameters in the camera parameters; using the inverse transformation matrix to transform the facial parameters from the camera coordinate system to the world coordinate system to obtain the converted parameters.

[0056] Specifically, in order to place the FLAME model in the world coordinate system, the model's position and rotation need to be inversely applied to the camera parameters. This can be achieved by the following steps: First, the inverse matrix of the camera's extrinsic parameters needs to be calculated, which includes inverse rotation and translation operations to determine the position and orientation of the camera relative to the world coordinate system. The position and rotation parameters of the FLAME model are then transformed using the inverse transformation matrix to ensure the correctness of the direction vectors provided by the data when fitting the SH (Spherical Harmonics) coefficients.

[0057] In some embodiments of the present application, the face in the image frame is three-dimensionally reconstructed based on the converted parameters and Gaussian points to obtain a face model in the following specific steps: binding each Gaussian point of a plurality of Gaussian points to each face mesh patch in the initial face model, wherein the attributes of the Gaussian point include at least one of the following: the center coordinates of the Gaussian point, the rotation quaternion parameter of the Gaussian point, the scale coefficient of the Gaussian point, the transparency parameter of the Gaussian point, the color coefficient of the Gaussian point, the Gaussian point patch weight coefficient and the binding patch index of the Gaussian point; determining the movement parameters of the face mesh patch in each image frame according to the face parameters; determining the movement parameters of each face mesh patch and the movement parameters of the face mesh patches adjacent to each face mesh patch from the movement parameters of the face mesh patch in each image frame; determining the attributes of each Gaussian point after movement according to the movement parameters of each face mesh patch and the movement parameters of the face mesh patches adjacent to each face mesh patch; and constructing the face model according to the attributes of each Gaussian point and the attributes of each Gaussian point after movement.

[0058] Wherein, a rendering equation is determined according to the attribute of each Gaussian point and the attribute of each Gaussian point after movement; and the face model is determined according to the rendering equation.

[0059] Specifically, bind the Gaussian points to the mesh patches in the FLAME model: During initialization, each FLAME mesh patch is bound to a Gaussian point, and the properties of the Gaussian point are as follows:

[0060]

[0061] Where μ is the center coordinate of the Gaussian point, r is the rotation quaternion parameter of the Gaussian point, s is the scale coefficient of the Gaussian point, α is the transparency parameter of the Gaussian point, h is the color SH (Spherical Harmonics) coefficient of the Gaussian point, w is the patch weight coefficient of the Gaussian point, and i is the bound patch index of the Gaussian point.

[0062] Based on Gaussian point movement: The estimated FLAME parameters for each frame in the data are as follows:

[0063]

[0064] Where β is the shape parameter, Δv is the fixed displacement of the FLAME vertex, t is the displacement parameter, θ is the joint position, and ψ is the expression parameter. Based on the above parameters, we can calculate the movement parameters of the FLAME patch i in each frame:

[0065] f i ={K,T,R}

[0066] Where K is the scale of the triangle patch (face mesh patch), T is the displacement parameter of the triangle patch, and R is the rotation parameter of the triangle patch. Correspondingly, we can obtain the transformed properties of the Gaussian point bound to the patch. At the same time, in order to achieve the transformation of the Gaussian point bound to the patch and the smooth movement of the Gaussian point, the movement of the Gaussian point depends not only on the bound patch, but also on the neighboring patches of the bound patch:

[0067]

[0068] in for:

[0069] μ′=kRμ+T

[0070] r′=Rr

[0071] s′=ks

[0072] where i k ,k=1,2,3 corresponds to the three adjacent patches of patch i, w k The weight of the kth adjacent patch, μ′ represents the center coordinate after the Gaussian point is moved, r′ represents the rotation quaternion parameter after the Gaussian point is moved, and s′ represents the scale coefficient after the Gaussian point is moved.

[0073] Through the rendering equation Render the reconstructed image using 3D Gaussian splashing.

[0074] Where C represents the reconstructed image, c i Indicates the color brightness of the i-th Gaussian point, α′ i represents the updated transparency of the i-th Gaussian point, α′ j Indicates the updated transparency of the j-th Gaussian point.

[0075] The rendered reconstructed image is compared with the dataset (original image) and the Gaussian point attributes are updated using gradient descent with the following loss:

[0076]

[0077] Where, Indicates the color loss value, represents the scale loss value, Represents the position loss value, λ represents the weight coefficient, which is used to adjust and The weight of represents the absolute error loss, represents the loss based on the difference structure similarity index, ∈ scaling represents the threshold of the scale factor, ∈ position Indicates the position coordinate threshold.

[0078] At the same time, the increase and decrease of Gaussian points are realized according to the adaptive density control strategy. Here, each patch retains at least one Gaussian point, and Implement the bound patch transformation of Gaussian points, where i represents the index of the patch and w k represents the weight of the kth Gaussian point, i k Represents the kth Gaussian point, and finds the Gaussian point i with the maximum weight score among all Gaussian points k The patch index i.

[0079] In some embodiments of the present application, the specific steps for driving the face model and generating the driven face video frame are as follows: receiving video data of the driving video, the video data including: face parameters and camera parameters of each frame image; determining the parameters after the position of the Gaussian points in the face model is changed according to the video data; and generating the driven face video frame based on the parameters after the position of the Gaussian points in the face model is changed.

[0080] Specifically, if Figure 4 As shown, the reconstructed 3D face model is driven according to the parameters corresponding to the video after pre-processing. Here f i The movement parameters of patch i corresponding to the FLAME parameters estimated for each frame of the driving video, Indicates the parameters after the Gaussian point position is changed (moved).

[0081] Linear Blend Skinning (LBS) is used to calculate the movement direction and position of each vertex in the face model based on the FLAME parameters of the driving video.

[0082] Finally, by fitting the Gaussian points based on the motion direction and post-motion position of each vertex in the 3D face model, we obtain the center position, covariance matrix, color, and transparency stored for each Gaussian point. Using rasterization technology, we quickly render the driven face video frames to synthesize a complete video.

[0083] It should be noted that the motion facial modeling method provided in the embodiments of this application can be deployed in scenarios such as digital human customer service. Users only need to upload a single-view facial video of several tens of seconds to generate their own 3D digital avatar. They can then use their own or someone else's facial video as a driving video to drive the reconstructed 3D facial model in real time, significantly increasing the personalization and interactivity of the digital human. For example, a user can capture single-view video data through a mobile phone camera, preprocess this data to obtain a corresponding 3D facial model, and then use their own or someone else's video as a driving video to drive the reconstructed facial model, ultimately obtaining a facial video with the target facial movements. This elevates the digital human from a static model to a virtual avatar that can be modified according to needs, providing a better user experience. In short, the present invention can generate a high-quality, drivable 3D facial model using only a single-view video, significantly improving the practicality and convenience of the application. This opens up greater potential for digital human application scenarios, enabling more personalized and interactive digital human applications and improving the user experience.

[0084] It is understandable that the method for modeling a moving face model provided in the embodiment of the present application optimizes the binding strategy of Gaussian points during retraining and introduces Gaussian points into the patch weight attributes, making the movement of Gaussian points according to the FLAME mesh patch smoother and more accurate. At the same time, this solution can realize the replacement of the binding relationship of Gaussian points between the patches of the FLAME model, making the adjustment of Gaussian points more flexible, and ultimately achieving a rendering effect with clearer boundaries and higher quality without significantly increasing training time. The FLAME model parameters and camera parameters corresponding to each frame of the single-view video are estimated using a three-dimensional face detection model, and the relative relationship between the camera and the FLAME model is converted, so that the single-view video data can be used as input to reconstruct the three-dimensional face model. This more accessible data greatly improves the practicality of the algorithm, eliminates the need for professional equipment, and increases the possibility of application implementation. The use of an adaptive low-pass filter to smooth the driving video data makes the rendered result more natural and smooth, improving the authenticity of the generated video. The method for modeling a moving face model provided in the embodiment of the present application greatly accelerates the model fitting speed compared to the neural radiation field, and the final face model can be obtained in a shorter time. Through the gradient descent method and density control strategy, this technology can quickly fit Gaussian points in space to those with suitable properties, and can achieve better rendering quality than the implicit neural radiation field. By pre-processing methods such as frame segmentation and parameter estimation of single-view facial videos, it is possible to obtain three-dimensional information from single-view videos for model reconstruction, and adjust and transform the estimated data to adapt to the input of the algorithm. The advantage is that data acquisition is easier and the algorithm system is more practical. It can be widely used in scenarios that require photo-level timely rendering, such as digital humans, virtual anchors, video conferencing, games and film production. Compared with existing methods, the present invention achieves a significant improvement in real-time multi-view rendering capabilities while ensuring generation quality.

[0085] Figure 5 A motion face modeling device according to an embodiment of the present application includes:

[0086] An acquisition module 50 is configured to acquire an original video of a target object and split the original video into a sequence of image frames, wherein the original video is a single-view video;

[0087] a parameter module 52, configured to obtain facial parameters of each image frame in the image frame sequence and determine camera parameters based on each image frame;

[0088] a reconstruction module 54 for applying the facial parameters to the camera parameters to obtain converted parameters, and performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a facial model;

[0089] The driving module 56 is used to drive the face model to generate driven face video frames.

[0090] Through the above-mentioned motion face modeling device, the original video of the target object is collected and the original video is split into an image frame sequence, wherein the original video is a single-view video; the facial parameters of each image frame in the image frame sequence are obtained, and the camera parameters are determined based on each image frame; the facial parameters are applied to the camera parameters to obtain converted parameters, and the face in the image frame is three-dimensionally reconstructed based on the converted parameters and Gaussian points to obtain a face model; the face model is driven to generate a driven face video frame, thereby achieving the purpose of constructing a face model only through a single-view video, thereby achieving the technical effect of simplifying the construction process of the motion face model, and further solving the technical problem that the construction process of the motion face model in the related art is relatively complicated.

[0091] The parameter module 52 includes: an acquisition submodule for acquiring facial parameters of each image frame in the image frame sequence, including: extracting a facial region image from each image frame to obtain a facial region image corresponding to each image frame; using a three-dimensional face detection model to identify each facial region image to obtain facial parameters corresponding to each image frame, wherein the facial parameters include at least one of the following: a position parameter for characterizing the position of the head, a posture parameter for characterizing the posture of the head, a shape parameter for characterizing the shape of the head, and an expression parameter for characterizing the facial expression.

[0092] The acquisition submodule includes: a camera unit, which is used to determine camera parameters based on each image frame, including: obtaining the coordinates of multiple feature points in the face area image corresponding to each image frame, and obtaining a standard three-dimensional face model; obtaining the position coordinates of the multiple feature points in the standard three-dimensional face model, and determining the camera parameters according to the coordinates of the multiple feature points and the position coordinates of the multiple feature points in the standard three-dimensional face model.

[0093] The reconstruction module 54 includes: a conversion submodule for inversely applying the facial parameters to the camera parameters to obtain converted parameters, including: determining an inverse transformation matrix based on the extrinsic parameters in the camera parameters; using the inverse transformation matrix to convert the facial parameters from the camera coordinate system to the world coordinate system to obtain the converted parameters.

[0094] The reconstruction module 54 includes a reconstruction submodule for performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a face model, including: binding each Gaussian point of a plurality of Gaussian points to each face mesh patch in the initial face model one by one, wherein the attributes of the Gaussian point include at least one of the following: the center coordinates of the Gaussian point, the rotation quaternion parameter of the Gaussian point, the scale coefficient of the Gaussian point, the transparency parameter of the Gaussian point, the color coefficient of the Gaussian point, the face weight coefficient of the Gaussian point, and the binding face patch index of the Gaussian point; determining the movement parameters of the face mesh patch in each image frame according to the face parameters; determining the movement parameters of each face mesh patch and the movement parameters of the face mesh patches adjacent to each face mesh patch from the movement parameters of the face mesh patch in each image frame; determining the attributes of each Gaussian point after movement according to the movement parameters of each face mesh patch and the movement parameters of the face mesh patches adjacent to each face mesh patch; and constructing the face model according to the attributes of each Gaussian point and the attributes of each Gaussian point after movement.

[0095] The reconstruction submodule includes: a rendering unit, which is used to determine a rendering equation according to the attributes of each Gaussian point and the attributes of each Gaussian point after movement; and determine the face model according to the rendering equation.

[0096] The driving module 56 includes: a driving submodule for driving the face model to generate driven face video frames, including: receiving video data of the driving video, the video data including: face parameters and camera parameters of each frame image; determining the parameters of the Gaussian point position in the face model after the change according to the video data;

[0097] A driven face video frame is generated based on the parameters of the changed Gaussian point positions in the face model.

[0098] It should be noted that Figure 5 The motion face modeling device shown is used to perform Figure 2 The motion face modeling method shown in the figure, therefore the relevant explanations in the above motion face modeling method are also applicable to the motion face modeling device, and will not be repeated here.

[0099] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned motion face modeling method.

[0100] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned motion face modeling method by running the computer program.

[0101] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the motion face modeling method in the present application.

[0102] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0103] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0106] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0108] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for modeling a moving face model, characterized in that: include: Capturing an original video of a target object and splitting the original video into a sequence of image frames, wherein the original video is a single-view video; Acquire facial parameters of each image frame in the image frame sequence, and determine camera parameters based on each image frame; Applying the facial parameters to the camera parameters to obtain converted parameters, and performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a facial model; The face model is driven to generate a driven face video frame.

2. The method according to claim 1, characterized in that Obtaining facial parameters of each image frame in the image frame sequence, including: Extracting a face region image from each image frame to obtain a face region image corresponding to each image frame; A three-dimensional face detection model is used to identify each face region image to obtain face parameters corresponding to each image frame, wherein the face parameters include at least one of the following: a position parameter for characterizing the head position, a posture parameter for characterizing the head posture, a shape parameter for characterizing the head shape, and an expression parameter for characterizing the facial expression.

3. The method according to claim 2, characterized in that Determining camera parameters based on each image frame includes: Obtaining the coordinates of multiple feature points in the face area image corresponding to each image frame, and obtaining a standard three-dimensional face model; The position coordinates of the multiple feature points in the standard three-dimensional face model are obtained, and the camera parameters are determined according to the coordinates of the multiple feature points and the position coordinates of the multiple feature points in the standard three-dimensional face model.

4. The method according to claim 1, wherein The face parameters are reversely applied to the camera parameters to obtain converted parameters, including: Determine an inverse transformation matrix according to the extrinsic parameters in the camera parameters; The inverse transformation matrix is ​​used to transform the facial parameters from the camera coordinate system to the world coordinate system to obtain the transformed parameters.

5. The method according to claim 1, wherein Performing three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a face model includes: Binding each Gaussian point of the plurality of Gaussian points to each face mesh patch in the initial face model, wherein the attributes of the Gaussian point include at least one of the following: the center coordinates of the Gaussian point, the rotation quaternion parameter of the Gaussian point, the scale coefficient of the Gaussian point, the transparency parameter of the Gaussian point, the color coefficient of the Gaussian point, the patch weight coefficient of the Gaussian point, and the binding patch index of the Gaussian point; Determining movement parameters of the face mesh patch in each image frame according to the face parameters; Determining the movement parameters of each face mesh patch and the movement parameters of face mesh patches adjacent to each face mesh patch from the movement parameters of the face mesh patches in each image frame; Determining the attributes of each Gaussian point after movement according to the movement parameters of each face mesh patch and the movement parameters of face mesh patches adjacent to each face mesh patch; The face model is constructed according to the attribute of each Gaussian point and the attribute of each Gaussian point after movement.

6. The method according to claim 5, characterized in that The method further comprises: Determine a rendering equation according to the attribute of each Gaussian point and the attribute of each Gaussian point after movement; The face model is determined according to the rendering equation.

7. The method according to claim 1, characterized in that Driving the face model to generate driven face video frames includes: Receive video data driving the video, the video data including: facial parameters and camera parameters of each frame image; Determine parameters of the Gaussian point positions in the face model after the changes according to the video data; A driven face video frame is generated based on the parameters of the changed Gaussian point positions in the face model.

8. A motion face modeling device, characterized in that: include: An acquisition module is used to acquire an original video of a target object and split the original video into a sequence of image frames, wherein the original video is a single-view video; a parameter module, configured to obtain facial parameters of each image frame in the image frame sequence and determine camera parameters based on each image frame; a reconstruction module, configured to apply the facial parameters to the camera parameters to obtain converted parameters, and perform three-dimensional reconstruction of the face in the image frame based on the converted parameters and Gaussian points to obtain a facial model; The driving module is used to drive the face model and generate a driven face video frame.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the motion face modeling method described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the motion face modeling method according to any one of claims 1 to 7 is implemented.

Images