Mesh sequence driven two-dimensional face animation generation method, device, equipment and medium

Abstract

The application discloses a kind of grid sequence drive two-dimensional face animation generation method, device, equipment and medium, wherein method includes: obtaining the 2D face image needing to be driven;The initial frame of grid sequence movement is bound using Gaussian splat expression with 2D face image, the topology of 3D grid and the identity information of 2D face are bound to Gaussian model, and initial Gaussian avatar is generated;Combining deformation field to represent the movement of grid sequence on Gaussian expression, the Gaussian avatar is optimized, and animation video sequence is generated by rendering;Wherein, 2D replay network processes input reference face image and grid current rendering frame as condition, generates gradient and optimizes the attitude and feature of avatar, so that the generated animation is highly consistent with the features of reference face image.The application can effectively reduce rendering cost and complexity by introducing Gaussian splat-based intermediate representation method and combining two-dimensional prior and gradient-guided Gaussian deformation field optimization method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, and in particular to a mesh sequence driven two-dimensional face animation generation method, device, equipment and medium. BACKGROUND

[0002] In the field of modern digital content creation, especially in the film, game and virtual reality industries, the production and rendering of three-dimensional assets has always been a time-consuming and expensive process. Traditional content creation processes usually require the creation of detailed three-dimensional assets for each static image, which not only increases production costs but also prolongs production cycles, making it difficult to meet the urgent demand for personalized and dynamic content in the market.

[0003] The current technical solution includes: (1) a replay network in a 2D scene - static image animation generation. This solution has the following defects: a) strong data dependency: since the replay network has not seen data in the form of 3D gray model mesh rendering, the driving effect under the 2D replay network is poor. b) poor time stability: when generating long video sequences, inconsistencies and frame jitter in time are prone to occur, affecting user experience. c) poor migration: the model is usually trained for a specific data set, making it difficult to migrate to different static images, limiting the universality of the application. (2) Dynamic portrait generation based on three-dimensional models. This solution has the following defects: a) high cost and resource demand: building high-precision three-dimensional face models requires expensive scanning equipment and high-performance computing resources, increasing production costs. b) complex rendering process: relying on a complex three-dimensional rendering process, rendering time is long, making it difficult to achieve real-time generation. c) high demand for manual adjustment: requiring a large amount of manual adjustment and professional knowledge, increasing the difficulty and time cost of content creation. d) identity limitation: each static image requires a corresponding three-dimensional model to be specially constructed, lacking flexibility and making it difficult to be applied on a large scale. SUMMARY

[0004] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a mesh sequence driven two-dimensional face animation generation method, device, equipment and medium based on Gaussian splash representation.

[0005] The first technical solution adopted by the present application is:

[0006] A mesh sequence driven two-dimensional face animation generation method, comprising the following steps:

[0007] Obtaining a 2D face image that needs to be driven;

[0008] Binding the initial frame of the mesh sequence motion and the 2D face image using Gaussian splash representation, binding the topology of the 3D mesh and the identity information of the 2D face to the Gaussian model, and generating an initial Gaussian avatar.

[0009] The motion of the grid sequence on the Gaussian expression is characterized by combining the deformation field, the Gaussian avatar is optimized, and an animated video sequence is generated by rendering;

[0010] The 2D replay network processes the input reference face image and the current rendering frame of the grid as a condition, generates a gradient, and optimizes the pose and features of the avatar, so that the generated animation is highly consistent with the features of the reference face image.

[0011] Further, a method based on Gaussian splash representation is used as an intermediate expression between 2D faces and 3D grids of different identities to realize the transition from grid sequence M to animated sequence video ; Where T is the number of frames of the animation.

[0012] Further, the initial frame of the grid sequence motion and the 2D face image are bound using Gaussian splash expression, the topology of the 3D grid and the identity information of the 2D face are bound to the Gaussian model, and the initial Gaussian avatar is generated, including:

[0013] Using 3D Gaussian splash as an intermediate, the connection between 3D grids and 2D face images under different identities is established;

[0014] For each unit triangle in the grid, a 3D Gaussian splash unit is bound, and the initial center position μ, rotation matrix r, and scaling factor s of the bound Gaussian are given according to the properties of the unit triangle.

[0015] Further, for each unit triangle in the grid, a 3D Gaussian splash unit is bound, and the initial center μ, rotation matrix r, and scaling factor s of the bound Gaussian are given according to the properties of the unit triangle, including:

[0016] For the center position μ of the bound Gaussian, the expression form based on the barycentric coordinates is:

[0017] For any point V in the space triangle ΔV1V2V3, there must be unique coefficients λ1, λ2, such that:

[0018]

[0019] Where, are the three vertices of the triangle, respectively;

[0020] For each triangle, the normal vector n is obtained by calculating the difference set of the two edge vectors:

[0021]

[0022] For the direction vector d, the edge composed of vertex 1 and vertex 2 is selected to calculate the direction vector d, that is:

[0023]

[0024] To describe the orientation of a triangle in global space, form a rotation matrix R with the direction vector d of one of its edges, its normal vector n, and their cross product d x n as columns:

[0025] R = [d, n, d x n]

[0026] Compute a scalar k by the length of the shortest edge l and its perpendicular direction h to describe the scaling of the triangle:

[0027]

[0028] For a paired 3D Gaussian, define its position μ, rotation r, and anisotropic scaling s in local space; initialize the position as the local origin, the rotation r as the identity rotation matrix, and the scaling s as the identity vector; at rendering time, convert these attributes to global space by:

[0029] r' = Rr

[0030] μ' = kRμ + T

[0031] s' = ks

[0032] where r' is the rotation of the unit Gaussian splash in global space, μ' is the center position of the unit Gaussian splash in global space, s' is the scaling of the unit Gaussian splash in global space, and T is the center position of the mesh that the Gaussian splash is in global space.

[0033] Further, the generating an animated video sequence by rendering comprises:

[0034] For any given frame i, the deformation of each vertex is defined as:

[0035]

[0036] where is the position of vertex j in the deformed mesh in the i-th frame, v j,can is the position of the same vertex in the canonical mesh;

[0037] For each Gaussian x i,can in canonical space, compute its representation of its deformation by an MLP network, and then transform the Gaussian from canonical space to deformed space:

[0038] x i,def = x i,can + Def(v j,can , δv j )

[0039] By directly using the deformed vertices of the mesh and the vertex positions of the canonical space, complex deformation in dynamic scenes can be more accurately captured to achieve high-quality dynamic scene rendering.

[0040] Further, the motion of the Gaussian avatar on the Gaussian representation is optimized by combining the deformation field to represent the sequence of meshes, including:

[0041] The 2D reference face image is input into a pre-trained 2D replay network, and the current frame provides a driven mesh state as a conditional drive. The 2D replay network can give the reference prior of the 2D reference face image driven under the current frame according to the current driving information. These reference priors provide guidance for keeping the consistency of the original image features and the replay motion.

[0042] These reference priors are combined with gradient-based optimization techniques to adjust the Gaussian deformation field to ensure accurate alignment of the motion and unique facial features and poses of the static image.

[0043] Further, in the training optimization process, a combination of L1 term and D-SSIM term is used to supervise the rendered image:

[0044] L rgb =(1-λ)L1+λL D-SSIM

[0045]

[0046] In the formula, and are the average values of the rendered image I render and the image I persudo generated by the pre-trained 2D replay network as a condition; and are the variances of the rendered image I render and the image I persudo generated by the pre-trained 2D replay network as a condition, is the covariance of the rendered image I render and the image I persudo generated by the pre-trained 2D replay network as a condition, C1 and C2 are small constants for stabilizing the denominator; L1 is the value obtained by loss calculation between each pixel of the rendered image and the image generated by the pre-trained 2D replay network as a condition; L D-SSIM is a structural similarity loss function, indicating the difference between images; λ is the weight size of L D-SSIM , specifically 0.2. The smaller the value of D-SSIM, the more similar the two images are.

[0047] A L1 term is also used to supervise the loss between the transparency produced by the rendering and the image mask, expecting to guarantee the pixel alignment:

[0048]

[0049] wherein, is a pre-trained 2D replay network as a condition to generate the mask image of the human face part, alpha is a mask image of the human face part of the rendering image;

[0050] The final loss function is:

[0051] L=L rgb +λ mask L mask

[0052] wherein, λ mask is the L mask The weight size in the total loss function, L mask is the absolute value difference of the two mask image pixels as a loss value.

[0053] The second technical solution adopted by the present application is:

[0054] A grid sequence driven two-dimensional face animation generation device, comprising:

[0055] A 2D image acquisition module is configured to acquire a 2D face image to be driven;

[0056] A Gaussian avatar generation module is configured to bind the initial frame of the grid sequence motion and the 2D face image using Gaussian splatting expression, bind the topology of the 3D grid and the identity information of the 2D face to the Gaussian model, and generate an initial Gaussian avatar;

[0057] A video sequence generation module is configured to combine a deformation field to represent the motion of the grid sequence on the Gaussian expression, optimize the Gaussian avatar, and generate an animation video sequence through rendering;

[0058] The 2D replay network processes the input reference face image and the current rendering frame of the grid as a condition to generate a gradient and optimize the pose and features of the avatar, so that the generated animation is highly consistent with the features of the reference face image.

[0059] The third technical solution adopted by the present application is:

[0060] An electronic device includes a processor and a memory having stored therein at least one instruction, at least one program, a code set or instruction set, which is loaded and executed by the processor to implement the grid sequence driven two-dimensional face animation generation method as described above.

[0061] The fourth technical solution adopted by the present application is:

[0062] A computer-readable storage medium having stored therein at least one instruction, at least one program, a code set or instruction set, which is loaded and executed by a processor to implement the grid sequence driven two-dimensional face animation generation method as described above.

[0063] The fifth technical solution adopted by the present application is:

[0064] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to make the computer device execute the above method.

[0065] The beneficial effects of the present application are: by introducing the intermediate representation method based on Gaussian Splatting, and combining the two-dimensional prior and gradient guided Gaussian deformation field optimization method, the rendering cost and complexity can be effectively reduced, the flexibility and efficiency of animation generation are improved, and the time stability of long video sequences is improved. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0067] Figure 1 is a flowchart of a grid sequence driven two-dimensional face animation generation method in an embodiment of the present application;

[0068] Figure 2 is a 3D Gaussian binding grid topology flowchart in an embodiment of the present application;

[0069] Figure 3is a 3D Gaussian modeling schematic diagram of a corresponding character id constructed according to a given 2D identity image and a 3D grid topology in an initial state in an embodiment of the present application;

[0070] Figure 4 is a visualization result schematic diagram of a grid-driven 2D character portrait model in an embodiment of the present application using 3D Gaussian as an intermediary and using a replay network as prior;

[0071] Figure 5 is a test comparison schematic diagram of the method provided by the embodiment of the present application and the current 2D replay method. DETAILED DESCRIPTION

[0072] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0073] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0074] In the description of the present application, several meanings are one or more, and multiple meanings are two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If it is described as first, second, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0075] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0076] Technical explanation:

[0077] (1) Mesh-Driven Animation: A method that uses a mesh sequence to drive animation generation, achieving dynamic representation of objects through continuous mesh deformation.

[0078] (2) Gaussian Splatting: A rendering method that represents a 3D scene as a point cloud defined by a Gaussian distribution. It achieves efficient image generation and rendering by projecting these Gaussian points onto a 2D image plane.

[0079] (3) Deformation Field: In Gaussian splashing technology, the deformation field is a mathematical model used to describe and control the deformation of the surface or volume of a three-dimensional object in space. It represents the local displacement and shape change of the object during the deformation process by using parameters such as rotation offset and center point offset in each point or voxel in a 3D Gaussian representation, thereby achieving smooth and continuous deformation of the object's shape.

[0080] (4) 2D Replay: 2D replay is a computer vision task designed to generate a replay image that combines information from two input images: it preserves the pose and expression of the driving image while retaining the identity features of the source image. The core task is to transfer facial movements and expressions from one image or video to another.

[0081] (5) Face Reenactment

[0082] This method generates a reenacted image by transferring facial movements and expressions from a given driven image to a given source image. It preserves pose information from the driven image while retaining identity information from the source image. The specific steps include: first, rendering a 3D mesh into a 2D animation, extracting facial pose information from the driven image, and extracting identity features from the source image; then, using a deep learning model to map the expression features of the driven image onto the source image, ensuring that the identity features of the source face are preserved; finally, the generated reenacted image displays how the source face appears with the expression and pose from the driven image, and performs necessary post-processing to improve image quality. This method achieves high-fidelity facial reenactment and is suitable for various application scenarios.

[0083] (6) Dynamic human portrait generation based on 3D model

[0084] Motion from one three-dimensional mesh is retargeted onto another mesh and finally rendered as a two-dimensional video. First, motion data, including joint positions and animation features, are extracted from the source mesh, then these data are mapped onto the target mesh using motion retargeting algorithms to ensure naturalness and fluidity of motion. Next, animation sequences for the target mesh are generated according to the retargeted motion, and a rendering engine is used to set camera perspectives and lighting conditions to generate high-quality two-dimensional video output. Finally, post-processing is applied to the rendered results to enhance visual effects.

[0085] (7) 3D Gaussian Splatting

[0086] The method of 3D Gaussian Splatting provides a solution for reconstructing static scenes based on images and camera parameters. The method represents a scene as a set of 3D anisotropic Gaussian Splatting points, where each Splatting point is defined by a covariance matrix centered at a point (mean) μ:

[0087]

[0088] It is worth noting that the covariance matrix only has physical meaning in the case of positive semi-definiteness, which cannot be guaranteed when using gradient descent for optimization. Therefore, Kerbl et al. defined a parametric ellipse with a scaling matrix S and a rotation matrix R, and then constructed the covariance matrix as follows:

[0089] Σ = RSS T R T .

[0090] Therefore, an ellipse can be represented by a position vector a scaling vector and a quaternion In this paper, we use the rotation matrix to represent the corresponding rotation.

[0091] During rendering, the color C of a pixel is calculated by blending all 3D Gaussian Splatting points that overlap with the pixel:

[0092]

[0093] where c i represents the color of each point, which is modeled by a third-order spherical harmonic function. The blending weight α' is determined by the 2D projection of the 3D Gaussian multiplied by the transparency α of each point. To maintain the visibility order, the Gaussian Splatting points are sorted according to the depth before blending.

[0094] In view of the existing technical problems, the application provides an innovative method based on Gaussian splash representation, which focuses on the field of two-dimensional face animation driven by grid motion sequence. The method simplifies the grid model and rendering process, and enhances the transferability of motion data, so that related work such as art and content creation no longer needs to create fine three-dimensional assets for each static picture. Motion data can be flexibly applied to any head avatar, realizing the efficiency and flexibility of animation generation.

[0095] Specifically, the method serves as an intermediary between two-dimensional portraits and three-dimensional grids, and realizes efficient and flexible animation generation of any static portrait by combining the gradient-guided optimization technology of two-dimensional prior. By significantly simplifying the traditional 3D driving and rendering process, the application reduces the resource demand of 3D face asset production and simplifies the rendering process from 3D to 2D. At the same time, excellent temporal stability is provided in the generation of long video sequences.

[0096] Embodiment 1

[0097] As shown in Figure 1 , the embodiment provides a grid sequence driven two-dimensional face animation generation method based on Gaussian splash representation, comprising the following steps:

[0098] S1, obtaining a 2D face image to be driven;

[0099] S2, binding the initial frame of the grid sequence motion and the 2D face image using Gaussian splash representation, binding the topology of the 3D grid and the identity information of the 2D face to the Gaussian model, and generating an initial Gaussian avatar;

[0100] S3, combining a deformation field to represent the motion of the grid sequence on the Gaussian representation, optimizing the Gaussian avatar, and generating an animation video sequence through rendering;

[0101] Among them, the 2D replay network processes the input reference face image and the current rendering frame of the grid as conditions to generate a gradient and optimize the pose and features of the avatar, so that the generated animation is highly consistent with the features of the reference face image.

[0102] The flow of the method of the embodiment is shown in Figure 1 , specifically, we use grid sequence driving technology to realize the animation generation of the portrait. First, a 2D portrait picture to be driven is given, then we can use a face grid sequence with any topological structure to make the 2D portrait move according to the specified grid sequence.

[0103] Specifically, first, the topology of the 3D mesh and the identity information of the 2D portrait are bound to the Gaussian model by using Gaussian splatting expression to bind the initial frame of the mesh sequence motion (Mesh Sequence Motion) and the 2D initial image, and an initial Gaussian avatar is generated. Then, the motion of the mesh sequence on the Gaussian expression is represented by combining the deformation field, the Gaussian splatting avatar is optimized, and an animated video sequence is generated by rendering.

[0104] Meanwhile, the 2D replay pre-training network processes the input reference image and the current rendering frame of the mesh as conditions to generate gradients and optimize the pose and features of the avatar, so that the generated animation is highly consistent with the features of the input image. The whole process ensures that the generated 3D dynamic avatar has high fidelity and accurate facial motion in vision.

[0105] wherein, Figure 1 、 Figure 4 and Figure 5 The face image in the above method is not an AI-processed face image, so there is no infringement of the portrait right of others.

[0106] The above method is supplemented by the following description of the drawings and specific embodiments.

[0107] (1) Network framework

[0108] The object of the present application is to generate a corresponding animated video sequence by driving the motion of the mesh given a static 2D portrait image I and a 3D human head mesh M via an arbitrary topology. wherein T is the number of frames of the animation. To this end, the embodiment of the present application proposes a method based on Gaussian splatting representation as an intermediate expression between 2D portraits of different identities and 3D meshes, which can realize the transition from mesh sequence M to animated sequence video .

[0109] (2) Gaussian binding

[0110] As Figure 2 shown, the core of the method of the present application is to use 3D Gaussian splatting as an intermediate to establish a connection between 3D meshes and 2D portraits of different identities. For each unit triangle in the mesh, we bind a 3D Gaussian unit to it, and give the initial center μ, rotation matrix r, and scaling factor s of the bound Gaussian according to the properties of the unit triangle.

[0111] Specifically, for the center position μ of the bound Gaussian, we are based on the expression form of the barycentric coordinates, that is:

[0112] For any point V of the spatial triangle ΔV1V2V3, there must be a unique λ1, λ2, such that:

[0113]

[0114] where, are the three vertices of the triangle, respectively.

[0115] For each triangle, we obtain the normal vector n by computing the difference set of two edge vectors:

[0116]

[0117] Then, we normalize the normal vector:

[0118]

[0119] For the direction vector d, we choose the edge formed by vertex 1 and vertex 2 to compute the direction vector d, i.e.,

[0120]

[0121] To describe the orientation of the triangle in the global space, we form a rotation matrix R with the direction vector d of an edge, the normal vector n of the triangle, and their cross product d x n as column vectors.

[0122] R = [d, n, d x n]

[0123] We also compute a scalar k to describe the scaling of the triangle by the length of the shortest edge l and its perpendicular direction h.

[0124]

[0125] For a paired 3D Gaussian, we define its position μ, rotation r, and anisotropic scaling s in the local space. We initialize the position μ as the local origin, the rotation r as the identity rotation matrix, and the scaling s as the identity vector. When rendering, we transform these attributes to the global space by:

[0126] r' = R r

[0127] μ' = k R μ + T

[0128] s' = k s

[0129] The constructed 3D mesh topology in the initial state and the Gaussian of a 2D character are shown in Figure 3 .

[0130] (3) Learnable Prior Deformed Gaussian Field

[0131] Standard 3D Gaussian splatting techniques do not support dynamic scene rendering, as the initial point cloud is assumed to be static. Our method, however, needs to handle dynamic scenes, where the subject constantly exhibits different facial expressions and head poses, while we also need to model the parts of the mesh that do not align with the 2D picture identity. We model this dynamic behavior as a deformation from a pre-defined canonical space to a deformed space. In the canonical space, the 3D Gaussians are located at positions x i，can , while in the deformed space, their positions are x i,def .

[0132] For any given frame i, the deformation of each vertex is defined as:

[0133]

[0134] where x is the position of vertex j in the deformed mesh in the i-th frame, v j,can is the position of the same vertex in the canonical mesh.

[0135] Next, for each Gaussian x i,can in the canonical space, we compute its representation of the deformation through an MLP network, and then we transform the Gaussian from the canonical space to the deformed space:

[0136] x i,def = x i，can + Def(v j,can , δv j )

[0137] By directly using the deformed vertices of the mesh and the vertex positions in the canonical space, our model can more accurately capture complex deformations in dynamic scenes, achieving high-quality dynamic scene rendering.

[0138] (4) Optimization with 2D priors and gradient guidance

[0139] To further optimize the driving effect, we introduce 2D priors derived from static images. We input the 2D reference portrait into a pre-trained 2D reenactment network, and provide the current frame's driven mesh state as a conditional drive. The 2D reenactment network can give the reference prior of the 2D reference image after being driven under the current driving information, which provides guidance for maintaining the consistency of the original image features and the reenacted motion, especially in areas prone to distortion, such as the eyes and mouth. We combine these priors with gradient-based optimization techniques to adjust the Gaussian deformation field, ensuring that the motion accurately aligns with the unique facial features and poses of the static image.

[0140] (5) Optimization and regularization

[0141] a) Color loss

[0142] We use a combination of L1 term and D-SSIM term to supervise the rendered image:

[0143] L rgb =(1-λ)L1+λL D-SSIM

[0144] In this embodiment, λ=0.2.

[0145] b) Mask loss

[0146] At the same time, we also use an L1 term to supervise the loss between the rendered transparency and the image mask, expecting to ensure pixel alignment:

[0147]

[0148] c) Total loss function

[0149] Our final loss function is:

[0150] L=L rgb +λ mask L mask

[0151] Where λ mask =1, it is worth noting that we only apply L mask to the visible Gaussian, so only the points are regularized when there is color loss L rgb .

[0152] (6) Experimental details

[0153] The Adam optimizer is used for parameter optimization. We set the position learning rate of the 3D Gaussian to 5e -3 , and the scaled learning rate to 1.7e -2 , and the learning rate of the remaining parameters is consistent with the 3D Gaussian splatting. We trained for 60,000 iterations, and the position learning rate of the Gaussian was exponentially decayed until it reached 0.01 times the initial value at the final iteration.

[0154] (7) Advantages and beneficial effects

[0155] The present application introduces a Gaussian splatting-based intermediate representation method, and combines a two-dimensional prior and gradient-guided Gaussian deformation field optimization method, which brings the following beneficial effects:

[0156] a) Reduce rendering cost and complexity:

[0157] 1. No need for fine 3D models: Traditional methods often rely on high-precision 3D models and complex rendering processes, resulting in high resource consumption and high cost. The present invention uses a coarse grid model and motion data without identity restrictions, achieving high-quality animation generation and significantly reducing the dependence on fine 3D assets.

[0158] 2. Simplify rendering process: By using Gaussian splatting as an intermediate representation between 2D portraits and 3D grids, the complexity of traditional 3D rendering is avoided, reducing rendering time and computational resource consumption.

[0159] b) Improve the flexibility and efficiency of animation generation:

[0160] 1. Enhance the transferability of motion data: The method of the present invention can transfer motion trajectories from any grid model to any static image, breaking the identity restriction. This allows motion data to be flexibly applied to different head avatars, meeting the needs of individualization and diversification.

[0161] 2. Reduce manual adjustments: Using a two-dimensional prior and gradient-guided optimization method, the optimization of Gaussian deformation fields is automatically completed, reducing the need for extensive manual adjustments and professional knowledge, and accelerating the content production process.

[0162] c) Improve the temporal stability of long video sequences: Natural temporal consistency: The grid sequence-driven method can maintain temporal continuity and stability when processing long video sequences, avoiding inter-frame jitter and inconsistency problems.

[0163] The present invention has the following advantages compared to the prior art:

[0164] a) Reduced resource occupation: No longer need for high-precision 3D models and expensive rendering equipment, reducing hardware requirements.

[0165] b) Improved temporal consistency.

[0166] c) Ability to maintain stability when processing long sequences.

[0167] The results of the method of the present invention are shown in Figure 4 It can be seen that through 3D Gaussian splatting modeling, good temporal consistency and spatial consistency can be provided for driving video, significantly improving the fidelity under large-angle rotation.

[0168] Comparison test: At the same time, we also compared with the current 2D reenactment method, the results are shown in Figure 5As shown, based on the mesh motion sequence driving this character, the visual results of comparing our method with the representative work of reenacting direction can be seen that the 2D reenacting method performs poorly at extreme view angles and has ghosting phenomenon, and our method provides higher temporal consistency and spatial consistency, significantly improving the fidelity of driving video. It can be seen that by modeling through 3D Gaussian Splatting, good temporal consistency and spatial consistency can be provided for the driving video, and the fidelity under a large rotation view angle is significantly improved.

[0169] In summary, the method of the present application uses motion data with a coarse mesh model, allowing the motion trajectory to be migrated to any static image, thereby generating a corresponding speaker avatar video sequence. This way effectively reduces the rendering cost and complexity, so that the art and content creation no longer need to create fine three-dimensional assets for each static picture, improving the efficiency and flexibility of animation generation. In addition, for long video sequences, the mesh sequence driving method can provide temporal stability to ensure high continuity and consistency of animation effects. The mesh sequence driving method still has obvious advantages in long sequence processing.

[0170] The method of the present application mainly focuses on migrating a mesh sequence to different character models to drive the same action given a mesh sequence. Specifically, this technology can be applied to a variety of products and scenarios:

[0171] First, in the field of animation production, animators can use this technology to quickly migrate the action sequence of one character to other characters, saving a lot of time and effort while maintaining the consistency and naturalness of the action. This is particularly important for the film and television production, game development and other industries.

[0172] Secondly, in virtual reality (VR) and augmented reality (AR) applications, this technology can enable users to experience the same action performance when switching between different virtual characters, enhancing immersion and interactivity. For example, users can choose different characters in VR games, but still enjoy consistent action experience.

[0173] In addition, in the field of education and training, this technology can be used to create diverse teaching characters, so that the same teaching content can be presented through different characters, enhancing the interest and participation of learning. Teachers can migrate the teaching actions of one character to other characters to maintain the consistency of teaching content.

[0174] Finally, on social media and content creation platforms, users can use this technology to migrate their own action performances to different virtual avatars, creating personalized content to attract more viewers and fans.

[0175] In summary, the technical solution of the present application has wide application potential in the fields of animation production, virtual reality, education and training, and social media, and can realize efficient motion transfer and consistent performance.

[0176] Embodiment 2

[0177] The embodiment provides a grid sequence driven two-dimensional face animation generation device, which comprises:

[0178] A 2D image acquisition module is configured to acquire a 2D face image to be driven.

[0179] A Gaussian avatar generation module is configured to bind an initial frame of the grid sequence motion and the 2D face image using Gaussian splashing expression, bind the topology of the 3D grid and the identity information of the 2D face to the Gaussian model, and generate an initial Gaussian avatar.

[0180] A video sequence generation module is configured to combine a deformation field to represent the motion of the grid sequence on the Gaussian expression, optimize the Gaussian avatar, and generate an animation video sequence through rendering.

[0181] The 2D replay network processes a reference face image and a current rendering frame of the grid as conditions, generates a gradient, and optimizes the pose and features of the avatar, so that the generated animation is highly consistent with the features of the reference face image.

[0182] Since the device is a grid sequence driven two-dimensional face animation generation device according to an embodiment of the present application, and the principle of solving the problem of the device is similar to that of the method, the implementation of the device can be referred to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be described here.

[0183] Embodiment 3

[0184] The embodiment of the present application further provides an electronic device, which comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to realize a grid sequence driven two-dimensional face animation generation method as shown in Figure 1

[0185] ​It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, and the like; and the data storage area can store data created according to the use of the server and the like.

[0186] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0187] Since the electronic device is an electronic device corresponding to the grid sequence driven two-dimensional face animation generation method of the embodiment of the application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0188] Embodiment 4

[0189] The embodiment of the application also provides a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the grid sequence driven two-dimensional face animation generation method as shown in Figure 1 ​

[0190] Those skilled in the art can understand that all or part of the steps in the above-mentioned various methods of the embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0191] Since the storage medium is a storage medium corresponding to the grid sequence driven two-dimensional face animation generation method of the embodiments of the present application, and the problem solving principle of the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0192] Embodiment 5

[0193] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the grid sequence driven two-dimensional face animation generation method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein the executable computer program codes or "codes" for executing various embodiments can be written in high-level programming languages such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0194] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0195] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0196] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for generating 2D face animation driven by a grid sequence, characterized in that, Includes the following steps: Acquire the 2D face image that needs to be driven; The initial frame of the mesh sequence motion and the 2D face image are bound together using Gaussian splashing representation. The topology of the 3D mesh and the identity information of the 2D face are bound to the Gaussian model to generate the initial Gaussian avatar. By combining deformation fields to characterize the motion of mesh sequences on Gaussian representations, Gaussian portraits are optimized and animated video sequences are generated through rendering. The process of binding the initial frame of the mesh sequence motion and the 2D face image using Gaussian splashing representation, and binding the topology of the 3D mesh and the identity information of the 2D face to the Gaussian model to generate the initial Gaussian avatar includes: Using 3D Gaussian splashing as an intermediary, a connection is established between 3D meshes and 2D face images under different identities; For each unit triangle in the mesh, bind a 3D Gaussian splash unit to it, and give the initial center position μ, rotation matrix r, and scaling factor s of the bound Gaussian according to the properties of the unit triangle; The method of combining deformation fields to characterize the motion of the mesh sequence on the Gaussian representation and optimizing the Gaussian head image includes: A 2D reference face image is input into a pre-trained 2D replay network, and the grid state provided by the current frame is used as a conditional driver. The 2D replay network can provide reference priors for the 2D reference face image in the current frame after being driven based on the current driving information. These reference priors provide guidance for maintaining the consistency between the original image features and the replay motion. By combining these reference priors with gradient-based optimization techniques, the Gaussian deformation field is adjusted to ensure accurate alignment of unique facial features and poses in both moving and static images. The 2D replay network processes the input reference face image and the current rendering frame of the mesh as conditions to generate gradients and optimize the pose and features of the avatar so that the generated animation is highly consistent with the features of the reference face image.

2. The method for generating two-dimensional face animation driven by a mesh sequence according to claim 1, characterized in that, A Gaussian splashing representation-based method is employed as an intermediary representation between 2D faces of different identities and 3D meshes, enabling the conversion from mesh sequence M to animation video sequence. The transition; where T is the number of frames in the animation.

3. The method for generating two-dimensional face animation driven by a mesh sequence according to claim 1, characterized in that, For each unit triangle in the mesh, bind it to a 3D Gaussian splash unit, and give the center position μ, rotation matrix r, and scaling factor s of the bound Gaussian unit based on the properties of the unit triangle, including: For the center position μ of the bound Gaussian, the expression based on the centroid coordinates is: For any point V in the spatial triangle ΔV1V2V3, there exist unique coefficients λ1 and λ2 such that: μ=λ1V1 fid +λ2V2 fid +(1-λ1-λ2)V3 fid Among them, V1 fid V2 fid V3 fid These are the three vertices of the triangle; For each triangle, the normal vector n is obtained by calculating the difference between the two side vectors: n=(V2 fid -V1 fid )×(V3 fid -V1 fid ) For the direction vector d, the edge formed by vertex 1 and vertex 2 is used to calculate the direction vector d, that is: To describe the orientation of a triangle in global space, the orientation vector d of one side, the normal vector n of the triangle, and their cross product d×n are used as column vectors to form a rotation matrix R: R = [d, n, d × n] The scalar k is calculated using the lengths of the shortest side l and its perpendicular direction h, to describe the scaling of the triangle: For a paired 3D Gaussian, define its center position μ, rotation matrix r, and anisotropic scaling factor s in local space; initialize the position to the local origin, the rotation r to the unit rotation matrix, and the scaling s to the unit vector; During rendering, these properties are converted to the global space in the following way: r’＝Rr μ'=kRμ+T s' = ks In the formula, r' is the rotation in the global space of a unit Gaussian splash, μ' is the center position in the global space of a unit Gaussian splash, s' is the scaling in the global space of a unit Gaussian splash, and T is the center position of the grid where the Gaussian splash is located in the global space.

4. The method for generating two-dimensional face animation driven by a mesh sequence according to claim 1, characterized in that, The process of generating animated video sequences through rendering includes: For any given frame i, the deformation of each vertex is defined as: In the formula, v is the position of vertex j in the deformed mesh in the i-th frame. j,can These are the positions of the same vertices within the canonical grid; For every Gaussian x in the gauge space i,can The deformation is represented by an MLP network, and then the Gaussian is transformed from the normal space to the deformation space. x i,def =x i,can +Def(v j,can ,δv j ) By directly using the deformable vertices of the mesh and the vertex positions in the normal space, complex deformations in dynamic scenes can be captured more accurately, thus achieving high-quality dynamic scene rendering.

5. The method for generating two-dimensional face animation driven by a mesh sequence according to claim 1, characterized in that, During training and optimization, a combination of L1 and D-SSIM terms is used to supervise the rendered images: L rgb =(1-λ)L1+λL D-SSIM In the formula, and It is the rendered image I render and pre-trained 2D replay network as conditionally generated image I persudo The average value; and It is the rendered image I render and pre-trained 2D replay network as conditionally generated image I persudo variance It is the rendered image I render and pre-trained 2D replay network as conditionally generated image I persudo The covariance of the denominator; C1 and C2 are small constants used to stabilize the denominator; L1 is the loss calculated by subtracting the absolute difference between each pixel between the rendered image and the image conditionally generated by the pre-trained 2D replay network; L D-SSIM Let λ be the structural similarity loss function; λ is L D-SSIM The weight size; An L1 term is also used to monitor the transparency produced by rendering and the loss between image masks, with the aim of ensuring pixel alignment: In the formula, Masked images of faces are generated conditionally using a pre-trained 2D replay network. I alpha The image includes a mask for the face portion of the rendered image; The final loss function is: L=L rgb +λ mask L mask In the formula, λ mask For L mask The weight it occupies in the total loss function.

6. A grid sequence-driven two-dimensional face animation generation device, characterized in that, include: The 2D image acquisition module is used to acquire the 2D face image that needs to be driven. The Gaussian avatar generation module is used to bind the initial frame of the mesh sequence motion and the 2D face image using Gaussian splashing representation, and bind the topology of the 3D mesh and the identity information of the 2D face to the Gaussian model to generate the initial Gaussian avatar. The video sequence generation module is used to combine the deformation field to characterize the motion of the mesh sequence on the Gaussian representation, optimize the Gaussian head, and generate an animated video sequence through rendering; The process of binding the initial frame of the mesh sequence motion and the 2D face image using Gaussian splashing representation, and binding the topology of the 3D mesh and the identity information of the 2D face to the Gaussian model to generate the initial Gaussian avatar includes: Using 3D Gaussian splashing as an intermediary, a connection is established between 3D meshes and 2D face images under different identities; For each unit triangle in the mesh, bind a 3D Gaussian splash unit to it, and give the initial center position μ, rotation matrix r, and scaling factor s of the bound Gaussian according to the properties of the unit triangle; The method of combining deformation fields to characterize the motion of the mesh sequence on the Gaussian representation and optimizing the Gaussian head image includes: A 2D reference face image is input into a pre-trained 2D replay network, and the grid state provided by the current frame is used as a conditional driver. The 2D replay network can provide reference priors for the 2D reference face image in the current frame after being driven based on the current driving information. These reference priors provide guidance for maintaining the consistency between the original image features and the replay motion. By combining these reference priors with gradient-based optimization techniques, the Gaussian deformation field is adjusted to ensure accurate alignment of unique facial features and poses in both moving and static images. The 2D replay network processes the input reference face image and the current rendering frame of the mesh as conditions to generate gradients and optimize the pose and features of the avatar so that the generated animation is highly consistent with the features of the reference face image.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 5.

Images