Expression base generation and hybrid driving method and system based on topological consistency and sparse constraint

Abstract

The invention provides an expression base generation and hybrid driving method and system based on topological consistency and sparse constraint. The method comprises the following steps: constructing a source template of an ARKit standard expression semantic hybrid shape; based on sparse key point guidance, wrapping the source template to the surface of the cartoon role three-dimensional model by using a non-rigid registration technology to realize topology consistency reconstruction; on the basis of a deformation migration technology, constructing an energy function containing key point constraint terms, solving and generating a whole set of expression bases of the target role, and performing set correction on a specific region in combination with radial basis function interpolation; a mixed shape coefficient of an input video is obtained through a visual capture module, and after time sequence smoothing processing, a final driving coefficient is obtained to drive a virtual character. According to the method, the problems of high difficulty and long time consumption of constructing a cartoon role expression base in a traditional method are effectively solved, low-cost and high-efficiency cartoon image real-time driving is realized, and the method is suitable for virtual live broadcast and meta-universe interaction scenes.

Description

Technical Field

[0001] This invention belongs to the fields of computer graphics and human-computer interaction technology. Specifically, it relates to a method and system for generating and hybridizing facial expressions based on topological consistency and sparse constraints, and more particularly to a method and system for generating and hybridizing cartoon character facial expressions based on topological consistency and sparse constraints using deformation transfer. Background Technology

[0002] With the rise of virtual reality (VR), augmented reality (AR), and the "metaverse" concept, stylized virtual avatars are increasingly being used in various fields such as live streaming, gaming, video conferencing, and film production. Virtual avatars can serve as virtual avatars for users, while stylization adds more interest to 3D faces, providing users with more personalized choices and enhancing the user experience. Currently, constructing blend shapes is a common method for creating high-quality facial animations for these characters. Blend shapes are a commonly used technique in computer graphics, often used to create facial expression animations. Specifically, a blend shape contains a set of predefined expression bases and a neutral expression. By linearly interpolating the neutral expression and the expression base, a smooth transition between them can be achieved, such as from open eyes to gradually closed eyes. Different expression bases, combined with different blending weights, can achieve a wide variety of expressions.

[0003] Traditional methods for constructing blended shapes rely on artists manually sculpting dozens to hundreds of blended facial expressions, a labor-intensive and costly process. Existing technologies, such as deformation transfer, can transfer the facial expressions of a source character to a target neutral model, thereby automating facial expression generation. However, when the source (e.g., a realistic human face) and target (e.g., an exaggerated cartoon face) have vastly different styles, direct transfer often results in semantic distortion and loss of stylistic features, requiring significant manual correction. Furthermore, existing solutions often address how to ensure compatibility between the generated facial expression assets and mainstream real-time driving solutions (e.g., ARKit, MediaPipe) and achieve natural, style-controllable driving, lacking a truly end-to-end solution.

[0004] Patent document "A Method and Device for Driving Animated Characters Based on Artificial Intelligence" (CN110517339A) discloses a method that obtains the expression base of the driving character and the expression base of the driven character, determines the mapping relationship between the two, and directly drives the driven character using the known expression parameters of the driving character. It focuses on solving the parameter conversion problem between the driving character (with a face-pinching base) and the driven character (without a face-pinching base), assuming that the expression bases of both the driving and driven characters already exist, and emphasizes the transmission of driving signals. Patent document "A Method, Electronic Device, and Storage Medium for Real-Time Expression Driving of Cartoon Digital Humans" (CN120472510A) discloses a method that uses the expression base coefficients and rotation feature representation of cartoon characters, combined with a preset BlendShape parameter model, and renders according to computer graphics knowledge to obtain the cartoon character expression animation image corresponding to the target image. This is continuously output to form a real-time facial expression driven animation of the cartoon digital human corresponding to the target user. It is an end-to-end deep learning solution, also assuming that the cartoon character expression base already exists.

[0005] Existing technologies primarily focus on the driving phase, assuming the target character already possesses qualified expression bases (Blendshapes). They address the question of "how to make the asset move," but construction must precede driving. None of the aforementioned patents involve construction from scratch, and they suffer from poor geometric adaptability to stylized / exaggerated characters. When the target character is a cartoon figure with drastically different topological structures and extremely exaggerated facial proportions, direct transfer or prediction often leads to geometric errors such as "unable to close the eyes," because they lack physical constraints and corrections for specific geometric meshes. If relying on pre-set expression bases, the output animation is relatively fixed, lacks diversity, and lacks a mechanism for linear interpolation or style fusion between "realistic" and "cartoonish" styles.

[0006] Therefore, there is an urgent need for a technical solution that can automatically and efficiently generate stylized character facial expression assets, seamlessly integrate with real-time driving pipelines, and allow for flexible adjustment between fidelity and stylization. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for generating expression bases and hybrid driving based on topological consistency and sparse constraints.

[0008] The expression basis generation and hybrid driving method based on topological consistency and sparse constraints provided by the present invention includes: Step S1: Construct a standardized target character model and generate a base set of target character expressions; Step S2: Set the style adjustment factor, obtain the blending shape coefficient of the input video, drive the target character's facial expression base set to calculate and generate animation mesh data, render and output.

[0009] Preferably, step S1 includes: Step S1.1: Construct a source template that includes the source topology of the hybrid shape source; Step S1.2: Obtain the target character model and perform topological normalization processing. Wrap the source template onto the surface of the target character model to generate a normalized target character model. Step S1.3: Calculate the deformation of the mesh patches in the source template and transfer them to the standardized target character model to generate the target character expression base set.

[0010] The source template It includes hybrid shapes that conform to ARKit standards, and the topology meets the specified requirements.

[0011] Preferably, step S1.2 includes: Step S1.2.1: In the source template and target character model Mark the corresponding facial key points on the top; Step S1.2.2: Using facial key points as constraints, a non-rigid registration algorithm is used to deform and fit the mesh vertices of the source template to the geometric surface of the target character model, thereby generating a standardized target character model.

[0012] The standardized target character model is generated by wrapping the source template onto the surface of the target character model, and has the same vertex index, topology, and UV distribution as the source template.

[0013] In step S2, facial data from the video stream is collected in real time, or offline input video data is extracted using a deep learning-based facial feature point detection algorithm to obtain the hybrid shape coefficients of each frame of the video. After time-domain smoothing filtering, the smoothed final driving coefficients are obtained. ; Set stylization adjustment factors, use the final driving coefficient to drive the target character's facial expression base set to fuse multiple sources of real-life characters and the target character, generate animation mesh data, and render and output the facial animation of the virtual character.

[0014] Preferably, step S1.3 includes: Step S1.3.1: Based on the deformation transfer algorithm, construct an energy function containing key point constraints, and minimize the energy function. Find the target facial expression basis set for the target character:

[0015]

[0016]

[0017]

[0018] in, Represents the weight coefficients of different corresponding items; Indicates deformation constraint terms; Indicates key point constraints; This represents the regularization term.

[0019] Triangular facets representing the source template The transformation from a neutral expression to the current expression; Represents the corresponding triangle facet of the target character model Deformation; Indicates the total number of mesh faces; This represents the Frobenius norm.

[0020] Indicates the first in the source template The displacement vectors of each keypoint from the neutral expression to the current expression; L represents the total number of keypoints; This represents the displacement vector of the corresponding key point in the target character from the neutral expression to the newly generated expression base; This represents a 3x3 matrix calculated independently for different facial regions. Represents the identity matrix.

[0021] Step S1.3.2: Extract the neutral expression from the source template to the eye rotation change of the expression with eye movement, and apply it to the eyes of the target character corresponding to the new expression.

[0022] Step S1.3.3: Combine radial basis function interpolation to perform geometric correction on the selected key regions to generate the target character's expression basis set.

[0023] Preferably, step S1.3.2 includes: Step S1.3.2.1: Calculate the rigid rotation matrix of the eyeball mesh in the source template relative to the neutral expression mesh; Step S1.3.2.2: Mark sparse key points on the eyeball, use singular value decomposition to solve the rigid transformation matrix that minimizes the mean square error, and extract the rotation component; Step S1.3.2.3: Convert the rotation components into Euler angles or quaternions in the local coordinate system of the target character's eyeball, and use them as rotation parameters for the target character's eyeball model to rotate the eyeball.

[0024] Step S1.3.3 includes: Step S1.3.3.1: Determine the base expression category of the current expression. If the category is not closed eyes, continue to determine the next expression. If the category is closed eyes, proceed to step S1.3.3.2. Step S1.3.3.2: Detect the distance between the vertices of the key regions of the generated target expression base. If the distance between the vertices of the key regions meets the preset closure threshold, then continue to judge the next expression. If the distance between the vertices of the key regions does not meet the preset closure threshold, then execute step S1.3.3.3. Step S1.3.3.3: Force the key points of the upper eyelid of the target expression base to the coordinate position of the corresponding vertex of the lower eyelid, re-optimize the key point constraint terms, construct the RBF interpolation field with the optimized key points as constraints, and perform local geometric correction on the vertex position; Step S1.3.3.4: Determine whether all target expression bases in the target expression base set have been traversed. If all target expression bases have been traversed, output the target expression base set. If the target expression bases have not been traversed, execute step S1.3.2.

[0025] Preferably, the multi-source fusion adopts pre-fusion at the expression base level or post-fusion at the animation mesh level.

[0026] The pre-fusion at the expression base level generates a hybrid expression base set in advance according to a formula:

[0027] in, The generated sequence number is The target character's facial expressions; Represents the corresponding expression base of a real face model with the same topology.

[0028] The mixed shape coefficients are applied directly to the mixed expression base set to calculate the animation mesh data.

[0029] The post-blending at the animation mesh level utilizes hybrid shape coefficients to drive the target character's facial expression base set, calculating the... Frames of cartoon character mesh Using 3D face video tracking technology to drive a real face model, the first... Realistic face mesh in frames .

[0030] According to the formula and Linearly blending is performed on the corresponding vertex positions to calculate the animation mesh data:

[0031] in, This is a style adjustment factor.

[0032] The present invention provides an expression basis generation and hybrid driving system based on topological consistency and sparse constraints, comprising: Module M1: Obtain the standardized target character model and generate the target character's facial expression base set; Module M2 sets the style adjustment factor, obtains the blending shape coefficient of the input video, drives the target character's facial expression base set to calculate and generate animation mesh data, renders and outputs it.

[0033] Preferably, the standardized target role model is a model of the target role registered with the source template topology, and the topology structure is reconstructed to be consistent with the source template.

[0034] The source template is a model with a hybrid shape and arbitrary topology that conforms to the ARKit standard.

[0035] Calculate the mesh surface deformation of each expression base in the source template relative to the neutral expression, and transfer them one by one to the standardized target character model to generate the target character expression base set.

[0036] Preferably, the module M2 acquires facial data from the video stream in real time, or extracts offline input video data through a deep learning-based facial feature point detection algorithm to obtain the hybrid shape coefficients of each frame of the video. The coefficient sequence in the time domain is smoothed by time-domain smoothing filtering to obtain the smoothed final driving coefficients. ; Set stylization adjustment factors, use the final driving coefficient to drive the target character's facial expression base set to fuse multiple sources of real-life characters and the target character, generate animation mesh data, and render and output the facial animation of the virtual character.

[0037] Preferably, the multi-source fusion adopts pre-fusion at the expression base level or post-fusion at the animation mesh level.

[0038] The pre-fusion at the expression base level generates a hybrid expression base set in advance according to a formula:

[0039] in, The generated sequence number is The target character's facial expressions; Represents the corresponding expression base of a real face model with the same topology.

[0040] The mixed shape coefficients are applied directly to the mixed expression base set to calculate the animation mesh data.

[0041] The post-blending at the animation mesh level utilizes hybrid shape coefficients to drive the target character's facial expression base set, calculating the... Frames of cartoon character mesh Using 3D face video tracking technology to drive a real face model, the first... Realistic face mesh in frames .

[0042] According to the formula and Linearly blending is performed on the corresponding vertex positions to calculate the animation mesh data:

[0043] in, This is a style adjustment factor.

[0044] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention uses topological standardization technology to achieve automated batch generation of facial expression bases with only one key point marking, which significantly reduces the manpower cost of art.

[0045] 2. By introducing key point constraints and combining them with RBF interpolation correction, this invention achieves automated batch generation of facial expression bases while ensuring the artistic quality of the generated results, effectively solving various defects of deformation transfer when dealing with cartoon characters.

[0046] 3. Through a dynamic style fusion mechanism, the present invention enables the same system to adapt to various application scenarios, such as realistic news broadcasts and exaggerated entertainment live broadcasts, thereby improving the user interaction experience.

[0047] 4. This invention constructs a standard and standardized expression base for the target character, which naturally conforms to mainstream standard driving protocols and seamlessly integrates with real-time facial capture technology, forming a complete solution from character modeling to final animation output. Attached Figure Description

[0048] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the expression basis generation and hybrid driving method based on topological consistency and sparse constraints; Figure 2 This is a schematic diagram of manually marking facial key points in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the fusion at the expression base level in an embodiment of the present invention; Figure 4 The image shows the base effect of the character's facial expression generated in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0050] The present invention provides an expression basis generation and hybrid driving method based on topological consistency and sparse constraints. Based on computer graphics geometric algorithms (topological normalization technology and deformation transfer), it constructs a set of topologically unified and semantically aligned cartoon expression basis geometric mesh from scratch. It is oriented towards stylization, especially for cartoon characters, and effectively solves the problems of high difficulty and long time consumption in constructing cartoon character expression basis in traditional methods. It realizes low cost and high efficiency in real-time driving of cartoon characters and is suitable for virtual live broadcast and metaverse interaction scenarios.

[0051] by Figure 1 For example, specifically including: Step S1: Construct a set of hybrid shape source topology templates that include standard facial expression semantics from Apple's Augmented Reality Framework (ARKit).

[0052] Specifically, a set of hybrid shapes containing ARKit standard facial expression semantics and a model whose topology meets the requirements is obtained as the baseline source template (source template).

[0053] As long as it has a hybrid shape that conforms to the ARKit standard, any topological knot (including commonly used FLAME topology) model can be used as a source template. Any topological model with high-precision anatomical structure and containing ARKit standard facial expression definitions can be used (such as the standard face mesh provided in Apple's developer documentation).

[0054] In more preferred examples, the source template serves only as a carrier of geometric shapes and facial semantics, and its specific number of vertices and triangle facet connections are not limited. A character's BlendShapes, constructed using traditional methods (modeling with Maya software), is used, with a high-precision real-life facial model of 52 blend shapes conforming to ARKit standards as the source template. And pre-registered to the FLAME topology.

[0055] Based on deterministic algorithms using mathematical optimization (such as energy function minimization) and geometric computation, a standardized target role model is constructed using non-rigid registration, deformation transfer, and RBF interpolation.

[0056] Step S2: Based on sparse keypoint guidance, non-rigid registration technology is used to wrap the source template onto the surface of the cartoon character's 3D model to achieve topological consistency reconstruction.

[0057] The target role model is subjected to topological normalization. Based on sparse keypoint guidance, non-rigid registration technology is used to generate a standardized target role model with the same vertex index and topological structure as the source template.

[0058] The standardized target character model is a target character model registered with the source template topology. The source template is wrapped around the surface of the target character model to generate a standardized target character model with a cartoonish appearance, but the topology has been reconstructed to be consistent with the source template.

[0059] Specifically, the corresponding facial key points are marked on the source template model and the target character model respectively.

[0060] Using facial key points as constraints, a non-rigid registration algorithm is employed to deform and fit the mesh vertices of the source template model to the geometric surface of the cartoon target character. The core algorithm used is the non-rigid iterative nearest point algorithm (Non-rigid ICP).

[0061] Obtain a model whose geometric surface closely matches the target cartoon character and has vertex indices, topology, and UV distribution consistent with the source template model.

[0062] by Figure 2 For example, manually mark the source templates respectively. and target cartoon characters Facial key points (including eyes, mouth, nose, etc.), guided by these key points, While maintaining its own topological structure, the deformation wraps to... The geometric surface. This process outputs a geometric surface that is consistent with... The topology is completely identical, and the geometric appearance is the same as... The standard topology target role is basically consistent. .

[0063] Since the two have the same topology, their surfaces naturally correspond one-to-one, and subsequent deformation migration can directly apply the surface correspondence between them.

[0064] Step S3: Calculate the mesh deformation of each expression base in the source template relative to the neutral expression, and transfer the deformation one by one to the standardized target character model to generate the target character expression base set.

[0065] To address the issue of insufficient deformation in traditional deformation transfer methods when handling exaggerated cartoon styles, an improved energy minimization framework is employed. This transforms the problem of solving for the target character's facial expression basis into an optimization problem of determining the vertex positions of the target mesh. Keypoint constraints and local RBF (Radial Basis Function) corrections are introduced to produce facial expression bases that semantically conform to the ARKit standard, ensuring the geometric correctness of the assets themselves.

[0066] Specifically, based on deformation transfer technology, an energy function containing keypoint constraints is constructed to generate a complete set of facial expression bases for the target character model. Radial basis function (RBF) interpolation is then used to perform set corrections on specific regions. The target character facial expression base set is derived from the standardized target character model through deformation transfer.

[0067] The deformation migration algorithm minimizes the following energy function. To solve for the target emoji base:

[0068]

[0069]

[0070]

[0071] in, These are the weighting coefficients for each corresponding item.

[0072] For each triangular facet on the source template, the deformation constraint term is defined. The deformation from a neutral expression to the current expression can be represented as a three-dimensional affine transformation. , Represents the corresponding face of the target character The deformation needs to be as close as possible to... , This represents the total number of mesh patches. This is the Frobenius Norm. This term ensures that if the source template performs an action at a certain location, the target character will perform the same action at the same location.

[0073] For key point constraints, where, Indicates the first in the source template L is the displacement vector of a keypoint from a neutral expression to the current expression, where L represents the total number of keypoints. This represents the displacement vector of the corresponding keypoint in the target character from the neutral expression to the newly generated expression basis. It is a 3x3 matrix calculated independently for different facial regions, used to adjust the magnitude and direction of keypoint displacement vectors, thereby compensating for potential scale and orientation differences (such as differences in eye size) between the source template and the target character. Scale differences are measured by the ratio of the region bounding box size, and orientation differences are measured by the orientation vector. This ensures that even cartoon characters that differ significantly from real people can have good deformation transfer effects.

[0074] Here, is the regularization term, The identity matrix is ​​used to constrain the deformation by penalizing the degree to which the affine transformation matrix deviates from the identity matrix.

[0075] Deformation transfer only applies to a single mesh volume, while 3D face models often include eyeballs, but deformation transfer cannot handle eyeballs simultaneously. To handle eyeballs and obtain a blended shape with eyes, deformation transfer is first performed on the facial expression to generate a new expression base containing only the face.

[0076] After processing the facial mesh using deformation transfer as usual, the eye rotation transformation from a neutral expression in the source template to an expression with eye movement is extracted. This rotation transformation is then applied to the eyes of the target character corresponding to the new expression, thus achieving the same eye movement and completing the eye deformation transfer. By combining the eye and facial expression base, the complete expression base of the target character can be obtained.

[0077] Specifically, traditional deformation transfer mainly handles vertex displacement and cannot handle rigid rotation of the eyeball. In the mixed shape data of the source template, for expressions involving eyeball movement (such as left eye looking down, eyeLookDownLeft), the rigid rotation matrix of the source template eyeball mesh relative to the neutral expression mesh is calculated.

[0078] By marking sparse keypoints on the eyeball, the rigid transformation matrix that minimizes the mean square error is solved using singular value decomposition (SVD) to extract the pure rotation component. This rotation component is then converted into Euler angles or quaternions in the local coordinate system of the target character's eyeball and directly used as the rotation parameters of the target character's eyeball model.

[0079] In more preferred embodiments, a post-processing correction mechanism based on specific keypoint constraints and radial basis functions (RBF) is introduced to address the common problem of incomplete eye and mouth closure in cartoon characters. The correction steps based on radial basis functions (RBF) include: For the "closed eyes" category of expression base, detect the key region vertex distance of the generated specific cartoon expression base (that is, the preliminary target expression base obtained by deformation transfer algorithm), such as calculating the Euclidean distance between corresponding vertices of the upper and lower eyelids.

[0080] A specific combination strategy was adopted: first, the key points of the upper eyelid were forced to conform to the lower eyelid, the key point constraints were re-optimized, and then RBF was used to smooth the surrounding geometry.

[0081] Specifically, if the distance between the vertices of the key area does not meet the preset closure threshold (e.g., 0.1mm), it is determined to be incompletely closed. A forced fitting strategy is executed, which forcibly shifts the key points of the upper eyelid to the coordinate position of the corresponding vertex of the lower eyelid, and re-optimizes the key point constraints to make the key points corresponding to the eyes of the target character fit together.

[0082] Using the key points after forced closure as constraints, an RBF interpolation field is constructed. The vertex positions of the key regions are locally geometrically corrected to ensure a smooth transition of the eyelid edge and surrounding skin, thereby generating a tightly closed final expression base.

[0083] Step S4: Obtain the blending shape coefficients of the input video through the visual capture module.

[0084] Specifically, the visual capture module uses existing mature technologies (such as the MediaPipe library) to predict BlendShapes coefficients as the input signal source for the system. It also acquires facial data from the video stream in real time, or uses MediaPipe to extract offline input video data to obtain the original blending shape coefficients for each frame of the video.

[0085] The original hybrid shape coefficients are obtained through a deep learning-based facial feature point detection algorithm (such as MediaPipe) and can undergo temporal smoothing filtering before application, including further jitter reduction, to obtain the smoothed final driving coefficients. To reduce animation jitter while driving virtual characters.

[0086] The time-domain smoothing filtering process specifically uses a moving average filter or a Kalman filter to smooth the coefficient sequence in the time domain.

[0087] Step S5: Perform the multi-source fusion driving step: Based on the consistency of the topology, set a style adjustment factor and mix the target character with any real-life character. Use the final driving coefficient to drive the target character's facial expression base set, calculate and generate the final animation mesh data; render and output the virtual character's facial animation.

[0088] Specifically, the real-life character, or real-life face model, is the source template, or any other real-life model with the same topology as the source template, which can be considered the input to the system. The standardized target character model has reconstructed the target character into a topological structure completely consistent with the source template, so the two are in one-to-one correspondence at the vertex level, which allows for linear blending adjustment of the stylization level. The target character is the desired system output, while the virtual character is the final actual output result of the system.

[0089] Taking the adopted FLAME topology as an example, the arbitrary real face model is the source template, or a FLAME model animation driven by other tools, both of which have the same topological structure.

[0090] Any real-life character can be linearly blended with the target character to adjust the degree of stylization, and it can be adapted to any method that uses facial expression base coefficients for driving.

[0091] The multi-source fusion driver employs pre-fusion at the facial expression base level or post-fusion at the animation mesh level. The generated assets have a unified topology, allowing the system to perform real-time blending between real-person and cartoon facial expression bases (or meshes), overcoming the rigidity problem of single-style driving.

[0092] The pre-fusion at the expression base level specifically includes: Before using the hybrid shape coefficients to drive the character, a hybrid expression base set is generated in advance according to the formula:

[0093] in, The sequence number generated in step S3 is Cartoon emoji base, For the corresponding expression base of a real face model with the same topology, This is a style adjustment factor. The blending shape coefficient is directly applied to the blending expression base set to calculate the final animation mesh data.

[0094] by Figure 3 For example, it shows a live model and a cartoon character with their mouths open, located in the upper left and lower right corners of the image, respectively. The geometry and materials of the two can be proportionally blended according to the style adjustment factor to achieve a style gradient effect.

[0095] The post-blending at the animation mesh level specifically includes: In each frame rendering loop, the blending shape coefficients are used to drive the target character's facial expression base set to calculate the... Frames of cartoon character mesh ; Using arbitrary 3D face video tracking technology to drive the real face model (such as a parametric FLAME model), the first... Realistic face mesh in frames ; Based on the consistency of the aforementioned topology, the corresponding vertex positions of the two meshes are linearly merged according to the formula to calculate the final animation mesh data:

[0096] in, This is a style adjustment factor.

[0097] Furthermore, taking the use of the FLAME parametric model as a real-world face reference as an example, since the FLAME model generates meshes through parameters, it is not convenient to directly extract the blended shape. Therefore, the animated meshes are merged. Based on the standard FLAME head model, a topology normalization step is performed, and the cartoon model is reconstructed into a structure that is completely consistent with the mesh topology output by the FLAME model through non-rigid registration.

[0098] It receives 52-dimensional blended shape coefficients from MediaPipe, drives the blended expression basis of the normalized cartoon model, and calculates the world coordinate set of cartoon vertices in the current frame. Using the FLAME driving tool, it predicts the FLAME expression coefficients for each frame of the input video, calculates the FLAME model, and obtains the world coordinate set of real-person vertices in the current frame. The vertex data from the two models are blended frame by frame. A stylization adjustment factor is then set. The vertex positions are calculated, and the merged mesh is used in conjunction with the corresponding material textures for rendering to generate animation.

[0099] In more preferred examples, a series of ARKit standard facial expression bases for characters are constructed, and three cases are selected to discuss the ability to construct facial expression bases. Figure 4 For example, the text showcases some facial expression templates for different characters. The first row shows the facial expression template used as the source template; the second row shows the real-life facial expression template with added eye movements and textures; the third and fourth rows show new facial expression templates constructed from two different cartoon characters.

[0100] In more preferred examples, an experimental environment was built using an AMD Ryzen 9 7940H processor and an NVIDIA GeForce RTX 4060 Laptop GPU to verify the technological advantages, specifically covering three dimensions: geometric accuracy, asset production efficiency, and online drive performance.

[0101] To assess the quality of topological consistency and sparse constraint generation, a quantitative test was conducted on the generated set of 52 ARKit standard facial expression bases, and the average geometric accuracy was 99.21%.

[0102] This data demonstrates that the model generated by this method exhibits extremely high geometrical surface overlap with the target original model (GroundTruth), which serves as the standard truth model, with an error accounting for only 0.79% of the model scale. To objectively evaluate the geometrical consistency of models at different scales and eliminate the influence of unit system differences, the geometrical accuracy calculation standard uses normalized Euclidean distance:

[0103] in, The diagonal length representing the outline of the original target model (the specific calculation uses the 3D axis-aligned bounding box standard) serves as the dimensional reference for the model, used to convert absolute errors into relative errors. To average the geometric error, since the expression basis model and the original target model have completely different topological structures, direct point-to-point distance calculation is not possible. Therefore, a point-to-surface sampling algorithm is used: First, iterate through and generate each vertex on the surface of the model. Find the distance on the geometry of the original model. Nearest spatial point ( (It can be located inside a triangular facet of the original model mesh, and does not have to be a vertex); calculation and The Euclidean distance between them is obtained by averaging the Euclidean distances over all vertices. .

[0104] The scale consistency verification yielded a true model scale of 25.83 (coordinate units) and a generated model scale of 25.35 (coordinate units).

[0105] While maintaining high accuracy, the model perfectly inherits the original scale of the target character (with a difference of only 1.8%), without the need for additional manual scaling correction.

[0106] To address the pain point of "time-consuming and labor-intensive manual sculpting of facial expression bases" in traditional processes, the total time from a static model to outputting a complete set of dynamic facial expression bases was statistically analyzed to compare asset production efficiency. Among other things: Preprocessing and standardization (steps S1-S2): 10 minutes and 51 seconds. (This stage includes not only the topology reconstruction of the target character's individual model, but also topological consistency registration processing for all 52 source template expressions to ensure the accuracy of subsequent deformation transfer. If only registration of a single model is performed, it would only take a few seconds.) Emoji base generation (step S3): took 3 minutes and 57 seconds (batch generation of 52 emoji bases).

[0107] The entire process takes only about 15 minutes, while in the traditional manual process, it usually takes 3-5 working days for a senior artist to manually carve 52 expressions of the same quality. This increases production efficiency by several orders of magnitude and enables the rapid generation of cartoon character expression assets.

[0108] For "hybrid-driven" and "virtual live streaming / metaverse scenario applications," this invention compares the driving performance and real-time capabilities with commonly used optimization-based driving algorithms (such as the Metrical-Tracker method, which drives the input video for the FLAME model). The Metrical-Tracker method runs in a GPU environment, outputting mesh geometry data and a rendered image for each frame. The expression base generation and hybrid-driven method provided in this invention runs in a CPU environment, similarly outputting and saving mesh geometry data and a rendered image for each frame (including disk write time).

[0109] The results showed that the average frame processing time of Metrical-Tracker (GPU accelerated) was 12.28 seconds per frame, which is only suitable for offline production, cannot be interactive in real time, and is time-consuming. In contrast, the method provided by this invention (based on MediaPipe, CPU environment, including IO time) has an average total frame processing time of 0.46 seconds per frame (including 0.39 seconds of OBJ file saving time + 0.07 seconds of algorithm calculation time).

[0110] In other words, even with inferior hardware and file saving (IO bottleneck), it is still 26 times faster than Metrical-Tracker.

[0111] The method provided by this invention (CPU environment, pure computation), if the step of saving the obj file for debugging is removed (simulating a live streaming scenario), has an average single-frame latency of approximately 0.03 to 0.07 seconds / frame, with an equivalent frame rate of >15 FPS (CPU real-time). That is, in pure computation mode, it is about 175 times faster than Metrical-Tracker.

[0112] The facial expression base generation and hybrid driving method provided by this invention achieves truly low-computing-power, low-latency driving, and can be deployed on ordinary PCs and even mobile devices without relying on expensive high-performance GPUs. Data proves that it has real-time driving capabilities, which can meet the application scenarios with strict low-latency requirements such as virtual live streaming and online meetings.

[0113] This invention also provides an expression base generation and hybrid driving system based on topological consistency and sparse constraints. The expression base generation and hybrid driving system based on topological consistency and sparse constraints can be implemented by executing the process steps of the expression base generation and hybrid driving method based on topological consistency and sparse constraints. That is, those skilled in the art can understand the expression base generation and hybrid driving method based on topological consistency and sparse constraints as a preferred embodiment of the expression base generation and hybrid driving system based on topological consistency and sparse constraints.

[0114] The present invention provides an expression basis generation and hybrid driving system based on topological consistency and sparse constraints, comprising: Module M1: Obtain the standardized target character model and generate the target character's facial expression base set; Module M2 sets the style adjustment factor, obtains the blending shape coefficient of the input video, drives the target character's facial expression base set to calculate and generate animation mesh data, renders and outputs it.

[0115] In more preferred embodiments, the standardized target role model is a model of the target role registered with the source template topology, and the topology structure is reconstructed to be consistent with the source template.

[0116] The source template is a model with a hybrid shape and arbitrary topology that conforms to the ARKit standard.

[0117] Calculate the mesh surface deformation of each expression base in the source template relative to the neutral expression, and transfer them one by one to the standardized target character model to generate the target character expression base set.

[0118] In more preferred embodiments, the module M2 acquires facial data from the video stream in real time, or extracts offline input video data through a deep learning-based facial feature point detection algorithm to obtain the hybrid shape coefficients of each frame of the video. The coefficient sequence in the time domain is smoothed by time-domain smoothing filtering to obtain the smoothed final driving coefficients. ; Set stylization adjustment factors, use the final driving coefficient to drive the target character's facial expression base set to fuse multiple sources of real-life characters and the target character, generate animation mesh data, and render and output the facial animation of the virtual character.

[0119] In more preferred embodiments, the multi-source fusion employs pre-fusion at the facial expression base level or post-fusion at the animation mesh level.

[0120] The pre-fusion at the expression base level generates a hybrid expression base set in advance according to a formula:

[0121] in, The generated sequence number is The target character's facial expressions; Represents the corresponding expression base of a real face model with the same topology.

[0122] The mixed shape coefficients are applied directly to the mixed expression base set to calculate the animation mesh data.

[0123] The post-blending at the animation mesh level utilizes hybrid shape coefficients to drive the target character's facial expression base set, calculating the... Frames of cartoon character mesh Using 3D face video tracking technology to drive a real face model, the first... Realistic face mesh in frames .

[0124] According to the formula and Linearly blending is performed on the corresponding vertex positions to calculate the animation mesh data:

[0125] in, This is a style adjustment factor.

[0126] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0127] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for generating expression basis and hybrid driving based on topological consistency and sparse constraints, characterized in that, include: Step S1: Construct a standardized target character model and generate a base set of target character expressions; Step S2: Set the style adjustment factor, obtain the blending shape coefficient of the input video, drive the target character's facial expression base set to calculate and generate animation mesh data, render and output.

2. The expression basis generation and hybrid driving method based on topological consistency and sparse constraints according to claim 1, characterized in that, Step S1 includes: Step S1.1: Construct a source template that includes the source topology of the hybrid shape source; Step S1.2: Obtain the target character model and perform topological normalization processing. Wrap the source template onto the surface of the target character model to generate a normalized target character model. Step S1.3: Calculate the deformation of the mesh patches in the source template and transfer them to the standardized target character model accordingly to generate the target character's expression base set; The source template It includes hybrid shapes that conform to ARKit standards, and the topology meets the specified requirements.

3. The expression basis generation and hybrid driving method based on topological consistency and sparse constraints according to claim 2, characterized in that, Step S1.2 includes: Step S1.2.1: In the source template and target character model Mark the corresponding facial key points on the top; Step S1.2.2: Using facial key points as constraints, a non-rigid registration algorithm is used to deform and fit the mesh vertices of the source template to the geometric surface of the target character model, thereby generating a standardized target character model. The standardized target character model is generated by wrapping the source template onto the surface of the target character model, and has the same vertex index, topology, and UV distribution as the source template; In step S2, facial data from the video stream is collected in real time, or offline input video data is extracted using a deep learning-based facial feature point detection algorithm to obtain the hybrid shape coefficients of each frame of the video. After time-domain smoothing filtering, the smoothed final driving coefficients are obtained. ; Set stylization adjustment factors, use the final driving coefficient to drive the target character's facial expression base set to fuse multiple sources of real-life characters and the target character, generate animation mesh data, and render and output the facial animation of the virtual character.

4. The expression basis generation and hybrid driving method based on topological consistency and sparse constraints according to claim 2, characterized in that, Step S1.3 includes: Step S1.3.1: Based on the deformation transfer algorithm, construct an energy function containing key point constraints, and minimize the energy function. Find the target facial expression basis set for the target character: in, Represents the weight coefficients of different corresponding items; Indicates deformation constraint terms; Indicates key point constraints; Represents the regularization term; Triangular facets representing the source template The transformation from a neutral expression to the current expression; Represents the corresponding triangle facet of the target character model Deformation; Indicates the total number of mesh faces; Represents the Frobenius norm; Indicates the first in the source template The displacement vectors of the key points from the neutral expression to the current expression; L represents the total number of key points; This represents the displacement vector of the corresponding key point in the target character from the neutral expression to the newly generated expression base; This represents a 3x3 matrix calculated independently for different facial regions; Represents the identity matrix; Step S1.3.2: Extract the neutral expression from the source template to the eye rotation change of the expression with eye movement, and apply it to the eyes of the target character corresponding to the new expression; Step S1.3.3: Combine radial basis function interpolation to perform geometric correction on the selected key regions to generate the target character's expression basis set.

5. The expression basis generation and hybrid driving method based on topological consistency and sparse constraints according to claim 4, characterized in that, Step S1.3.2 includes: Step S1.3.2.1: Calculate the rigid rotation matrix of the eyeball mesh in the source template relative to the neutral expression mesh; Step S1.3.2.2: Mark sparse key points on the eyeball, use singular value decomposition to solve the rigid transformation matrix that minimizes the mean square error, and extract the rotation component; Step S1.3.2.3: Convert the rotation components into Euler angles or quaternions in the local coordinate system of the target character's eyeball, and use them as rotation parameters for the target character's eyeball model to rotate the eyeball. Step S1.3.3 includes: Step S1.3.3.1: Determine the base expression category of the current expression. If the category is not closed eyes, continue to determine the next expression. If the category is closed eyes, proceed to step S1.3.3.

2. Step S1.3.3.2: Detect the distance between the vertices of the key regions of the generated target expression base. If the distance between the vertices of the key regions meets the preset closure threshold, then continue to judge the next expression. If the distance between the vertices of the key regions does not meet the preset closure threshold, then execute step S1.3.3.

3. Step S1.3.3.3: Force the key points of the upper eyelid of the target expression base to the coordinate position of the corresponding vertex of the lower eyelid, re-optimize the key point constraint terms, construct the RBF interpolation field with the optimized key points as constraints, and perform local geometric correction on the vertex position; Step S1.3.3.4: Determine whether all target expression bases in the target expression base set have been traversed. If all target expression bases have been traversed, output the target expression base set. If the target expression bases have not been traversed, execute step S1.3.

2.

6. The expression basis generation and hybrid driving method based on topological consistency and sparse constraints according to claim 3, characterized in that, The multi-source fusion mentioned above adopts pre-fusion at the facial expression base level or post-fusion at the animation mesh level; The pre-fusion at the expression base level generates a hybrid expression base set in advance according to a formula: in, The generated sequence number is The target character's facial expressions; Represents the corresponding expression base of a real face model with the same topology; The mixed shape coefficients are directly applied to the mixed expression base set to calculate the animation mesh data; The post-blending at the animation mesh level utilizes hybrid shape coefficients to drive the target character's facial expression base set, calculating the... Frames of cartoon character mesh Using 3D face video tracking technology to drive a real face model, the first... Realistic face mesh in frames ; According to the formula and Linearly blending is performed on the corresponding vertex positions to calculate the animation mesh data: in, This is a style adjustment factor.

7. A facial expression basis generation and hybrid driving system based on topological consistency and sparse constraints, characterized in that, include: Module M1: Obtain the standardized target character model and generate the target character's facial expression base set; Module M2 sets the style adjustment factor, obtains the blending shape coefficient of the input video, drives the target character's facial expression base set to calculate and generate animation mesh data, renders and outputs it.

8. The expression basis generation and hybrid driving system based on topological consistency and sparse constraints according to claim 7, characterized in that, The standardized target role model is a model of the target role registered with the source template topology, and the topology structure is reconstructed to be consistent with the source template. The source template is a model with a hybrid shape and arbitrary topology that conforms to the ARKit standard; Calculate the mesh surface deformation of each expression base in the source template relative to the neutral expression, and transfer them one by one to the standardized target character model to generate the target character expression base set.

9. The expression basis generation and hybrid driving system based on topological consistency and sparse constraints according to claim 7, characterized in that, The module M2 collects facial data from the video stream in real time, or extracts offline input video data through a deep learning-based facial feature point detection algorithm to obtain the hybrid shape coefficients of each frame of the video. The coefficient sequence in the time domain is smoothed by time-domain smoothing filtering to obtain the smoothed final driving coefficients. ; Set stylization adjustment factors, use the final driving coefficient to drive the target character's facial expression base set to fuse multiple sources of real-life characters and the target character, generate animation mesh data, and render and output the facial animation of the virtual character.

10. The expression basis generation and hybrid driving system based on topological consistency and sparse constraints according to claim 7, characterized in that, The multi-source fusion mentioned above adopts pre-fusion at the facial expression base level or post-fusion at the animation mesh level; The pre-fusion at the expression base level generates a hybrid expression base set in advance according to a formula: in, The generated sequence number is The target character's facial expressions; Represents the corresponding expression base of a real face model with the same topology; The mixed shape coefficients are directly applied to the mixed expression base set to calculate the animation mesh data; The post-blending at the animation mesh level utilizes hybrid shape coefficients to drive the target character's facial expression base set, calculating the... Frames of cartoon character mesh Using 3D face video tracking technology to drive a real face model, the first... Realistic face mesh in frames ; According to the formula and Linearly blending is performed on the corresponding vertex positions to calculate the animation mesh data: in, This is a style adjustment factor.

Images