A method, system, electronic device, and storage medium for constructing a head model.

Abstract

This application relates to the field of computer vision, and more specifically, to a method, system, and electronic device for constructing a head model. The method includes: acquiring point cloud data of a head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speech; performing coarse alignment of the point cloud data with a preset head template model; constructing a main function, which includes identity parameters and expression parameters; constructing a total loss function; iteratively optimizing the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data; iteratively optimizing the expression parameters based on the total loss function, the coarsely aligned point cloud data at the moment of speech, and the iteratively optimized identity parameters; and obtaining a topologically unified head model based on the iteratively optimized identity parameters, expression parameters, and main function. This method enables the construction of a topologically accurate and unified head model, making the head model more realistic and natural when driven by speech data.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a method, system, electronic device, and storage medium for constructing a head model. Background Technology

[0002] Recently, speech-driven datasets are datasets constructed by capturing facial dynamics during real human speech, typically represented as point cloud data. Their core objective is to provide high-quality training samples for speech-driven tasks. Speech-driven tasks aim to generate realistic lip-sync animations that match the speech data by driving a pre-built and trained 3D head model. To achieve this, speech-driven datasets need to contain accurate 3D facial geometry information. Current technologies typically employ multi-view stereo reconstruction techniques to capture head point cloud data from different time points and orientations. However, due to the complexity of natural facial expressions, the topological structure of point cloud data in each frame often differs, while speech-driven tasks require point cloud data with a unified topology to ensure the construction and training of more accurate head models. In existing technologies, mainstream methods employ the Nonrigid Iterative Closest Point (NICP) algorithm, which aligns the topology of point cloud data by deforming a mesh of expressionless face templates to match the point cloud data of each frame. The core advantage of the NICP algorithm lies in its ability to finely adjust local deformations of the face and relies on the initial similarity between the input point cloud data and the face template mesh. However, when encountering point cloud data corresponding to expressions that deviate significantly from the face template mesh, such as exaggerated mouth shapes, the NICP algorithm cannot effectively handle the corresponding global deformations, which can easily lead to a decrease in the topological uniformity accuracy of the point cloud data. This limits the quality of the speech-driven dataset and the realism of the subsequent driving effect. Summary of the Invention

[0003] This invention provides a topology-unified head model construction method, system, electronic device, and storage medium for constructing a topology-unified head model, making the head model more realistic and natural when driven by speech data.

[0004] According to a first aspect of this application, a method for constructing a head model is provided, the method comprising:

[0005] Collect point cloud data of a head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speaking;

[0006] The point cloud data is coarsely aligned with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

[0007] Construct a main function, which includes identity parameters and expression parameters;

[0008] Construct a total loss function, iteratively optimize the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data, and iteratively optimize the expression parameters based on the total loss function, the coarsely aligned speaking point cloud data, and the iteratively optimized identity parameters.

[0009] Based on the main function and the identity parameters and expression parameters that have been iteratively optimized in the main function, a topologically unified head model is obtained.

[0010] Understandably, by collecting point cloud data of the same head object during both expressionless and speaking moments, and after coarse alignment preprocessing with a preset head template model, a main function containing identity parameters and expression parameters is constructed. Based on the total loss function, the identity parameters are iteratively optimized in stages to accurately represent individual facial features. Then, the expression parameters are further optimized in combination with the optimized identity parameters to capture dynamic expression details. Finally, a high-precision head model with topological consistency is generated, which effectively achieves decoupled control of identity features and expression changes, providing an efficient and reliable technical solution for personalized 3D head modeling and dynamic expression synthesis.

[0011] Optionally, the step of coarsely aligning the point cloud data with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speech includes:

[0012] Obtain the key points of the point cloud data;

[0013] Obtain the mesh vertices of the head template model, and determine the mesh key points from the mesh vertices;

[0014] Calculate the transformation matrix between the key points of the point cloud and the key points of the mesh, and transform the point cloud data based on the transformation matrix to obtain coarsely aligned point cloud data. The coarsely aligned point cloud data includes coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

[0015] Understandably, by extracting key points from point cloud data and a pre-defined head template model, and calculating a transformation matrix based on the spatial distribution of these key points, rapid coarse alignment of point cloud data during both expressionless and speaking moments is achieved. This process unifies the original point cloud data to the head template model coordinate system through rigid transformations, i.e., rotation and / or translation. This preserves the original morphological features of the point cloud data while eliminating overall pose differences between different acquisition states. This provides a precise input data foundation for subsequent refined iterative optimization of identity and expression parameters, significantly improving modeling efficiency and robustness in complex deformation scenarios.

[0016] Optionally, constructing the total loss function includes:

[0017] Constructing a distance loss function Specifically:

[0018]

[0019] in, For the head template model, the first Each grid vertex Let be the rotation matrix of the head template model. For the grid vertices The translation vector, This represents the distance from the grid vertex after rotation and translation in the point cloud data. The most recent point cloud vertex, Distance weights Represents the mesh vertices and the point cloud vertices The square of the distance; This is a header template model;

[0020] Constructing the keypoint loss function Specifically:

[0021]

[0022] In the formula, For the first The aforementioned key points of the grid, This is a set of matching relationships between the grid keypoints and the point cloud keypoints, including those pre-defined based on the grid keypoints. and the corresponding point cloud key points The obtained matching relationship, and ;

[0023] The total loss function is constructed based on the distance loss function and the keypoint loss function.

[0024] Understandably, high-precision topology alignment is achieved by constructing a total loss function that integrates distance loss and keypoint constraints. The distance loss function quantifies global shape similarity based on the weighted sum of squared Euclidean distances between mesh vertices after rigid transformation (rotation and translation) and the nearest point cloud vertex. The keypoint loss function, on the other hand, achieves precise alignment of keypoints through predefined matching relationships between mesh keypoints and point cloud keypoints. This combined total loss function ensures both the overall shape approximation accuracy and enhances the reliability of key region localization. It effectively solves the nonlinear registration problem between point cloud data and the head template model in complex deformation scenarios, providing a head framework that combines global consistency with accurate local details for subsequent decoupling optimization of identity and expression parameters.

[0025] Optionally, constructing the total loss function based on the distance loss function and the keypoint loss function includes:

[0026] The total loss function Specifically:

[0027]

[0028] in, The symbol representing the regularization term. For identity parameters, For facial expression parameters, Euler angles, , , , , These are the weights of the keypoint loss function, identity parameter regularization term, facial expression parameter regularization term, Euler angle regularization term, and translation vector regularization term, respectively.

[0029] Euler angles Through the rotation matrix Obtained.

[0030] Understandably, precise and controllable head model optimization is achieved by constructing a total loss function with multidimensional constraints. Distance loss measures the overall similarity between the head template model and the point cloud, while keypoint loss enhances the accurate alignment of anatomical keypoints. Regularization terms for identity parameters, expression parameters, Euler angles, and translation vectors are introduced, and data fitting and parameter stability are dynamically balanced through weight coefficients. This multi-objective joint optimization mechanism constrains global shape consistency, explicitly separates the parameter spaces of identity features and expression changes, and suppresses overfitting risks through regularization terms, ensuring that the optimized head model maintains geometric accuracy and providing a robust mathematical framework for subsequent optimization.

[0031] Optionally, the iterative optimization of the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data includes:

[0032] The facial expression parameters are preset to initial fixed values, and the identity parameters are iteratively optimized using the following formula:

[0033]

[0034] in, For the first Identity parameters during the second optimization. For the first Identity parameters during the second optimization. This indicates the preset learning rate. Represents the total loss function;

[0035] And / or, the iterative optimization of the expression parameters based on the total loss function, the coarsely aligned speech time point cloud data, and the identity parameters after iterative optimization includes:

[0036] The identity parameters that have undergone iterative optimization are taken as fixed values, and the expression parameters are iteratively optimized using the following formula:

[0037]

[0038] in, For the first Identity parameters during the second optimization. For the first Identity parameters during the second optimization. This indicates the preset learning rate. Represents the total loss function;

[0039] And / or, while iteratively optimizing the identity parameters and the expression parameters, simultaneously iteratively optimize the Euler angles and translation vectors using the following formula:

[0040]

[0041] in, For the first Euler angles in suboptimal optimization For the first Euler angles in suboptimal optimization For the first Translation vector during suboptimal optimization For the first Translation vector during suboptimal optimization This indicates the preset learning rate. This represents the total loss function.

[0042] Understandably, a phased iterative optimization strategy is employed to achieve high-precision head model construction. When optimizing identity parameters, expression parameters are set to fixed values, and the identity parameters are updated sequentially based on the total loss function, allowing the head template model to gradually adapt to the individual features of expressionless point cloud data. Subsequently, the optimized identity parameters are fixed, and expression parameters are iteratively optimized to capture dynamic deformations during speech. During this process, Euler angles and translation vectors are simultaneously optimized and updated to achieve precise global pose calibration. This multi-parameter decoupled optimization mechanism, by alternately constraining identity features, expression changes, and spatial transformation parameters, combined with the data fitting and regularization terms in the total loss function, ensures both the stability of individual identity features and achieves flexible expression-driven deformation, ultimately generating a head model with topological consistency and high-precision detail retention.

[0043] Optionally, the main function is specifically:

[0044]

[0045] in, The head model is the one that represents the topologically unified structure. The head template model, These are the preset principal component analysis baseline parameters. These are the preset facial expression parameters. For identity parameters, For expression parameters.

[0046] Understandably, by constructing a main function, the parameter decoupling and efficient reconstruction of the head model are achieved. Based on the head template model, the combination of principal component analysis base parameters and identity parameters is used to characterize the unique facial morphological features of an individual. At the same time, the synergistic effect of expression base parameters and expression parameters is used to represent dynamic expression changes. The identity attributes and expression changes are separated into an independent low-dimensional parameter space, preserving the topological structure of the head template model. This allows the generated head model to reflect the uniqueness of the individual's identity and flexibly drive rich expression dynamics, thereby generating a high-precision head model with a unified topology.

[0047] Optionally, the method further includes:

[0048] The facial expression parameters obtained from each iteration of optimization are input into a low-pass filter to smooth the facial expression parameters, and the smoothed facial expression parameters are used as the input data for the next iteration of optimization.

[0049] Understandably, by introducing a low-pass filter to smooth the optimized facial expression parameters, high-frequency noise and abrupt changes in the facial expression parameters are effectively suppressed, making the generated facial expressions more natural and smooth. At the same time, the smoothed facial expression parameters are used as input data for the next iteration of optimization, forming a closed-loop feedback mechanism. This not only preserves the continuity of facial expression changes, but also improves the stability of the parameter sequence through filtering operations, significantly enhancing the robustness and visual quality of the head model synthesis.

[0050] According to a second aspect of this application, a head model construction system is provided, the system comprising:

[0051] The acquisition module is used to acquire point cloud data of the head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speaking;

[0052] The coarse alignment module is used to coarsely align the point cloud data with the preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

[0053] The main function construction module is used to construct the main function, which includes identity parameters and expression parameters;

[0054] The total loss function construction module is used to construct a total loss function, iteratively optimize the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data, and iteratively optimize the expression parameters based on the total loss function, the coarsely aligned speaking time point cloud data, and the iteratively optimized identity parameters.

[0055] The acquisition module is used to acquire a topologically unified head model by obtaining the identity parameters and expression parameters that have been iteratively optimized in the main function and the main function.

[0056] According to a third aspect of this application, an electronic device is provided, comprising:

[0057] Memory, used to store one or more computer programs;

[0058] A processor, when the one or more computer programs are executed by the processor, implements the head model construction method described in the first aspect above.

[0059] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the head model construction method described in the first aspect above.

[0060] Based on any of the above aspects, embodiments of this application provide a head model construction method, system, electronic device, and storage medium. The method involves collecting point cloud data of a head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speech; coarsely aligning the point cloud data with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speech; constructing a main function, which includes identity parameters and expression parameters; constructing a total loss function; iteratively optimizing the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data; iteratively optimizing the expression parameters based on the total loss function, the coarsely aligned point cloud data at the moment of speech, and the iteratively optimized identity parameters; and obtaining a topologically unified head model based on the main function and the iteratively optimized identity and expression parameters within the main function. This application can achieve the following benefits:

[0061] • Compatible with exaggeratedly deformed point cloud data, achieving higher accuracy: By introducing a dual mechanism of coarse alignment and fine-grained iterative optimization, this application effectively addresses the limitations of traditional algorithms when dealing with exaggerated deformations in point cloud data. Unlike traditional algorithms that rely solely on local fine-tuning, this application achieves global coarse alignment through point cloud keypoints and mesh keypoints, eliminating initial pose deviations. Subsequently, a total loss function incorporating distance and keypoint losses is constructed. Through iterative optimization of identity and expression parameters, complex deformation features are accurately captured, avoiding the convergence failure problem caused by initial matching errors in traditional algorithms and significantly improving the accuracy of the model after topological unification.

[0062] • Utilizing lightweight parameters significantly reduces network transmission costs: This application employs a Three-Dimensional Morphable Model (3DMM) to unify the topology of point cloud data, compressing the thousands of vertex coordinates that would otherwise need to be transmitted into a small number of identity and expression parameters, achieving an exponential reduction in data volume. Compared to traditional methods that transmit complete mesh vertex coordinates frame by frame, this application only needs to transmit a compact sequence of personality parameters to drive head model generation, greatly reducing network bandwidth usage, memory and GPU memory consumption, significantly shortening transmission latency, and improving real-time interaction efficiency.

[0063] Multi-parameter collaborative optimization and dynamic smoothing ensure the quality and temporal consistency of the head model: By constructing a total loss function that includes identity parameters, expression parameters, Euler angles, and translation vectors, and introducing a regularization term to constrain the parameter space, joint optimization of multi-dimensional parameters is achieved. The decoupling design of identity parameters and expression parameters allows independent control of individual features and dynamic expressions, while the synchronous optimization of Euler angles and translation vectors enhances the stability of the global pose. Furthermore, by inputting the optimized expression parameters into a low-pass filter for smoothing, jitter caused by high-frequency noise is effectively suppressed, ensuring a natural transition in expression changes. This allows the generated head model to maintain consistency in identity features while presenting smooth and physically consistent dynamic expressions, significantly improving the realism and visual quality of the synthesized results. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is an illustrative application scenario diagram of the head model construction method provided in this embodiment.

[0066] Figure 2 This is a flowchart of a head model construction method provided in this embodiment.

[0067] Figure 3 This is a flowchart of a coarse alignment method provided in this embodiment.

[0068] Figure 4 This is a schematic diagram of the functional modules of a head model construction system provided in this embodiment.

[0069] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Detailed Implementation

[0070] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0071] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0073] A 3D speech-driven dataset is a collection of data that reconstructs a head model with a unified topological structure from multiple frames of facial point cloud data. Its core objective is to enable the generation of realistic head animations by simply inputting a speech signal. In existing technologies, the construction of a topologically unified head model for a 3D speech-driven dataset typically relies on multi-view stereo reconstruction techniques to obtain multiple frames of head point cloud data during speech. However, since the topological structures of each frame of point cloud data are inherently inconsistent, processing is required to obtain a topologically unified head model. Current technologies often employ the NICP algorithm to align each frame with a neutral facial mesh template to achieve topological uniformity. However, the NICP algorithm is essentially a local fine-tuning method, only applicable to scenarios where the initial shape of the input point cloud data is similar to that of the neutral facial mesh template. When encountering point cloud data with significant deformations such as exaggerated mouth shapes, the accumulated initial matching error can easily lead to head model distortion or convergence failure, making it difficult to meet the requirements of high-precision speech-driven tasks for complex expression modeling.

[0074] This embodiment provides a technical solution that can solve the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.

[0075] An exemplary diagram illustrating an application scenario of a head model construction method provided in this application embodiment is shown below. Figure 1As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has data processing and data optimization functions; the terminal 200 has data acquisition, data optimization, and data display functions.

[0076] Understandably, the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smartphone terminal, personal computer, tablet computer, vehicle terminal, etc., but is not limited to these.

[0077] In one possible implementation, server 100 and terminal 200 may each execute a header model construction method provided in the embodiments of this application. Alternatively, the header model construction method provided in the embodiments of this application may be executed partly in server 100 and partly in terminal 200.

[0078] like Figure 2 As shown, this embodiment provides a method for constructing a head model, which can be further divided into the following steps:

[0079] S110. Collect point cloud data of the head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speaking;

[0080] In this embodiment, the head object can be the subject whose point cloud data needs to be recorded. Preferably, multi-view stereo reconstruction technology is used to collect the point cloud data of the head object. Multi-view stereo reconstruction technology is a computer vision technology that simultaneously acquires images of the same object using multiple cameras or multiple perspectives, and calculates the three-dimensional spatial information of objects in the scene based on the principle of parallax. Its core idea is to deduce the three-dimensional coordinates of the object's surface by simulating the depth perception mechanism of human binocular vision and utilizing the image differences or parallax under different perspectives. Preferably, in this embodiment, multiple cameras are used to surround the head of the object for shooting, and the shutter trigger time and frame rate of the cameras are preset. For example, the shutter trigger time can be preset to within 5ms, and the frame rate can be set to 30fps.

[0081] Understandably, it is necessary to capture the head of the subject in a neutral, expressionless state in order to obtain two-dimensional point cloud data of the subject without facial expressions. Simultaneously, the head of the subject needs to read a sentence in front of the camera, covering all phonemes of the language, to capture the deformation changes of the head, especially the mouth, during the reading, so that two-dimensional point cloud data of the speaking moment can be obtained. The languages ​​mentioned include, but are not limited to, Mandarin and English.

[0082] Understandably, after acquiring two-dimensional expressionless point cloud data and two-dimensional speaking point cloud data, multi-view stereo vision technology (MVS) is needed to perform dense reconstruction on the two-dimensional expressionless point cloud data and the two-dimensional speaking point cloud data respectively, to obtain millions of vertices corresponding to the two-dimensional expressionless point cloud data and the two-dimensional speaking point cloud data, thereby forming three-dimensional expressionless point cloud data and speaking point cloud data, which are the expressionless point cloud data and speaking point cloud data obtained in this embodiment.

[0083] S120. Perform coarse alignment between the point cloud data and the preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

[0084] In this embodiment, the preset head template model is a standard head model without corresponding facial expressions, and the corresponding head structure data are preset standard values. This head template model provides a model framework for the constructed topologically unified head model, allowing point cloud data containing the identity features and facial expression features of the head object to be modified to obtain the corresponding topologically unified head model.

[0085] In this embodiment, when collecting point cloud data of the head object, the head object may inevitably deviate from the center point during collection. Since the point cloud data with this deviation is not in the preset coordinate system, it increases the difficulty of subsequently constructing a topologically unified head model and reduces the accuracy of the construction. Therefore, it is necessary to coarsely align the point cloud data with the preset head template model to ensure that the point cloud data and the head template model are in the same coordinate system, thereby improving the efficiency of subsequently constructing a topologically unified head model.

[0086] Specifically, such as Figure 3 As shown, the step of coarsely aligning the point cloud data with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speech may include the following steps:

[0087] S121. Obtain the key points of the point cloud data;

[0088] In this embodiment, the acquired point cloud data consists of a large number of point cloud data points distributed around the head object as a framework. Understandably, these point cloud data points cover the entire head of the head object, and some point cloud data points have positioning and location reference significance, such as point cloud data points corresponding to the mouth and eyes. In this embodiment, these point cloud data points with positioning and location reference significance are used as point cloud key points, and these point cloud key points serve as data credentials for coarse alignment to complete the alignment of key areas of the head object. Preferably, the point cloud key points are obtained by detecting corresponding 2D key points in the 2D expressionless point cloud data and the 2D speaking point cloud data, respectively, and then sparsely reconstructing the 2D key points to obtain the corresponding 3D point cloud key points in the expressionless point cloud data and the speaking point cloud data.

[0089] S122. Obtain the mesh vertices of the head template model, and determine the mesh key points from the mesh vertices;

[0090] In this embodiment, it is understood that the head template model contains several mesh vertices, used to complete the subsequent transformation based on point cloud data to form the corresponding head model. It is also understood that some of these mesh vertices will have positioning and location reference significance, such as the mesh vertices corresponding to the mouth and eyes of the head template model. In this embodiment, these mesh vertices with positioning and location reference significance are used as mesh key points, and these mesh key points are used as data credentials for coarse alignment to complete the alignment of key areas of the head object.

[0091] S123. Calculate the transformation matrix between the key points of the point cloud and the key points of the mesh, and transform the point cloud data based on the transformation matrix to obtain coarsely aligned point cloud data, wherein the coarsely aligned point cloud data includes coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

[0092] In this embodiment, there is a positional offset between the point cloud keypoints and the mesh keypoints. Therefore, it is necessary to calculate the transformation matrix between the point cloud keypoints and the mesh keypoints to quantify their positional deviation. The point cloud data is then transformed according to this transformation matrix so that the point cloud keypoints are aligned with the mesh keypoints, thus obtaining coarsely aligned point cloud data. Understandably, the expressionless point cloud data and the point cloud data at the moment of speech need to be coarsely aligned with the head template model respectively to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speech.

[0093] S130. Construct the main function, which includes identity parameters and expression parameters;

[0094] In this embodiment, the main function is constructed based on a three-dimensional deformable model (3DMM), which includes identity parameters and expression parameters. The identity parameters reflect the overall head shape of the head object when it is expressionless, while the expression parameters reflect the changes in the head object's expression when it speaks. Optimizing the identity and expression parameters to construct a topologically unified head model can both take into account the overall shape features of the head object and highlight the muscle deformation of the head and face when the head object makes expressions and / or speaks, making the constructed head model more realistic and natural.

[0095] Specifically, the main function is as follows:

[0096]

[0097] in, The head model is the one that represents the topologically unified structure. The head template model, These are the preset principal component analysis baseline parameters. These are the preset facial expression parameters. For identity parameters, For expression parameters.

[0098] In this embodiment, Principal component analysis (PCA) basis parameters can be shape variation rules extracted in advance from massive amounts of human face and head data, and capture the core laws of human facial diversity in extremely simple mathematical language. These are the expression base parameters, which typically include 51 facial expressions. The construction of these parameters is guided by a Facial Action Coding System (FACS), which describes the correspondence between different facial muscle movements and different expressions. The expression base parameters are also called blendshapes. As long as the head model contains the same blendshapes, the expression parameters can drive the head model. Preferably, the expression base parameters can take values ​​in the range [0,1], representing the corresponding expression intensity from weak to strong.

[0099] S140. Construct a total loss function, iteratively optimize the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data, and iteratively optimize the expression parameters based on the total loss function, the coarsely aligned speaking time point cloud data, and the iteratively optimized identity parameters.

[0100] In this embodiment, the identity parameters are first iteratively optimized using the total loss function and coarsely aligned expressionless point cloud data. Then, the expression parameters are iteratively optimized using the total loss function, coarsely aligned speaking point cloud data, and iteratively optimized identity parameters. This allows the identity parameters and expression parameters to be optimized separately. In the identity parameter optimization stage, the focus can be on extracting static features in the expressionless state, effectively removing expression interference. In the expression parameter optimization stage, since the identity parameters are fixed, the optimization space is limited to a reasonable range of physiological movements, avoiding the pathological solution caused by the entanglement of identity features and expression features in traditional methods. This decoupling mechanism not only improves optimization efficiency but also allows the expression parameters to focus on capturing dynamic features such as lip shape changes and muscle stretching, especially significantly improving the modeling accuracy of exaggerated lip shapes.

[0101] Specifically, the construction of the total loss function includes:

[0102] Constructing a distance loss function Specifically:

[0103]

[0104] in, For the head template model, the first Each grid vertex Let be the rotation matrix of the head template model. For the grid vertices The translation vector, This represents the distance from the grid vertex after rotation and translation in the point cloud data. The most recent point cloud vertex, Distance weights Represents the mesh vertices and the point cloud vertices The square of the distance; This is a header template model;

[0105] In this embodiment, each mesh vertex on the head template model needs to find a corresponding point cloud vertex in the point cloud data, and the distance between the two should be as small as possible. When the distance between the two is greater than a preset threshold, it is considered that the mesh vertex has no corresponding point cloud vertex in the point cloud data. The mesh vertex and its corresponding point cloud vertex can be used to calculate the loss value using the aforementioned distance loss function. Preferably, in this embodiment, more attention is paid to the data of the mouth region of the head object; therefore, when the mesh vertex... Distance weight when a vertex belongs to the mouth region of the grid. It can be set to 1 when the grid vertex... Distance weight when a grid vertex belongs to another region It can be set to 0.1.

[0106] Preferably, rotation matrix The Euler angles of the three coordinate axes corresponding to the coordinate system are as follows:

[0107]

[0108] in, express Rotation matrices corresponding to the coordinate axes express Rotation matrices corresponding to the coordinate axes express Rotation matrices corresponding to the coordinate axes 、 、 Let the Euler angles of the three coordinate axes be denoted as , and let the Euler angles be denoted as . .

[0109] Constructing the keypoint loss function Specifically:

[0110]

[0111] In the formula, For the first The aforementioned key points of the grid, This is a set of matching relationships between the network key points and the point cloud key points, including those pre-defined based on the grid key points. and the corresponding point cloud key points The obtained matching relationship, and ;

[0112] In this embodiment, during the initial optimization stage, the point cloud data and the head template model differ significantly. Relying solely on the distance loss function to calculate the loss and obtain the optimization result is inaccurate. Therefore, a keypoint loss function needs to be set as one of the factors affecting the total loss value. Because the acquired mesh keypoints and point cloud keypoints contain additional semantic information—for example, mesh keypoints and / or point cloud keypoints near the mouth region are pre-marked with semantic information related to the "mouth"—fine-grained alignment of keypoints is achieved using this additional semantic information, and the loss value is calculated using the aforementioned keypoint loss function. Understandably, the matching relationship set... It includes the pre-acquired matching relationship between grid keypoints and their corresponding point cloud keypoints, and there is one matching relationship between each grid keypoint and its corresponding point cloud keypoint.

[0113] The total loss function is constructed based on the distance loss function and the keypoint loss function.

[0114] Specifically, constructing the total loss function based on the distance loss function and the keypoint loss function includes:

[0115] The total loss function Specifically:

[0116]

[0117] in, The symbol representing the regularization term. For identity parameters, For facial expression parameters, Euler angles, , , , , These are the weights of the keypoint loss function, identity parameter regularization term, facial expression parameter regularization term, Euler angle regularization term, and translation vector regularization term, respectively.

[0118] Euler angles Through the rotation matrix Obtained.

[0119] Preferably, considering the low accuracy of keypoint alignment, the keypoint loss function weights are adjusted during the iteration process in this embodiment. It needs to be continuously lowered to reduce errors caused by inaccurate alignment of key points.

[0120] S150. Based on the main function and the identity parameters and expression parameters that have been iteratively optimized in the main function, obtain a topologically unified head model.

[0121] Specifically, the iterative optimization of the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data includes:

[0122] The facial expression parameters are preset to initial fixed values, and the identity parameters are iteratively optimized using the following formula:

[0123]

[0124] in, For the first Identity parameters during the second optimization. For the first Identity parameters during the second optimization. Indicates the preset learning rate;

[0125] In this embodiment, it is understood that before the iteration begins, the identity parameters, expression parameters, Euler angles, and translation vectors all need to be set to initial values, preferably 0. During the identity parameter optimization phase, the identity parameters, Euler angles, and translation vectors can be iteratively optimized, while the expression parameters remain at their initial fixed values ​​throughout the optimization process. This reduces confusion and misleading information during the simultaneous optimization of identity and expression parameters, thereby obtaining more accurate identity parameters.

[0126] And / or, the iterative optimization of the expression parameters based on the total loss function, the coarsely aligned speech time point cloud data, and the identity parameters after iterative optimization includes:

[0127] The identity parameters that have undergone iterative optimization are taken as fixed values, and the expression parameters are iteratively optimized using the following formula:

[0128]

[0129] in, For the first Identity parameters during the second optimization. For the first Identity parameters during the second optimization;

[0130] In this embodiment, it is understood that during the expression parameter optimization stage, the expression parameters, Euler angles, and translation vectors can be iteratively optimized. The expression parameters remain at the already optimized fixed values ​​throughout the optimization process, thereby reducing confusion and misleading information during the optimization of identity parameters and expression parameters together, and thus obtaining more accurate expression parameters.

[0131] And / or, while iteratively optimizing the identity parameters and the expression parameters, simultaneously iteratively optimize the Euler angles and translation vectors using the following formula:

[0132]

[0133] in, For the first Euler angles in suboptimal optimization For the first Euler angles in suboptimal optimization For the first Translation vector during suboptimal optimization For the first The translation vector during sub-optimization.

[0134] Specifically, the method further includes:

[0135] The facial expression parameters obtained from each iteration of optimization are input into a low-pass filter to smooth the facial expression parameters, and the smoothed facial expression parameters are used as the input data for the next iteration of optimization.

[0136] In this embodiment, the relationship between frames in the point cloud data at the speaking time was not considered during the optimization stage of the facial expression parameters, which may result in the optimized facial expression parameters not being smooth enough. Therefore, a low-pass filter is needed to smooth each facial expression parameter to obtain smooth facial expression parameters, and the smooth facial expression parameters are used as the input data for the next optimization.

[0137] like Figure 4 As shown in the embodiments of this application, a head model construction system is also provided. Optionally, the system includes:

[0138] The module comprises: acquisition module 211, coarse alignment module 212, main function construction module 213, total loss function module 214, and acquisition module 215, wherein:

[0139] The acquisition module 211 is used to acquire point cloud data of the head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speaking;

[0140] In this embodiment, the acquisition module 211 can be used to perform... Figure 2 For a detailed description of the acquisition module 211, please refer to the description of step S110 shown.

[0141] The coarse alignment module 212 is used to coarsely align the point cloud data with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

[0142] In this embodiment, the coarse alignment module 212 can be used to perform... Figure 2 For a detailed description of the coarse alignment module 212, please refer to the description of step S120 shown.

[0143] The main function construction module 213 is used to construct the main function, which includes identity parameters and expression parameters;

[0144] In this embodiment, the main function construction module 213 can be used to execute... Figure 2 For a detailed description of the main function construction module 213 shown in step S130, please refer to the description of step S130.

[0145] The total loss function construction module 214 is used to construct a total loss function, iteratively optimize the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data, and iteratively optimize the expression parameters based on the total loss function, the coarsely aligned speaking time point cloud data, and the iteratively optimized identity parameters.

[0146] In this embodiment, the total loss function construction module 214 can be used to perform... Figure 2 For a detailed description of step S140, the total loss function construction module 214 can be found in the description of step S140.

[0147] The acquisition module 215 is used to acquire a topologically unified head model based on the main function and the identity parameters and expression parameters that have been iteratively optimized in the main function.

[0148] In this embodiment, the acquisition module 215 can be used to perform... Figure 2 For a detailed description of the acquisition module 215 shown in step S150, please refer to the description of step S150.

[0149] This application also provides an electronic device, the structure of which is as follows: Figure 5 As shown, the electronic device includes a memory 311, a processor 312, a communication module 313, and an input / output interface 314, etc. Optionally, the memory 311, the processor 312, the communication module 313, and the input / output interface 314 can be connected and communicate with each other through a bus 315.

[0150] The memory 311 is used to store one or more computer programs and to transfer the code of the computer programs to the processor 312; when the one or more computer programs are executed by the processor 312, a head model construction method in this application embodiment is implemented.

[0151] Optionally, the electronic device can be connected to a network via communication module 313 to communicate with other devices, such as terminals or servers, to achieve data interaction. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workbenches, mainframes, or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smartphones, tablets, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0152] Optionally, the electronic device can connect to desired input / output devices, such as a keyboard or display device, via the input / output interface 314. The electronic device itself may have a display device, and other display devices can also be connected externally via the input / output interface 314. Optionally, a storage device, such as a hard disk, can also be connected via the input / output interface 314 to store data from the electronic device, read data from the storage device, or store data from the storage device in the memory 311. It is understood that the input / output interface 314 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 314 can be a component of the electronic device or an external device connected to the electronic device when needed.

[0153] Optionally, the memory 311 may be a volatile memory and / or a non-volatile memory. The volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0154] Optionally, the computer program stored in the memory 311 can be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to perform the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0155] Optionally, the processor 312 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and can also be any suitable controller, microcontroller, processor, etc. The processor 312 executes the various methods and processes of this embodiment, exemplarily, such as a head model construction method according to an embodiment of this application.

[0156] Optionally, the bus 315 may include a path for transmitting information. Depending on its function, the bus 315 may be classified as an address bus, a data bus, a control bus, etc.

[0157] In an optional implementation, this application embodiment also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above-described method embodiments. Part or all of the computer program can be loaded and / or installed on the memory 311 of an electronic device. When the computer program is executed by the processor 312, one or more steps of a header model construction method according to an embodiment of this application can be performed.

[0158] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0159] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for constructing a head model, characterized in that, The method includes: Collect point cloud data of a head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speaking; The point cloud data is coarsely aligned with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking. Construct a main function, which includes identity parameters and expression parameters; A total loss function is constructed, and the identity parameters are iteratively optimized based on the total loss function and the coarsely aligned expressionless point cloud data. The expression parameters are then iteratively optimized based on the total loss function, the coarsely aligned speech time point cloud data, and the iteratively optimized identity parameters. The construction of the total loss function includes: constructing a distance loss function. Specifically: ;in, For the head template model, the first Each grid vertex Let be the rotation matrix of the head template model. For the grid vertices The translation vector, This represents the distance from the grid vertex after rotation and translation in the point cloud data. The most recent point cloud vertex, Distance weights Represents the mesh vertices and the point cloud vertices The square of the distance; For the head template model; construct the keypoint loss function. Specifically: In the formula, For the first The key mesh points in the head template model. This is a set of matching relationships between the grid keypoints and the point cloud keypoints in the point cloud data, including those pre-defined based on the grid keypoints. and the corresponding point cloud key points The obtained matching relationship, and The total loss function is constructed based on the distance loss function and the keypoint loss function; a topologically unified head model is obtained based on the main function and the identity parameters and expression parameters that have been iteratively optimized in the main function.

2. The method according to claim 1, characterized in that, The step of coarsely aligning the point cloud data with a preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speech includes: Obtain the key points of the point cloud data; Obtain the mesh vertices of the head template model, and determine the mesh key points from the mesh vertices; Calculate the transformation matrix between the key points of the point cloud and the key points of the mesh, and transform the point cloud data based on the transformation matrix to obtain coarsely aligned point cloud data. The coarsely aligned point cloud data includes coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

3. The method according to claim 1, characterized in that, The step of constructing the total loss function based on the distance loss function and the keypoint loss function includes: The total loss function Specifically: in, The symbol representing the regularization term. For identity parameters, For facial expression parameters, Euler angles, , , , , These are the weights of the keypoint loss function, identity parameter regularization term, facial expression parameter regularization term, Euler angle regularization term, and translation vector regularization term, respectively. Euler angles Through the rotation matrix Obtained.

4. The method according to claim 1, characterized in that, The iterative optimization of the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data includes: The facial expression parameters are preset to initial fixed values, and the identity parameters are iteratively optimized using the following formula: in, For the first Identity parameters during the second optimization. For the first Identity parameters during the second optimization. This indicates the preset learning rate. Represents the total loss function; And / or, the iterative optimization of the expression parameters based on the total loss function, the coarsely aligned speech time point cloud data, and the identity parameters after iterative optimization includes: The identity parameters that have undergone iterative optimization are taken as fixed values, and the expression parameters are iteratively optimized using the following formula: in, For the first Identity parameters during the second optimization. For the first Identity parameters during the second optimization. This indicates the preset learning rate. Represents the total loss function; And / or, while iteratively optimizing the identity parameters and the expression parameters, simultaneously iteratively optimize the Euler angles and translation vectors using the following formula: in, For the first Euler angles in suboptimal optimization For the first Euler angles in suboptimal optimization For the first Translation vector during suboptimal optimization For the first Translation vector during suboptimal optimization This indicates the preset learning rate. This represents the total loss function.

5. The method according to any one of claims 1 to 3, characterized in that, The main function is specifically: in, The head model is the one that unifies the topology. The head template model, These are the preset principal component analysis baseline parameters. These are the preset facial expression parameters. For identity parameters, For expression parameters.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The facial expression parameters obtained from each iteration of optimization are input into a low-pass filter to smooth the facial expression parameters, and the smoothed facial expression parameters are used as the input data for the next iteration of optimization.

7. A head model construction system, characterized in that, The system includes: The acquisition module is used to acquire point cloud data of the head object, wherein the point cloud data includes expressionless point cloud data and point cloud data at the moment of speaking; The coarse alignment module is used to coarsely align the point cloud data with the preset head template model to obtain coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking. The main function construction module is used to construct the main function, which includes identity parameters and expression parameters; The total loss function construction module is used to construct a total loss function, iteratively optimize the identity parameters based on the total loss function and the coarsely aligned expressionless point cloud data, and iteratively optimize the expression parameters based on the total loss function, the coarsely aligned speech time point cloud data, and the iteratively optimized identity parameters; wherein, constructing the total loss function includes: constructing a distance loss function. Specifically: ;in, For the head template model, the first Each grid vertex Let be the rotation matrix of the head template model. For the grid vertices The translation vector, This represents the distance from the grid vertex after rotation and translation in the point cloud data. The most recent point cloud vertex, Distance weights Represents the mesh vertices and the point cloud vertices The square of the distance; For the head template model; construct the keypoint loss function. Specifically: In the formula, For the first The key mesh points in the head template model. This is a set of matching relationships between the grid keypoints and the point cloud keypoints in the point cloud data, including those pre-defined based on the grid keypoints. and the corresponding point cloud key points The obtained matching relationship, and The total loss function is constructed based on the distance loss function and the keypoint loss function. The acquisition module is used to acquire a topologically unified head model by obtaining the identity parameters and expression parameters that have been iteratively optimized in the main function and the main function.

8. The system according to claim 7, characterized in that, The coarse alignment module also includes: Obtain key points of the point cloud data; obtain the mesh vertices of the head template model, and determine the mesh key points from the mesh vertices; calculate the transformation matrix between the key points of the point cloud and the mesh key points, and transform the point cloud data based on the transformation matrix to obtain coarsely aligned point cloud data, wherein the coarsely aligned point cloud data includes coarsely aligned expressionless point cloud data and point cloud data at the moment of speaking.

9. An electronic device, characterized in that, include: Memory, used to store one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements a head model construction method as described in any one of claims 1-6.

10. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the processor to implement a head model construction method as described in any one of claims 1-6.

Images