Face animation generation method and device based on voice driving

Abstract

The invention discloses a face animation generation method based on voice driving. The method comprises the following steps: receiving a first voice and a first emotion tag from a client; the first voice and the first emotion label are input into a voice-to-geometric network to obtain a dynamic three-dimensional point cloud sequence with emotion, and the point cloud is a face point cloud or a head point cloud; based on a first emotion label, the three-dimensional point cloud sequence is optimized, a face appearance sequence is obtained, and the first emotion label is used for guiding facial geometry and / or appearance generation. Therefore, in the process of generating the human face appearance, the human face appearance is guided to be generated from the three-dimensional point cloud sequence through the first emotion label, so that the digital human expression matched with the first emotion label can be presented on the generated human face appearance, and thus emotion-controllable digital human face animation synthesis is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for generating voice-driven facial animation. Background Technology

[0002] Three-dimensional (3D) driven digital human technology refers to the technology of accurately modeling and driving virtual characters using 3D technology and computer graphics methods. It can create and control highly realistic and interactive 3D digital humans. This technology combines techniques from multiple fields such as 3D modeling, animation, speech recognition, facial expression recognition, and sentiment analysis, enabling digital humans to interact with users in a more natural and realistic way.

[0003] Motion capture technology captures the movements of real actors and converts them into the movements of digital models, generating realistic facial animations of 3D digital humans. This method makes the expressions and lip movements of 3D digital humans more natural and realistic, and possess a certain degree of emotional expressiveness. However, motion capture relies on specialized equipment and real actors, which not only increases costs but also raises the complexity of the operation. Therefore, how to conveniently generate facial animations of 3D digital humans is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a voice-driven face animation generation method, apparatus, computing device cluster, computer storage medium, and computer product that can conveniently generate 3D digital human face animations under different emotions.

[0005] In a first aspect, this application provides a voice-driven face animation generation method, comprising: receiving a first voice and a first emotion tag from a client; inputting the first voice and the first emotion tag into a speech-to-geometry network to obtain a dynamic three-dimensional point cloud sequence with attached emotion, wherein the point cloud is a facial point cloud or a head point cloud; optimizing the three-dimensional point cloud sequence based on the first emotion tag to obtain a face appearance sequence, wherein the first emotion tag is used to guide the generation of facial geometry and / or appearance.

[0006] In this way, during the process of generating a human face appearance, the first emotion tag guides the generation of the human face appearance from the 3D point cloud sequence, so that the generated human face appearance can present a digital human expression that matches the first emotion tag, thereby realizing the synthesis of digital human facial animation with controllable emotions.

[0007] In one possible implementation, a 3D point cloud sequence is optimized based on a first sentiment label to obtain a facial appearance sequence. This includes: predicting the first attribute change of a third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment label, a first Gaussian point cloud, and a second head Gaussian point cloud in a regularized space. The first Gaussian point cloud is fitted based on any frame of the 3D point cloud sequence, and the third head Gaussian point cloud is related to the first Gaussian point cloud. The second head Gaussian point cloud is updated based on the first attribute change to obtain the third head Gaussian point cloud. The third head Gaussian point cloud is then rendered to obtain the facial appearance related to any frame of the point cloud. In this way, the attribute change of the second head Gaussian point cloud can be controlled through the sentiment label, thereby guiding the generation of facial geometry and / or appearance, and achieving optimization of the 3D point cloud sequence.

[0008] In one possible implementation, the point cloud is a facial point cloud, and the first Gaussian point cloud is a facial Gaussian point cloud fitted based on any frame of the point cloud. Then, based on the first emotion label, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space, the prediction of the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud includes: calculating the first positional difference between points on the first Gaussian point cloud and points on the facial Gaussian point cloud in the second head Gaussian point cloud; determining the first weight of points on the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the distance between points on the non-facial Gaussian point cloud in the second head Gaussian point cloud and points on the first Gaussian point cloud; and predicting the first attribute change based on the first emotion label, the second head Gaussian point cloud, the first positional difference, and the first weight. Since the first Gaussian point cloud does not contain non-facial point clouds, the positional difference between it and points on the non-facial point cloud in the second head Gaussian point cloud cannot be obtained. Therefore, a parameter to replace this positional difference is needed. The first weight is this parameter that replaces the positional difference. Furthermore, by analyzing the first sentiment label, the second head Gaussian point cloud, the first positional difference, and the first weight, the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud can be predicted.

[0009] In one possible implementation, the first attribute change includes: a first sub-attribute change of the facial Gaussian point cloud and a second sub-attribute change of the non-facial Gaussian point cloud in the second head Gaussian point cloud. Then, based on the first attribute change, the second head Gaussian point cloud is updated to obtain the third head Gaussian point cloud, including: updating the facial Gaussian point cloud in the second head Gaussian point cloud based on the first sub-attribute change to obtain the facial Gaussian point cloud in the third head Gaussian point cloud; updating the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the second sub-attribute change to obtain the first non-facial Gaussian point cloud; updating the opacity of points on the first non-facial Gaussian point cloud based on a first weight, and updating the opacity of points on the non-facial Gaussian point cloud in the second head Gaussian point cloud; obtaining the non-facial Gaussian point cloud in the third head Gaussian point cloud based on the first non-facial Gaussian point cloud and the non-facial Gaussian point cloud in the second head Gaussian point cloud after updating opacity; and fusing the facial Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud to obtain the third head Gaussian point cloud. Since the first Gaussian point cloud does not contain non-facial point clouds, the predicted attribute changes of the two non-facial Gaussian point clouds are obtained through simulation calculations. These attribute changes may contain errors. Therefore, in order to reduce errors, the non-facial Gaussian point clouds can be optimized in this way to obtain a clear and complete head appearance.

[0010] In one possible implementation, the point cloud is a head point cloud, and the first Gaussian point cloud is a head Gaussian point cloud fitted based on any frame of point cloud. In this case, based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space, the prediction of the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud includes: calculating the second positional difference between points on the first Gaussian point cloud and points on the second head Gaussian point cloud; and predicting the first attribute change based on the first sentiment label, the second head Gaussian point cloud, and the second positional difference. Since the first Gaussian point cloud is a head Gaussian point cloud, its points have a one-to-one mapping relationship with the points on the second head Gaussian point cloud. Therefore, the first attribute change can be predicted using the positional difference between the points on both, the first sentiment label, and the second head Gaussian point cloud.

[0011] In one possible implementation, the first attribute change includes one or more of the following: rotation change, color change, or position change.

[0012] Secondly, this application provides a voice-driven face animation generation device, comprising: a communication module and a processing module. The communication module is used to receive a first voice and a first emotion tag from a client. The processing module is used to input the first voice and the first emotion tag into a speech-to-geometry network to obtain a dynamic three-dimensional point cloud sequence with attached emotion, wherein the point cloud is a facial point cloud or a head point cloud; and to optimize the three-dimensional point cloud sequence based on the first emotion tag to obtain a face appearance sequence, wherein the first emotion tag is used to guide facial geometry and / or appearance generation.

[0013] In one possible implementation, when the processing module optimizes the 3D point cloud sequence based on the first sentiment label to obtain a face appearance sequence, it specifically performs the following steps: based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in the regularized space, predicts the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud, wherein the first Gaussian point cloud is obtained by fitting any frame of the point cloud in the 3D point cloud sequence, and the third head Gaussian point cloud is related to the first Gaussian point cloud; updates the second head Gaussian point cloud based on the first attribute change to obtain the third head Gaussian point cloud; and renders the third head Gaussian point cloud to obtain the face appearance related to any frame of the point cloud.

[0014] In one possible implementation, the point cloud is a facial point cloud, and the first Gaussian point cloud is a facial Gaussian point cloud fitted based on any frame of point cloud. In this case, when the processing module predicts the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first emotion label, the first Gaussian point cloud, and the second head Gaussian point cloud in the regularized space, it specifically performs the following: calculating the first positional difference between points on the first Gaussian point cloud and points on the facial Gaussian point cloud in the second head Gaussian point cloud; determining the first weight of points on the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the distance between points on the non-facial Gaussian point cloud in the second head Gaussian point cloud and points on the first Gaussian point cloud; and predicting the first attribute change based on the first emotion label, the second head Gaussian point cloud, the first positional difference, and the first weight.

[0015] In one possible implementation, the first attribute change includes: the first sub-attribute change of the facial Gaussian point cloud in the second head Gaussian point cloud and the second sub-attribute change of the non-facial Gaussian point cloud. At this point, when the processing module updates the second head Gaussian point cloud based on the change of the first attribute to obtain the third head Gaussian point cloud, it specifically performs the following: updates the facial Gaussian point cloud in the second head Gaussian point cloud based on the change of the first sub-attribute to obtain the facial Gaussian point cloud in the third head Gaussian point cloud; updates the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the change of the second sub-attribute to obtain the first non-facial Gaussian point cloud; updates the opacity of the points on the first non-facial Gaussian point cloud based on the first weight, and updates the opacity of the points on the non-facial Gaussian point cloud in the second head Gaussian point cloud; obtains the non-facial Gaussian point cloud in the third head Gaussian point cloud based on the first non-facial Gaussian point cloud and the non-facial Gaussian point cloud in the second head Gaussian point cloud after updating the opacity of the first non-facial Gaussian point cloud and the non-facial Gaussian point cloud in the second head Gaussian point cloud; and merges the facial Gaussian point cloud in the third head Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud to obtain the third head Gaussian point cloud.

[0016] In one possible implementation, the point cloud is a head point cloud, and the first Gaussian point cloud is a head Gaussian point cloud fitted based on any frame of point cloud. In this case, when the processing module predicts the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space, it specifically performs the following: calculates the second positional difference between points on the first Gaussian point cloud and points on the second head Gaussian point cloud; and predicts the first attribute change based on the first sentiment label, the second head Gaussian point cloud, and the second positional difference.

[0017] In one possible implementation, the first attribute change includes one or more of the following: rotation change, color change, or position change.

[0018] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0020] Fifthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0021] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of a voice-driven face animation generation system provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram illustrating the process of generating the facial appearance corresponding to the point cloud of frame t, provided in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram illustrating the process of training a G2ANet according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of another voice-driven face animation generation system provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating an interaction pattern between a tenant and a cloud computing platform, as provided in an embodiment of this application.

[0027] Figure 6 This is a flowchart illustrating a voice-driven face animation generation method provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram illustrating the steps for obtaining the facial appearance corresponding to any frame of point cloud provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram illustrating a step for predicting a change in a first attribute, as provided in an embodiment of this application.

[0030] Figure 9 This is a schematic diagram illustrating another step in predicting a change in the first attribute provided in an embodiment of this application;

[0031] Figure 10 This is a schematic diagram illustrating the steps for obtaining a third-head Gaussian point cloud according to an embodiment of this application;

[0032] Figure 11 This is a schematic diagram of a voice-driven face animation generation device provided in an embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0035] Figure 14 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0036] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0037] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0038] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0040] Generally, voice-driven methods can also be used in the field of facial animation for 3D digital humans. When driving facial animation of a 3D digital human through voice, the system can analyze features such as speech rate, volume, and pitch changes to obtain the emotional state in the speech, such as happiness, sadness, or anger. Based on the results of the emotion analysis, the system can dynamically adjust the facial expressions and movements of the 3D digital human. For example, when a sad voice is recognized, the 3D digital human's expression may become more depressed or sad. While this driving method can achieve facial animation for 3D digital humans, the emotion it uses is recognized from the speech, and it cannot decouple the emotion from the speech. Therefore, this driving method struggles to control the diverse results of emotion generation. For example, if a speaker conveys information in a calm, stable, and neutral tone, but the speaker is actually very angry, the emotion analyzed from that speech will be calm. Therefore, the 3D digital human will ultimately display a calm expression, not an angry one.

[0041] In view of this, embodiments of this application provide a voice-driven face animation generation system. This system can predict a dynamic 3D head / face point cloud sequence with emotion based on the input voice and emotion tags. The predicted point cloud sequence is then optimized using the emotion tags to synthesize a realistic and emotionally rich face appearance sequence. The emotion tags can be used to guide facial geometry and / or appearance generation. Thus, by decoupling voice and emotion, and guiding facial geometry and / or appearance generation through emotion tags, emotion-controllable face animation generation is achieved.

[0042] For example, Figure 1 This illustration shows a schematic diagram of the architecture of a voice-driven face animation generation system provided in an embodiment of this application. Figure 1 As shown, the voice-driven face animation generation system 100 may include: a speech-to-geometry network (S2GNet) 110 and a geometry-to-appearance network (G2ANet) 120.

[0043] S2GNet110 is primarily used to process the input speech and emotion tags to predict dynamic 3D face point cloud sequences with emotional connotations, i.e., facial points in the diagram. S2GNet110 first encodes the input speech into feature vectors (content c in the diagram) using its internal audio encoder. Then, S2GNet110 processes the encoded feature vectors and the input emotion tags using its gated recurrent unit blocks (GRUBlocks) to convert the speech features into geometric representations, thereby predicting dynamic 3D face point cloud sequences with emotional connotations. For example, face point clouds can also be called "facial point clouds." For example, the predicted face point cloud sequence may include at least one frame of point cloud. During the process of converting speech features into geometric representations, emotion tags help S2GNet110 better understand the user's intent and generate geometric representations that better match the user's emotions. For example, emotion tags are labels used to describe emotional tendencies. It can include, but is not limited to, any of the following: positive, negative, neutral, surprised, disgusted, angry, fearful, sad, and happy.

[0044] G2ANet120 is primarily used to combine the sentiment labels input to the system with the head Gaussian point cloud in canonical space to model the facial appearance sequence of a 3D digital human from the face point cloud sequence output by S2GNet110. This effectively synthesizes emotionally rich facial details such as wrinkles and subtle expressions, thereby achieving the goal of synthesizing a realistic and emotionally rich facial appearance. Specifically, G2ANet120 can optimize the face point cloud sequence output by S2GNet110 based on the sentiment label 'e' to obtain the facial appearance sequence. For example, G2ANet120 can estimate the attribute changes of the head Gaussian point cloud corresponding to each frame of the face point cloud sequence output by S2GNet110 relative to the head Gaussian point cloud in canonical space, such as color changes, rotation changes, and / or position changes. For example, the attributes of the Gaussian point cloud can include parameters such as position, scale, opacity, rotation, and color. The color of a Gaussian point cloud can be expressed using a spherical harmonic function. For example, a head Gaussian point cloud in a regularized space can be obtained through pre-training or modeling. Here, the regularized space refers to a pose-independent canonical space.

[0045] After obtaining the attribute changes of the head Gaussian point cloud corresponding to each frame in the face point cloud sequence output by S2GNet110 relative to the head Gaussian point cloud in the regular space, the head Gaussian point cloud in the regular space can be updated using these attribute changes to obtain the head Gaussian point clouds corresponding to other frames. Finally, by rendering the head Gaussian point clouds corresponding to each frame, the face appearance corresponding to each frame can be obtained, thus obtaining the face appearance sequence.

[0046] For ease of understanding, the following example uses the point cloud P of the t-th frame in the face point cloud sequence output by S2GNet110. t Taking the corresponding human face appearance as an example, the working process of G2ANet120 will be introduced.

[0047] For example, please refer to Figure 2 In S201, using 3D Gaussian sputtering (3DGS) technology, the point cloud P in frame t is obtained. t Fit the point cloud P t The corresponding Gaussian point cloud of the facial region.

[0048] In S202, calculate the point cloud P. t The positional difference λ between the corresponding Gaussian point cloud of the facial region and the corresponding points on the Gaussian point cloud of the facial region of the head in regular space. Wherein, cloud P... t There is a one-to-one mapping between the points on the Gaussian point cloud of the corresponding facial region and the points on the Gaussian point cloud of the facial region of the head in regular space. Furthermore, there is a mapping between the points on the Gaussian point cloud of the non-facial region in regular space and the point cloud P. t The distance between the nearest points on the Gaussian point cloud of the corresponding facial region is used to calculate the weight W of each point on the Gaussian point cloud of the non-facial region in the regularized space. p For example, the formula for calculating the weight of a point p in a Gaussian point cloud of a non-facial region in regular space can be as follows:

[0049]

[0050] Where d represents the distance between point p on the Gaussian point cloud of a non-facial region in regular space and the nearest point P in the point cloud. t The distance between points on the Gaussian point cloud corresponding to the facial region, d0 and d th It is a transition factor.

[0051] In S203, using G2ANet120, the attribute parameters of the head Gaussian point cloud in regular space are calculated for the sentiment label e, as well as the positional difference λ and weight W. p Process the data to estimate the point cloud P in frame t. tThe corresponding attribute changes of the head Gaussian point cloud relative to the head Gaussian point cloud in the regular space. G2ANet120 can include: a rotation network (RotationNet) and a color change estimation network (FeatureNet). RotationNet mainly processes the sentiment label e, the rotation of the facial Gaussian point cloud in the head Gaussian point cloud, and the positional difference λ to estimate the point cloud P in frame t. t The corresponding facial Gaussian point cloud is rotated relative to the facial Gaussian point cloud in the regularized space. Additionally, RotationNet can also handle the emotion label e, the rotation of non-facial Gaussian point clouds within the head Gaussian point cloud, as well as the positional differences λ and weights W. p Processing is performed to estimate the point cloud P in frame t. t The rotational transformation of the corresponding non-face Gaussian point cloud relative to the non-face Gaussian point cloud in the regular space.

[0052] Feature Net primarily processes the emotion tag e, the color of the facial Gaussian point cloud in the head Gaussian point cloud, and the positional difference λ to estimate the point cloud P in frame t. t The corresponding facial Gaussian point cloud shows color changes relative to the facial Gaussian point cloud in the regularized space. Additionally, FeatureNet can also analyze the emotion label e, the colors of non-facial Gaussian point clouds within the head Gaussian point cloud, as well as the positional differences λ and weights W. p Processing is performed to estimate the point cloud P in frame t. t The corresponding color changes of the non-face Gaussian point cloud in the relative regular space.

[0053] For the point cloud P in frame t t The positional change of the corresponding facial Gaussian point cloud relative to the facial Gaussian point cloud in the regularized space can be calculated based on the position of the facial Gaussian point cloud in the regularized space and the positional difference λ. For example, for point Q in the facial Gaussian point cloud, if its position is (x, y, z) and its corresponding positional difference λ is (a, b, c), then the positional change of point Q can be (x+a, y+b, z+c). For the point cloud P in frame t... t The positional change of the corresponding non-facial Gaussian point cloud relative to the non-facial Gaussian point cloud in the regularized space can be determined based on the position of the non-facial Gaussian point cloud in the regularized space, as well as the positional difference λ and the weight W. p Calculated. For example, for a point S in a non-facial Gaussian point cloud, if its position is (x, y, z), its corresponding position difference λ is (a, b, c), and the weight W... p If the value is 0.2, then the position of point S can change to (x+0.2a, y+0.2b, z+0.2c).

[0054] In S204, based on the estimated attribute changes, the head Gaussian point cloud in the regular space is updated to obtain the point cloud P in frame t. t The corresponding head Gaussian point cloud. For example, for point Q in the head Gaussian point cloud in regular space, if its position in regular space is (x,y,z), and the estimated position change is (x+a,y+b,z+c), then the position of point Q can be updated from (x,y,z) to (x+a,y+b,z+c).

[0055] In S205, for the point cloud P in frame t... t Render the corresponding Gaussian point cloud of the head to obtain the point cloud P of frame t. t The corresponding facial appearance.

[0056] It should be understood that any of the steps in S201 to S204 can be implemented using G2ANet120, depending on the specific circumstances, and no restrictions are imposed here.

[0057] Furthermore, since a human face includes both facial and non-facial areas of the head, and the S2GNet110 outputs only facial point clouds, excluding non-facial point clouds, therefore, combining... Figure 2 It can be seen that when generating the non-facial Gaussian point cloud corresponding to the face point cloud output by S2GNet110, only the facial point cloud in the face point cloud is used as a reference, and there is no real non-facial point cloud for reference. Therefore, the non-facial appearance obtained from the subsequent rendering of the non-facial Gaussian point cloud may have problems such as blurring. To solve this problem, in this embodiment, in Figure 2 Before S205 in the process, the weights W of each point on the Gaussian point cloud of the non-facial region in the regular space can be calculated. p Given the opacity α of each point on the Gaussian point cloud of the non-facial region in the regular space, and the resulting point cloud P on frame t. t The opacity β of each point in the corresponding non-facial Gaussian point cloud is updated. Furthermore, based on the updated opacity, the point cloud P in frame t is... t The corresponding optimization of the non-facial Gaussian point cloud is thus achieved, thereby realizing the point cloud P in frame t. t The corresponding natural fusion of non-facial Gaussian point clouds and facial Gaussian point clouds. For example, for point K in the non-facial Gaussian point cloud, if its weight is W... k The opacity in the regular space is α, and the point cloud P in frame t is... t If the opacity in the corresponding non-facial Gaussian point cloud is β, then after optimizing α, we can obtain: α×W k After optimizing β, we get: β×(1-W k The opacity of the final point K can be α×W. k+β×(1-W k ).

[0058] It should be understood that in G2ANet120, the emotion label can better fit the changes in facial geometry and / or appearance under different emotions. For example, when angry, the face will frown, and the emotion label will be input into the color change estimation network that controls the color change of the Gaussian point cloud to simulate the frowning details. In some embodiments, the positional difference between the Gaussian point cloud of the facial region in the 3D face point cloud sequence used by G2ANet120 and the corresponding points on the Gaussian point cloud of the facial region of the head in regular space can also be replaced by the difference between the facial expression parameters corresponding to the two point clouds. The facial expression parameters corresponding to each frame of the point cloud can all be derived from S2GNet110.

[0059] It should be understood that, in the above Figure 1 In the described speech-driven face animation generation system 100, G2ANet120 can be pre-trained to improve the accuracy of face appearance synthesis. The training process of G2ANet120 can be as follows: Figure 3 As shown, in S301, the first frame point cloud predicted by S2GNet110 (i.e., the first frame point cloud) is processed using 3DGS technology to obtain a complete head Gaussian point cloud, which includes Gaussian point clouds of the facial and non-facial regions. The head Gaussian point cloud fitted in S301 can be understood as a head Gaussian point cloud in a regularized space. Since the face point cloud output by S2GNet110 can only fit the Gaussian point cloud of the facial region, in order to obtain the Gaussian point cloud of the non-facial region, in this embodiment, starting from some randomly initialized points and the first frame point cloud P1 output by S2GNet110, a complete head Gaussian point cloud can be fitted using 3DGS technology to obtain the head and face in a regularized space. Thus, the point clouds of the facial region and the non-facial region are obtained. Since the first frame point cloud P1 is only the point cloud of the face and does not contain the point cloud of the non-facial region, some points need to be randomly initialized in order to fit a complete head Gaussian point cloud.

[0060] In S302, G2ANet120 is used to estimate the attribute changes of the head Gaussian point cloud corresponding to the first frame point cloud, the emotion label e, the positional differences between the facial Gaussian point cloud corresponding to the first frame point cloud and the facial Gaussian point cloud corresponding to other frames in the face point cloud sequence, and the weights of each point on the facial Gaussian point cloud corresponding to the first frame point cloud, relative to the head Gaussian point cloud corresponding to the first frame point cloud.

[0061] In S303, the Gaussian point cloud of the head corresponding to the point cloud of the first frame is updated based on the calculated attribute changes to obtain the Gaussian point clouds of the head corresponding to the point clouds of other frames. Furthermore, by rendering the Gaussian point clouds of the head corresponding to each frame, the facial appearance corresponding to each frame's point cloud can be obtained.

[0062] In S304, the parameters of G2ANet120 are updated based on the differences between the face appearance and the corresponding face appearance label corresponding to each frame of point cloud, for example, by calculating the difference between the two using a loss function. Alternatively, the calculated differences can be used to update the random points required for fitting the non-facial Gaussian point cloud in S301 to obtain the non-facial Gaussian point cloud in regular space.

[0063] Furthermore, regarding the aforementioned description of "the point cloud P in frame t", t The corresponding optimization of the non-facial Gaussian point cloud is thus achieved, thereby realizing the point cloud P in frame t. t "Natural fusion of corresponding non-facial Gaussian point clouds and facial Gaussian point clouds", by Figure 3 As described above, the complete Gaussian point cloud of the head comes only from the first frame point cloud P1 output by S2Gnet110. The facial region of the head is obtained by optimizing the predicted 3D face point cloud, while the non-facial region is obtained from randomly initialized points. In the process of generating the complete Gaussian point cloud of the head, the non-facial region is obtained through simulation calculation from the facial region; therefore, there is a possibility of blurred reconstructed appearance due to inaccurate pose prediction. Therefore, in order to obtain a clear and complete head appearance, the aforementioned optimization method can be used to optimize the point cloud P1 in frame t. t Optimization of the corresponding non-facial Gaussian point cloud.

[0064] The above is a description of the voice-driven face animation generation system 100 provided in the embodiments of this application. As can be seen from the above description, the voice-driven face animation generation system 100 can first predict a dynamic 3D face point cloud sequence with emotion by inputting voice and emotion tags into the system. Then, based on the emotion tags, the predicted 3D face point cloud sequence can be optimized to generate a realistic and emotionally rich face appearance sequence. Thus, by decoupling voice and emotion, and guiding the geometric and / or appearance synthesis of the face through emotion tags, emotion-controllable face animation generation is achieved.

[0065] Furthermore, in the voice-driven face animation generation system 100 described above, the S2GNet 110 predicts a sequence of face point clouds. This method subsequently requires the use of non-facial regions that are not closely related to the face point cloud sequence, making the processing relatively complex. To simplify the face animation generation process, this application also provides another voice-driven face animation generation system. For example... Figure 4As shown, the voice-driven face animation generation system 400 may include S2GNet410 and G2ANet420. S2GNet410 can predict a dynamic head point cloud sequence (including facial and non-facial regions) with emotion based on the voice and emotion label e input to the system. G2ANet420 can optimize the head point cloud sequence output by S2GNet410 based on the emotion label e to obtain a face appearance sequence. For example, G2ANet120 can estimate the attribute changes of the head Gaussian point cloud corresponding to each frame of the head point cloud sequence output by S2GNet110 relative to the head Gaussian point cloud in the regularized space, based on the emotion label, the head Gaussian point cloud in the regularized space, and the head point cloud sequence output by S2GNet110. In estimating the attribute changes, since S2GNet110 outputs a head point cloud, the head Gaussian point cloud can be fitted from its output, so the aforementioned weight W is not needed in subsequent calculations. p Only point cloud P in the head point cloud sequence is used t The positional difference between the corresponding head Gaussian point cloud and the corresponding point on the head Gaussian point cloud in regular space is sufficient.

[0066] Furthermore, after obtaining the attribute changes of the head Gaussian point cloud corresponding to each frame in the head point cloud sequence output by S2GNet110 relative to the head Gaussian point cloud in the regular space, the head Gaussian point cloud in the regular space can be updated using these attribute changes to obtain the head Gaussian point clouds corresponding to other frames. Finally, by rendering the head Gaussian point clouds corresponding to each frame, the facial appearance corresponding to each frame can be obtained, thus obtaining the facial appearance sequence.

[0067] It should be understood that, Figure 4 The voice-driven face animation generation system 400 shown is... Figure 1 The main difference between the voice-driven face animation generation system 100 shown is:

[0068] 1) Figure 4 The S2GNet410 output is a complete head point cloud sequence, while Figure 1 The S2GNet110 output is a sequence of face point clouds. In other words, Figure 4 The S2GNet410 output contains point clouds of both the facial and non-facial regions of the head. Figure 1 The data output by S2GNet110 only contains point clouds of the facial region of the head.

[0069] 2) Figure 4The input to S2GNet420 includes: the sentiment label e, the attribute parameters of the head Gaussian point cloud in regular space, and the output of S2GNet110 is the positional difference between the head Gaussian point cloud and the head Gaussian point cloud in regular space; while Figure 1 The inputs of G2ANet120 include: sentiment label e, attribute parameters of the head Gaussian point cloud in regular space, and the outputs of S2GNet110 are the positional differences of each point on the face Gaussian point cloud corresponding to the face point cloud and the face Gaussian point cloud in regular space, as well as the weights of each point on the non-face Gaussian point cloud in regular space.

[0070] From the comparison of the two systems above, it can be seen that Figure 4 The system shown requires fewer processing parameters than Figure 1 The processing parameters required by the system shown.

[0071] The above is a description of the voice-driven face animation generation system provided in this application embodiment. It should be understood that the voice-driven face animation generation system described above can, but is not limited to, be configured on a cloud computing platform. For example, it can be deployed on at least one virtual machine or container instance, enabling the cloud computing platform to provide voice-driven face animation generation services. Of course, the voice-driven face animation generation system can also be configured on nodes other than the cloud computing platform. For example, it can be deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud computing platform can provide pages related to public cloud services for tenants to remotely access public cloud services. In this embodiment, the tenant (also referred to as a "user") can pre-purchase the face animation generation services provided by the voice-driven face animation generation system on the cloud computing platform. For ease of understanding, the interaction between the tenant and the cloud computing platform is described below. Figure 5As shown, the interaction between the tenant and the cloud computing platform mainly includes: the tenant logs into the cloud computing platform 500 through a client webpage, selects and purchases cloud services related to the voice-driven face animation generation system (i.e., face animation generation service) on the cloud computing platform 500. After purchase, the tenant can perform voice-driven face animation generation on the cloud computing platform 500 based on the functions provided by the face animation generation service. The cloud computing platform 500 is primarily used to manage the infrastructure for running the voice-driven face animation generation service. For example, the infrastructure for running this face animation generation service may include multiple data centers located in different regions, each data center including multiple servers. The data centers can provide basic resources for the voice-driven face animation generation service, such as computing resources and storage resources. Therefore, when purchasing and using the voice-driven face animation generation service, the tenant mainly pays for the resources used. When using the voice-driven face animation generation service, tenants can input voice and emotion tags through the configuration interface, application program interface (API), or interface for interaction with the tenant provided by the cloud computing platform 500. The cloud computing platform 200 can then generate the corresponding face appearance sequence according to the voice and emotion tags input by the tenant.

[0072] Next, based on the above, a voice-driven face animation generation method provided in the embodiments of this application will be introduced.

[0073] For example, Figure 6 This illustration shows a flowchart of a voice-driven face animation generation method according to an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, the method can be executed by a voice-driven face animation generation device, which can be implemented by software and / or hardware, and can be configured in, but is not limited to, an electronic device or server; typically, it can be configured on a cloud computing platform. For ease of description, the following description will use a cloud computing platform as the execution entity. Figure 6 As shown, the voice-driven face animation generation method may include the following steps:

[0074] S601 receives the first voice and first emotion tag from the client.

[0075] In this embodiment, the cloud computing platform can receive the first voice and the first emotion tag from the client. The user can input the first voice and the first emotion tag on the client. The client can then transmit the first voice and the first emotion tag to the cloud computing platform.

[0076] S602. Input the first speech and the first emotion label into the speech-to-geometry network to obtain a dynamic three-dimensional point cloud sequence with attached emotion, wherein the point cloud is a facial point cloud or a head point cloud.

[0077] In this embodiment, the cloud computing platform can input the first speech and the first emotion label into a speech-to-geometry network to process the first speech and the first emotion label, thereby obtaining a dynamic 3D point cloud sequence with attached emotion. The emotion attached to the dynamic 3D point cloud sequence is the same as the emotion expressed by the first emotion label, and it can geometrically represent the corresponding facial expression. The point cloud in the dynamic 3D point cloud sequence is a facial point cloud or a head point cloud. For example, when the point cloud in the dynamic 3D point cloud sequence is a facial point cloud, the speech-to-geometry network here can be the aforementioned… Figure 1 The S2GNet110 shown here; when the point cloud in the dynamic 3D point cloud sequence is a head point cloud, the speech-to-geometry network here can be the aforementioned Figure 4 The S2GNet410 shown.

[0078] S603. Based on the first sentiment label, optimize the 3D point cloud sequence to obtain a face appearance sequence, wherein the first sentiment label is used to guide the generation of facial geometry and / or appearance.

[0079] In this embodiment, after obtaining the 3D point cloud sequence, the cloud computing platform can optimize the 3D point cloud sequence using a first sentiment label to obtain a facial appearance sequence. The first sentiment label is used to guide the generation of facial geometry (e.g., shape) and / or appearance (e.g., wrinkles, skin).

[0080] In this way, during the process of generating a human face appearance, the first emotion tag guides the generation of the human face appearance from the 3D point cloud sequence, so that the generated human face appearance can present a digital human expression that matches the first emotion tag, thereby realizing the synthesis of digital human facial animation with controllable emotions.

[0081] In some embodiments, the specific steps of S603 can be as follows: Figure 7 As shown. Specifically, S603 may include:

[0082] S6031. Based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in the regular space, predict the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud. The first Gaussian point cloud is obtained by fitting any frame of point cloud in the three-dimensional point cloud sequence, and the third head Gaussian point cloud is related to the first Gaussian point cloud.

[0083] In this embodiment, for any frame of point cloud in the 3D point cloud sequence, it can be fitted into a first Gaussian point cloud using 3DGS technology. Then, based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space, the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud can be predicted. The third head Gaussian point cloud is related to the first Gaussian point cloud. For example, the first attribute change includes one or more of the following: rotation change, color change, or position change. For example, the head Gaussian point cloud in regularized space can be obtained, but is not limited to, through pre-training or modeling.

[0084] Since the point cloud in the 3D point cloud sequence is a facial point cloud or a head point cloud, the first Gaussian point cloud can be either a facial Gaussian point cloud or a head Gaussian point cloud. These two cases will be described separately below.

[0085] 1) The point cloud in the 3D point cloud sequence is a facial point cloud.

[0086] When the point cloud in the 3D point cloud sequence is a facial point cloud, the first Gaussian point cloud is the facial Gaussian point cloud. At this time, as... Figure 8 As shown, the process of predicting the change of the first attribute may include: In S801, calculating the first positional difference between points on the first Gaussian point cloud and points on the face Gaussian point cloud in the second head Gaussian point cloud. For any point on the face Gaussian point cloud, the first positional difference associated with that point can be determined by the deviation between its position on the first Gaussian point cloud and its position on the face Gaussian point cloud in the second head Gaussian point cloud. Each point on the face Gaussian point cloud may correspond to one positional difference. In S802, determining the first weight of points on the non-face Gaussian point cloud in the second head Gaussian point cloud based on the distance between points on the non-face Gaussian point cloud in the second head Gaussian point cloud and points on the first Gaussian point cloud. For example, for any point on the non-face Gaussian point cloud in the second head Gaussian point cloud, the distance between that point and each point on the first Gaussian point cloud can be calculated, and then, based on the calculated minimum distance and the aforementioned "Formula 1", the first weight of that point can be determined. Each point on the non-face Gaussian point cloud may correspond to one first weight. In S803, based on the first sentiment label, the second head Gaussian point cloud, the first positional difference, and the first weight, the change in the first attribute is predicted. Specifically, the attribute parameters of the first sentiment label and the second head Gaussian point cloud, the first positional difference, and the first weight can be input into the aforementioned... Figure 1 The G2ANet120 described in the paper is used to predict changes in the first attribute.

[0087] 2) The point cloud in the 3D point cloud sequence is the head point cloud.

[0088] When the point cloud in the 3D point cloud sequence is the head point cloud, the first Gaussian point cloud is the head Gaussian point cloud. At this time, as... Figure 9 As shown, the process of predicting the change in the first attribute may include: in S901, calculating the second positional difference between points on the first Gaussian point cloud and points on the second head Gaussian point cloud. This step can be referred to the aforementioned... Figure 8 The description of calculating the first positional difference is omitted here. S902, based on the first sentiment label, the second head Gaussian point cloud, and the second positional difference, predict the change in the first attribute. Here, the attribute parameters of the first sentiment label, the second head Gaussian point cloud, and the second positional difference can be input into the aforementioned... Figure 4 The G2ANet420 described in the paper is used to predict changes in the first attribute.

[0089] After predicting the change in the first attribute, S6032 can be executed.

[0090] S6032. Based on the change of the first attribute, update the second head Gaussian point cloud to obtain the third head Gaussian point cloud.

[0091] In this embodiment, since the first Gaussian point cloud can be either a facial Gaussian point cloud or a head Gaussian point cloud, this step can be executed in two different ways. Specifically, when the first Gaussian point cloud is a head Gaussian point cloud, the attribute parameters of the second head Gaussian point cloud can be directly updated through changes to the first attribute to obtain the third head Gaussian point cloud.

[0092] When the first Gaussian point cloud is a facial Gaussian point cloud, the first attribute change can include: the first sub-attribute change of the facial Gaussian point cloud in the second head Gaussian point cloud and the second sub-attribute change of the non-facial Gaussian point cloud. For example, Figure 10 As shown, the process of updating the second head Gaussian point cloud may include: S1001, updating the facial Gaussian point cloud in the second head Gaussian point cloud based on the change of the first sub-attribute, to obtain the facial Gaussian point cloud in the third head Gaussian point cloud. S1002, updating the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the change of the second sub-attribute, to obtain the first non-facial Gaussian point cloud. S1003, based on the aforementioned... Figure 8 The first weights of each point on the non-facial Gaussian point cloud are calculated, and the opacity of the points on the first non-facial Gaussian point cloud is updated. The opacity of the points on the non-facial Gaussian point cloud in the second head Gaussian point cloud is also updated. In step S1004, based on the updated opacity of the first and second non-facial Gaussian point clouds, the non-facial Gaussian point cloud in the third head Gaussian point cloud is obtained. In step S1005, the facial Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud are fused to obtain the third head Gaussian point cloud. For steps S1001 to S1004, please refer to the previous explanation. Figure 2and in Figure 2 Before S205, the point cloud P in frame t. t The relevant descriptions of the optimization of the corresponding non-facial Gaussian point cloud are not repeated here. For S1005, the facial Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud can be stitched together according to the position of each point in the Gaussian point cloud to obtain the third head Gaussian point cloud.

[0093] After obtaining the third head Gaussian point cloud, S6033 can be executed.

[0094] S6033. Render the Gaussian point cloud of the third head to obtain the facial appearance related to any frame of the point cloud.

[0095] In this embodiment, after obtaining the third head Gaussian point cloud, the facial appearance associated with any frame of the point cloud can be obtained by rendering the third head Gaussian point cloud. This completes the generation of the facial appearance corresponding to a frame of the point cloud. This process is performed on each frame of the point cloud sequence. Figure 7 The process in which the three-dimensional point cloud sequence corresponds to the human face appearance sequence can be obtained.

[0096] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0097] The above is an introduction to the voice-driven face animation generation method provided in the embodiments of this application. Next, based on the method in the above embodiments, a voice-driven face animation generation device provided in the embodiments of this application will be introduced.

[0098] For example, Figure 11 A schematic diagram of a voice-driven face animation generation device provided in an embodiment of this application is shown. Figure 11 As shown, the voice-driven face animation generation device 1100 includes a communication module 1101 and a processing module 1102. The communication module 1101 receives a first voice message and a first emotion tag from a client. The processing module 1102 inputs the first voice message and the first emotion tag into a speech-to-geometry network to obtain a dynamic 3D point cloud sequence with accompanying emotion, wherein the point cloud is a facial point cloud or a head point cloud; and optimizes the 3D point cloud sequence based on the first emotion tag to obtain a face appearance sequence, wherein the first emotion tag guides the generation of facial geometry and / or appearance.

[0099] In some embodiments, when the processing module 1102 optimizes the 3D point cloud sequence based on the first sentiment label to obtain a face appearance sequence, it specifically performs the following steps: based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in the regularization space, predicts the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud, wherein the first Gaussian point cloud is obtained by fitting any frame of point cloud in the 3D point cloud sequence, and the third head Gaussian point cloud is related to the first Gaussian point cloud; updates the second head Gaussian point cloud based on the first attribute change to obtain the third head Gaussian point cloud; and renders the third head Gaussian point cloud to obtain a face appearance related to any frame of point cloud.

[0100] In some embodiments, the point cloud is a facial point cloud, and the first Gaussian point cloud is a facial Gaussian point cloud fitted based on any frame of point cloud. At this time, when the processing module 1102 predicts the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first emotion label, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space, it specifically performs the following: calculating the first positional difference between points on the first Gaussian point cloud and points on the facial Gaussian point cloud in the second head Gaussian point cloud; determining the first weight of points on the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the distance between points on the non-facial Gaussian point cloud in the second head Gaussian point cloud and points on the first Gaussian point cloud; and predicting the first attribute change based on the first emotion label, the second head Gaussian point cloud, the first positional difference, and the first weight.

[0101] In some embodiments, the first attribute change includes: a first sub-attribute change of the facial Gaussian point cloud and a second sub-attribute change of the non-facial Gaussian point cloud in the second head Gaussian point cloud. At this time, when processing module 1102 updates the second head Gaussian point cloud based on the change of the first attribute to obtain the third head Gaussian point cloud, it is specifically used to: update the facial Gaussian point cloud in the second head Gaussian point cloud based on the change of the first sub-attribute to obtain the facial Gaussian point cloud in the third head Gaussian point cloud; update the non-facial Gaussian point cloud in the second head Gaussian point cloud based on the change of the second sub-attribute to obtain the first non-facial Gaussian point cloud; update the opacity of the points on the first non-facial Gaussian point cloud based on the first weight, and update the opacity of the points on the non-facial Gaussian point cloud in the second head Gaussian point cloud; obtain the non-facial Gaussian point cloud in the third head Gaussian point cloud based on the first non-facial Gaussian point cloud and the non-facial Gaussian point cloud in the second head Gaussian point cloud after updating the opacity of the first non-facial Gaussian point cloud and the non-facial Gaussian point cloud in the second head Gaussian point cloud; and fuse the facial Gaussian point cloud in the third head Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud to obtain the third head Gaussian point cloud.

[0102] In some embodiments, the point cloud is a head point cloud, and the first Gaussian point cloud is a head Gaussian point cloud fitted based on any frame of point cloud. At this time, when the processing module 1102 predicts the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment label, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space, it specifically performs the following: calculating the second positional difference between points on the first Gaussian point cloud and points on the second head Gaussian point cloud; and predicting the first attribute change based on the first sentiment label, the second head Gaussian point cloud, and the second positional difference.

[0103] In some embodiments, the first attribute change includes one or more of the following: rotation change, color change, or position change.

[0104] In some embodiments, Figure 11 Both the communication module 1101 and the processing module 1102 shown can be implemented in software or in hardware. For example, the implementation of the communication module 1101 will be described below. Similarly, the implementation of the processing module 1102 can refer to the implementation of the communication module 1101.

[0105] As an example of a software functional unit, the communication module 1101 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the communication module 1101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0106] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0107] As an example of a hardware functional unit, the communication module 1101 may include at least one computing device, such as a server. Alternatively, the communication module 1101 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0108] The multiple computing devices included in communication module 1101 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in communication module 1101 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in communication module 1101 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0109] It should be noted that, in other embodiments, the communication module 1101 can be used to execute any step in the voice-driven face animation generation method described in the above embodiments, and the processing module 1102 can also be used to execute any step in the voice-driven face animation generation method described in the above embodiments. Furthermore, the steps implemented by the communication module 1101 and the processing module 1102 can be specified as needed, and different steps in the voice-driven face animation generation method described in the above embodiments can be implemented by the communication module 1101 and the processing module 1102 respectively. Figure 11 The full functionality of the voice-driven face animation generation device 1100 shown is illustrated.

[0110] This application also provides a computing device 1200. For example... Figure 12 As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0111] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1204 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1204 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0112] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0113] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0114] The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned functions. Figure 11 The communication module 1101 and processing module 1102 shown herein perform their functions to implement the voice-driven face animation generation method described in the above embodiments. That is, the memory 1206 stores instructions for executing the voice-driven face animation generation method described in the above embodiments.

[0115] Alternatively, the memory 1206 stores executable code, and the processor 1204 executes the executable code to implement the aforementioned functions respectively. Figure 11The voice-driven face animation generation device 1100 shown herein functions to implement the voice-driven face animation generation method described in the above embodiments. That is, the memory 1206 stores instructions for executing the voice-driven face animation generation method described in the above embodiments.

[0116] The communication interface 1208 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0117] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0118] like Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the voice-driven face animation generation method described in the above embodiments.

[0119] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the voice-driven face animation generation method described in the above embodiments. In other words, a combination of one or more computing devices 1200 can jointly execute instructions for executing the voice-driven face animation generation method described in the above embodiments.

[0120] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 11 The illustrated device 1100 is a voice-driven face animation generation device. Specifically, the instructions stored in the memory 1206 of different computing devices 1200 can implement the functions of one or more modules in the communication module 1101 and processing module 1102.

[0121] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14As shown, two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of communication module 1101. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of processing module 1102.

[0122] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0123] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 13 and Figure 14 The connection method of the computing device cluster is different in that the memory 1206 of one or more computing devices 1200 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0124] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the aforementioned voice-driven face animation generation method. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the aforementioned voice-driven face animation generation method.

[0125] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0126] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0127] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices containing at least one computing device, cause the cluster of computing devices to perform the methods in the above embodiments.

[0128] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0129] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0130] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0131] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A voice-driven face animation generation method, characterized in that, The method includes: Receive the first voice and first emotion tag from the client; The first speech and the first emotion label are input into a speech-to-geometry network to obtain a dynamic three-dimensional point cloud sequence with emotion attached, wherein the point cloud is a facial point cloud or a head point cloud. Based on the first emotion tag, the 3D point cloud sequence is optimized to obtain a face appearance sequence, wherein the first emotion tag is used to guide the generation of facial geometry and / or appearance.

2. The method according to claim 1, characterized in that, The step of optimizing the 3D point cloud sequence based on the first emotion tag to obtain a face appearance sequence includes: Based on the first sentiment tag, the first Gaussian point cloud, and the second head Gaussian point cloud in the regular space, the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud is predicted, wherein the first Gaussian point cloud is obtained by fitting any frame of point cloud in the three-dimensional point cloud sequence, and the third head Gaussian point cloud is related to the first Gaussian point cloud. Based on the change in the first attribute, the second head Gaussian point cloud is updated to obtain the third head Gaussian point cloud; Render the third head Gaussian point cloud to obtain the face appearance related to any frame of the point cloud.

3. The method according to claim 2, characterized in that, The point cloud is a facial point cloud, and the first Gaussian point cloud is a facial Gaussian point cloud obtained by fitting the point cloud of any frame. The step of predicting the change of the first attribute of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment tag, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space includes: Calculate the first positional difference between points on the first Gaussian point cloud and points on the face Gaussian point cloud in the second head Gaussian point cloud; Based on the distance between points on the non-face Gaussian point cloud in the second head Gaussian point cloud and points on the first Gaussian point cloud, a first weight of the points on the non-face Gaussian point cloud in the second head Gaussian point cloud is determined. Based on the first sentiment tag, the second head Gaussian point cloud, the first positional difference, and the first weight, the change in the first attribute is predicted.

4. The method according to claim 3, characterized in that, The first attribute change includes: the first sub-attribute change of the facial Gaussian point cloud and the second sub-attribute change of the non-facial Gaussian point cloud in the second head Gaussian point cloud; The step of updating the second head Gaussian point cloud based on the first attribute change to obtain the third head Gaussian point cloud includes: Based on the change of the first sub-attribute, update the facial Gaussian point cloud in the second head Gaussian point cloud to obtain the facial Gaussian point cloud in the third head Gaussian point cloud. Based on the change of the second sub-attribute, update the non-face Gaussian point cloud in the second head Gaussian point cloud to obtain the first non-face Gaussian point cloud. Based on the first weight, update the opacity of the points on the first non-facial Gaussian point cloud, and update the opacity of the points on the non-facial Gaussian point cloud in the second head Gaussian point cloud. Based on the non-face Gaussian point cloud in the first non-face Gaussian point cloud and the second head Gaussian point cloud after updating the opacity, the non-face Gaussian point cloud in the third head Gaussian point cloud is obtained. The facial Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud are fused to obtain the third head Gaussian point cloud.

5. The method according to claim 2, characterized in that, The point cloud is a head point cloud, and the first Gaussian point cloud is a head Gaussian point cloud obtained by fitting the point cloud of any frame. The step of predicting the change of the first attribute of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment tag, the first Gaussian point cloud, and the second head Gaussian point cloud in regularized space includes: Calculate the second positional difference between points on the first Gaussian point cloud and points on the second head Gaussian point cloud; Based on the first sentiment label, the second head Gaussian point cloud, and the second positional difference, the change in the first attribute is predicted.

6. The method according to any one of claims 2-5, characterized in that, The first attribute change includes one or more of the following: Rotation changes, color changes, or position changes.

7. A voice-driven face animation generation device, characterized in that, include: The communication module is used to receive the first voice and the first emotion tag from the client; The processing module is used to input the first speech and the first emotion tag into the speech-to-geometry network to obtain a dynamic three-dimensional point cloud sequence with emotion attached, wherein the point cloud is a facial point cloud or a head point cloud. The processing module is further configured to optimize the three-dimensional point cloud sequence based on the first emotion tag to obtain a face appearance sequence, wherein the first emotion tag is used to guide the generation of facial geometry and / or appearance.

8. The apparatus according to claim 7, characterized in that, When the processing module optimizes the 3D point cloud sequence based on the first emotion tag to obtain a face appearance sequence, it is specifically used for: Based on the first sentiment tag, the first Gaussian point cloud, and the second head Gaussian point cloud in the regular space, the first attribute change of the third head Gaussian point cloud relative to the second head Gaussian point cloud is predicted, wherein the first Gaussian point cloud is obtained by fitting any frame of point cloud in the three-dimensional point cloud sequence, and the third head Gaussian point cloud is related to the first Gaussian point cloud. Based on the change in the first attribute, the second head Gaussian point cloud is updated to obtain the third head Gaussian point cloud; Render the third head Gaussian point cloud to obtain the face appearance related to any frame of the point cloud.

9. The apparatus according to claim 8, characterized in that, The point cloud is a facial point cloud, and the first Gaussian point cloud is a facial Gaussian point cloud obtained by fitting the point cloud of any frame. When the processing module predicts the change in the first attribute of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment tag, the first Gaussian point cloud, and the second head Gaussian point cloud in a regularized space, it is specifically used for: Calculate the first positional difference between points on the first Gaussian point cloud and points on the face Gaussian point cloud in the second head Gaussian point cloud; Based on the distance between points on the non-face Gaussian point cloud in the second head Gaussian point cloud and points on the first Gaussian point cloud, a first weight of the points on the non-face Gaussian point cloud in the second head Gaussian point cloud is determined. Based on the first sentiment tag, the second head Gaussian point cloud, the first positional difference, and the first weight, the change in the first attribute is predicted.

10. The apparatus according to claim 9, characterized in that, The first attribute change includes: the first sub-attribute change of the facial Gaussian point cloud and the second sub-attribute change of the non-facial Gaussian point cloud in the second head Gaussian point cloud; When the processing module updates the second head Gaussian point cloud based on the first attribute change to obtain the third head Gaussian point cloud, it is specifically used for: Based on the change of the first sub-attribute, update the facial Gaussian point cloud in the second head Gaussian point cloud to obtain the facial Gaussian point cloud in the third head Gaussian point cloud. Based on the change of the second sub-attribute, update the non-face Gaussian point cloud in the second head Gaussian point cloud to obtain the first non-face Gaussian point cloud; Based on the first weight, update the opacity of the points on the first non-facial Gaussian point cloud, and update the opacity of the points on the non-facial Gaussian point cloud in the second head Gaussian point cloud. Based on the non-face Gaussian point cloud in the first non-face Gaussian point cloud and the second head Gaussian point cloud after updating the opacity, the non-face Gaussian point cloud in the third head Gaussian point cloud is obtained. The facial Gaussian point cloud and the non-facial Gaussian point cloud in the third head Gaussian point cloud are fused to obtain the third head Gaussian point cloud.

11. The apparatus according to claim 8, characterized in that, The point cloud is a head point cloud, and the first Gaussian point cloud is a head Gaussian point cloud obtained by fitting the point cloud of any frame. When the processing module predicts the change in the first attribute of the third head Gaussian point cloud relative to the second head Gaussian point cloud based on the first sentiment tag, the first Gaussian point cloud, and the second head Gaussian point cloud in a regularized space, it is specifically used for: Calculate the second positional difference between points on the first Gaussian point cloud and points on the second head Gaussian point cloud; Based on the first sentiment label, the second head Gaussian point cloud, and the second positional difference, the change in the first attribute is predicted.

12. The apparatus according to any one of claims 8-11, characterized in that, The first attribute change includes one or more of the following: Rotation changes, color changes, or position changes.

13. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-6, wherein the cluster of computing devices includes at least one computing device.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-6, wherein the computing device cluster includes at least one computing device.

Images