Three-dimensional face animation automatic generation method and device, medium and product

Abstract

The invention discloses a three-dimensional face animation automatic generation method and device, a medium and a product, and relates to the technical field of knowledge maps and image processing. The method comprises the following steps: acquiring an emotion label of a face animation image to be generated, converting the emotion label into text description by applying a language model, and converting the emotion label into emotion features by applying a prompt adapter module; according to the priori knowledge of the to-be-generated face animation image, acquiring face features of the to-be-generated face animation image by using a face encoder; acquiring facial geometric data of a to-be-generated facial animation image, and fusing the text description, the emotion features, the facial features and the facial geometric data to obtain a facial digital body of the to-be-generated facial animation image; and adjusting the face digitized body by applying a point cloud operation technology, and rendering the adjusted face digitized body to obtain a final digitized body of the to-be-generated face animation image. According to the method, the digital human face avatar with rich emotion expression can be obtained.

Description

Technical Field

[0001] The present invention relates to the fields of knowledge graphs and image processing technology, and in particular to a method, device, medium and product for automatically generating three-dimensional facial animation. Background Art

[0002] Although current facial avatar generation methods face many technical challenges in the pursuit of realistic and emotionally rich digital representations. Traditional methods are mostly based on fixed 3D morphological models or simple expression parameterization methods. Although they can generate basic facial morphology and expression changes, they perform poorly in terms of delicate emotional expression and dynamic capture. Because these methods lack accurate simulation of the complex movements of facial muscles, the generated avatars usually appear stiff and unnatural, and cannot match the rich emotions expressed by humans.

[0003] In particular, existing technologies have significant deficiencies in capturing subtle differences in facial expressions. For example, while some methods can generate smiling or laughing facial expressions, they often fail to distinguish the differences in intensity and emotional levels in these expressions. Although a smile and a big smile both convey a happy emotion, they involve different degrees of facial muscle engagement and dynamic changes. Failure to accurately capture these subtle differences will cause the avatar to appear unreal when expressing complex emotions, affecting the user's immersion and interactive experience.

[0004] In addition, existing methods also face challenges in lip synchronization and consistency of facial expressions of facial avatars. There are often temporal differences and mismatches between lip movements and expressions in the driving video, resulting in the generated avatar being unable to synchronize with the facial expressions in the actual video. This inconsistency not only weakens the expressiveness of the avatar, but also destroys the emotional resonance between the user and the avatar, reducing the overall credibility of the digital avatar.

[0005] A further challenge is how to maintain the uniqueness of individual identity features during the facial avatar generation process. As facial expressions and emotions change, it becomes increasingly difficult to keep the avatar consistent with the original individual's identity features. Existing methods often lose precise control of identity features when generating different expressions, resulting in the generated avatar not being able to accurately reflect the individual's true appearance and identity. This limitation will significantly affect the user's sense of identity and usage experience of the avatar in practical applications.

[0006] Therefore, how to break through these technical bottlenecks and develop a facial avatar generation method that can accurately capture the dynamics of human facial expressions, maintain individual identity consistency, and achieve high-quality lip synchronization has become an important topic in the current research field. By solving these problems, the authenticity and expressiveness of facial avatars in virtual reality, games, and online communications can be greatly improved, thereby enhancing user experience and promoting the development of related technologies. Summary of the invention

[0007] The purpose of the present invention is to provide a method, device, medium and product for automatically generating three-dimensional facial animation, which can solve the limitations of the prior art in the field of facial avatar generation and realize a highly realistic digital human facial avatar with rich emotional expression.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A method for automatically generating a three-dimensional facial animation, the method comprising:

[0010] Obtaining the emotion label of the face animation image to be generated, converting the emotion label into a text description by using a language model, and converting the emotion label into an emotion feature by using a prompt adapter module; the prompt adapter module includes a CLIP model and a multi-layer perceptron;

[0011] According to the prior knowledge of the facial animation image to be generated, a facial encoder is applied to obtain facial features of the facial animation image to be generated;

[0012] Acquire the facial geometry data of the facial animation image to be generated, and fuse the text description, the emotional features, the facial features and the facial geometry data to obtain a facial digital avatar of the facial animation image to be generated;

[0013] The facial digital avatar is adjusted by applying point cloud operation technology, and the adjusted facial digital avatar is rendered to obtain a final digital avatar of the facial animation image to be generated.

[0014] Optionally, a language model is applied to convert the sentiment label into a text description, which is expressed as:

[0015] Demotion=F ChatGPT (E);

[0016] Among them, E is the emotional label; Demotion is the text description; F ChatGPT Data processing using the language model ChatGPT.

[0017] Optionally, a prompt adapter module is applied to convert the emotion label into an emotion feature, expressed as:

[0018] Demotion=MLP(p ij );

[0019] (v i ,p ij )=(g CLIP (e i ),hCLIP (v i ,c j ));

[0020] Among them, Demotion is the emotional feature; MLP is a multi-layer perceptron; v i is the i-th sentiment label e i The embedded vector of the mapping; p ij is the i-th sentiment label e i The corresponding j-th detailed text prompt; g CLIP The emotion label e in the CLIP model i The mapping function that maps to the embedding vector; h CLIP Use the embedding vector v for the CLIP model i and the context vector c j Language generation function that generates detailed text prompts; c j A context vector used when generating detailed text hints.

[0021] Optionally, according to the prior knowledge of the facial animation image to be generated, applying a facial encoder to obtain facial features of the facial animation image to be generated specifically includes:

[0022] Acquire standard facial features of the face animation image to be generated through the FLAME model; the standard facial features include posture and expression;

[0023] The facial encoder is used to encode the standard facial features to obtain the facial features of the facial animation image to be generated.

[0024] Optionally, the facial encoder is applied to encode the standard facial features, which is expressed as:

[0025] G initial = FaceEncoder(β,α);

[0026] Among them, G initial The facial features of the face animation image to be generated are FaceEncoder, which is a facial encoder; α is an expression parameter; and β is a posture parameter.

[0027] Optionally, the text description, the emotional features, the facial features and the facial geometry data are fused and expressed as:

[0028]

[0029] in, A digital facial avatar of the face animation image to be generated; is the facial feature of the input linear layer MOD; G initialis the facial geometry data; Demotion is the text description and the emotional characteristics.

[0030] Optionally, the point cloud operation technology is applied to adjust the digital facial avatar, specifically including:

[0031] P′=DeformRender(P,E,Demotion);

[0032] Among them, P is the initial point cloud of the facial digital avatar; P′ is the adjusted facial digital avatar; E is the emotion label; Demotion is the text description and the emotion feature; DeformRender is rendering the point cloud.

[0033] A computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-mentioned methods for automatically generating three-dimensional facial animation.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above-mentioned methods for automatically generating three-dimensional facial animation.

[0035] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for automatically generating three-dimensional facial animation described in any one of the above items is implemented.

[0036] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0037] The present invention discloses a method, device, medium and product for automatic generation of 3D facial animation, which uses point cloud data and layered text prompt knowledge enhancement to capture and express emotions through a layered structure, from broad categories to subtle changes, to generate digital human representations with high expressiveness and authenticity; it combines a large language model with the most advanced neural network architecture to parse and translate text prompts into corresponding facial expressions, lip movement patterns and individual identity features, ensuring that the generated avatar maintains a unique identity within a wide range of emotions, providing highly realistic and consistent performance. Through extensive comparative evaluation, it has shown significant superiority in generating delicate, realistic and personalized facial animations, creating a new standard for facial avatar generation, and paving the way for more immersive and interactive virtual experiences. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of a flow chart of a method for automatically generating a 3D facial animation according to Embodiment 1 of the present invention;

[0040] Figure 2 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The purpose of the present invention is to provide a method, device, medium and product for automatically generating three-dimensional facial animation, aiming to solve the limitations of the prior art in the field of facial avatar generation and realize a digital human facial avatar that is highly realistic and has rich emotional expressions.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1

[0045] like Figure 1 As shown, the method for automatically generating a 3D facial animation in this embodiment includes:

[0046] Step S1: Obtain the emotion label of the face animation image to be generated, apply the language model to convert the emotion label into a text description, and apply the prompt adapter module to convert the emotion label into an emotion feature; the prompt adapter module includes a CLIP model and a multi-layer perceptron.

[0047] In practical applications, the initialization Hierarchical Prompt Enhancement module of the present invention uses the GPT language model to generate detailed text descriptions from a wide range of emotion tags; introduces a prompt adapter and integrates it with the CLIP model to obtain multi-level emotion features and improve the encoding ability of fine-grained emotion descriptions, that is, GPT is used to generate descriptions, text features are generated through CLIP, and then emotion features are generated through the prompt adapter.

[0048] The hierarchical prompt enhancement module converts the input emotion label into a detailed text description containing specific facial component movements through ChatGPT-3.5, and the formula is:

[0049] Demotion=F ChatGPT (E);

[0050] Among them, (E) represents the input emotion label, and (Demotion) is the detailed description of the output.

[0051] The prompt adapter module maps high-level sentiment labels to corresponding detailed prompts through the CLIP model, and converts these detailed prompts into sentiment features through a multi-layer perceptron (MLP), as follows:

[0052] Demotion=MLP(p ij );

[0053] Among them, (p ij ) are detailed prompts generated by the CLIP model.

[0054] Step S2: Based on the prior knowledge of the facial animation image to be generated, a facial encoder is applied to obtain facial features of the facial animation image to be generated.

[0055] S2 specifically includes:

[0056] Step S21: acquiring standard facial features of the face animation image to be generated through the FLAME model; the standard facial features include posture and expression.

[0057] Step S22: applying the facial encoder to encode the standard facial features to obtain the facial features of the face animation image to be generated.

[0058] In practical applications, a face encoder is used to integrate image prior information into the digital avatar generation process to further enhance the expressiveness of the avatar.

[0059] The facial encoder combines the FLAME model parameters (β, α) to encode the posture and expression of the image, further enhancing the expressiveness of the digital avatar. The formula is:

[0060] G initial =FaceEncoder(β,α).

[0061] Step S3: Acquire the facial geometry data of the facial animation image to be generated, and fuse the text description, the emotional features, the facial features and the facial geometry data to obtain a facial digital avatar of the facial animation image to be generated.

[0062] In practical applications, the Knowledge Prompt Modulation module is introduced to effectively integrate multimodal features, and these features are converted into geometric transformations through a multi-layer fusion mechanism to achieve delicate facial expressions.

[0063] The knowledge prompt modulation module maps the detailed emotion description (emotional features) (Demotion) to the corresponding geometric transformation through a multi-layer fusion mechanism to adjust the point cloud geometry so that the digital avatar can dynamically express emotions, which is formulated as:

[0064]

[0065] Among them, (G modulated ) represents the modulated geometric data, (MOD(·)) is the linear layer, (G initial ) represents the geometric configuration of the avatar’s face.

[0066] Step S4: applying point cloud operation technology to adjust the facial digital avatar, and rendering the adjusted facial digital avatar to obtain the final digital avatar of the facial animation image to be generated.

[0067] In practical applications, point cloud deformation and rendering are performed to adjust and render the point cloud to accurately express the target expression and generate the final digital avatar.

[0068] Through the point cloud deformation and rendering module, the initial point cloud (P) is adjusted to the final point cloud (P′) of the target expression, which is formulated as:

[0069] P'=DeformRender(P,E,Demotion).

[0070] In summary, the input hierarchical text prompt information is obtained, parsed and encoded into a hierarchical structure, and input into the neural network model to generate text semantic latent code. Specifically, the input hierarchical text prompt information is further semantically layered through the deep learning model to enhance the fine-grainedness of emotional expression. The obtained hierarchical text semantic latent code is combined with the point cloud data and input into the convolutional neural network to extract and fuse feature information, build an emotion recognition model, and generate preliminary expression dynamic information. Specifically, the adaptive instance normalization technology is used to stylize the expression dynamic information to ensure the delicacy and richness of the expression. The extracted expression dynamic information and individual features are input into the multimodal fusion network to generate an emotional expression model with personalized characteristics. By combining language and visual information, the matching degree of facial expressions and lip movements is further optimized. The generated personalized emotional expression model is input into the decoding module of the neural network to decode and generate high-precision three-dimensional facial expression animations while ensuring the authenticity and coherence of facial expressions. Specifically, the emotion recognition ability of the semantic latent code is enhanced through a large-scale pre-trained language model to improve the expressiveness of the virtual avatar. The generated 3D facial expression animations are subjected to multiple rounds of comparative training and optimization to ensure that the generated virtual avatars can maintain high-fidelity visual performance and individual uniqueness in various emotional expression scenarios.

[0071] The following is a specific implementation process of a method for automatically generating a 3D facial animation:

[0072] Step 1: Get input data. Extract relevant sentiment and geometric information from the text description, image or video input by the user. The natural language processing module preprocesses the text description and generates a hierarchical sentiment vector (E), which is expressed as a high-level sentiment (E high ) and low-level emotions (E low ). At the same time, the point cloud data (P) is extracted and used for subsequent geometric processing.

[0073] Step 2: Hierarchical prompt enhancement module. Through the hierarchical prompt enhancement module, the input emotion vector (E) is further refined into multiple granular emotion descriptions to generate emotion features (Demotion). high ) generates strong expressions, such as joy and anger; while low-level emotion labels (E low ) to generate subtle expressions, such as smiles.

[0074] Demotion=P ChatGPT (E);

[0075] Step 3: Prompt Adapter Module. The prompt adapter module uses the CLIP model to map the emotion label (E) to the corresponding detailed text prompt. For each emotion label (e i), the CLIP model generates an embedding vector (v i ) and detailed instructions (p ij ). The generated text prompts are mapped to sentiment features (Demotion) and further processed by a multi-layer perceptron (MLP):

[0076] v i =g CLIP (e i );

[0077] p ij =h CLIP (v i ,c j );

[0078] Demotion=MLP(p ij );

[0079] Among them, Demotion is the emotional feature; MLP is a multi-layer perceptron; v i is the i-th sentiment label e i The embedded vector of the mapping; p ij is the i-th sentiment label e i The corresponding j-th detailed text prompt; g CLIP The emotion label e in the CLIP model i The mapping function that maps to the embedding vector; h CLIP Use the embedding vector v for the CLIP model i and the context vector c j Language generation function that generates detailed text prompts; c j A context vector used when generating detailed text hints.

[0080] Step 4: Face encoder module. The face encoder module combines the prior information in the image (e.g., facial geometry) and further optimizes the facial features using the FLAME model parameters (β) and (α). This module converts the face encoder result (G initial ) is input into the geometry deformation module:

[0081] G initial =FaceEncoder(β,α).

[0082] The physical meaning of FaceEncoder is to convert the geometric information, expression changes and posture information of the face into a digital representation (embedded vector) that can be calculated and processed. This encoding method not only ensures the accurate modeling of facial expressions, but also converts every detail of the face (such as facial contours, muscle movements and expression details) into an abstract mathematical form, which is convenient for subsequent facial animation generation and real-time rendering.

[0083] Step 5: Knowledge prompt modulation module. The knowledge prompt modulation module fuses the emotional features (Demotion) and the initial geometric data (G initial ), generate the modulated geometric data (G modulated ). This process is completed through a multi-layer fusion mechanism, and the formula is as follows:

[0084]

[0085] in, It is used to adjust the facial expression and emotion features by inputting feature vector The facial geometry and emotional expressions are carefully adjusted to ensure that the generated facial avatars appear more natural and realistic visually.

[0086] Step 6: Point cloud deformation and rendering module. The point cloud deformation module performs geometric adjustments on the point cloud data (P) and renders the final 3D face animation. The emotion vector and geometric data are used to generate the final facial avatar through the rendering module:

[0087] P′=DeformRender(P,E,Demotion).

[0088] Among them, P′ represents the facial point cloud data after emotion-driven deformation and rendering, which contains the facial expressions after emotion changes, and generates realistic images or animations through rendering. This process combines the geometric changes of facial expressions with the generation of visual effects, and is a key part in facial animation generation. DeformRender generates the final facial expression image or animation by performing emotion-driven geometric deformation on the input facial point cloud data and combining rendering technology. It is a comprehensive operation that closely combines the geometric features, emotion changes and visual effects of the face to ensure that the generated facial avatar is not only natural and realistic in form, but also can accurately express the emotional state. E represents the emotional feature, which contains the emotion label and the corresponding facial expression change pattern. It determines the generation process of facial expressions and ensures that the generated facial avatar can accurately reflect different emotional states by adjusting the movement pattern and strength of facial muscles. This process ensures that the generated 3D facial animation has high expressiveness, emotional accuracy and maintenance of individual identity.

[0089] The present invention provides an innovative method combining point cloud operations and natural language processing technology to generate highly realistic and expressive facial avatars, covering the complete process from emotion label input to final digital avatar generation, highlighting the functions and interrelationships of each technical module, and ensuring that complex emotional expressions and individual identity characteristics are preserved. Compared with the prior art, the present invention has the following advantages:

[0090] The present invention improves the expressiveness and naturalness of facial avatars in different emotional states, and solves many challenges of traditional methods in capturing facial expressions, expressing subtle differences in emotions, maintaining identity features, and lip synchronization. This innovative framework provides more realistic and emotionally rich digital human facial avatars for user experience in fields such as virtual reality, games, and online communications. In addition, the present invention achieves the generation of highly realistic and expressive facial avatars through the refined rendering of point cloud data, layered text prompt analysis, the synergy of language and visual representation, and the application of multimodal consistency technology, providing a new solution for research and application in related fields.

[0091] Example 2

[0092] Based on Example 1, the present invention also provides a 3D facial animation automatic generation system, including: an acquisition module, a prompt enhancement module, an adapter module, a facial encoder module, a modulation module and a rendering module. Among them:

[0093] The acquisition module is used to obtain input emotion labels and image data.

[0094] The prompt enhancement module is connected to the acquisition module; the prompt enhancement module is used to generate a detailed text description using GPT.

[0095] The adapter module is connected to the acquisition module; the adapter module is used to convert high-level emotion tags into detailed prompts and encode them into emotion features.

[0096] The facial encoder module is connected to the acquisition module; the facial encoder module is used to encode the image prior information into geometric features.

[0097] The modulation module is connected to the facial encoder module, the prompt enhancement module and the adapter module respectively; the modulation module is used to integrate emotional description and geometric features to generate a highly expressive digital avatar.

[0098] The rendering module is connected to the modulation module; the modulation module is used to adjust and render the point cloud to generate the final digital avatar.

[0099] In practical applications, the present invention combines point cloud operations with natural language processing.

[0100] (1) The present invention combines advanced point cloud manipulation techniques with sophisticated natural language processing techniques to generate facial avatars with unparalleled expressiveness and fidelity. During the generation process, the capabilities of natural language models (such as GPT) are first applied to generate detailed text descriptions from broad emotion labels through a hierarchical prompt enhancement module. This module enhances the expressiveness of the generated facial avatar by refining simple emotion labels (such as happiness and anger) into multi-level facial descriptions. Assuming the input emotion label is (E), the detailed description obtained by the text generation model is (Demotion):

[0101] Demotion=F ChatGPT (E);

[0102] (2) This paper proposes a hierarchical cue enhancement module. This module uses the knowledge of the language model to refine the simple emotion label into three levels of refined facial description. The emotion label (E) is encoded as a high-level emotion (E high ) and low-level emotions (E low ), respectively, to express strong expressions (such as happiness, anger) and subtle expressions (such as smile, contemplation):

[0103] E=Encode(E high ,E low );

[0104] The high-level emotion and the low-level emotion are encoded to obtain the emotion label E again, and the description generated in the first step is obtained by obtaining these two emotions. This hierarchical emotion encoding process enables the present invention to generate a facial avatar that not only covers a wide range of emotions, but also can smoothly transition between different emotional states.

[0105] (3) In order to better encode fine-grained emotional descriptions, the present invention introduces a prompt adapter module. This module is integrated with the CLIP model to obtain multi-level emotional features. Assume that the high-level emotional label set is (E = {e1, e2, …, e n}), prompt the adapter to generate detailed text prompts (P = {p1, p2, ..., p m}):

[0106] P i =f CLIP (e i );

[0107] Among them, (f CLIP ) means that the emotion label (e i ) is mapped to the CLIP model corresponding to the text prompt. Each emotion label (e i ) is mapped to the corresponding embedding vector (v i ):

[0108] (v i ,p i )=(g CLIP (e i ),h CLIP (v i ,c j ));

[0109] The detailed text description finally generated by the prompt adapter is further mapped into sentiment features through a multi-layer perceptron (MLP):

[0110] Demotion=MLP(p i );

[0111] (4) The present invention also proposes a facial encoder module that uses prior knowledge in the image to further enhance the expressiveness of the facial avatar. Standard facial features are obtained through the FLAME model, and the facial features include pose and expression. These features are then further encoded using the facial encoder of the present invention:

[0112] G initial = FaceEncoder(β,α);

[0113] Wherein, (β) and (α) are the posture and expression parameters of the FLAME model. The facial encoder of the present invention is composed of two fully connected layers and is a lightweight MLP network.

[0114] (5) The knowledge prompt modulation module is an important part of the present invention. It combines the emotion description with the facial geometry data to achieve delicate facial expressions. The modulation process adopts a multi-layer fusion mechanism to process the input features through a linear layer (MOD). and the initial geometric representation (G initial ), input features For point cloud, get the modulated geometric data

[0115] This module adjusts vertex positions, normals, and applies subtle deformations so that the generated facial avatars are not only physically similar to real faces, but also able to dynamically express a variety of human emotions.

[0116] (6) In the final stage, the present invention generates the final facial avatar (P′) by adjusting the point cloud (P) to accurately model the desired expression:

[0117] P′=DeformRender(P,E,Demotion);

[0118] Among them, (P) and (P′) represent the initial and final point cloud representations, respectively, reflecting the detailed adjustments of emotional expression. Physically based rendering technology is used in the rendering process, combined with lighting and material characteristics, so that the generated facial avatars are highly realistic in both geometry and expression.

[0119] Through these complex technical components, the present invention establishes a new standard for generating digital avatars with excellent expressiveness and individual identity preservation, significantly improving the realism and immersion of virtual interactions.

[0120] Example 3

[0121] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for automatically generating three-dimensional facial animation in Example 1.

[0122] Example 4

[0123] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for automatically generating three-dimensional facial animation in embodiment 1.

[0124] Example 5

[0125] A computer program product includes a computer program, which, when executed by a processor, implements the method for automatically generating three-dimensional facial animation in embodiment 1.

[0126] Example 6

[0127] A computer device, which may be a database, may have an internal structure as shown in Figure 2 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the three-dimensional facial animation automatic generation method in Example 1 is implemented.

[0128] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for automatically generating a three-dimensional facial animation, characterized in that: The method comprises: Obtaining the emotion label of the face animation image to be generated, converting the emotion label into a text description by using a language model, and converting the emotion label into an emotion feature by using a prompt adapter module; the prompt adapter module includes a CLIP model and a multi-layer perceptron; According to the prior knowledge of the facial animation image to be generated, a facial encoder is applied to obtain facial features of the facial animation image to be generated; Acquire the facial geometry data of the facial animation image to be generated, and fuse the text description, the emotional features, the facial features and the facial geometry data to obtain a facial digital avatar of the facial animation image to be generated; The facial digital avatar is adjusted by applying point cloud operation technology, and the adjusted facial digital avatar is rendered to obtain a final digital avatar of the facial animation image to be generated.

2. The method for automatically generating three-dimensional facial animation according to claim 1, characterized in that: The language model is applied to convert the sentiment label into a text description, which is expressed as: Demotion=F ChatGPT (E); Among them, E is the emotional label; Demotion is the text description; F ChatGPT Data processing using the language model ChatGPT.

3. The method for automatically generating three-dimensional facial animation according to claim 1, characterized in that: The application prompt adapter module converts the emotion label into emotion features, which are expressed as: Demotion=MLP(p ij ); (v i ,p ij )=(g CLIP (e i ),h CLIP (v i ,c j )); Among them, Demotion is the emotional feature; MLP is a multi-layer perceptron; v i is the i-th sentiment label e i The embedded vector of the mapping; p ij is the i-th sentiment label e i The corresponding j-th detailed text prompt; g CLIP The emotion label e in the CLIP model i The mapping function that maps to the embedding vector; h CLIP Use the embedding vector v for the CLIP model i and the context vector c j Language generation function that generates detailed text prompts; c j A context vector used when generating detailed text hints.

4. The method for automatically generating three-dimensional facial animation according to claim 1, characterized in that: According to the prior knowledge of the face animation image to be generated, a face encoder is applied to obtain the facial features of the face animation image to be generated, specifically including: Acquire standard facial features of the face animation image to be generated through the FLAME model; the standard facial features include posture and expression; The facial encoder is used to encode the standard facial features to obtain the facial features of the facial animation image to be generated.

5. The method for automatically generating 3D facial animation according to claim 4, characterized in that: Apply the facial encoder to encode the standard facial features, expressed as: G initial =FaceEncoder(β,α); Among them, G initial is the facial feature of the face animation image to be generated; FaceEncoder is the facial encoder; α is the expression parameter; β is the posture parameter.

6. The method for automatically generating 3D facial animation according to claim 1, characterized in that: The text description, the emotional features, the facial features and the facial geometry data are fused and expressed as: in, A digital facial avatar of the face animation image to be generated; is the facial feature of the input linear layer MOD; G initial is the facial geometry data; Demotion is the text description and the emotional characteristics.

7. The method for automatically generating three-dimensional facial animation according to claim 1, characterized in that: Applying point cloud operation technology to adjust the digital facial avatar includes: P′=DeformRender(P, E, Demotion); Among them, P is the initial point cloud of the facial digital avatar; P′ is the adjusted facial digital avatar; E is the emotion label; Demotion is the text description and the emotion feature; DeformRender is rendering the point cloud.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for automatically generating three-dimensional facial animation according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically generating three-dimensional facial animation described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for automatically generating three-dimensional facial animation described in any one of claims 1 to 7 is implemented.

Images